Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data

Authors: zhilin  zhao
longbing  cao
xuhui  fan
wei-shi  zheng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 06:11:25

Abstract: There are increasing cases where the class labels of test samples are unavailable, creating a significant need and challenge in measuring the discrepancy between training and test distributions. This distribution discrepancy complicates the assessment of whether the hypothesis selected by an algorithm on training samples remains applicable to test samples. We present a novel approach called Importance Divergence (I-Div) to address the challenge of test label unavailability, enabling distribution discrepancy evaluation using only training samples. I-Div transfers the sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error of the distribution discrepancy as transformed through the two ratios. Experimentally, I-Div accurately quantifies the distribution discrepancy, as evidenced by a wide range of complex data scenarios and tasks.

Main paper: 
 introduction: 
 The assumption that data are independently and identically distributed (IID) is staple in statistical machine learning. It suggests that a hypothesis selected by an algorithm, after observing several training samples, should perform effectively on test samples from the same unknown distribution. However, this assumption often oversimplifies the intricate and diverse nature of real-world data, particularly in non-IID scenarios [1, 2] . Thus, if training samples are considered in-distribution (ID), there is a risk that test samples may deviate from this distribution, characterized as out-of-distribution (OOD) [3] . This distribution discrepancy between training and test distributions poses a critical and challenging non-IID learning [2] question: How to quantify the applicability of a hypothesis derived from training samples to test samples?. This question is generally applicable to areas like OOD generalization [4] , OOD detection [5, 6] , domain adaptation [7, 8] , transfer learning [9] , semi-supervised learning [10] , robust learning [11] , and adversarial learning [12] .The applicability of a hypothesis can be determined by the distribution discrepancy between training and test distributions. When the two distributions align, meaning both training and test samples are ID, the hypothesis derived from training samples can be seamlessly applied to test samples. However, in reality, particularly when test samples fall OOD, this scenario rarely holds. Under such circumstances, decisions must be based on the extent of the distribution discrepancy. This may require enhancing the hypothesis generalization capability towards test samples or rejecting these samples outright. These actions are fundamental to the principles of OOD generalization and OOD detection, respectively [13] .Evaluating the distribution discrepancy between training and test distributions for a selected hypothesis presents a significant challenge, as training samples are typically labeled, whereas test samples often are not [14] . This discrepancy means that conventional statistical distances, such as F-divergences [15] , integral probability metrics [16] , and total variation [17] , are not suitable for this situation. Furthermore, density ratio methods [18, 19] offer a partial solution by disregarding label information and estimating the ratio between input distributions. Moreover, more performance prediction techniques [20] aim to navigate this challenge by examining the uncertain outcomes of the hypothesis, potentially leading to unreliable results. A detailed discussion of related work is provided in Appendix A.To tackle the issue of unavailable test labels, we introduce the concept of Importance Divergence (I-Div), which measures the training-test distribution discrepancy w.r.t. the difference between the expected risks of the selected hypothesis on training and test distributions. To estimate the expected risk on test distributions without label access, the core strategy involves importance sampling to transfer the sampling patterns from the test distribution to the training distribution. This process requires the estimation of density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error related to the distribution discrepancy as transformed through the two ratios. As a result, I-Div leverages the estimated density and likelihood ratios to quantitatively measure the distribution discrepancy between training and test distributions, eliminating the need for test class labels, and thus quantifying the applicability of the hypothesis across different datasets. 
preliminaries: 
 Let X denote the input space, and Y represent the set of labels. The joint probability distributions are represented by P for training samples and Q for test samples. Assume we observe a labeled training dataset P and a unlabeled test dataset Q containing N IID samples from P and Q, respectively. Q X denotes the marginal distribution of Q over X . P and Q are described asEQUATIONwhere the label space Y encompasses all labels of samples from both P and Q. We also define H as the hypothesis space and L(•, •) ∈ (0, B L ) as the bounded loss function. The expected and empirical risks [21] for a hypothesis h ∈ H : X → Y on distribution P and the dataset P are defined as P (h) and P (h), respectively, i.e.,P (h) = E P [L(h(x), y)] = X YL(h(x), y) P(x, y) dx dy,EQUATIONwhere E and E represent the expectation with respect to a data distribution and the sample average over a dataset, respectively. Accordingly, an algorithm A aims to select the empirical risk minimizer h P after observing the samples from the training dataset P byEQUATIONto approximate the optimal hypothesis h * P selected from the distribution P through h * P ∈ arg min h∈H P (h).Our central research question is formulated as follows: How can we quantify the applicability of the minimizer h P , originated from the training samples P, to the unlabeled test samples Q? To address this question quantitatively, we delve into methodologies for assessing the distribution discrepancy between training and test distributions, particularly focusing on doing so without the need to access class labels from test samples. 
importance divergence: 
 For a hypothesis chosen by an algorithm after observing training samples, to assess its applicability to test samples in the absence of ground truth labels, we introduce the concept of Importance Divergence (I-Div). I-Div estimates the distribution discrepancy between training and test distributions for the given hypothesis. To address the issue of unavailable ground truth labels, it leverages importance sampling, density ratios, and likelihood ratios, facilitating the sampling transfer in the test distribution back to the training distribution.In this section, we first present the distribution discrepancy with importance sampling, which transfers the data sampling from the test distribution to the training distribution. Following this, we discuss the methodologies for estimating the hypothesis-oriented density and adaptive likelihood ratios, essential components of this discrepancy, to facilitate this sampling transfer. The hypothesis-oriented density ratio is specifically tailored to each hypothesis, as it assesses the suitability of a specific hypothesis based on the resulting distribution discrepancy. The adaptive likelihood ratio is adjusted according to the density ratio to expedite the convergence of the distribution discrepancy. Lastly, we utilize an empirical estimator of the distribution discrepancy to evaluate the applicability of a hypothesis selected from training samples to test samples.For the hypothesis h P , I-Div evaluates the distribution discrepancy between training and test distributions without requiring the ground truth labels of the test samples. A smaller discrepancy implies that the training and test samples could be considered as drawn from the same distribution with respect to the given hypothesis, and vice versa. By using variational divergence, we can express this discrepancy as the difference between the expected risks of the hypothesis on training and test distributions byEQUATIONIf the training and test distributions are aligned, i.e., P = Q, then the expected risks for the hypothesis h P are similar, resulting in a minimal distribution discrepancy. This condition suggests that the test samples are likely ID for the hypothesis h P . Conversely, a notable difference between P and Q indicates a larger distribution discrepancy, implying that the hypothesis h P perceives the training and test samples as originated from distinct distributions, thus categorizing the test samples as OOD.To make the expected risk difference in Eq. ( 4) more pronounced, I-Div employs Jensen's inequality to consider its upper bound byEQUATIONthereby more distinctly highlighting the differences between the training and test distributions.Recall that class labels in test samples are inaccessible, thus, the principal challenge is evaluating the expected risk for the test distribution with respective to the given hypothesis without access to ground truth labels. Since direct sampling from the test distribution is not feasible, an alternative is sampling from the training distribution. To overcome this limitation, we employ the importance sampling technique [22] , converting the data sampling from test to training distributions, i.e.,EQUATIONwhere r and v denote a density ratio and a likelihood ratio, respectively. Thus, I-Div estimates distribution discrepancy between training and test distributions by merely sampling from the training one without accessing the class labels of the test samples. According to Eq. ( 5) and Eq. ( 6), we haveEQUATIONAccordingly, for a given labeled training dataset P and a unlabeled test dataset Q, we can then construct an empirical estimator for estimating the distribution discrepancy in Eq. ( 7) byEQUATIONHowever, to estimate the distribution discrepancy I-Div, the emerging challenge involves determining the density ratio r and likelihood ratio v from observed samples, which are discussed in Section 3.2 and Section 3.3, respectively.The density ratio in the distribution discrepancy should be hypothesis-oriented. That is, it should depend on the specific hypothesis h P ∈ H selected by an algorithm A on P. This is because the criteria for judging the discrepancy between distributions P and Q vary across different algorithms.For instance, whether two datasets of cats and dogs respectively come from the same distribution depends on whether the algorithm aims to identify if the subjects are biological entities or to distinguish between these two species. Accordingly, we apply a deep neural network to model a density ratio r from the hypothesis space R( h P ) depending on the specific hypothesis h P . Then, we select a density ratio r ∈ R( h P ) to minimize the Kullback-Leibler divergence between the actual distribution and the estimated distribution based on this density ratio.To construct the hypothesis space R( h P ), we utilize the output representations from h P with a learnable component ω to model a density ratio r. Specifically, we decompose the hypothesis into a backbone ψ and a softmax layer φ, represented ash P (x) = (φ • ψ) (x), where the output dimension of the backbone is O ψ . Using a learnable component ω, r is constructed asr(x) = (ω • ψ) (x),where the learnable component ω(•) = (ω SP • ω AD )(•) contains an adapter ω AD [23] to introduce learnable parameters and a Softplus layer ω SP [24] to ensure strictly positive outputs. Specifically, the adapter ω AD , which follows the bottleneck ψ, comprises two fully connected layers with a Gaussian Error Linear Units (GELU) [25] activation layer in between. Furthermore, the Softplus layer is adopted to the adapter output, effectively mapping it to the range (0, +∞]. Furthermore, weight matrices areW 1 ∈ W 1 ∈ R O ψ ×Om and W 2 ∈ W 2 ∈ R Om×Oω in the fully connected layers.The activation functions GELU and Softplus are β 1 -and β 2 -Lipschitz, respectively. Additionally, we assumeτ 1 = sup W1∈V1 W 1 1,∞ and τ 2 = sup W2∈V2 W 2 1,∞ .Thus, the density ratio r ∈ R( h P ) can be modeled asEQUATIONwhere O ω represents the output dimensionality of the adapter. Without loss of generality, we further assume that r(x) ∈ (b r , B r ) for any r ∈ R( h P ) and x ∈ X .To select a density ratio r ∈ R( h P ), we use it to estimate the density P and Q byEQUATIONDrawing on the inspiration of importance estimation methods [26, 27] , we construct objectives and constraints around P(x) and Q(x). The estimated probability distributions are designed to approximate their actual counterparts, suggesting the minimization of the two KL divergences with normalization constraintsEQUATIONFor convenience, we assume U = P/2 + Q/2 and define C = {1, -1} as labels for training and test samples, respectively. A label c ∈ C, corresponding to a sample from U, indicates its distribution origin. The assignment c = 1 indicates that a sample originates from distribution P, while c = -1 signifies that a sample comes from distribution Q. We can then obtain the following objective function for learning the density ratioEQUATION)where λ ≥ 0 balances the KL divergence and normalization constraints, as the detailed derivation shown in Appendix B. We further assume that f (r) is L f -lipschitz continuous with respect to r ∈ R( h P ). This objective function Eq. ( 12) can be estimated byEQUATIONwhere U = P ∪ Q. An empirical risk minimizer r is selected byEQUATIONwhich aims to approximate the population risk minimizer r ∈ arg min r∈R( h P ) f (r). The convergence rate of r can be guaranteed by the following theorem. Theorem 3.1. Let λ ≥ 1 and µ be a constant related to the function f . With a probability of at least 1 -δ,E P |r(x) -r (x)| 2 ≤ 64L f (β 1 β 2 τ 1 τ 2 O ψ + 1 + B r ln 4/δ) µ √ N + 8β 2 τ 2 L f µN := B(δ, N ).The presence of N in the denominators of both terms suggests that the bound tightens with an increasing sample size, which aligns with the general understanding that more data can lead to more accurate estimates in statistical learning.Without making further assumptions, it is infeasible to estimate the likelihood ratio v due to its dependence on the unknown joint distribution Q(x, y). Instead of pursuing the true likelihood ratio, our goal is to approximate an adaptive likelihood ratio v ∈ V( r) that enables a swift convergence of the distribution discrepancy, guided by the hypothesis-oriented density ratio r ∈ R( h P ). This strategy is valid since the density ratio captures the input distribution discrepancy between training and test distributions. It indicates that utilizing even a basic form of the covariate shift assumption, i.e., v(x, y) = 1 for all (x, y) ∼ P, allows the distribution discrepancy, as calculated by Eq. ( 8), to approximate the difference between distributions to a reasonable degree. Furthermore, since the density ratio serves primarily to gauge the distribution discrepancy, the corresponding likelihood ratio must be specifically adapted to this density ratio for precisely assessing the distribution discrepancy.Accordingly, we reveal the generalization error bound of the distribution discrepancy, leveraging both density and likelihood ratios, through the convergence rate of the hypothesis-oriented density ratio, Rademacher complexity [28] and Talagrand's contraction lemmas [29] , which is shown as follows. Theorem 3.2. Based on the conditions and results outlined in Theorem 3.1, with a probability of at least 1 -δ, d P, Q | h P , r, v -d P, Q | h P , r, v is bounded byB L ln(2/δ) (x,y)∼ P | r(x)v(x, y) -1| 2 N + B L E P [v(x, y)] B(δ/2, N ) µ .The result shows that the estimated distribution discrepancy converges quickly with increasing the sample size. Additionally, the bound is associated with the values of the likelihood ratio. Moreover, as per Eq. ( 6), for any r ∈ R( h P ), we haveEQUATIONConsidering the terms in Theorem 3.2 related to v and the average error over the samples from P , we have min v∈V( r)EQUATIONwhere γ > 0 acts as a regularization parameter, influencing the trade-off, and the optimal solution is v. By using an proximal algorithm [30] , we can obtain the following approximate solutionEQUATION)We observe that in the two boundary cases where γ → 0 and γ → +∞, v consistently equals 1, thus adhering to the covariate shift assumption. Even under these extreme conditions, it is feasible to calculate the distribution discrepancy using both the hypothesis-oriented density ratio and adaptive likelihood ratio to overcome the challenge of unlabelled test samples. This approach is viable because the hypothesis-oriented density ratio r quantifies the distribution differences without relying on class labels, while employing γ ∈ (0, +∞) utilizes the class labels of training samples. Although this method may not precisely determine the likelihood ratio for each instance, it is designed in accordance with a hypothesis-oriented density ratio such that the estimated distribution discrepancy aligns with the actual value, fulfilling our primary objective.I-Div quantifies the applicability of the hypothesis h P selected by the algorithm A from the training dataset P to the test dataset Q. Specifically, I-Div employs the empirical estimator in Eq. ( 8) with hypothesis-oriented density ratio r in Eq. ( 14), and adaptive likelihood ratio v in Eq. (17) to estimate the distribution between training and test distributions byEQUATION)where r is chosen from the hypothesis space R( h P ) based on h P , and v is selected from the space V( r) based on r. A smaller discrepancy indicates that the training and test samples are likely drawn from the same distribution relative to h P . Since h P minimizes the empirical risk on P, a reduced distribution discrepancy improves the transferability of the hypothesis from training to test samples. On the other hand, a greater discrepancy suggests a reduced likelihood of hypothesis applicability. The I-Div methodology is detailed in Algorithm 1.Algorithm 1 Importance divergence 1: Input:2:-Training samples P = {(x i , y i , c i = 1)} N i=1 ∼ P 3:-Test samplesQ = {(x i , c i = -1)} N i=1 ∼ Q 4:-Empirical minimizer h P , Hyperparameters λ and γ 5: Merge datasets: U = P ∪ Q 6: Estimate the hypothesis-oriented density ratio on U:r ∈ arg min r∈R( h P ) E (x,c)∼ U c log r(x) + λ (r (x)) c -1 2 7:Estimate the adaptive likelihood ratio on P:v(x, y) = (N γ r(x) -N ) / γ( r(x)) 2 • E x∈ P γ( r( x)) 2 / (γ r( x) -1)8: Estimate the distribution discrepancy with importance sampling on P:d P, Q | h P , r, v = E P | r (x) v(x, y) -1| L( h P (x), y) 9: Output: empirical estimator d P, Q | h P , r, vQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental results: 
 This section presents a comparative analysis of I-Div2 against existing methods for evaluating the distribution discrepancy between training and test samples. The detailed experimental setups are presented in Appendix D.1. Our initial experiments focus on a relatively straightforward task: assessing the applicability of a hypothesis obtained on the training dataset to the test dataset with distinctly different class labels. We utilize two datasets, CIFAR10 [31] and SVHN [32] , each comprising ten semantically unique classes.For our experiments, we select samples from one class to serve as the test dataset, with the samples from the remaining nine classes forming the training dataset. This setup clearly illustrates that the knowledge learned in the training dataset cannot be transferred to the test dataset.The results for CIFAR10 and SVHN are detailed in Table 1 and Table 6 (Appendix D.2), respectively. Our proposed I-Div algorithm consistently achieves perfect scores (100%) in both AUROC and AUPR metrics across all classes of both datasets. This demonstrates its exceptional capability in distinguishing between training and test datasets, aligned with our initial hypothesis. The results unequivocally support the premise that the knowledge transfer from the training to the test datasets is ineffective, as evidenced by the flawless performance of I-Div. This starkly contrasts with the varying effectiveness of other algorithms, including NNBD and MMD-D. Notably, I-Div, MSP, and R-Div all yielded similarly impressive results. A key commonality among these algorithms is their reliance on a specific hypothesis to calculate distribution discrepancy, as opposed to NNBD and MMD-D, which use independent hypotheses. This highlights the significance of considering a particular hypothesis when evaluating distribution discrepancies. The rationale is that the hypothesis applicability depends on the specific design and its intended task. We now turn to a more complex scenario where the training and test datasets may share semantic similarities in class labels, indicating an overlap in the class label spaces. In cases where semantics differ significantly, we expect the algorithm to clearly differentiate the two kinds of samples. Conversely, if their semantics are similar, the algorithm may find it challenging to distinguish them. This outcome would suggest that the knowledge acquired from the training dataset is transferable to the test dataset, or it may indicate potential pathways to enhance the hypothesis generalization for the test distribution. We leverage CLIP [33] to align class labels between the training and test datasets, using the prompt template "A photo of a {label}." This helps adapt class labels across domains and captures semantic relationships between different datasets.We conduct experiments using classic domain adaptation datasets: PACS [34] and Office-Home [35] , each containing four domains. We designate one domain as the training dataset and merge the remaining three as the test dataset. The results, presented in Table 2 , indicate that each hypothesis selected from a dataset performs significantly better than randomly selected hypotheses, demonstrating its applicability. Our I-Div algorithm aptly reflects this, in contrast to other algorithms that overly emphasize distribution discrepancies, thereby rigidly categorizing the difference between training and test datasets. We use CIFAR10 [31] as the training dataset and evaluate on diverse test datasets including Randomly Generated Images (RGI), SVHN [32] , DTD [36] , Flowers102 [37] , OxfordIIITPet [38] , SEMEION [39] , Caltech256 [40] , CIFAR100 [31] , CIFAR10.1 [41] , and STL10. Since CIFAR10.1 and STL10 share similar category spaces with CIFAR10, the model shows minimal differentiation for these datasets, with ACC values of 73.4% and 42.1%, respectively. This suggests partial knowledge transferability. However, I-Div demonstrates lower AUROC values of 43.7% and 37.2%, indicating reduced discrimination. Other algorithms like MSP and R-Div show higher AUROC values, near 100% across all datasets, but these results suggest an overemphasis on distribution discrepancy rather than semantic similarity. I-Div, in contrast, better captures semantic relationships between datasets, providing a more nuanced view of class label semantics. We use ImageNet [42] as the training dataset and evaluate on diverse test datasets using ResNet50 [43] and ViT-B/16 [44] . The test datasets include the Open Images Dataset v4 (OIDv4) [45] , Cal-tech256 [40] , Flowers102 [37] , and DTD [36] . The experimental results presented in Table 4 show that the AUROC values of I-Div effectively capture the semantic similarity between ImageNet and the test datasets, yielding results that closely align with human intuition. For example, I-Div demonstrates lower AUROC values for OID and Caltech256, reflecting their semantic overlap with ImageNet, as these datasets share common object categories and scene types. In contrast, datasets such as Flowers102 and DTD, which focus on more specialized object categories and textures, show higher AUROC values with I-Div, indicating greater divergence from ImageNet. On the other hand, algorithms like MSP, NNBD, and MMD-D show consistently high AUROC values across most datasets, implying they emphasize distribution discrepancies over semantic relationships. This limits their effectiveness in distinguishing nuanced semantic differences compared to I-Div, which provides a more human-aligned understanding of dataset relationships. This section discusses experimental results on corrupted datasets. We progressively introduce noise into a dataset that serves as the training one, treating the resultant corrupted samples as the test dataset. Intuitively, as the noise level increases, the hypothesis performs worse, which indicates the hypothesis becomes less applicable to the corrupted data. To conduct this experiment, CIFAR10 serves as the training dataset, with incremental addition of noises to the original dataset to create the test dataset. The types of noises [46] used include Gaussian, Salt & Pepper, Uniform, and Speckle, with the noise rate increasing from 0.1 to 0.9 with a 0.1 interval. The methods for comparison include Hypothesis-oriented Density Ratio (HDR) in I-Div and R-Div [47] . Fig. 1 presents our experimental findings, showing key performance metrics as influenced by varying noise rates. Notably, the classification performance declines with increasing noise, impacting the hypothesis predictive accuracy. Interestingly, our proposed I-Div algorithm demonstrates robustness against these challenges, with its discrimination power inversely related to the classification accuracy of the standard network in noisy conditions. A brief comparative analysis hints at the superior performance of I-Div over HDR, especially in relation to hypothesis applicability in noisy test datasets. In this experiment, we delve into a specific scenario involving adversarial samples [48] . We designate one dataset as training and its corresponding adversarial samples as the test dataset. It is a well-known phenomenon that a minimal adversarial perturbation, though visually imperceptible, can drastically alter the classification performance of a network. This suggests potential issues with the direct applicability of the hypothesis selected for a training dataset to a test dataset. However, based on human perception, which fails to distinguish original and adversarial samples visually, we would expect a negligible distribution discrepancy between the distributions of the original and adversarial samples. This outcome could guide us in enhancing network robustness against adversarial attacks and in generalizing the hypothesis to adversarial contexts. For this purpose, we use the CIFAR10 dataset to train a standard network, with adversarial perturbation magnitudes selected from the set {0.001, 0.01, 0.1, 0.3}. The results in Fig. 2 indicate a marked decrease in standard network accuracy against adversarial perturbations, while the consistent AUROC of I-Div suggests its limited differentiation capability. Detailed results and analyses are provided in Appendix D.4. We examine the impact of different sample sizes on the performance of the I-Div algorithm, focusing on its ability to generalize hypotheses from training to test datasets. Fig. 3 show that I-Div tends to maintain low AU-ROC values for semantically similar datasets like CIFAR10.1 and STL10, indicating effective hypothesis applicability. Conversely, for datasets with significant semantic differences, the performance of I-Div improves with larger sample sizes, highlighting its capacity to recognize non-transferable knowledge. Additionally, we investigate the effect of varying network architectures as shown in Table 5 . Detailed results are provided in Appendix D.5. 
limitations: 
 I-Div relies on density and likelihood ratios to achieve the sampling transfer in unlabeled test data, allowing for the estimation of distribution discrepancies between training and test datasets with labeled training samples. Although the density ratio can be accurately estimated using inputs from both training and test samples, the likelihood ratio cannot be estimated precisely due to the unavailability of class labels of test samples. The strategy used by I-Div targets the estimation of distribution discrepancies between training and test distributions. It optimizes a likelihood ratio that adapts to this density ratio to ensure a rapid convergence of the distribution discrepancy, by minimizing the upper bound of the generalization error based on the density ratio. Our future research includes refining the estimation methods for likelihood ratios and exploring distribution discrepancy estimation methods that can bypass the likelihood ratio. 
conclusion: 
 In the realm of complex data and machine learning tasks, a crucial question arises regarding the applicability of a hypothesis derived from a training dataset to a test dataset. This uncertainty, especially challenging when test samples lack class labels, significantly determining the hypothesis generalization. To address this, we introduce the I-Div measure for estimating the distribution discrepancy between training and test distributions. I-Div involves the hypothesis-oriented density ratio and adaptive likelihood ratio in expected risk difference to shift the sampling problem from test to training distributions. Experimentally, we validate that I-Div can effectively assess the hypothesis capability of handling test samples, yielding results consistent with prior human knowledge. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: The paper solely focuses on technical aspects and does not address potential societal impacts, such as ethical considerations or possible misuse of the technology. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] , Justification: The paper does not discuss any specific safeguards related to the responsible release of data or models, particularly those that might have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The paper properly credits the creators or original owners of all used assets, including data and code. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we briefly review the methods for distribution discrepancy estimation, density ratio estimation, and performance prediction.Mean Embedding (ME) [49] and Smooth Characteristic Functions (SCF) [50] utilize differences in Gaussian mean embeddings at optimized points and frequencies, respectively, to quantify distribution discrepancy. Building on ME, MMD-O [51] calculates Maximum Mean Discrepancy (MMD) using a Gaussian kernel [52] , while MMD-D [53] enhances the performance of MMD-O by substituting the Gaussian Kernel with a learnable deep kernel. In contrast, Classifier Two-Sample Tests, including C2ST-S [54] and C2ST-L [55] , classify samples from one dataset as positive and from another as negative, leveraging the classification accuracy of a binary classifier to differentiate between them. H-Divergence (H-Div) [20] identifies optimal hypotheses for both the mixture distribution and individual distributions within a specific model, positing that if two distributions are identical, the expected risk on the training samples from the mixture distribution exceeds that from each individual distribution. R-Divergence (R-Div) [47] tackles the overfitting problem of H-Div by suggesting that two distributions are likely identical if their optimal hypotheses yield the same expected risk for each. However, to estimate the discrepancy between training and test distributions, the aforementioned methods assume symmetry between these datasets, meaning either both sets are labeled or both are unlabeled.To estimate the density ratio, Kernel Mean Matching (KMM) [56] offers direct estimates of importance at training inputs by efficiently aligning two distributions, leveraging a unique characteristic of universal reproducing kernel Hilbert spaces. The Kullback-Leibler Importance Estimation Procedure (KLIEP) [19] seeks an importance estimate to minimize the Kullback-Leibler divergence from the true test input density to its estimate. Least-Squares Importance Fitting (LSIF) [57] addresses the direct importance estimation issue as a least-squares function fitting problem, transforming the optimization challenge into a convex quadratic program, which is solvable efficiently with standard quadratic programming solvers. Non-Negative Bregman Divergence (NNBD) [58] employs deep neural networks alongside empirical Bregman divergence minimization, addressing the train-loss hacking issue by adjusting the empirical Bregman divergence estimator. Although these methods can estimate the distribution discrepancy between test and training inputs by calculating the density ratio even when class labels for test samples are not available, this implies disregarding the class labels of the training data. Consequently, such density ratios are not suitable for evaluating the applicability of a hypothesis.Maximum over Softmax Probability (MSP) [59] gauges the class label confidence on each sample by a given model to distinguish ID and OOD samples, which can be extend by computing the average class label confidence across both training and test datasets and consider the confidence disparity as indicative of distribution discrepancy. Average Thresholded Confidence (ATC) [60] identifies a threshold such that the proportion of training samples exceeding this confidence threshold aligns with their accuracy, and then it predicts accuracy as the proportion of unlabeled test samples surpassing this threshold. Agreement-on-the-Line (AL) [61] capitalizes on the observation that if the accuracy of models on training samples linearly correlates with their accuracy on test samples, then a similar linear relationship exists between the training and test agreement of model predictions. Projection Norm (PN) [62] employs model predictions to pseudo-label test samples and trains a new model on these pseudo-labels. The variation in parameters between this new model and the original model is utilized to estimate the test error. The aforementioned algorithms estimate the effectiveness of a hypothesis selected from the training samples on test samples based on the confidence output of the hypothesis to judge its applicability. However, since the hypothesis is chosen based on a limited set of training samples, it exhibits significant uncertainty towards unseen test samples, especially when these test samples are OOD, leading to misleading results.According to the KL divergence between P(x) and P(x), we haveEQUATION) Similarly, for the KL divergence between Q(x) and Q(x), we haveKL Q(x) Q(x) = x∈X Q(x) log Q(x) r(x)P(x) dx = KL (Q(x) P(x)) -E Q [log r(x)] .(20) Note that the terms KL P(x) P(x) and KL Q(x) Q(x) are constants independent of the density ratio r. To normalize P(x) and Q(x) estimated by r, we have x∈X P(x) dx = 1 and x∈X Q(x) dx = 1. According to Eq. ( 10), we haveEQUATIONBy considering Eq. ( 19), Eq. ( 20) and Eq. ( 21), we can obtain an optimization problem with constraints for learning a density ratio r, i.e., min r∈R( h P )EQUATIONHowever, solving such a problem with constraints is difficult. Inspired by the method of Lagrange multipliers [30] , we can introduce a hyperparameter λ ≥ 0, which allows us to relax this constraint and balance the loss function of r with the constraint conditions, i.e., min r∈R( h P )EQUATION)Accordingly to the Cauchy-Schwarz inequality [63] , we haveEQUATIONApplying Eq. ( 24) and the Jensen's inequality, we haveEQUATION)We obtain the objective function by combining Eq. ( 23) and Eq. ( 25).C.1 Proof of Theorem 3.1We rewrite f (r) by applying the density ratio and obtainEQUATIONAccording to Eq. ( 12), we know that λ is used to ensure the constraints are satisfied as much as possible. Intuitively, its value should be relatively large. More specifically, we provide the following lemma to determine the lower bound of λ to ensure that f (r) is strongly convex.Lemma C.1. f (r) is µ-strongly convex if λ ≥ 1.Thus, we haveEQUATIONThe first inequality arises from the µ-strong convexity of u(r(x)). The second inequality is because r is the minimizer of f (r). According to r(x) ∈ (b r , B r ) and f (r) is L f -lipschitz continuous on (b r , B r ), we know that the absolute value of the difference caused by altering one data point in|f (r) -f (r)| is bounded by 2L f B r .To proceed, we introduce the following two lemmas.Lemma C. R(l • H • P) ≤ LR(H • P),where R(H• P) = 1 N E σ∼{±1} N sup h∈H x∈ P σ i h(x i ) with random choice of σ.Accordingly, with probability of at least 1 -δ, for any r ∈ R, we haveEQUATION)Recall that ψ is a frozen network backbone and ω is a learnable adapter containing two fully connected layers and an activation layer in the middle. The activation function used in the middle and the last layers are GELU and Softplus, respectively. According to the Rademacher bound [64] , we haveEQUATION)We complete the proof by combining Jensen's inequality, Eq. ( 27), Eq. ( 28) and Eq. ( 29).Applying the triangle inequality, for and any v ∈ V, we haveEQUATION)To bound B 1 , with probability of at least 1 -δ, we haveEQUATIONwhere the first inequality follows from the triangle inequality, the second is a consequence of Hölder's inequality, and the final inequality results from Theorem 3.1. Further, we knowEQUATIONBy applying Hoeffding's inequality [65] , with probability of at least 1 -δ, we haveEQUATION)We complete the proof by combining Eq. ( 30), Eq. ( 31), and Eq. (33) .The second derivative of the function f (r) with respect to r isEQUATIONEQUATION)To ensure K(r(x)) is strongly convex [66] , we can simply assume λ ≥ 1.Our experiments are designed to align with established knowledge and intuition. We hypothesize that I-Div will indicate a minimal distribution discrepancy when the training and test distributions share semantically similar class labels, meaning that samples from both distributions can be treated as ID. This suggests the applicability of the hypothesis selected by an algorithm from the training samples to the test samples. Conversely, if class labels significantly differ in semantics, with samples from the training and test distributions being categorized as ID and OOD, respectively, we expect I-Div to reveal a more pronounced distribution discrepancy. This implies that the knowledge learned from the training samples may not be transferable to the test samples.In our study, we explore whether a hypothesis selected from a training dataset retains its capacity when applied to a test dataset. Our quantitative analysis utilizes the I-Div metric to measure the distribution discrepancy between training and test distributions, pertinent to the hypothesis applicability. Unless otherwise specified, we set λ = 1 and γ = 1. We hypothesize that data with closer semantic relationships will exhibit smaller distribution discrepancies, as opposed to those with distinct semantic differences.For the given training dataset P and test dataset Q, we consider using subsets of these datasets to estimate the distribution discrepancy instead of the entire datasets. This is because using too many samples can trivialize the estimation task and, realistically, obtaining the entire test dataset at once is impractical. Accordingly, we randomly draw samples from P and Q, creating a subset of three smaller datasets ( P , P , Q ). Adhering to the two-sample test framework [53, 67] , each subset consists of M N samples. We then form positive pairings ( P , P ) and negative pairings ( P , Q ). Unless otherwise noted, our experiments use a standard sample size of M = 1000. We generate 100, 000 tuples to achieve significant distribution discrepancy in the positive pairs and minimal discrepancy in the negative pairs. The effectiveness is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC) [68] and the Area Under the Precision-Recall Curve (AUPR) [69] , with higher values indicating more effective differentiation between pair types. To illustrate the effectiveness of our I-Div algorithm, we compare it with a selection of representative contrastive algorithms, each adapted to our specific task. All approaches quantitatively evaluate hypothesis applicability by measuring the distribution discrepancy between training and test distributions. These algorithms, which include Maximum over Softmax Probability (MSP) [59] , Non-Negative Bregman Divergence (NNBD) [58] , Maximum Mean Discrepancy with Deep kernels (MMD-D) [53] , and R-Div [47] , assess hypothesis applicability through the lens of distribution discrepancy. Fig. 1 (a) shows a steady decline in the classification performance of the standard network in the test dataset as the noise rate increases. This decline results from the loss of class label-relevant information in the samples as noises increase, hindering the ability of the hypothesis to predict class labels accurately. The expectation is that the distribution discrepancy between the training and test distributions will increase with added noises, suggesting a decrease of hypothesis applicability from the clean training dataset to the corrupted test dataset without accessing the ground truth labels of test samples. However, the results in Fig. 1 rate, its performance is not as consistent as that of I-Div, whose discrimination power more closely aligns with changes in the classification ability of the standard network. Fig. 1 (d) indicates that R-Div is extremely sensitive to data variation, achieving a complete separation of the two datasets at any noise level. R-Div can differentiate between clean and corrupted datasets even with minimal noises, but this does not indicate whether the hypothesis derived from training dataset is applicable to the test dataset.The comparison between Fig. 1 (b) and Fig. 1 (d) can be seen as an ablation study, since HDR is a component of I-Div. From this analysis, it is clear that I-Div outperforms HDR in determining the hypothesis applicability in the test dataset more accurately. HDR relies only on input data for estimating the input density ratio, a metric unrelated to specific model performance. In contrast, I-Div uses this density ratio to estimate the disparity of hypothesis performance between two datasets, offering a more accurate reflection of hypothesis applicability in the test dataset.As shown in Fig. 2 (a), the classification accuracy of a standard network on adversarial samples markedly diminishes with increasing perturbation magnitude. However, as depicted in Fig. 2 (d), I-Div maintains a consistently low AUROC, indicating its inability to differentiate between the training and test datasets, akin to human visual perception. This suggests that, even when the classification performance of the standard network is compromised, I-Div still perceives the knowledge from training dataset as transferable to the test dataset. This finding underscores the need to focus on improving the generalization of a standard network to adversarial samples, corroborating our empirical understanding and existing knowledge in the field. Fig. 2 (b) presents the AUROC of HDR, which remains high across various perturbation magnitudes, indicating its effectiveness in distinguishing original and adversarial samples. This observation might suggest that, while HDR is responsive to distribution changes induced by adversarial perturbations, it may not accurately reflect the hypothesis applicability from the training to the test datasets. Fig. 2(c ) shows the AUROC of R-Div, exhibiting a distinct pattern where the AUROC is relatively lower at the smallest perturbation but improves with larger perturbation. This indicates that R-Div could potentially be used to assess the hypothesis applicability to the test dataset. Nonetheless, in the context of adversarial samples, the results of I-Div imply that enhancing the hypothesis generalization to adversarial samples should be a priority. This conclusion aligns more closely with our intuitive understanding and is supported by numerous advanced research efforts in this area.We explore the influence of varying sample sizes, denoted as M , on the experimental results. This investigation is conducted under the same experimental setup as the one used for the semantically similar datasets. The findings illustrated in Fig. 3 indicate that for test datasets like CIFAR10.1 and STL10, I-Div consistently exhibits a relatively low AUROC. This pattern suggests that I-Div considers the hypothesis derived from the training dataset to be applicable to these test datasets, a conclusion further supported by the classification results in Table 3 . In contrast, for other test datasets with class labels semantically dissimilar to the training dataset, the knowledge learned by the model is deemed non-transferable. As shown in Fig. 3 , the ability of I-Div to distinguish these datasets, as quantified by AUROC and AUPR, improves with increasing sample size, eventually reaching a plateau.Given the hypothesis-oriented nature of the I-Div algorithm, which bases its analysis of hypothesis applicability from training to test datasets, it is pertinent to investigate the impact of varying network architectures on the performance. The findings of these experiments are summarized in Table 5 . The data reveal that, across all tested hypotheses, there are consistently low AUROC and AUPR scores for test samples with semantic similarities to the training datasets, such as CIFAR10.1 and STL10. However, a notable variance of performance is observed when examining test datasets with distinct semantic differences, like SVHN and DTD. This highlights the significant role of network architecture in determining a hypothesis capability to generalize and apply learned knowledge to novel, unseen dataset. The robust and consistent performance of I-Div across diverse network architectures further attests to the algorithm stability and broad applicability.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).The source code is publicly available at: https://github.com/Lawliet-zzl/I-div. 


Decision: This paper introduces a discrepancy measure to quantify the difference between two distributions, i.e., the labeled training set distribution and the unlabeled test set distribution. This discrepancy is derived from the expected risk difference between these distributions, built on a model pre-trained on the training data. The experimental results show the general applicability of the proposed method. The reviewers acknowledge the novelty and the effectiveness of the proposed method. I recommend it for acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a discrepancy to measure the difference between two distributions within a common scenario: the labeled training set distribution and the unlabeled test set distribution. This discrepancy arises from the expected risk difference between these two distributions, considering a model pre-trained on the training samples, and involves the estimation of both the density ratio and the likelihood ratio. Experiments were conducted on different types of data splits and data corruptions to demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. My primary concern revolves around the significant impact of the pre-trained classifier $\hat{h}$ on the proposed I-Div. After obtaining the pre-trained model from the training set, the estimated density ratio is predicted by appending a branch to the last layer of this classifier, and the subsequent Adaptive Likelihood Ratio is further estimated based on the predicted $\hat{r}(x)$. Therefore, I am concerned that inaccuracies in the initial estimation by $\hat{h}$ may amplify errors in the final discrepancy. (Although Theorem 3.2 provides a bound, which measures the upper limit of the distance within the same classifier.) I am unsure whether this could limit the practical effectiveness of the proposed I-Div.
   
2. The compared methods are not presented under consistent settings regarding label utilization and network parameterization. Aligning these settings could enhance clarity for comparison, although direct application to the scenario proposed by the authors might not be feasible.
questions: questions: 1. Could the authors provide a more detailed explanation of how Eq.16 is derived?
2. In Figure 3, what is the sample size of the data in the first column, and why does it perform better than the sample size of 500?
3. Considering my first concern in Cons, what would happen if we were to swap the training set and testing set in the Corrupted Data Experiment, or if the training set itself is inherently challenging?
4. Experiment 4.4 is intriguing; the proposed I-Div appears to be completely unaffected by adversarial samples. Could you provide a more insightful explanation?
5. It would be beneficial to analyze the sensitivity of hyperparameters （e.g. $\sigma$ and $\gamma$）
limitations: limitations: The proposed I-DIV, rooted in expectation risk, bears similarities to existing measures like R-Div. And due to the unknown labels of the test distribution, the authors propose to estimate the density ratio and likelihood ratio, which introduces uncontrollable errors, especially in complex distributions. Thus, I-DIV is affected not only by the estimation error of the likelihood ratio, as noted by the authors, but also the empirical minimizer, which is difficult to mitigate. I believe this limitation somewhat restricts the practical effectiveness of I-DIV.
paper_summary: paper_summary: This paper presents a novel approach called Importance Divergence (I-Div) to address the challenge of measuring the discrepancy between training and test distributions when test sample class labels are unavailable. I-Div transfers sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. The density ratio is obtained by minimizing the Kullback-Leibler divergence, while the likelihood ratio is adjusted to reduce the generalization error. Experimentally, I-Div accurately quantifies distribution discrepancy across a wide range of complex data scenarios and tasks.
strength_weakness: strength_weakness: 1. **Limitations of Likelihood Ratio Estimation**: As the authors mention, accurately estimating the likelihood ratio is challenging since test sample class labels are unavailable. The proposed adaptive adjustment only partially mitigates this issue and may not fully resolve it.

2. **Computational Complexity**: Estimating the density and likelihood ratios can be computationally expensive and potentially impractical for large datasets or complex models.

3. **Limited Scope of Experiments**: The experiments still focus primarily on simple datasets. Additional validation on other types of datasets is necessary to confirm the generalizability of I-Div.
questions: questions: This paper presents a methodological advancement in evaluating distribution discrepancies without test sample class labels. However, the fundamental challenge of accurately estimating the likelihood ratio remains unresolved, potentially compromising the method's accuracy. Additionally, the computational complexity may hinder practical application, especially with large or complex datasets.

1. Since the Likelihood Ratio significantly influences Importance Sampling, it greatly affects the quality of the samples. Consequently, this often results in high variance in the outcomes. I am curious about how the author addressed this stability issue during the implementation.
For example, Fig. 3 illustrates the effect of sample size but does not explain which factor is crucial to controlling the quality of the samples. And the size > 1000 means it requires high computational costs. So please elaborate on some more challenges on this matter.

2. What challenges might be faced in extreme cases, such as when the Adaptive Likelihood Ratio is large? While proving convergence is important, discussing practical implementation difficulties, such as when the test dataset size is relatively bigger or when many classes exist, would improve this paper.
limitations: limitations: I believe this paper does not have any negative societal impact.
paper_summary: paper_summary: This paper investigates the applicability of hypotheses derived from training datasets to distinctly different test datasets through a series of experiments using various datasets, including CIFAR10, SVHN, PACS, and Office-Home. T
strength_weakness: strength_weakness: Outdated Baselines: The baseline comparison, particularly using MMD-D from 2020, is quite outdated. This diminishes the paper's novelty and effectiveness compared to state-of-the-art methods. To enhance the paper's relevance and credibility, it would be beneficial to include more recent benchmark algorithms, preferably those introduced after 2021, or popular models like ViT (Vision Transformer) and CLIP (Contrastive Language-Image Pre-training), which have shown significant impact in computer vision.


Backbone and Feature Encoder: Indeed, the choice of backbone in the paper does not align well with recent developments in computer vision. Modern visual tasks increasingly rely on Transformer-based architectures such as ViT and CLIP, which excel in handling interactions between visual and language modalities. Introducing these modern feature encoders in the experiments would allow the algorithm to be tested on more complex and diverse datasets, better reflecting its generalization ability and practical utility.


Performance of I-DIV in Table 1: I concern about I-DIV achieving 100% performance in Table 1 is valid. In practical applications, achieving 100% accuracy is nearly impossible, especially on complex datasets and real-world scenarios. This might suggest some idealization or overfitting in the experimental setup, particularly in how the model or algorithm is tailored to specific datasets or test conditions. Therefore, it's advisable to provide a more detailed explanation of these experimental results in the paper, discussing their applicability and generalization capability in real-world environments.
questions: questions: see weaknesses
limitations: limitations: see weaknesses
paper_summary: paper_summary: The paper proposes a novel concept called Importance Divergence (I-Div) to address the lack of labeling of test samples and to measure the difference between training and testing distributions. Through the computation of importance samples, density ratios, and likelihood ratios, I-Div is able to assess the applicability of hypotheses on different datasets without the need for test labels. The main contribution of this method is that it provides a new way to quantify and reduce the expected risk difference between training and testing distributions, thus enhancing the generalization capability of machine learning models in the face of unknown test data.
strength_weakness: strength_weakness: Since this method relies on accurate estimation of the density ratio and likelihood ratio, this may limit the general feasibility of the method in practical applications.
questions: questions: It is recommended that in future research, more simplified or optimized algorithmic processes be explored to reduce the computational cost and improve the practical feasibility of the method.
limitations: limitations: yes

Comments: title: Thank you for raising the score
Corrected Results on Noisy Data
Response to Reviewer yuQt
Thank You for Increasing the Score
comment: We sincerely appreciate the insightful experimental suggestion you provided, which has led us to explore more intriguing research questions and inspired our ongoing research. We have incorporated the complete experiment in the revised version and will continue to refine the paper.

If you have any further concerns or suggestions, we would be eager to discuss them.
I understand your point. What truly impacts the method's performance is actually the classifier's generalization ability. A classifier that performs less well on the training set might somehow perform better with better generalization. This makes sense to me. Therefore, I am willing to raise my score to 6. Additionally, I believe that the discussion about Q3 is important for this method, so please include more detailed experiments and analysis in the revised version.
**Noisy Data and Q3:**

Thank you for your observation. We realized there was a small mistake in our previous experiment, which we have now corrected.

As you can see, the corrected results show that when the noise level is low, the model maintains strong generalization ability, making it difficult to distinguish between clean and noisy data. However, as the noise level increases, the model ability to generalize decreases, and the distinction between clean and noisy data becomes more evident. This allows our algorithm to more effectively differentiate between the two distributions.

We hope this resolves your concern, and if you have any further questions, please don’t hesitate to ask.

|Train|0.1|0.3|0.5|0.7|0.9|
|-------|-------|-------|-------|-------|-------|
|Uniform|62.3|89.8|100.0|100.0|100.0|
|Salt & Pepper|76.9| 100.0|100.0|100.0|100.0|
Thank you for your insightful questions. We have carefully considered your comments and would like to provide the following responses:

**Noisy Data:** 

There is indeed a turning point as you suggested. As shown in the table below, this turning point appears at a higher noise rate, such as 0.8.

We previously only presented results for the 0.3 to 0.7 noise range to highlight this interesting finding. In this range, it seems that our algorithm is not very effective at distinguishing between the training and test samples. However, this aligns with our intuition because even with these noise levels, the model still maintains a certain level of generalization to the test data. Specifically, since we added noise to the original data, this can be considered a form of data augmentation, where a certain amount of noise might even enhance generalization. The proposed I-Div relies on a pre-trained classification, which is closely tied to the model generalization ability.

When the noise level is significantly higher, such as at 0.8, the situation changes. At this point, the noise overwhelms the original data, making it difficult for the model to generalize. The noise introduces large variations that no longer match the true data distribution, leading to a clear separation between the training and test sets.

We are truly grateful for you pointing out this interesting aspect. We will continue to delve into the relationship between model generalization and data discrepancy in future research, especially in the context of noisy data.

| Train|0.3|0.5|0.7|0.8|
|-------|-------|-------|-------|-------|
|Uniform|57.9|53.5|48.6|100.0|
|Salt & Pepper|50.7| 48.7|42.1|100.0|

**Error Propagation:** 

$\widehat{r}$ and $\widehat{v}$ do indeed rely on the pre-trained classifier. I-Div is designed to measure the difference between training and test data for any given pre-trained network, which in turn helps assess its applicability on test data. This does not involve the type of error propagation typically seen in non-end-to-end training. If I-Div shows a significant discrepancy between two semantically similar datasets when applied to a given pre-trained classifier, this is not due to error propagation. Rather, it indicates the weak generalization ability of the classifier itself, meaning the classifier struggles to generalize well between those two datasets.

To further clarify, you can think of $\widehat{r}$ and $\widehat{v}$ as tools for analyzing the sensitivity of a given network to different data, similar to using a detector to identify out-of-distribution samples for a specific network. Therefore, the nature of the pre-trained classifier, whether it is highly accurate, somewhat flawed, or even based on randomly generated data, does not affect the validity of our approach. I-Div effectively evaluates the model generalization ability regardless of the specific characteristics of the pre-trained classifier. Even with a classifier trained on random data, our method can still measure how well the model generalizes across different datasets.

That said, in our experiments, we used a network trained on the training dataset. While this network generalization ability may not be exceptionally strong, it does possess some degree of generalization, allowing us to measure the differences between its training and test datasets. Naturally, different training methods and network architectures will result in different pre-trained classifiers due to varying levels of generalization ability. Consequently, this will also lead to different data discrepancy measurement results, as demonstrated in our Table 3.

**Adversarial Samples:** 

The robustness of I-Div compared to R-Div is not simply due to differences in label settings. The following table highlights that the additional experiments provide further clarification on your point

Regardless of whether labels are present, many current methods, including R-Div, tend to maximize the differences between data (based on our experimental observations). These methods assume that any difference in data implies different distributions. This behavior persists even in labeled settings, where the methods detect even the slightest variations in input data and conclude they belong to different distributions.

However, I-Div does not fall into the trap of detecting these minor differences. Instead, it evaluates data discrepancies based on the model generalization ability. When given adversarial samples, I-Div focuses on the overall information that the model considers for generalization. If the model deems it should generalize to these samples, the perceived difference between the datasets becomes smaller.

| Method |Adversarial(0.01)|
|------------|-------|
|R-Div w/o labels |100.0|
|R-Div w/ labels |100.0|
|I-Div|38.7|


Thank you for your feedback. If you have any more questions or concerns, feel free to reach out.
Thank you very much for the response. I still have some questions regarding using noisy labels as the training set and clean labels as the test set. Can we assume that when the noise rate is small, they should belong to the same distribution, but as the noise gradually increases, they should be considered as two different distributions, which is consistent with the results in Figure 1(d). On the other hand, when noisy labels are used as the training set, the classification accuracy will decrease as the noise rate increases, but the distance between the two distributions is also increasing. Therefore, there should be some kind of turning point in the results (I am not sure).

And what I want to express in W1 is that whether it's your estimated $\hat{r}(x)$ or $\hat{v}(x, y)$, they both depend on your Pre-trained Classifier. If the Pre-trained Classifier has errors, will these errors be propagated to the estimation of $\hat{r}$ or $\hat{v}$, eventually leading to a significant discrepancy between the measured discrepancy and the actual value?

Additionally, regarding Experiment 4.4, if I-Div is robust against adversarial attacks merely because it fails to detect the existence of adversarial samples, can we understand that I-Div appears more robust compared to R-Div simply due to the difference in label settings, rather than the method itself?
Thank you very much for your thoughtful response and for taking the time to thoroughly review our answers. We appreciate your acknowledgment of our methodology and your decision to increase the score. Regarding your suggestions:

**Code Availability:** We fully agree with the importance of code transparency. If the paper is accepted, we are committed to releasing the code used in all experiments to ensure transparency and facilitate further research in this area.

**Scope of Experiments:** We understand your concern about the scope of the experiments. In this revised version, in addition to addressing your concerns with expanded experiments, we have also conducted further experiments on more complex datasets (ImageNet, OID, Places365, StanfordCars, Caltech101, and DTD) and models (ResNet50, ViT, and CLIP) to validate the effectiveness of our algorithm. Detailed results can be found in Figure 1 of the PDF provided in the global response. These new experiments have also been incorporated into the revised version of the paper.

If you have any further suggestions or require additional experiments, we are more than happy to incorporate them. Thank you again for your constructive feedback and for your consideration.
Additionally, I suggest the authors make the code used in all experiments publicly available (if accepted).
Thank you very much for the response. I have thoroughly reviewed the answers, and most of my concerns have been well-addressed. Although the scope of the experiments presented in the paper still is somewhat limited, I would like to give more credit to the methodology proposed in the paper. Therefore, I increase the score one step higher.