Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data
Authors: Zhilin Zhao
Longbing Cao
Xuhui Fan
Wei-Shi Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Non-IID Data; Distribution Discrepancy; Density Ratio; Likelihood Ratio; Generalization
Primary area: evaluation
TL;DR: This paper introduces a method to evaluate distribution discrepancies between training and test distributions without needing class labels in the test samples.

Abstract: There are increasing cases where the class labels of test samples are unavailable, creating a significant need and challenge in measuring the discrepancy between training and test distributions. This distribution discrepancy complicates the assessment of whether the hypothesis selected by an algorithm on training samples remains applicable to test samples. We present a novel approach called Importance Divergence (I-Div) to address the challenge of test label unavailability, enabling distribution discrepancy evaluation using only training samples. I-Div transfers the sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error of the distribution discrepancy as transformed through the two ratios. Experimentally, I-Div accurately quantifies the distribution discrepancy, as evidenced by a wide range of complex data scenarios and tasks.

Revealing Distribution Discrepancy by Sampling Transfer in Unlabeled Data
Zhilin Zhao 2.2
Longbing Cao
Xuhui Fan'
Wei-Shi Zheng2.3+
School of Computing, Macquarie University, Australia 2 School of Computer Science and Engineering, Sun Yat-sen University, China 3 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China zhaozh170he /0totmaoltalll com, (longbing cao, xuhui. .an @mq. edu. au, wszhengdieee org
Abstract
There increasing cases where the class labels of test samples unavailable, creating a significant need and challenge in measuring the discrepancy between training and test distributions. This distribution discrepancy complicates the as- sessment of whether the hypothesis selected by an algorithm on training samples remains applicable to test samples. We present novel approach called Importance Divergence (I-Div) to address the challenge of test label unavailability, enabling distribution discrepancy evaluation using only training samples. I-Div transfers the sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. Specifically, the density ratio, informed by the select- ed hypothesis, is obtained by minimizing the Kullback-Leibler divergence between the actual and estimated input distributions. Simultaneously. the likelihood ratio is adjusted according to the density ratio by reducing the generalization error of the distribution discrepancy as transformed through the two ratios. Experimentally, I-Div accurately quantifies the distribution discrepancy, evidenced by wide range of complex data scenarios and tasks.
Introduction
The assumption that data are independently and identically distributed (IID) is staple in statistical machine learning. It suggests that hypothesis selected by an algorithm, after observing several training samples, should perform effectively on test samples from the same unknown distribution. However, this assumption often oversimplifies the intricate and diverse nature of weal-world data, particularly in non-lll) scenarios 1 Thus, if training samples are considered in-distribution (ID), there is a risk that test samples may deviate from this distribution, characterized as out-of- distriibtntnonn (OOD) B. This distribution discrepancy between training and test distributions poses critical and challenging non-IlD learning (2] question: How to quantify the applicability of hypothesis derived from training samples to test samples?. This question is generally applicable to areas like
OOD generalization 4 OOD detection [5] 6), domain adaptation 7 8 transfer learning [9]
semi-supervised learning 10, robust learning
and adversarial learning 11
The applicability of a hypothesis can be determined by the distribution discrepancy between training and test distributions. When the two distributions align, meaning both training and test samples are II), the hypothesis derived from training samples can be seamlessly applied to test samples. However, reality, particularly when test samples fall OOD, this scenario rarely holds. Under such circumstances, decisions must be based on the extent of the distribution discrepancy. This may require enhancing the hypothesis generalization capability towards test samples or rejecting these
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

samples outright. These actions are fundamental to the principles of OOD generalization and OOD
detection, respectively [3]
Evaluating the distribution discrepancy between training and test distributions for a selected hy- pothesis presents a significant challenge, training samples are typically labeled, whereas test samples often are not [14] This discrepancy means that conventional statistical distances, such as F-divergences (15). integral probability metries [16], and total variation (17), are not suitable for this situation. Furthermore, density ratio methods [18 19] offer a partial solution by disregarding
label information and estimating the ratio between input distributions. Moreover, more performance prediction techniques [20] aim to navigate this challenge by examining the uncertain outcomes of the hypothesis, potentially leading to unreliable results. A detailed discussion of related work is provided in Appendix
To tackle the issue of unavailable test labels, we introduce the concept of Importance Divergence (I-Div), which measures the training-test distribution discrepancy w.r.t. the difference between the expected risks of the selected hypothesis on training and test distributions. To estimate the expected risk on test distributions without label access, the core strategy involves importance sampling to transfer the sampling patterns from the test distribution to the training distribution. This process requires the estimation of density and likelihood ratios. Specifically, the density ratio, informed by the selected hypothesis, is obtained by minimizing the Kullback-Leible divergence between the actual and estimated input distributions. Simultaneously, the likelihood ratio is adjusted according to the density ratio by reducing the generalization error related to the distribution discrepancy as transformed through the two ratios. As result, I-Div leverages the estimated density and likelihood ratios to quantitatively measure the distribution discrepancy between training and test distributions, eliminating the need for test class labels, and thus quantifying the applicability of the hypothesis across different datasets.
2 Preliminaries
Let X' denote the input space, and Y' represent the set of labels. The joint probability distributions are represented by P for training samples and Q for test samples. Assume we observe a labeled training dataset P and a unlabeled test dataset Q containing N IID samples from P and Q respectively. 2x denotes the marginal distribution of Q over X. / and ② are described as
(1) where the label space )' encompasses all labels of samples from both P and Q. We also define H. as the hypothesis space and -(.,)) E (0, Bc) as the bounded loss function. The expected and empirical risks [21 for hypothesis h E H X + y on distribution P
p = ((x; ))) HD P. Ō= (x Qx:
and the dataset P are defined as (h)
and éΦ (h). respectively, i.e.
(h) = Ep (Ep[S(((x),u) = 1. I,, 2(h(x),)) P(())))))()))))) y) dx dy.
(h) [=)()((x).))) PP
Σ 2(h(x),)).
(X,D)E
where E and È represent the expectation with respect to data distribution and the sample average over dataset, respectively. Accordingly, an algorithm A aims to select the empirical risk minimizer his after observing the samples from the training dataset P by
h. E arg min (p)), hEH
(3)
to approximate the optimal hypothesis hip selected from the distribution P through hip E arg minkeN (h).
Our central research question is formulated as follows: How can we quantify the applicability of the minimizer lug, originated from the training sumples P, to the unlabeled test sumples Q To address this question quantitatively, we delve into methodologies for assessing the distribution discrepancy between training and test distributions, particularly focusing on doing so without the need to access class labels from test samples.

---

3 Importance divergence
For hypothesis chosen by an algorithm after observing training samples, to assess its applicability to test samples in the absence of ground truth labels, we introduce the concept of Importance Divergence (I-Div). I-Div estimates the distribution discrepancy between training and test distributions for the given hypothesis. To address the issue of unavailable ground truth labels, it leverages importance sampling, density ratios, and likelihood ratios, facilitating the sampling transfer in the test distribution back to the training distribution.
In this section, first present the distribution discrepancy with importance sampling, which transfers the data sampling from the test distribution to the training distribution. Following this, we discuss the methodologies for estimating the appotheiis-oiientee density and adaptive likelihood ratios, essential components of this discrepancy, to facilitate this sampling transfer. The hypothesis-orrentte density ratio is specifically tailored to each hypothesis, it assesses the suitability of specific hypothesis based on the resulting distribution discrepancy. The adaptive likelihood ratio is adjusted according to the density ratio expedite the convergence of the distribution discrepancy. Lastly, we utilize an empirical estimator of the distribution discrepancy to evaluate the applicability of a hypothesis selected from training samples to test samples.
3.1 Distribution discrepancy with importance sampling
For the hypothesis h.g. I-Div evaluates the distribution discrepancy between training and test distribu- tions without requiring the ground truth labels of the test samples. A smaller discrepancy implies that the training and test samples could be considered as drawn from the same distribution with respect to the given hypothesis, and vice versa. By using variational divergence, we can express this discrepancy as the difference between the expected risks of the hypothesis on training and test distributions by
d(P, Q hT)) = (hpp) <c(h))
(4)
If the training and test distributions are aligned, i.e., P = 2, then the expected risks for the hypothesis h are similar, resulting in minimal distribution discrepancy. This condition suggests that the test samples are likely ID for the hypothesis Peg. Conversely, a notable difference between P and Q indicates a larger distribution discrepancy, implying that the hypothesis he perceives the training and test samples as originated from distinct distributions, thus categorizing the test samples as OOD,
To make the expected risk difference in Eq. 4 more pronounced, I-Div employs Jensen's 's inequality to consider its upper bound by
d(p, 2 hup) S
(xx v) e
(hs(x)))) dx dy.
(5)
thereby more distinctly highlighting the differences between the training and test distributions.
Recall that class labels test samples are inaccessible, thus, the principal challenge is evaluating the expected risk for the test distribution with respective to the given hypothesis without access to ground truth labels. Since direct sampling from the test distribution not feasible, an alternative sampling from the training distribution. To overcome this limitation, sampling technique (22), converting the data sampling from test to training distributions, i.c.,
employ the importance
Q(x,y) 2(x, y) v) P(x, y)
2(x)
x)
y)
P(x)
P(y
x
r(x)
<(xy)
where r and v denote a density ratio and a likelihood ratio, respectively. Thus, I-Div estimates distribution discrepancy between training and test distributions by merely sampling from the training one without accessing the class labels of the test samples. According to Eq. (5) and Eq. (6). we have
d (P. 2 r.p.r.) A Ep [l(())((xx v) 1 (h (x)(x)))))
Accordingly, for given labeled training dataset P. and unlabeled test dataset Q, we then construct an empirical estimator for estimating the distribution discrepancy in Eq. (7) by
ā (P, @ Rip.r. r. A E.p [Ir (x) v) 20h (x).
(8)
3

---

However, to estimate the distribution discrepancy I-Div, the emerging challenge involves determining the density ratio and likelihood ratio y from observed samples, which are discussed in Section 3.2 and Section 3.3 respectively.
3.2 Hypothesii-orriented density ratio
The density ratio in the distribution diserepancy should be hypothesis-oriented. That is, it should depend on the specific hypothesis hig E 'Hl selected by an algorithm on P. This is because the criteria for judging the discrepancy between distributions P and Q vary across different algorithms. For instance, whether two datasets of cats and dogs respectively come from the same distribution depends on whether the algorithm aims to identify if the subjects are biological entities or to distinguish between these two species. Accordingly, we apply deep neural network model density ratio r from the hypothesis space RR(g)) depending on the specific hypothesis hip- Then, we select density ratio r E R(hs)) minimize the Kullback-Leibler divergence between the actual distribution and the estimated distribution based on this density ratio.
To construct the hypothesis space Rhhs), we utilize the output representations from hip with a learnable component w model a density ratio r. Specifically, we decompose the hypothesis into
backhone and softmax layer Φ represented as lx5(x) = (6b v)) (x), where the output dimension
of the backbone is Op- Using learnable component w. is constructed as r(x) (w o +b) (x), where the learnable component w(.) = (Wsp (.) contains an adapter WAD (33 to introduce learnable parameters and Softplus layer wsp [24] to ensure strictly positive outputs. Specifically, the
adapter WAD, which follows the bottleneck ve, comprises two fully connected layers with a Gaussian Error Linear Units (GELU) [25] activation layer in between. Furthermore, the Softplus layer is adopted to the adapter output, effectively mapping it to the range (0, too]. Furthermore, weight matrices W1 W1 E Ro, x0. and W₂ W2 E Ro,x00 in the fully connected layers. The activation functions GELU and Softplus are 31- and 32-Lipschitz, respectively. Additionally, assume Ti supw,ew, || oo and 72 supw,ev, |2 Thus, the density ratio r E RR(ss) can be modeled as
Σ (WAD o {b) (x); 1):
r(x) (wsp WAD o v) (x) = In c |0
(56))
iE(Ow]
where Ow, represents the output dimensionality of the adapter. Without loss of generality, we further assume that r(x) E (b,, B,) for any E Rhp)) and x E X.
To select density ratio e R(hp), we use estimate the density P and Q by
P(x) ((x)/r(x) =(x)= = r(x).
(10)
Drawing on the inspiration of importance estimation methods [26] (2). we construct objectives and constraints around P(x) and Q(x). The estimated probability distributions are designed to approximate their actual counterparts, suggesting the minimization of the two KL divergences with normalization constraints
KI. (P(x) 11 KI. (2(x) @(x))
min (ER(sp)
(11)
dx /
/ (((x)
dx 1.
s.t.
For convenience, we assume / = P/2 2/2 and define C = {1, 11 as labels for training and test samples, respectively. label E C. corresponding to a sample from U. indicates its distribution origin. The assignment = 1 indicates that sample originates from distribution P. while -1 signifies that a sample comes from distribution Q. We can then obtain the following objective function for learning the density ratio
f(r)
E (x,e=~ll (clor(xx) ( -113]
(12)
where À 2 balances the KI divergence and normalization constraints, as the detailed derivation We further assume that f(r) is -ipischh continuous with respect to
shown in Appendix B

---

r E R(hØ). This objective function Eq. 12) can be estimated by
f(r) E (x.c)~u clogr(x) + A (x))* 1112]
(13)
where 77 Ō. An empirical risk minimizer is selected by
f E arg min f(r), rER(hg)
(14)
which aims approximate the population risk minimizer r arg min (ER(())))))) f(r). The convergence rate f can be guaranteed by the following theorem.
Theorem 3.1. Let À 2 1 and μ be constant related the function f. With probability of least 1 1-6, 6,
Ep -xx)) 63L +I+ B, 1n/0/ 832T2Ls B)6 UN
NVN
The presence of N in the denominators of both terms suggests that the bound tightens with an increasing sample size, which aligns with the general understanding that more data can lead to more accurate estimates in statistical learning.
3.3 Adaptive likelihood ratio
Without making further assumptions, it is infeasible to estimate the likelihood ratio e due to its dependence on the unknown joint distribution Q(x, y). Instead of pursuing the true likelihood ratio, our goal is to approximate an adaptive likelihood ratio v E VFF) that enables a swift convergence of the distribution discrepancy, guided by the hypothesis-oriented density ratio 7 E RR(s)). This strategy valid since the density ratio captures the input distribution discrepancy between training and test distributions. It indicates that utilizing even basic form of the covariate shift assumption, i.e. e(x, 3/ for all (x, y) ~ P. allows the distribution discrepancy, as calculated by Eq. (8). to approximate the difference between distributions
reasonable degree. Furthermore, since the density ratio serves primarily to gauge the distribution discrepancy, the corresponding likelihood ratio must be specifically adapted to this density ratio for precisely assessing the distribution discrepancy.
Accordingly, we reveal the generalization error bound of the distribution discrepancy, leveraging both density and likelihood ratios, through the convergence rate of the aypothesis-orientee density ratio, Rademacher complexity (28) and Talagrand's contraction lemmas [29], which is shown as follows.
Theorem 3.2. Based on the conditions and results outlined in Theorem with a probability of at 6, (P, ( least
(P, hep v. is bounded by
In(2/8) E(EW) ((x))((x)))))))))))()))))) 9) 112 + BsE (u(x,)),
B(8/2,N)
Bey
The result shows that the estimated distribution discrepancy converges quickly with increasing the sample size. Additionally, the bound is associated with the values of the likelihood ratio. Moreover, as per Eq. 66 for any r E R(hp), have
ʃ [(x,x) dx = Ep (x/rr())(xx v)] = 1.
(15)
Considering the terms in Theorem have
3.2 related to and the average error over the samples from P.
min Es [( v) + 2 (F(x))((x v) - 1)²].
(16)
s.t. E ((x))xx( v)] =
where 7 > 0 acts as regularization parameter, influencing the trade-off, and the optimal solution is By using an proximal algorithm [30], we can obtain the following approximate solution
o(x, y) = [x) - 1) 7((()))
)
E Exx
(17)
5

---

We observe that in the two boundary cases where - 0 and - +oo, consistently equals 1, thus adhering to the covariate shift assumption. Even under these extreme conditions. feasible to calculate the distribution discrepancy using both the hypothesis-orienteed density ratio and adaptive likelihood ratio to overcome the challenge unlabelled test samples. This approach is viable because the sypothesis-orientedd density ratio F quantifies the distribution differences without relying on class labels, while employing 7 E (0, +oo) utilizes the class labels of training samples. Although this method may not precisely determine the likelihood ratio for each instance, it is designed in accordance with hypothesis-oriented density ratio such that the estimated distribution discrepancy aligns with the actual value, fulfilling primary objective.
3.4 Hypothesis applicability evaluation
I-Div quantifies the applicability of the hypothesis h, selected by the algorithm A from the training dataset / to the test dataset ②. Specifically, 1-Div employs the empirical estimator in Eq. with hypothesis-oriennted density ratio f' in Eq. (I4), and adaptive likelihood ratio in Eq. estimate the distribution between training and test distributions by
Esp [xx) - 2 (x),
(18)
where F is chosen from the hypothesis space Rh₂z) based on hgg and is selected from the space V(FF) based on F. A smaller discrepancy indicates that the training and test samples likely drawn from the same distribution relative to h.p. Since he minimizes the empirical risk on P. a reduced distribution discrepancy improves the transferability of the hypothesis from training to test samples. On the other hand, a greater discrepancy suggests a reduced likelihood of hypothesis applicability. The I-Div methodology is detailed in Algorithm I
Algorithm 1 Importance divergence
I: Input:
2: Training samples P Vi, 1) ~p
Vi
3: -Test samples @ = [xx; Gi = - -1]] º
4: Empirical minimizer hip. Hyperparameters À and γ
5: Merge datasets: a - P Q
6: Estimate the hypothesis-oriented density ratio on ú7:
Bxoa) [clogr(x) + X + (x)) - 112]
F arg min (E)()))
7: Estimate the adaptive likelihood ratio on P:
6xx, y) = (N~f(x) - N) / (x(((x))) EEER [(>(((x))) (xF(x) - 1)]
8: Estimate the distribution discrepancy with importance sampling P:
d p θ h.p p, p, Es [(x())((x))) 21
9: Output: empirical estimator d (P. P.T)
4 Experimental results
This section presents a comparative analysis of I-Diy P against existing methods for evaluating the distribution discrepancy between training and test samples. The detailed experimental setups presented in Appendix D.I]
'The source code is publicly available httpss///ggtttooccm/////l/ez////

---

Table 1: Distribution discrepancy of different classes in CIFARIO. The larger the values of AUROC and AUPR, the better the performance.
DATASET
TARGET
MSP
NNBD
MMD-D
R-DIV
1-Drv
AUTRC:
AUROC
AUROC
100.0
100.0
93.1
97.4
100.9
100.0
100.0
AUTOMOBILE
100.0
96.2
100.0
100.0
100.00
100.0
BIRD
100.0
100.0
90.4
90.0
97.2
97.6
100.0
100.0
100.0
100.0
CAT
100.0
100.0
94.0
93.9
86.9
57.9
100.0
101.0
100.0
100.0
DEEER
100.0
100.0
100.0
100.0
100.0
100.0
Dou
100.0
100.0
100.0
100.0
Faon
100.0
100.0
91.7
91.6
96.0
96.5
100.0
100.9
100.0
100.0
HORSE:
100.0
100.0
91.9
91.8
82.8
83.4
100.0
100.0
100,0
160.0
TRUCK
100.0
96.9
96.7
100.0
100.0
100.0
100.0
4.1 Experiments on different classes
Our initial experiments focus relatively straightforward task: assessing the applicability of hypothesis obtained on the training dataset the test dataset with distinctly different class labels. We utilize two datasets, CIFARIO 10 [1 and SVHN 32,, each comprising ten semantically unique classes. For our experiments, we select samples from one class to serve as the test dataset, with the samples from the remaining nine classes forming the training dataset. This setup clearly illustrates that the knowledge learned in the training dataset cannot be transferred to test dataset.
The results for CIFARIO and SVHN are detailed in Table and Table 6 (Appendix D.2), respectively, Our proposed I-Div algorithm consistently achieves perfect scores (100%) in both AUROC and AUPR metrics across all classes of both datasets. This demonstrates its exceptional capability in distinguishing between training and test datasets, aligned with our initial hypothesis. The results unequivocally support the premise that the knowledge transfer from the training the test datasets is ineffective, evidenced by the flawless performance of I-Div. This starkly contrasts with the varying effectiveness of other algorithms, including NNBD and MMD-D. Notably. I-Div, MSP, and R-Div all yielded similarly impressive results. key commonality among these algorithms is their reliance on specific hypothesis to calculate distribution discrepancy, as opposed to NNBD and MMD-D, which use independent hypotheses. This highlights the significance of considering a particular hypothesis when evaluating distribution discrepancies. The rationale is that the hypothesis applicability depends on the specific design and its intended task.
4.2 Experiments on different datasets
We now turn to a more complex scenario where the training and test datasets may share semantic similar- ities in class labels, indicat- ing an overlap in the class label spaces. In cases where semantics differ significant- ly, we expect the algorithm clearly differentiate the two kinds of samples. Con- versely, if their semantics are similar, the algorithm may find challenging distinguish them. This outcome would suggest that the knowledge acquired from the training dataset
Table 2: Distribution discrepancy of domain adaptation data.
AUROC
DATASET
SOURCE
ACC
P
94.7
100.0
99.4
95.8
100.0
39.7
77.4
98.1
74.3
100 0
98.2
95.2
78.9
100.0
99.6
94.6
100.0
49.5
76.1
OFRer-HoME
58.6
48.5
100.0
100 0
98.7
98.8
95.8
96.5
100.0
100.0
51.4
48.0
74.1
95.8
100.0
49.5
transferable to test dataset, or it may indicate potential pathways to enhance hypothesis generalization for the test distribution. We leverage CLIP R3 to align class labels between the training and test datasets, using the prompt template "A photo of a (label)." This helps adapt class labels across domains and captures semantic relationships between different datasets.
7

---

We conduct experiments using classic domain adaptation datasets: PACS [4 and Offece-Home [35] each containing four domains. We designate one domain as the training dataset and merge the remaining three as the test dataset. The results, presented in Table 2 indicate that each hypothesis selected from dataset performs significantly better than randomly selected hypotheses, demonstrating its applicability. Our I-Div algorithm aptly reflects this, in contrast to other algorithms that overly emphasize distribution discrepancies, thereby rigidly categorizing the difference between training and test datasets.
Table 3: Distribution discrepancy of different datasets on ResNet18. For CIFARIO.1 and STL10, smaller values of AUROC and AUPR indicate better performance. However, for other test datasets, larger values are better.
TARGET
ACC (CLIP)
MSP
NNBD
MMD-D
R-DIv
I-DIV
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AURRO
AIIPR
AUROC
A1IPR
100.0
100.0
99.2
99.3
SYHN
17.0
100.0
100.0
100.0
100.0
94.8
95.7
100.0
100.0
100.0
100.0
DID
1.9
100.0
100.0
100.0
100.0
100.0
160.0
100.0
100.0
100.0
100.0
FLOWERS102
1.6
100.0
100.0
100.0
100.0
97.0
98.8
100.0
100.0
100.0
100.0
2.3
100.0
100.0
100.0
98.7
103.0
SEMEION
8.7
100.0
100.0
100.0
100.0
98.5
99.0
100.0
100.0
100.0
100.0
CAUTEC1256
2.4
100.0
100.0
100.0
101.0
92.3
92.4
100.0
100.0
99.9
99.9
CIFAR1O0
2.2
100.0
100.0
CIFARIO.1
73.4
100.0
92.1
93.7
92.9
93.1
STI,10
63.0
100.0
100.0
94.0
93.3
90.4
91.1
100.0
100.0
37.2
41.9
We use CIFAR10 31
as the training dataset and evaluate diverse test datasets including Ran- domly Generated Images (RGI), SVHN [32], DTD [36]. Flowers 102 (37], OxfordIlITPet [38], SEMEION (39, Caltech256 44).. CIFARI00 31,, CIFARIO.1 [1], and STL10. Since 1IFAR10.1 and 10 share similar category spaces with CIFARIO, the model shows minimal differentiation for these datasets, with ACC values of 73.4% and 42.1% 1%, respectively. This suggests partial knowledge transferability. However, I-Div demonstrates lower AUROC values of 43.7% and 37. 37,2% indicating reduced discrimination. Other algorithms like MSP and R-Div show higher AUROC values, near 100% across all datasets, but these results suggest an overemphasis on distribution discrepancy rather than semantic similarity. I-Div, in contrast, better captures semantic relationships between datasets, providing more nuanced view of class label semantics.
Table 4: Distribution discrepancy between ImageNet and other test datasets.
TRAINING
NETWORK
TEST
ACC (CLIP)
MSP
NNBD
MMD-D
H-DIV
R-DIV
1-Div
O1Dv4
43.9
100.0
91.7
94.6
100.0
94.6
69.3
RESNET50
CALTECU1256
36.6
100.0
91.4
95.6
100.0
100.0
72.4
FLOWERS102
5.1
100.0
98.6
100.0
100.0
IMAGENET
DTD
11.9
100.0
98.7
100.0
100.0
100.0
100.0
OIDv4
50.6
100.0
88.6
92.6
100.0
92.6
62.6
B/16
CALTECH256
40.4
100.0
94.8
100.0
100.0
100.0
71.9
FLOWERS102
5.1
100.0
98.1
100.0
100.0
100.0
100.0
DTD
13.9
100.0
99.7
100.0
100.0
100.0
100.0
We use ImageNet 42
as the training dataset and evaluate on diverse test datasets using ResNet50
43
and ViT-B/16 44 The test datasets include the Open Images Dataset v4 (OIDv4) 45
Cal-
tech256 [40]. Flowers 102
337. and DTD 36). The experimental results presented in Table 4 show
that the AUROC values of I-Div effectively capture the semantic similarity between ImageNet and the test datasets, yielding results that closely align with human intuition. For example, I-Div demonstrates lower AUROC values for OID and Caltech256, reflecting their semantic overlap with ImageNet, as these datasets share common object categories and scene types. In contrast, datasets such as Flowers 102 and DTD, which focus on more specialized object categories and textures, show higher AUROC values with I-Div, indicating greater divergence from ImageNet. On the other hand, algorithms like MSP, NNBD, and MMD-D show consistently high AUROC values across most datasets, implying they emphasize distribution discrepancies over semantic relationships. This limits their effectiveness distinguishing nuanced semantic differences compared to I-Div, which provides more human-alie -aiined understanding of dataset relationships.
8

---

Loafeer
(a) Classification
(b) Discrepancy: HDR (c) Discrepancy: R-Div (d) Discrepancy: I-Div
Figure 1: Distribution discrepancy between original data and its corrupted variants with different noise rate. (a) shows the classification performance of the standard network for the test datasets containing corrupted samples. (b)(c)(d present the distribution discrepancy in terms of AUROC.
100
33
00
90
MEDOC
02
AUROC
8
23
22
20
0.003 3.01
0.
3001 D0L 03
0.3
0001 0.33 33
0,011 0.01
8.1 0.3
Petlaibarion Magrlude
Partabasion Magnitade
Persabation Magriaale
Partu Magritude
(a) Standard Network
(b) HDR
(c) R-Div
(d) I-Div
Figure 2: Distribution discrepancy between original data and adversarial data.
4.3 Experiments corrupted data
This section discusses experimental results on corrupted datasets. We progressively introduce noise into a dataset that serves as the training one, treating the resultant corrupted samples as the test dataset. Intuitively, as the noise level increases, the hypothesis performs worse, which indicates the hypothesis becomes less applicable the corrupted data. To conduct this experiment, CIFAR10 serves as the training dataset, with incremental addition of noises to the original dataset to create the test dataset. The types noises (46) used include Gaussian, Salt & Pepper, Uniform, and Speckle, with the noise rate increasing from 0.1 to 0.9 with 0.1 interval. The methods for comparison include Hypothesis-orientee Density Ratio (HDR) I-Div and R-Div [47]. Fig. [ presents our experimental findings, showing key performance metrics as influenced by varying noise rates. Notably, the classification performance declines with increasing noise, impacting the hypothesis predictive accuracy. Interestingly, proposed I-Div algorithm demonstrates robustness against these challenges, with its discrimination power inversely related to the classification accuracy of the standard network in noisy conditions. A brief comparative analysis hints the superior performance of I-Div over HDR, especially in relation to hypothesis applicability in noisy test datasets. Fig. [ Appendix D.3 shows the results when noise is added the training data instead of the test data. The results are consistent with the above, as the classification accuracy decreases with increasing noise, and I-Div becomes more effective distinguishing between clean training data and noisy test data. For a comprehensive discussion and full experimental results, please see Appendix D.3
4.4 Experiments on adversarial data
In this experiment, we delve into a specific scenario involving adversarial samples [48] We designate one dataset training and its corresponding adversarial samples the test dataset. It a well-known phenomenon that minimal adversarial perturbation, though visually imperceptible, can drastically alter the classification performance of a network. This suggests potential issues with the direct applicability of the hypothesis selected for a training dataset to a test dataset. However, based on human perception, which fails distinguish original and adversarial samples visually, we would expect negligible distribution discrepancy between the distributions of the original and adversarial samples. This outcome could guide us in enhancing network robustness against adversarial attacks and in generalizing the hypothesis to adversarial contexts. For this purpose, we use the CIFAR 10 dataset to train a standard network, with adversarial perturbation magnitudes selected from the set {0.001, 0.01, 0.1, 0.33. The results in Fig, 2 indicate a marked decrease in standard network

---

Table 5: Effect of different network architectures.
RESNETiB
MOBILENET
EFFICIENTNET
ACC
AUROC
AUPR
ACC
AUROC
AUPR
ACC
AUROC
AUPR
ACC
AUROC
AUPR
RGI
0.0
100.0
100.0
0.0
73.0
70.0
0.0
100 0
100.0
0.0
99.6
98.2
SVHN
17.0
100.0
100.0
15.6
74.5
69.8
20.0
100.0
100.0
19.8
98.1
93.7
71.6
100
1.6
100.0
100.0
78.3
73.5
2.7
96.7
OXP
2.3
100.0
100.0
0.9
69.8
67.3
1.5
100.0
100.0
1,7
97.4
91.8
SEMEION
8.7
100.0
100.0
78.4
73.3
9.6
100.0
100.0
10.4
98.1
93.5
CALTECH256
CIFAR100
2.2
94.5
94.7
2.6
66.0
60.3
2.9
83.3
83.0
1.9
89.5
79.7
CIFARIO1
73.4
43.4
45.2
82.5
34.8
40.7
70.0
45.8
47.1
74.2
41.4
49.8
STL.10
63.0
37.2
41.9
72.0
50.1
48.5
60 6
43.8
31.1
63.9
39.5
47.5
accuracy against adversarial perturbations, while the consistent AUROC of I-Div suggests its limited differentiation capability. Detailed results and analyses are provided in Appendix D.4
4.5 Experiments with different sample sizes and network architectures
We examine the impact of different sample sizes on the performance of the I-Div algorithm, focusing on its ability to generalize hypotheses from training to test datasets. Fig. show that I-Div tends to maintain low AU- ROC values for semantically similar datasets like CIFAR10. and STL10, indicating effective hypothesis appli- cability. Conversely, for datasets with significant semantic differences, the performance I-Div improves with larger sample sizes, highlighting its capacity to recognize con-transfereble knowledge. Additionally, we investigate the effect of varying network architectures shown in Table Detailed results are provided in Appendix D.5
1030
CIARR0
100
CPARLO
150EDS CITARIEO
Tolk
CIAARB
S1113
$1.33
15
300
1893
340
1000
Figure 3: Effect of different sample sizes.
5
Limitations
I-Div relies on density and likelihood ratios to achieve the sampling transfer in unlabeled test data, allowing for the estimation of distribution discrepancies between training and test datasets with labeled training samples. Although the density ratio can be accurately estimated using inputs from both training and test samples, the likelihood ratio cannot be estimated precisely due the unavailability of class labels of test samples. The strategy used by I-Div targets the estimation of distribution discrepancies between training and test distributions. It optimizes likelihood ratio that adapts to this density ratio to ensure rapid convergence of the distribution discrepancy, by minimizing the upper bound of the generalization error based on the density ratio. Our future research includes refining the estimation methods for likelihood ratios and exploring distribution discrepancy estimation methods that can bypass the likelihood ratio.
Conclusion
In the realm of complex data and machine learning tasks, crucial question arises regarding the applicability of a hypothesis derived from a training dataset to a test dataset. This uncertainty, especially challenging when test samples lack class labels, significantly determining the hypothesis generalization. To address this, we introduce the I-Div measure for estimating the distribution discrepancy between training and test distributions. I-Div involves the hypothesis-oriented density ratio and adaptive likelihood ratio expected risk difference to shift the sampling problem from test training distributions. Experimentally, we validate that I-Div can effectively assess the hypothesis capability of handling test samples, yielding results consistent with prior human knowledge.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the Australian Research Council Linkage Grant LP230201022, the Australian Research Council Discovery Grant DP240102050, and the Australian Research Council Linkage Infrastructure, Equipment and Facilities Grant LE240100131.
References
[1] Ilia Nouretdinov, Volodya Vovk, Michael V. Vyugin, and Alex Gammerman. Pattern recognition and density estimation under the general i..d. assumption. In COLT, volume 2111, pages 337- 353, 2001.
[2] Longbing Cao. Beyond i.i.d.: Non-iid thinking, informatics, and learning. IEEE Intell. Syst., 37(4):5-17, 2022.
[3] Zhilin Zhao, Longbing Cao, and Kun- Yu Lin. Revealing the distributional vulnerability of discriminators by implicit generators. IEEE Trans. Pattern Anal. Mach. Intell., 45(7):8888 8901, 2023.
[4] David Krueger, Ethan Caballero, Jömn-Heniik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrapolation (rex). JCML, volume 139, pages 5815-5826, 2021.
[5] Zhilin Zhao and Longbing Cao. Dual representation learning for out-of defdiiittitutonn detection. Trans. Mach. Learn. Res., pages 1-21, 2023.
[6] Zhen Fang, Yixuan Li, Feng Liu, Bo Han, and Jie Lu. On the learnability of out-of-distribution detection. J. Mach. Learn. Res., 25:1 84, 2024.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17:1-59, 2016.
[8] Kun-Yu Yu Lin, Jiaming Zhou, Yukun Qiu, and Wei-Shi Zheng. Adversarial partial domain adaptation by cycle inconsistency. In ECCV, pages 330-548, 2022.
[9] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
[10] Jesper E Van Engelen and Holger Hoos. A survey on cemi-supervised learning. Mach. Learn., 109(2):373 2020.
[11] Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. ICMI., volume 97, pages 3488-3498, 2019.
[12] Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theoretical analysis adversarial learning: minimax approach. NeurIPS, 32: 222:12299-1266 2019.
[13] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Trans. Mach. Learn. Res., pages 1-81, 2022.
[14] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: survey. IEEE Trans. Pattern Anal. Mach. Intell., 45(4):4396 2023.
[15] Imre Csiszár. Eine informationsthherrttchhe ungleichung und ihre anwendung auf beweis der ergodizitact von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
[16] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral proba- bility metries. J. Mach. Learn. Res., 22:1-128, 2021.

---

[17] Anuran Makur, Marios Mertzanidis, Alexandros Psomas, and Athina Terzoglou. On the robustness of mechanism design under total variation distance. NeurIPS, pages 1-10, 2023.
[18] Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. J. Mach. Learn. Res., 10:2137-2155, 2009.
[19] Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, NeurIPS, pages 1433-1440, 2007.
[20] Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, and Stefano Ermon. Comparing distributions by measuring differences that affect decision making. In ICLR, pages 1-20, 2022.
[21] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
[22] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for domain adaptation. ICML, volume 97, pages
2019.
[23] Taojiannan Yang. Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. AIM: adapting image models for efficient video action recognition. In ICLR, pages 1-18, 2023.
[24] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, volume 15, pages 315 2011.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), CoRR, hbss//006.08415..
10, 2016.
[26] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation machine learning. Cambridge University Press, 2012.
[27] Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. Efficient direct density ratio estimation for non-stationarity adaptation and outlier detection. In NeurlPS, pages 809-816, 2008.
[28] Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3:463-482, 2002.
[29] Mehryar Mohri, Afshin Rostamizadeh, and Amect Talwalkar, Foundations of Machine Learning. MIT Press, 2018.
[30] Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends( in Optimiza- tion, 1(3):127-239, 2014.
[31] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
[32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, pages 1-9, 2011.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., volume 139, pages 8748-8763, 2021.
[34] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, pages 5543-5551, 2017.
[35] Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5018-502) 2017.
12

---

[36] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In CVPR, pages 3606-3613, 2014.
[37] Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR, pages 1447-1454, 2006.
[38] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, pages 3498-3393 30003,55,400 2012.
[39]
Semeion Handwritten Digit. UCI Machine Learning Repository, 2008
[40] Greg Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
[41] Antonio Torralba, Robert Fergus, and William T. Freeman. 80 million tiny images: A large data for nonparametrie object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 30(11):1988-1970 2008.
[42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255 2009.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[44] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, pages 1-21, 2021.
[45] The Open Images Dataset V4. Alina kuznetsova and hassan rom and neil alldrin and jasper r. uijlings and ivan krasin and jordi pont-tuset and shahab kamali and stefan popov and matteo malloci and alexander kolesnikov and tom duerig and vittorio ferrari. Int. J. Comput. Vis., 128(7):1966-1881 2020.
[46] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt, Test-time training with self-supervision for generalization under distribution shifts. In ICML, volume 119, pages 922999229999 9248, 2020.
[47] Zhilin Zhao and Longbing Cao. R-divergence for estimating model-oriented distribution discrepancy. In NeurlPS, pages 1-19, 2023.
[48] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In ICLR, pages 1-11, 2015.
[49] Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. Fast two-sample testing with analytic representations of probability measures. In NeurIPS, pages 1981-1989, 2015.
[50] Wittawat Jitkrittum, Zoltán Szabó, Kacper Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In NeurlPS, pages 181-189, 2016.
[51] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723-773, 2012.
[52] Mehmet Gönen and Ethem Alpaydin. Multiple kernel learning algorithms. J. Mach. Learn. Res., 2-22211-226, 2011.
[53] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML, volume 119, pages 6316-63266 2020.
[54] David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In ICLR, pages 1-15, 2017.
13

---

[55] Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Trans. Inf. Theory, (8(10)663316662 2022.
[56] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. 2008.
[57] Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. J. Mach. Learn. Res., 10: 10:1391-1445 2009.
[58] Masahiro Kato and Takeshi Teshima. Non-negative bregman divergence minimization for deep direct density ratio estimation. In ICML, volume 139, pages 5320-5333, 2021.
[59] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and -iittii-ittitttionn examples neural networks. ICLR, pages 1-12, 2017.
[60] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distributi performance. In ICLR, pages 1-29, 2022.
[61] Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-the ooee-ee--nteeeee Predicting the performance of neural networks under distribution shift. In NeurIPS, pages 1-16, 2022.
[62] Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out- of-distribution error with the projection norm. In ICML, volume 162, pages 25721-257465 2022.
[63] Elias M Stein. Singular integrals and differentiability properties of functions. Princeton university press, 1970.
[64] Lan V. Truong. On rademacher complexity-bule generalization bounds for deep learning. CoRR, htss2228..0224:1-33 2022.
[65] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[66] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[67] Matthias Kirchler, Shahryar Khorasani, Marius Kloft, and Christoph Lippert. Two-sample testing using deep learning. In AISTATS, volume 108, pages 1387-1398, 2020.
[68] Tom Faweett. An introduction to ROC analysis. Pattern recognition letters, 27(8):861-874, 2006.
[69] Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In ICML,, volume 148, pages 233-240, 2006.
14

---

A Related work
In this section, we briefly review the methods for distribution discrepancy estimation, density ratio estimation, and performance prediction.
A.1 Distribution discrepancy estimation
Mean Embedding (ME) 49 and Smooth Characteristic Functions (SCF) 500 utilize differences in
Gaussian mean embeddings at optimized points and frequencies, respectively, to quantify distribution discrepancy. Building on ME, MMD-O [5T] calculates Maximum Mean Discrepancy (MMD) using a Gaussian kernel [52]. while MMD-D [53] enhances the performance of MMD-O by substituting
the Gaussian Kernel with a learnable deep kernel. In contrast, Classifier Two-Sample Tests, including C2ST-S [54] and C2ST-L [55], classify samples from one dataset as positive and from another as negative, leveraging the classification accuracy of binary classifier to differentiate between them. H-Divergence (H-Div) [20] identifies optimal hypotheses for both the mixture distribution and individual distributions within specific model. positing that if two distributions are identical, the expected risk on the training samples from the mixture distribution exceeds that from each individual distribution. R-Divergence (R-Div) 447 tackles the overfitting problem of H-Div by suggesting that two distributions are likely identical if their optimal hypotheses yield the same expected risk for each. However, to estimate the discrepancy between training and test distributions, the aforementioned methods assume symmetry between these datasets, meaning either both sets are labeled or both are unlabeled.
A.2 Density ratio estimation
To estimate the density ratio, Kernel Mean Matching (KMM) [56] offers direct estimates of importance training inputs by efficiently aligning two distributions. leveraging a unique characteristic of universal reproducing kernel Hilbert spaces. The Kullback-Leibler Importance Estimation Procedure (KLIEP) [9] seeks an importance estimate to minimize the Kullback-Leibler divergence from the true test input density to its estimate. Least-Squares Importance Fitting (LSIF) (57) addresses the direct importance estimation issue as a least-squares function fitting problem, transforming the optimization challenge into convex quadratic program, which is solvable efficiently with standard quadratic programming solvers. Non-Negative Bregman Divergence (NNBD) (58) employs deep neural networks alongside empirical Bregman divergence minimization, addressing the train-loss hacking issue by adjusting the empirical Bregman divergence estimator. Although these methods can estimate the distribution discrepancy between test and training inputs by calculating the density ratio even when class labels for test samples are not available, this implics disregarding the class labels of the training data. Consequently, such density ratios are not suitable for evaluating the applicability of a hypothesis.
A.3 Performance prediction
Maximum over Softmax Probability (MSP) [59 gauges the class label confidence each sample by a given model to distinguish ID and OOD samples, which can be extend by computing the average class label confidence across both training and test datasets and consider the confidence disparity as indicative of distribution discrepancy. Average Thresholded Confidence (ATC) (60] identifies a threshold such that the proportion of training samples exceeding this confidence threshold aligns with their accuracy, and then it predicts accuracy as the proportion of unlabeled test samples surpassing this threshold. Agreement-on -ontte-------eeee (AL) (6] capitalizes on the observation that if the accuracy of models on training samples linearly correlates with their accuracy on test samples, then a similar linear relationship exists between the training and test agreement of model predictions. Projection Norm (PN) 62) employs model predictions to pseudo-label test samples and trains a new model on these pseudo-labels. The variation in parameters between this new model and the original model is utilized to estimate the test error. The aforementioned algorithms estimate the effectiveness of a hypothesis selected from the training samples on test samples based on the confidence output of the hypothesis to judge its applicability. However, since the hypothesis is chosen based on limited set of training samples, it exhibits significant uncertainty towards unseen test samples, especially when these test samples are OOD, leading to misleading results.
15

---

B Objective function for estimating density ratio
According to the KL. divergence between P(x) and P(xx, we have
lox) log P(x)r(x) dx (L [(x)) Ep [x)r(()) 2(x)
KL (((x) (1
JEX
(19)
Similarly, for the KL. divergence between (xx) and @(x), we have
((x) dx KL (Q(x) P(x)) Ee [log rxx)
KL (o(x) @(x))
,xEX ( log
r(x)P(x)
(20)
Note that the terms KL (P(x) || P(x)) and KL ((x)) ((x) are constants independent of the
density ratio r. To normalize P(x) and @(xx) estimated by we have SxEx P(x) dx = 1 and
LEE 2(x) dx 1. According to Eq. (00), we have
1
(xx)
1
dx +
(x)
(21)
XEX r(x)
2 XEX
By considering Eq. 19 Eq. 20 and Eq. 21p. we can obtain an optimization problem with constraints
for learning density ratio I.C.,
min rE)(
Ep [x)rr(x]]= Eg (- log r(x)]
(22)
(xx) s.t. XXX r(x)
dx +
r(x)
dx
2 XEX
However, solving such problem with constraints is difficult. Inspired by the method of Lagrange multipliers [30], we can introduce a hyperparameter 2 O, which allows us to relax this constraint and balance the loss function of with the constraint conditions, i.e.,
min rER(h))
Ep [log r (x)] + Eg log rx)] + XJ.
(23)
s.t. J= 1
((x) XEX r(x)
dx + r(x))(x) -
xEX
Accordingly to the Cauchy-Schward inequality 663 we have
((x)
dx 1
((x) 111
VJ
<<
XEX
r(x)
XEX
dx + ( Xxex
Q(x)
P(x) |||(x) 1/| dx
(24)
r(x)
XEX
= E
1|
+ Ep [|r(x)) - 1ll
r(x)
Applying Eq.
24 and the Jensen's inequality, have
I 5
Eg
+ (Ep [|r(xx) 11)2
r(x)
(25)
2
<
Egg
+ Eyp
[|((x) 113]
(xx
We obtain the objective function by combining Eq.
23 and Eq. 25
16

---

C
Proofs
C.1 Proof of Theorem 3.1
We rewrite f(r) by applying the density ratio and obtain
[(r) - Bp [logr(x) + |r(x(( 111]
1
A
1
-
1
log (x)
2
r(x)
1 Bp logr(x) 1 [r((x) 11|]
} Rp
1
2
2
Xr(x)
r(x) log r(x)
(26)
r(x)
1 -2Bp logr(x)+ + - 1112 + Xr(x)
2 r(x) log r(xx
r(x)
According to Eq. (12), we know that is used to ensure the constraints are satisfied as much as possible. Intuitively, its value should be relatively large. More specifically, we provide the following lemma to determine the lower bound of À to ensure that f() is strongly convex.
Lemma C.I. f(r) is j-sttronly convex if X 2 1.
Thus, we have
<f(F) f(F) f() (()))()--Fo f(F) M "EP EE fxx) -(x)) 2
(27)
- f(F) sup rER(N)
[()) - Rr)(.
The first inequality arises from the ju-strong convexity of u(((x)). The second incquality is because F is the minimizer of f(r). According to r(x) (b,, B,) and f(r) is Ly-lisschitz continuous on (br, B,), we know that the absolute value of the difference caused by altering one data point in
(s(r) f(r) is bounded by 2I,,B.
To proceed, we introduce the following two lemmas.
Lemma C.2 ((21), Theorem 26.5). Assume that P. contains N i..dd. samples from P and [((h, 2)1 S for all z and E H. With probability of at least for all E H,
(ep(h) - 2p(()) 29R(1 HoP) + doy In(4//)
where R(I H. P) is the Rademacher complexity of with respect to P. Lemma C.3 (129), Talagrand's Contraction Lemma). For any L-Lipschitz loss function I(;))) and
hypothesis space H. we obtain
R( H o P) < LRRH P),
where R(H P) (zI)) SuPRER Exe (z)()z)) with random choice of
Accordingly, with probability of at least 1 - 8, for any r E R, we have
In 4/8
f(r) - f(r) ((R(h) ú + B, V
(28)
M
Recall that v is a frozen network backbone and w is a learnable adapter containing two fully connected layers and an activation layer in the middle. The activation function used in the middle and the last layers are GELU and Softplus, respectively. According to the Rademacher bound [64), we have
O +
8272
R(R(h) o Ū < 4,1
(29)
2M
M
We complete the proof by combining Jensen's 's inequality, Eq.
Eq.
28 and Eq. 29
17

---

C.2 Proof of Theorem 3.2
Applying the triangle inequality, for and any v E V, we have
d P hip. r, - a (P,P (.p.F.))
Ep F 3) - 1 (nx(()), 1) ]
-Bpp [F(x) v) - 1/ (ns(x))))
(30)
Ep [(x))((x v) 1/2 (hs(()),)
- Esg ( 1/2 (hx((x),) (
Ba
To bound B₁, with probability of at least 1 - 6, we have
B = Ep ((F())((x,)) y) 11 | (x) ux, v) - 11) & (ns(()),))
- Ep u(( v) rx(x) -(x)] 2 (Tis(x). w)
(31)
< BsEp [u(x, y)] Ep [|F(x) - -x))l]
B)
< 2B [o(xx, w)]
/
where the first inequality follows from the triangle inequality, the second is a consequence of Hölder's inequality, and the final inequality results from Theorem 3.1 Further, we know
S Fxx))((x,)) y) - 1|2 (h v) s [F(x)((x, y) B|
(32)
By applying Hoeffding's 's inequality 65 with probability of at least - 8, we have
(2(2/)) Σ [xxx)>
[xx) y) 1
B2 < B&
(33)
N
We complete the proof by combining Eq.
30 Eq.
31
and Eq. 33
C.3 Proof of Lemma
The second derivative of the function fr) with respect to is
(2(r(x))3 (r(x))2-(((x)))) (r(x)) +2
(r(x))2 (r(x)) +
V f(r -Ep
2 > Ep
23r(x))3
2(r(x))3
(34)
We know that
(r(x)) (r(x))² (r(x)) +
-3 v7 (V7 1)2
$ 0.27.
(35)
2(r(x))3
2(V7 - 1)3
To ensure K(((()) is strongly convex (66], simply assume À 1.
D Experiments
Our experiments are designed to align with established knowledge and intuition. We hypothesize that I-Div will indicate minimal distribution discrepancy when the training and test distributions share semantically similar class labels, meaning that samples from both distributions can be treated ID. This suggests the applicability of the hypothesis selected by an algorithm from the training samples to the test samples. Conversely, class labels significantly differ semantics, with samples from the training and test distributions being categorized as ID and OOD, respectively, we expect I-Div to reveal more pronounced distribution discrepancy. This implies that the knowledge learned from the training samples may not be transferable to the test samples.
18

---

D.1 Experimental setup
In our study, we explore whether hypothesis selected from training dataset retains its capacity when applied to test dataset. Our quantitative analysis utilizes the I-Div metric to measure the distribution discrepancy between training and test distributions, pertinent the hypothesis applicability. Unless otherwise specified, we set À 1 and 7 1. We hypothesize that data with closer semantic relationships will exhibit smaller distribution discrepancies, as opposed to those with distinct semantic differences.
For the given training dataset P and dataset ②, consider using subsets of these datasets to estimate the distribution discrepancy instead of the entire datasets. This is because using too many samples can trivialize the estimation task and, realistically, obtaining the entire test dataset at once is impractical. Accordingly, we randomly draw samples from P and Ô creating subset of three smaller datasets (P', ,,", Q'). Adhering to the two-sample test framework [53] 67], each subset consists of M < N samples. We then form positive pairings (P' and negative pairings (P', 2'). Unless otherwise noted, our experiments use standard sample size of M 1000. We generate 100, 000 tuples to achieve significant distribution discrepancy in the positive pairs and minimal discrepancy in the negative pairs.
The effectiveness is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUROC) [68] and the Area Under the Precision-Recht Curve (AUPR) [69], with higher values indicating more effective differentiation between pair types. To illustrate the effectiveness of our I-Div algorithm, compare it with selection of representative contrastive algorithms, each adapted to our specific task. All approaches quantitatively evaluate hypothesis applicability by measuring the distribution discrepancy between training and test distributions. These algorithms, which include Maximum over Softmax Probability (MSP) [59], Non-Negative Bregman Divergence (NNBD) [58], Maximum Mean Discrepancy with Deep kernels (MMD-D) [53], and R-Div (7), assess hypothesis applicability through the lens of distribution discrepancy.
D.2 Additional experiments on semantically dissimilar data
Table 6: Distribution discrepancy of different classes in SVHN. The larger the values of AUROC and AUPR, the better the performance.
TARgeT
MSP
NNBD
MMD-D
R-DIV
I-11iv
AUROC
AUPR
AUROC
ALPR
AUBOC
ALPR
AUBOC
AUPR
AUROC
AUPR
DiGrr
100.0
100.0
95.0
98.2
83.1
$4.1
100.0
100.0
100.0
100.0
DIGIT
100.0
100.0
99.8
99.8
79.3
$1.7
100.0
100.0
100.0
100.0
DIGIT
100.0
100.0
98.0
98.3
70.0
71.3
100.0
100.0
100.0
100.0
71.0
106.0
100.0
SVHN
DIGIT
DIGIT
100.0
100.0
100.0
100.0
92.3
96.2
92.4
96.6
80.4
71.2
81.8
73.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Diarz 100.0
100.0
82.5
106.0
100.0
100.0
100.0
75.2
Diari 100.0
100.0
96.3
97.0
78.1
80.0
100.0
100.0
100.0
100.0
DiGrz
100.0
100.0
97.8
98.0
75.5
77.4
100.0
100.0
100.0
100.0
D.3 Additional experiments on corrupted data
Fig. I(a] shows steady decline in the classification performance of the standard network in the test dataset as the noise rate increases. This decline results from the loss of class label-relevant information the samples as noises increase, hindering the ability of the hypothesis predict class labels accurately. The expectation is that the distribution discrepancy between the training and test distributions will increase with added noises, suggesting a decrease of hypothesis applicability from the clean training dataset to the corrupted test dataset without accessing the ground truth labels of test samples. However, the results Fig. 1(b] Fig. [(c] and Fig. [(d) show that the proposed I-Div algorithm performs well in this respect. As the classification accuracy of the standard network in the test samples decreases, the ability I-Div to discriminate between the two datasets increases. While Fig. I(c) shows the improving ability of HDR to distinguish the two datasets with increasing noise
19

---

80
Noise Added Data
Noisa Added to Training Data
Gouss on Sett & Pepper
Unitom
Spackil
50
70
80
AG0
50
AUROO
Gsussian Unitorm
Perpor
30
Spociie
90
10
0.1
0.2
0.3
0.4
0.5
0.8
0.7
0.8
0.9
0.1
02
0.3
0.4
0.5
0.8
0.7
0.8
0.9
Noise Rate
Noise Rate
(a) Classification
(b) Discrepancy: I-Div
Figure 4: Distribution discrepancy between original data and its corrupted variants with different noise rate. (a) shows the classification performance of the standard network for the test datasets containing corrupted samples. (b)(c)() present the distribution discrepancy in terms of AUROC.
rate, its performance is not consistent that of I-Div, whose discrimination power more closely aligns with changes in the classification ability of the standard network. Fig. [(d) indicates that R-Div is extremely sensitive to data variation, achieving complete separation of the two datasets at any noise level. R-Div can differentiate between clean and corrupted datasets even with minimal noises, but this does not indicate whether the hypothesis derived from training dataset is applicable to the test dataset.
The comparison between Fig. ((b) and Fig. I(d) can be seen ablation study, since HDR is a component of I-Div. From this analysis, it is clear that I-Div outperforms HDR in determining the hypothesis applicability in the test dataset more accurately. HDR relies only on input data for estimating the input density ratio, a metric unrelated to specific model performance. In contrast, I-Div uses this density ratio to estimate the disparity of hypothesis performance between two datasets, offering a more accurate reflection of hypothesis applicability in the test dataset.
D.4 Additional experiments on adversarial data
As shown in Fig. 2(a) the classification accuracy of standard network on adversarial samples markedly diminishes with increasing perturbation magnitude. However, as depicted in Fig. 2 I-Div maintains consistently low AUROC, indicating its inability differentiate between the training and test datasets, akin to human visual perception. This suggests that, even when the classification performance of the standard network is compromised, I-Div still perceives the knowledge from training dataset as transferable to the test dataset. This finding underscores the need to focus on improving the generalization of standard network to adversarial samples, corroborating our empirical understanding and existing knowledge in the field, Fig. [ presents the AUROC of HDR, which remains high across various perturbation magnitudes, indicating its effectiveness in distinguishing original and adversarial samples. This observation might suggest that, while HDR is responsive to distribution changes induced by adversarial perturbations, it may not accurately reflect the hypothesis applicability from the training to the test datasets. Fig. 2(c) shows the AUROC of R-Div, exhibiting distinct pattern where the AUROC is relatively lower the smallest perturbation but improves with larger perturbation. This indicates that R-Div could potentially be used to assess the hypothesis applicability to the test dataset. Nonetheless, in the context of adversarial samples, the results of I-Div imply that enhancing the hypothesis generalization to adversarial samples should be priority. This conclusion aligns more closely with our intuitive understanding and is supported by numerous advanced research efforts in this area.
D.5 Additional experiments with different sample sizes and network architectures
We explore the influence of varying sample sizes, denoted as M, on the experimental results. This investigation is conducted under the same experimental setup as the used for the semantically
20

---

similar datasets. The findings illustrated in Fig. indicate that for test datasets like CIFARIO.1 and STL10, I-Div consistently exhibits relatively low AUROC. This pattern suggests that I-Div considers the hypothesis derived from the training dataset to be applicable to these test datasets, conclusion further supported by the classification results 3 In contrast, for other test datasets with class labels semantically dissimilar to the training dataset, the knowledge learned by the model deemed non-transferable. As shown Fig. 3 as quantified by AUROC and AUPR, improves with increasing sample size, eventually reaching plateau,
the ability of I-Div to distinguish these datasets,
Given the hypothesis sppothesis-orrented nature of the I-Div algorithm, which bases its analysis of hypothesis applicability from training to test datasets, pertinent investigate the impact of varying network architectures on the performance. The findings of these experiments are summarized in Table 5] The data reveal that, across all tested hypotheses, there are consistently low AUROC and AUPR scores for test samples with semantic similarities to the training datasets, such CIFAR1O. 10.1 and STL10. However, notable variance of performance observed when examining test datasets with distinet semantic differences, like SVHN and DTD. This highlights the significant role of network architecture determining a hypothesis capability to generalize and apply learned knowledge to novel, unseen dataset. The robust and consistent performance of I-Div across diverse network architectures further attests to the algorithm stability and broad applicability.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly outline the challenge evaluating distri-
bution discrepancies between training and test distributions when test labels unavailable.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The paper discusses the limitations of the work performed by the authors. It specifically addresses the challenges in estimating likelihood ratios due the absence of class labels in test samples and outlines future work to refine these methods.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: The paper does provide the full set of assumptions for each theoretical result and includes complete and detailed derivations in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all the necessary information needed to reproduce the main experimental results, detailing the experimental setup comprehensively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper utilizes publicly available datasets and makes the code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
inips.cc/publice.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all necessary training and test details, including data splits and experimental parameters, in the appendix, providing sufficient information understand the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper reports error bars other appropriate measures to indicate the statistical significance of the experimental results, ensuring that the findings are presented with necessary reliability assessments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for caleulating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All experiments are conducted using the Py PyTorch framework on a single 64GB GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////eeerpps.cccuauba//////
Question:
Answer: [Yes]
Justification: The paper adheres to the NeurlPS Code of Ethics, as it does not include any content or practices that violate ethical guidelines, and ensures that all experimental procedures are appropriately conducted and reported.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper solely focuses on technical aspects and does not address potential socictal impacts, such ethical considerations possible misuse of the technology.
25

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA],
Justification: The paper does not discuss any specific safeguards related to the responsible release of data or models, particularly those that might have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits the creators original owners of all used assets, including data and code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
26

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprsspppessiite thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
The paper does not involve crowdsourcing experiments research with human subjects, thus no details about participant instructions, screenshots, or compensation are applicable required.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The paper does not involve research with human subjects, therefore discussions of potential risks, disclosures to subjects, or the need for Institutional Review Board (IRB) approvals are not applicable.
Justification:
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "FztX1sRdUX", "reviewer": "Reviewer_yuQt", "summary": "This paper proposes a discrepancy to measure the difference between two distributions within a common scenario: the labeled training set distribution and the unlabeled test set distribution. This discrepancy arises from the expected risk difference between these two distributions, considering a model pre-trained on the training samples, and involves the estimation of both the density ratio and the likelihood ratio. Experiments were conducted on different types of data splits and data corruptions to demonstrate its effectiveness.", "questions": "1. Could the authors provide a more detailed explanation of how Eq.16 is derived?\n2. In Figure 3, what is the sample size of the data in the first column, and why does it perform better than the sample size of 500?\n3. Considering my first concern in Cons, what would happen if we were to swap the training set and testing set in the Corrupted Data Experiment, or if the training set itself is inherently challenging?\n4. Experiment 4.4 is intriguing; the proposed I-Div appears to be completely unaffected by adversarial samples. Could you provide a more insightful explanation?\n5. It would be beneficial to analyze the sensitivity of hyperparameters （e.g. $\\sigma$ and $\\gamma$）", "limitations": "The proposed I-DIV, rooted in expectation risk, bears similarities to existing measures like R-Div. And due to the unknown labels of the test distribution, the authors propose to estimate the density ratio and likelihood ratio, which introduces uncontrollable errors, especially in complex distributions. Thus, I-DIV is affected not only by the estimation error of the likelihood ratio, as noted by the authors, but also the empirical minimizer, which is difficult to mitigate. I believe this limitation somewhat restricts the practical effectiveness of I-DIV.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The scenario considered for this discrepancy is very practical.\n2. The definition of discrepancy is intuitive, and the paper is well-written.\n3. The experiments demonstrate the robustness of the discrepancy across various scenarios, providing solid evidence of its effectiveness.", "weaknesses": "1. My primary concern revolves around the significant impact of the pre-trained classifier $\\hat{h}$ on the proposed I-Div. After obtaining the pre-trained model from the training set, the estimated density ratio is predicted by appending a branch to the last layer of this classifier, and the subsequent Adaptive Likelihood Ratio is further estimated based on the predicted $\\hat{r}(x)$. Therefore, I am concerned that inaccuracies in the initial estimation by $\\hat{h}$ may amplify errors in the final discrepancy. (Although Theorem 3.2 provides a bound, which measures the upper limit of the distance within the same classifier.) I am unsure whether this could limit the practical effectiveness of the proposed I-Div.\n   \n2. The compared methods are not presented under consistent settings regarding label utilization and network parameterization. Aligning these settings could enhance clarity for comparison, although direct application to the scenario proposed by the authors might not be feasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for constructive suggestions. We appreciate the opportunity to address your concerns.\n\n**W1. Impact of Pre-trained Classifier** \n\nAs discussed in Line 24, our goal is to measure the discrepancy between training and test datasets for a given pre-trained classifier, thereby assessing its applicability on the test data. Therefore, our work does not amplify errors. Rather, as shown in Eqs. (5) and (18), we use density and likelihood ratios to indirectly estimate and magnify the discrepancy.\n\nHowever, different classifiers have varying generalization abilities, which affect their judgment on this matter. For instance, a classifier with strong generalization might consider samples from different domains as coming from the same distribution, while a classifier with weak generalization would not. Therefore, different pre-trained classifiers will result in different density and likelihood ratios, but this is expected. This can help us bypass the issue of unknown test sample class labels while amplifying the sensitivity to the distribution discrepancy.\n\n**W2. Consistent Setting**\n\nWe made every effort to ensure consistency in the experimental setup. When conducting comparative experiments, we used the same network structure for all algorithms. Additionally, we provide the performance of the comparative algorithms with and without using class labels, as shown in the table below.\n\nSpecifically, in Tables 1, 4, and 5, the different algorithms all use ResNet18 as the backbone, while in Table 2, they use AlexNet. The following experiments on CIFAR10 show that I-Div achieves better performance in both settings. This is mainly because it is based on a pre-trained classifier and combined with the sampling transfer, indirectly estimating the network applicability on test samples. In contrast, the comparative algorithms directly measure divergence without considering the impact of the pre-trained classifier, leading to results that do not align with intuitive expectations.\n\n| Method| CIFAR10.1|STL10|\n|------------|-------|-------|\n|R-Div w/o labels|84.6|94.5| \n|R-Div w/ labels |100.0|100.0| \n|I-Div |**65.3**|**78.9**|\n\n**Q1. Derivation of Eq. (16)**\n\nTheorem 3.2 derives an upper bound on the generalization error for distribution discrepancy w.r.t. the likelihood ratio. To minimize the generalization error bound, we extract the two terms related to the likelihood ratio and use $\\gamma$ to balance them. With the constraints revealed by Eq. (15), we then derive Eq. (16).\n\n**Q2. Sample Size**\n\nIn Figure 3, the first column represents 100 samples, but the overall performance is worse than that with 500 samples. The values of CIFAR10.1 and STL10 (where lower values are better), which are semantically related to CIFAR10, are similar for 100 and 500 samples and decrease with increasing the sample size. However, for other datasets unrelated to CIFAR10 (where higher values are better), the AUROC and AUPR values are higher with 500 samples.\n\n**Q3. Data Swap and Challenging Data**\n\nThis is indeed a very interesting idea. In the revised version, we add experiments with data swapping and challenging datasets. By swapping the data, the network trains on noisy data and tests on clean data. Results show that higher noise levels reduce the ability to distinguish between datasets, aligning with our intuition. For experiments with more challenging data (ImageNet, OID, DTD, etc.) on complex models (ResNet50, ViT, and CLIP), please refer to Table R1 in the PDF within the global response, or the table in our response to Reviewer fpGs.\n\n| Train|0.3|0.5|0.7|\n|------------|-------|-------|-------|\n|Uniform|57.9|53.5|48.6|\n|Salt & Pepper|50.7| 48.7|42.1|\n\n**Q4. Intriguing Experiments**\n\nI-Div aligns more closely with human intuition, as one also struggles to differentiate adversarial samples. I-Div, based on a pre-trained network without adversarial defense, perceives adversarial samples as ``normal\" and assigns them with high confidence scores, making it hard to distinguish between original and adversarial samples. Other algorithms detect subtle differences, concluding these samples are from different distributions. \n\n**Q5. Hyperparameter Analysis**\n\nPer your suggestion, in the revised version, we add parameter analysis for $\\lambda$ and $\\gamma$ using ResNet18 and CIFAR10. The results are shown below. The Lagrange coefficient $\\lambda$ stabilizes beyond a certain value, as explained in Appendix B. The algorithm is not very sensitive to $\\gamma$, with slightly better results at larger values, consistent with Theorem 3.2. Based on these findings, both parameters are set to 1 by default, as noted in Appendix D1.\n\n| Parameter|Value|SVHN|CIFAR10.1| Adver.|\n|-----------|-------|------|-----------|--------------------|\n|$\\lambda$|0.01|67.5|78.9|27.6|\n||0.1|100.0|44.5|27.9|\n||1|100.0|43.4|28.6|\n| $\\gamma$|0.1|100.0|54.6|28.4|\n||1|100.0|43.4|28.6|\n||10|100.0|45.6|29.1|\n\n**L1. Measurement Differences**\n\nAs discussed in Appendix A1, I-Div and R-Div are fundamentally different. R-Div measures the difference between training and test datasets using hypotheses from mixed data, requiring known test labels. I-Div measures this difference for a given hypothesis using density and likelihood ratios, even without test labels.\n\nWhile it is impossible to address unknown test labels without any assumptions, our algorithm easily mitigates this by estimating density and likelihood ratios. In Eq. (13), the density ratio is straightforward to estimate using a lightweight adaptor. The likelihood ratio, derived analytically by minimizing the generalization error bound as shown in Eq. (17), can be estimated with a weak assumption. Even if the likelihood ratio is set to 1, as shown in the CIFAR10 experiments below, our algorithm remains effective.\n\n| Method| SVHN| SEMEION| CIFAR10.1| CIFAR10(Gaussian)|Adver.|\n|------------|-------|-------|-------|-------|-------|\n|I-Div(1)|100.0|100.0|43.4|46.5|28.6|\n|I-Div(Eq. (17))|100.0|100.0|48.6| 49.2|38.7|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3n7UfDGI0", "reviewer": "Reviewer_1gtU", "summary": "This paper presents a novel approach called Importance Divergence (I-Div) to address the challenge of measuring the discrepancy between training and test distributions when test sample class labels are unavailable. I-Div transfers sampling patterns from the test distribution to the training distribution by estimating density and likelihood ratios. The density ratio is obtained by minimizing the Kullback-Leibler divergence, while the likelihood ratio is adjusted to reduce the generalization error. Experimentally, I-Div accurately quantifies distribution discrepancy across a wide range of complex data scenarios and tasks.", "questions": "This paper presents a methodological advancement in evaluating distribution discrepancies without test sample class labels. However, the fundamental challenge of accurately estimating the likelihood ratio remains unresolved, potentially compromising the method's accuracy. Additionally, the computational complexity may hinder practical application, especially with large or complex datasets.\n\n1. Since the Likelihood Ratio significantly influences Importance Sampling, it greatly affects the quality of the samples. Consequently, this often results in high variance in the outcomes. I am curious about how the author addressed this stability issue during the implementation.\nFor example, Fig. 3 illustrates the effect of sample size but does not explain which factor is crucial to controlling the quality of the samples. And the size > 1000 means it requires high computational costs. So please elaborate on some more challenges on this matter.\n\n2. What challenges might be faced in extreme cases, such as when the Adaptive Likelihood Ratio is large? While proving convergence is important, discussing practical implementation difficulties, such as when the test dataset size is relatively bigger or when many classes exist, would improve this paper.", "limitations": "I believe this paper does not have any negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Approach**: I-Div offers a novel method to evaluate distribution discrepancies between training and test data without needing test sample class labels, addressing a significant challenge existing methods struggle with.\n\n2. **Enhanced Generalization Capability**: By transferring sampling patterns and adjusting ratios, I-Div enhances the model's generalization capability, which is particularly beneficial for applications such as out-of-distribution (OOD) sample detection.\n\n3. **Validation**: The paper demonstrates the effectiveness of I-Div through various datasets and scenarios, showing that the proposed method works well across different conditions.", "weaknesses": "1. **Limitations of Likelihood Ratio Estimation**: As the authors mention, accurately estimating the likelihood ratio is challenging since test sample class labels are unavailable. The proposed adaptive adjustment only partially mitigates this issue and may not fully resolve it.\n\n2. **Computational Complexity**: Estimating the density and likelihood ratios can be computationally expensive and potentially impractical for large datasets or complex models.\n\n3. **Limited Scope of Experiments**: The experiments still focus primarily on simple datasets. Additional validation on other types of datasets is necessary to confirm the generalizability of I-Div.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback and thoughtful questions. Here are our responses to your raised concerns.\n\n**W1. Likelihood Ratio**\n\nIt is impossible to completely solve this problem because the class labels of the test samples are unknown. However, we theoretically decompose this problem to identify solvable aspects and necessary assumptions. The density ratio can be estimated using Eq. (13) with a lightweight adapter by freezing the pre-trained network, while the likelihood ratio can be derived analytically under the weak assumption of minimizing the generalization error, as shown in Eq. (17). We conduct additional experiments on more complex data (ImageNet, OID, DTD, etc.) and networks (ResNet50, ViT, and CLIP) to demonstrate the effectiveness. Please refer to Table R1 in the PDF within the global response, or the table in our response to Reviewer fpGs.\n\nWe don't even need to estimate very accurately. Under a stronger covariate shift assumption, where all sample likelihood ratios are set to 1, reasonable results can still be achieved in practice, as shown in the experiments on CIFAR10 below.\n\n| Method| SVHN| CIFAR10.1| Adver.|\n|------------|-------|-------|-------|\n|I-Div(1)|100.0|43.4|28.6|\n|I-Div(Eq. (17))|100.0|48.6|38.7|\n\n**W2. Computational Complexity** \n\nEstimating the density and likelihood ratios is actually very fast. This is because the density ratio can be estimated using a lightweight adapter, and the likelihood ratio can be quickly derived analytically. We add the following throughput experiments on large data and complex models to validate this.\n\nFor the density ratio in Eq. (9), the primary training time is spent on the adaptor for a given pre-trained network. The main bottleneck is the feature dimensions, which is the input size of the adaptor and the output size of the pre-trained network. The results below show the throughput for different dimensions across various pre-trained network architectures. ResNet18 has an input dimension of 28x28, while ViT-B/16 has 224x224. This demonstrates the efficiency for large datasets and complex models, as only the downstream adaptor is trained while the pre-trained network remains frozen. For the likelihood ratio, we can directly obtain its analytical solution using Eq. (17), making its time complexity minimal.\n\n| Dimension|512|1024|2048|\n|------------|-------|-------|-------|\n| ResNet18 | 276.14 | 142.97 | 79.02 |\n| ViT-B/16 | 243.62 | 136.70 | 68.04 |\n\n**W3. Limited Scope** \n\nTo address your concern, we add ResNet50 and ViT trained on ImageNet and CLIP to estimate the differences between this dataset and OID, Places365, StandfordCars, Caltech101, and DTD. For the complete results, please refer to Table R1 in the PDF within the global response. A portion of the experiments with ResNet50 is shown below. The outcomes align with human intuition.\n\n| Model| OID|Places365|StandfordCars|Caltech101|DTD|\n|------------|-------|-------|-------|-------|-------|\n|MSP|100.0|100.0|100.0|100.0|100.0|\n| NNBD|91.7|90.5|96.6|95.3|98.7| \n|R-Div|94.6|100.0|100.0|100.0|100.0|\n|I-Div|**69.3**|**78.9**|**86.5**|**100.0**|**100.0**|\n\n**Q1. Sample Quality** \n\nI-Div is highly stable and fast, and the experimental results in the paper are averaged over 100 runs. The experimental variance on CIFAR10 is shown in the table below. The lightweight adapter used to estimate the density ratio requires only a small number of samples, and the subsequent likelihood ratio can be directly derived analytically according to Eq. (17). Additionally, as shown in Figure 3, processing $M=1000$ samples on a single GPU takes only 7.09 seconds.\n\nAccording to the experimental results in the table, if there is a significant difference between the training and test data distributions, the algorithm can easily identify the differences and converge quickly. When the test data is semantically similar to the training data, the stability is lower, but it improves rapidly as the number of samples increases. For more discussion on efficiency, please refer to our response to your W2. \n\n| Test| 100 | 500|1000|\n|------------|-------|-------|-------|\n|SVHN|99.8±0.2 |100.0±0.0|100.0±0.0|\n|CIFAR100|67.6±0.3 |89.1±0.1|93.9±0.0|\n|CIFAR10.1|48.51±0.6|51.81±0.2|43.21±0.0|\n\n**Q2. Extreme Cases**\n\nIn fact, our implementation already considers the issue of extreme values. Our core strategy is to ensure that the density ratio remains within a reasonable range by using appropriate activation functions and subsequently applying a proximal algorithm to keep the likelihood ratio within our defined interval. Additionally, we explain that the sample size does not affect the range of the likelihood. We also add experimental explanations in the table below regarding the impact of the number of samples $M$ on data with many class labels.\n\nSpecifically, in constructing the adapter (Eq. (9)), we introduce Softplus and GELU to ensure that the density ratio stays within a reasonable range (experimentally within (0.5, 5)). According to Eq. (17), the likelihood ratio depends on the density ratio and can be analytically solved using this formula. In practice, we project the final values to ensure they fall within our predefined interval, which is set to (0.5, 5) in our experiments.\n\nRegarding the large data issue, the $M$ in Eq. (17) cancels out in the denominator of the second term, so the number of samples does not affect the likelihood value. Also, the number of class labels does not impact the likelihood value since the formula does not include them. Semantically different datasets have minimal bias and can be distinguished. However, semantically similar datasets may be misclassified as different distributions, presenting a lower AUROC, indicating that I-Div cannot distinguish between two subsets from CIFAR100.\n\n| Train| Test | 10 | 50 | 100 | 1000 |\n|------------|------------|-------|-------|-------|-------|\n|SVHN|CIFAR100|94.2±2.6 |97.7±1.1|100±0|100±0|\n|CIFAR100|CIFAR100|83.6±3.2|73.6±1.6|62.4±0.4|50±0|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCg6jSWKz4", "reviewer": "Reviewer_fpGs", "summary": "This paper investigates the applicability of hypotheses derived from training datasets to distinctly different test datasets through a series of experiments using various datasets, including CIFAR10, SVHN, PACS, and Office-Home. T", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "New Algorithm Introduction: The paper introduces the I-Div algorithm, which demonstrates a novel approach to evaluating distribution discrepancies between training and test datasets. This innovation contributes to the understanding of hypothesis applicability in machine learning.\n\nDiverse Experimental Scenarios: The study explores various scenarios, including semantically dissimilar data, semantically similar data, corrupted data, and adversarial data. This comprehensive approach offers a fresh perspective on the generalization and robustness of machine learning algorithms.\n\nRobustness and Generalization: The paper’s exploration of the algorithm’s robustness against noise and adversarial attacks, as well as its performance across different sample sizes and network architectures, adds depth to the quality of the research.", "weaknesses": "Outdated Baselines: The baseline comparison, particularly using MMD-D from 2020, is quite outdated. This diminishes the paper's novelty and effectiveness compared to state-of-the-art methods. To enhance the paper's relevance and credibility, it would be beneficial to include more recent benchmark algorithms, preferably those introduced after 2021, or popular models like ViT (Vision Transformer) and CLIP (Contrastive Language-Image Pre-training), which have shown significant impact in computer vision.\n\n\nBackbone and Feature Encoder: Indeed, the choice of backbone in the paper does not align well with recent developments in computer vision. Modern visual tasks increasingly rely on Transformer-based architectures such as ViT and CLIP, which excel in handling interactions between visual and language modalities. Introducing these modern feature encoders in the experiments would allow the algorithm to be tested on more complex and diverse datasets, better reflecting its generalization ability and practical utility.\n\n\nPerformance of I-DIV in Table 1: I concern about I-DIV achieving 100% performance in Table 1 is valid. In practical applications, achieving 100% accuracy is nearly impossible, especially on complex datasets and real-world scenarios. This might suggest some idealization or overfitting in the experimental setup, particularly in how the model or algorithm is tailored to specific datasets or test conditions. Therefore, it's advisable to provide a more detailed explanation of these experimental results in the paper, discussing their applicability and generalization capability in real-world environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and constructive criticisms. Here are our responses to your identified questions.\n\n**Q1&2. Baselines and Backbones** \n\nThe paper has already considered post-2021 algorithms, such as NNBD (ICML’21) [R1] and R-Div (NeurIPS’23) [R2] shown in Tables 1, 2, 4 and 5. To further address your concerns, we add H-Div (ICLR’22) [R3] and utilized updated network architectures (ResNet50, ViT, and CLIP) on diverse datasets (ImageNet, OID, Places365, StanfordCars, Caltech101, and DTD) to revise the paper. The additional experimental results are shown below.\n\nIt is important to note that the training data used for the pre-trained CLIP model is not accessible. According to Figure 4 in the CLIP paper [R4], CLIP achieves strong zero-shot performance on ImageNet. We, therefore, consider this dataset as part of the training data to estimate the differences between this dataset and the other test datasets. \n\nThe experimental results show that the AUROC values of I-Div better reflect the semantic similarity between datasets and ImageNet, aligning with human intuition. Specifically, ImageNet is more similar to OID and Places365 due to their shared everyday objects and scenes, leading to lower AUROC values of I-Div. Conversely, ImageNet is quite different from Caltech101 and DTD, which focus on specific object categories and textures, resulting in higher AUROC values of I-Div. However, the other algorithms show higher AUROC values across most datasets, indicating they struggle to capture the nuanced semantic and visual similarities between ImageNet and more diverse datasets. This results in less accurate distinctions compared to the I-Div algorithm, which aligns better with human intuition.\n\n| Backbone| Method | OID  | Places365 | StandfordCars | Caltech101 | DTD  |\n|----------|--------|------|-----------|---------------|------------|------|\n| ResNet50 | MSP | 100.0| 100.0| 100.0 | 100.0 | 100.0|\n|| NNBD   | 91.7 | 90.5  | 96.6 | 95.3 | 98.7 |\n||MMD-D  | 94.6 | 93.2| 98.6| 96.5| 100.0|\n||H-Div  | 100.0| 100.0| 100.0| 100.0| 100.0|\n||R-Div  | 94.6 | 100.0| 100.0| 100.0| 100.0|\n||I-Div  | **69.3**| **78.9** | **86.5**| **100.0**| **100.0**|\n| ViT      | MSP | 100.0| 100.0 | 100.0  | 100.0| 100.0|\n||NNBD   | 88.6 | 98.3 | 95.5 | 96.3| 99.7 |\n||MMD-D  | 92.6 | 100.0  | 96.3  | 95.6| 100.0|\n||H-Div  | 100.0| 100.0| 100.0| 100.0 | 100.0|\n||R-Div  | 92.6 | 100.0| 100.0| 100.0| 100.0|\n||I-Div  | **62.6**| **74.5** | **79.6**| **100.0**| **100.0**|\n| CLIP     | MSP | 100.0| 100.0| 100.0| 100.0| 100.0|\n||NNBD   | 93.2 | 91.6| 91.3| 95.9| 100.0|\n||MMD-D  | 96.3 | 93.7| 94.5| 98.6| 100.0|\n||H-Div  | 100.0| 100.0| 100.0| 100.0| 100.0|\n||R-Div  | 91.2 | 100.0| 100.0| 100.0| 100.0|\n||I-Div  | **61.3**| **71.8** | **83.5**| **100.0**| **100.0**|\n\n**Q3. Performance of I-DIV** \n\nIt appears there may have been a misunderstanding regarding the objective of our algorithm. Our goal is for the distribution discrepancy to align with human prior knowledge. As demonstrated in Tables 1 and 2, the effectiveness of the algorithm should be evaluated within the context of specific data. Achieving 100% distinction is not always feasible and does not necessarily indicate superior performance.\n\nIn cases where I-Div achieves 100% distinction, it suggests that, according to a given pre-trained network, the training and test data are certainly not from the same distribution. While we agree that achieving 100% classification accuracy in general is unrealistic, if the goal is to simply determine whether data samples come from the same distribution and have clear semantic differences, then achieving 100% distinction can be straightforward.\n\nSpecifically, in Table 1, the datasets show clear semantic differences, making it normal and straightforward for algorithms to achieve 100% distinction. This result is not due to model overfitting but rather reflects the inherent separability of the data categories. When datasets have distinct semantic boundaries, algorithms can easily detect these differences, especially with large dataset sizes that provide ample data for learning and differentiation. This phenomenon is consistent with the findings in the literature [R1-R3], where similar distinctions were observed, demonstrating that such results are typical in cases of clearly defined semantic classes. The clarity and volume of the data naturally lead to high accuracy in distinguishing between categories, confirming that the observed distinctions are a standard outcome under these conditions.\n\nConversely, in Table 2, which involves semantically similar datasets, I-Div achieves around 50% instead of 100% accuracy. This indicates that I-Div struggles to distinguish these samples, which is more intuitive. For example, the PACS dataset includes four domains with different styles, but consistent class semantics. Other algorithms like R-Div and MSP can distinctly separate all datasets, possibly because they do not rely on a pre-trained network and continually seek out differences during training. However, I-Div measures the discrepancy between two datasets based on a pre-trained network, considering the network predictive ability on the test data. If the network performs well on the test data, I-Div finds it challenging to distinguish between training and test data, aligning with our intended outcome.\n\n[R1] Kato et al., Non-negative bregman divergence minimization for deep direct density ratio estimation, ICML, 2021.\n\n[R2] Zhao et al, R-divergence for estimating model-oriented distribution discrepancy, NeurIPS, 2023.\n\n[R3] Zhao et al., Comparing distributions by measuring differences that affect decision making, ICLR, 2022.\n\n[R4] Radford et al., Learning transferable visual models from natural language supervision, ICML, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbQwQZ9iNl", "reviewer": "Reviewer_f37S", "summary": "The paper proposes a novel concept called Importance Divergence (I-Div) to address the lack of labeling of test samples and to measure the difference between training and testing distributions. Through the computation of importance samples, density ratios, and likelihood ratios, I-Div is able to assess the applicability of hypotheses on different datasets without the need for test labels. The main contribution of this method is that it provides a new way to quantify and reduce the expected risk difference between training and testing distributions, thus enhancing the generalization capability of machine learning models in the face of unknown test data.", "questions": "It is recommended that in future research, more simplified or optimized algorithmic processes be explored to reduce the computational cost and improve the practical feasibility of the method.", "limitations": "yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The study proposes a novel concept, which helps to estimate the difference between training and testing distributions, and adapts to the assessment of generalization ability between different datasets. This method can be tested for applicability on different datasets and task types, and can be widely used in a variety of machine learning scenarios, especially in practical applications when the training and testing data distributions are inconsistent.", "weaknesses": "Since this method relies on accurate estimation of the density ratio and likelihood ratio, this may limit the general feasibility of the method in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and suggestions on the limitations and potential improvements of our method. Here are our responses.\n\n**W1. General Feasibility**\n\nThe estimation of the density ratio and likelihood ratio does not limit the applicability of our method. We transform the issue of unknown test sample labels into a problem of estimating these ratios, which are both straightforward to estimate. The density ratio can be estimated using a lightweight adaptor and Eq. (13) within any given hypothesis. The likelihood ratio can be derived analytically by minimizing the upper bound of the generalization error, as shown in Eq. (17), or it can be simplified to a constant value, along with the following experiments.\n\nSpecifically, in practical applications, we can simplify the algorithm using only the density ratio and setting the likelihood ratio to a constant value. As discussed in Line 188 of the paper, setting the likelihood ratio to a constant (e.g., 1) is equivalent to assuming a covariate shift in the data. In contrast, using the adaptive likelihood ratio estimation in Eq. (17) assumes a stronger semantic shift.\n\nTo clarify, we add corresponding experiments on comparing the performance before and after simplifying the likelihood ratio, and test them using ResNet18 trained on CIFAR10, with test datasets including SVHN, SEMEION, CIFAR10.1, CIFAR10 corrupted by Gaussian noise and adversarial samples (perturbation factor 0.01). The results show little difference in performance. For instance, both setups clearly distinguish between SVHN and CIFAR10. However, with the more accurate likelihood ratio estimation from Eq. (17), I-Div aligns more closely with human prior knowledge, such as recognizing smaller differences between CIFAR10, CIFAR10.1, and adversarial samples. These results demonstrate that even with a simplified likelihood ratio, our method remains effective and practical for various applications. The density ratio and likelihood ratio estimations are both robust and easy to implement, ensuring the general feasibility of our method in real-world scenarios.\n\n| Method| SVHN| SEMEION| CIFAR10.1| CIFAR10 corrupted by Gaussian noise |Adversarial(0.01)|\n|------------|-------|-------|-------|-------|-------|\n|I-Div(constant)| 100.0 |100.0|43.4|46.5 |28.6|\n|I-Div(adaptive)|100.0|100.0|48.6| 49.2 |38.7|\n\n**Q1. Optimization Process**\n\nYour suggestion aligns well with our current direction, and we indeed have been considering similar approaches.\n\nThe current algorithm addresses the challenge of unknown class labels in the test samples using density and likelihood ratios for sampling transfer, as detailed in Eq. (4). This approach makes computational effort on estimating these ratios using an adaptor. Our preliminary idea is to treat both training and test samples as originating from an unknown labeling hypothesis. While we cannot know this labeling hypothesis, we do know the ground truth labels for the training data. We can explore the hypothesis space where this labeling hypothesis resides to derive an upper bound for Eq. (4). By applying the Legendre-Fenchel duality, we can simplify this upper bound and then minimize it to estimate the discrepancy between the two distributions. We would be happy to discuss any suggestions or thoughts you might have on this approach.\n\nWe appreciate the constructive suggestions and are committed to incorporating these insights into our future research to improve the applicability and efficiency. Thank you for your thoughtful review and guidance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for constructive comments and are encouraged by the overall positive feedback from the review. Specifically, the reviewers found that our work addresses an important and practical problem (Reviewers yuQt, 1gtU, fpGs, f37S), introduces a novel and intuitive approach (Reviewers yuQt, 1gtU, fpGs, f37S), and demonstrates robustness and generalization through comprehensive experiments (Reviewers yuQt, 1gtU, fpGs). Additionally, the reviewers appreciated the presentation of the paper (Reviewers yuQt, 1gtU, f37S).

We have made significant changes in creating a new version of this paper by addressing all the comments and substantially improving the paper quality. The main changes include:

1. Added extensive experiments on more complex datasets (ImageNet, Open Images Dataset (OID), Places365, StanfordCars, Caltech101, and DTD), models (ResNet50, ViT, and CLIP), and additional comparison algorithms (H-Div). The experimental table is included in the PDF;

2. Added parameter analysis ($\lambda$ and $\gamma$), time complexity evaluation, and ablation studies (with and without class label information, likelihood ratio estimation based on weak and strong assumptions);

3. Expanded discussions on the limitations, potential solutions, experimental setups, formula derivations, and experimental results, along with the relevant experiments;

4. Included a detailed explanation to the impact of pre-trained classifier and the likelihood ratio derived from the generalization error bound.


We sincerely thank the reviewers for their time and effort in evaluating our work and providing valuable suggestions for improvement. We are committed to continuously enhancing our research and its value to the community. Please feel free to reach out if there are any further questions or clarifications needed. We look forward to presenting our improved work to you.

Meta-review: This paper introduces a discrepancy measure to quantify the difference between two distributions, i.e., the labeled training set distribution and the unlabeled test set distribution. This discrepancy is derived from the expected risk difference between these distributions, built on a model pre-trained on the training data. The experimental results show the general applicability of the proposed method. The reviewers acknowledge the novelty and the effectiveness of the proposed method. I recommend it for acceptance.
Decision: Accept (poster)