SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\epsilon_ {u} (\widehat {h} _ {u}) - \epsilon_ {u} (h _ {u} ^ {*}) \leq 2 \mathcal {R} _ {H} ^ {l} (\widehat {u}) + 5 c \sqrt {\frac {\ln (8 / \delta)}{N}}.
$$
Lemma A.3 ([44], Talagrand's contraction lemma). For any  $L$ -Lipschitz loss function  $l(\cdot, \cdot)$ , we obtain,
$$
\mathcal {R} (l \circ \mathcal {H} \circ \widehat {u}) \leq L \mathcal {R} (\mathcal {H} \circ \widehat {u}),
$$
where  $\mathcal{R}_H^l (\widehat{u}) = \frac{1}{2N}\mathbb{E}_{\pmb {\sigma}\sim \{\pm 1\}^{2N}}\left[\sup_{h\in \mathcal{H}}\sum_{x\in \widehat{u}}\sigma_ih(x_i)\right]$
Lemma A.4 ([18], Theorem 1). Let  $\mathcal{H}$  be the class of real-valued networks of depth  $D$  over the domain  $\mathcal{X}$ . Assume the Frobenius norm of the weight matrices are at most  $M_1, \ldots, M_d$ . Let the activation function be 1-Lipschitz, positive-homogeneous, and applied element-wise (such as the ReLU). Then,
$$
\mathbb {E} _ {\sigma \in \{\pm 1 \} ^ {N}} \left[ \sup _ {h \in \mathcal {H}} \sum_ {i = 1} ^ {N} \sigma_ {i} h (x _ {i}) \right] \leq \sqrt {N} B (\sqrt {2 D \ln 2} + 1) \prod_ {i = 1} ^ {D} M _ {i}.
$$
A.2 Main Results
$$
| \mathcal {D} _ {R} (p \| q) - \widehat {\mathcal {D}} _ {R} (\widehat {p} \| \widehat {q}) | \leq 2 \mathcal {D} _ {\mathrm {L}} (p \| q) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {p}) + 2 \mathcal {R} _ {\mathcal {H}} ^ {l} (\widehat {q}) + 1 2 c \sqrt {\frac {\ln (8 / \delta)}{N}}.
$$
$$
\begin{array}{l} \left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \\ = \left| | \epsilon_ {q} (h _ {u} ^ {*}) - \epsilon_ {p} (h _ {u} ^ {*}) | - | \widehat {\epsilon} _ {\widehat {q}} (\widehat {h} _ {u}) - \widehat {\epsilon} _ {\widehat {p}} (\widehat {h} _ {u}) | \right| \\ \leq \left| \epsilon_ {q} \left(h _ {u} ^ {*}\right) - \epsilon_ {p} \left(h _ {u} ^ {*}\right), - \widehat {\epsilon} _ {\widehat {q}} \left(\widehat {h} _ {u}\right) + \widehat {\epsilon} _ {\widehat {p}} \left(\widehat {h} _ {u}\right) \right| \\ = \left| \epsilon_ {q} \left(h _ {u} ^ {*}\right) - \epsilon_ {p} \left(h _ {u} ^ {*}\right) + \widehat {\epsilon} _ {\widehat {p}} \left(\widehat {h} _ {u}\right) - \epsilon_ {p} \left(\widehat {h} _ {u}\right) + \epsilon_ {p} \left(\widehat {h} _ {u}\right) - \epsilon_ {q} \left(\widehat {h} _ {u}\right) + \epsilon_ {q} \left(\widehat {h} _ {u}\right) - \widehat {\epsilon} _ {\widehat {q}} \left(\widehat {h} _ {u}\right) \right| \tag {7} \\ \leq \left| \epsilon_ {q} \left(h _ {u} ^ {*}\right) - \epsilon_ {p} \left(h _ {u} ^ {*}\right) \right| + \left| \epsilon_ {p} \left(\widehat {h} _ {u}\right) - \epsilon_ {q} \left(\widehat {h} _ {u}\right) \right| + \left| \widehat {\epsilon_ {\widehat {p}}} \left(\widehat {h} _ {u}\right) - \epsilon_ {p} \left(\widehat {h} _ {u}\right) \right| + \left| \epsilon_ {q} \left(\widehat {h} _ {u}\right) - \widehat {\epsilon_ {\widehat {q}}} \left(\widehat {h} _ {u}\right) \right| \\ \leq 2 \mathcal {D} _ {\mathrm {L}} (p \| q) + \left| \widehat {\epsilon} _ {\widehat {p}} (\widehat {h} _ {u}) - \epsilon_ {p} (\widehat {h} _ {u}) \right| + \left| \epsilon_ {q} (\widehat {h} _ {u}) - \widehat {\epsilon} _ {\widehat {q}} (\widehat {h} _ {u}) \right|. \\ \end{array}
$$
The first two inequalities are owing to the triangle inequality, and the third inequality is due to the definition of L-divergence Eq.(5). We complete the proof by applying Lemma A.1 to bound  $\left|\widehat{\epsilon_{\widehat{p}}} (\widehat{h}_u) - \epsilon_p(\widehat{h}_u)\right|$  and  $\left|\epsilon_q(\widehat{h}_u) - \widehat{\epsilon}_{\widehat{q}}(\widehat{h}_u)\right|$  in Eq.(7).
Corollary 4.2. Following the conditions of Theorem 4.1, the upper bound of  $\sqrt{\operatorname{Var}\left[\widehat{\mathcal{D}}_R(\widehat{p} \| \widehat{q})\right]}$  is
Proof. For convenience, we assume  $\mathcal{C} = 2\mathcal{D}_{\mathrm{L}}(p\| q) + 2\mathcal{R}_{\mathcal{H}}^{l}(\widehat{p}) + 2\mathcal{R}_{\mathcal{H}}^{l}(\widehat{q})$ . Following to proof of Corollary 1 in [62], we have
$$
\begin{array}{l} \operatorname {V a r} \left[ \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right] \leq \mathbb {E} \left[ \left(\mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q})\right) ^ {2} \right] \\ = \int_ {t = 0} ^ {\infty} \Pr \left[ \left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \geq \sqrt {t} \right] d t \\ \stackrel {s = \sqrt {t}} {=} \int_ {s = 0} ^ {\infty} \Pr \left[ \left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \geq s \right] 2 s d s \\ \leq \int_ {s = 0} ^ {\mathcal {C}} 2 s d s + \int_ {s = 0} ^ {\infty} \Pr \left[ \left| \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) - \mathcal {D} _ {\mathrm {R}} (p \| q) \right| \geq \mathcal {C} + s \right] 2 (\mathcal {C} + s) d s \\ \leq \mathcal {C} ^ {2} + 1 6 \int_ {s = 0} ^ {\infty} (\mathcal {C} + s) e ^ {\frac {- N s ^ {2}}{1 4 4 c ^ {2}}} d s \tag {8} \\ = \mathcal {C} ^ {2} + 1 6 \mathcal {C} \int_ {s = 0} ^ {\infty} e ^ {\frac {- N s ^ {2}}{1 4 4 c ^ {2}}} d s + 1 6 \int_ {s = 0} ^ {\infty} s e ^ {\frac {- N s ^ {2}}{1 4 4 c ^ {2}}} d s \\ \end{array}
$$
$$
\begin{array}{l} { } ^ { t = \frac { s } { c } } \overset { \sqrt [ N ] { 1 4 4 } } { = } \mathcal { C } ^ { 2 } + 1 6 \mathcal { C } c \sqrt { \frac { a } { N } } \int _ { t = 0 } ^ { \infty } e ^ { - t ^ { 2 } } d t + \frac { 2 3 0 4 c ^ { 2 } } { N } \int _ { s = 0 } ^ { \infty } t e ^ { - t ^ { 2 } } d t \\ = \mathcal {C} ^ {2} + 9 6 \mathcal {C} c \sqrt {\frac {\pi}{N}} + \frac {1 1 5 2 c ^ {2}}{N} \\ \leq \left(\frac {3 4 c}{\sqrt {N}} + \mathcal {C} \sqrt {2 \pi}\right) ^ {2}, \\ \end{array}
$$
where the third inequality is due to
$$
\Pr \left[ \left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \geq \mathcal {C} + s \right] \leq 8 e ^ {\frac {- N s ^ {2}}{1 4 4 c ^ {2}}}, \tag {9}
$$
obtained from the results of Theorem 4.1.
Proof. We complete the proof by applying Lemma A.3 and Lemma A.4 to bound the Rademacher complexity of deep neural networks in Theorem 4.1.  $\square$
$$
| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) | \leq \frac {8 L B (\sqrt {2 D \ln 2} + 1) \prod_ {i = 1} ^ {D} M _ {i}}{\sqrt {N}} + 2 2 c \sqrt {\frac {\ln (1 6 / \delta)}{N}}.
$$
Proof. Following the proof of Theorem 4.1, we have
$$
\begin{array}{l} \left| \mathcal {D} _ {\mathrm {R}} (p \| q) - \widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) \right| \\ \leq \left| \epsilon_ {q} (h _ {u} ^ {*}) - \epsilon_ {p} (h _ {u} ^ {*}) + \widehat {\epsilon} _ {\widehat {p}} (\widehat {h} _ {u}) - \epsilon_ {p} (\widehat {h} _ {u}) \right| + \left| \widehat {\epsilon} _ {\widehat {p}} (\widehat {h} _ {u}) - \epsilon_ {p} (\widehat {h} _ {u}) \right| + \left| \epsilon_ {q} (\widehat {h} _ {u}) - \widehat {\epsilon} _ {\widehat {q}} (\widehat {h} _ {u}) \right| \\ = \max  \left(\left| \epsilon_ {q} \left(h ^ {*}\right) - \epsilon_ {p} \left(h ^ {*}\right) + \epsilon_ {p} (\widehat {h}) - \epsilon_ {q} (\widehat {h}) \right|, \left| \left(\epsilon_ {q} \left(h ^ {*}\right) + \epsilon_ {p} \left(h ^ {*}\right)\right) - \left(\epsilon_ {p} (\widehat {h}) - \epsilon_ {q} (\widehat {h})\right) \right|\right) \\ + \left| \widehat {\epsilon_ {p}} (\widehat {h} _ {u}) - \epsilon_ {p} (\widehat {h} _ {u}) \right| + \left| \epsilon_ {q} (\widehat {h} _ {u}) - \widehat {\epsilon_ {q}} (\widehat {h} _ {u}) \right| \\ \leq 2 \max  \left(D _ {\mathrm {L}} (p \| q), \left| \epsilon_ {u} \left(h ^ {*}\right) - \epsilon_ {u} (\widehat {h}) \right|\right) + \left| \widehat {\epsilon} _ {\widehat {p}} \left(\widehat {h} _ {u}\right) - \epsilon_ {p} \left(\widehat {h} _ {u}\right) \right| + \left| \epsilon_ {q} \left(\widehat {h} _ {u}\right) - \widehat {\epsilon} _ {\widehat {q}} \left(\widehat {h} _ {u}\right) \right| \\ \leq 2 \left| \epsilon_ {u} (h ^ {*}) - \epsilon_ {u} (\widehat {h}) \right| + \left| \widehat {\epsilon} _ {\widehat {p}} (\widehat {h} _ {u}) - \epsilon_ {p} (\widehat {h} _ {u}) \right| + \left| \epsilon_ {q} (\widehat {h} _ {u}) - \widehat {\epsilon} _ {\widehat {q}} (\widehat {h} _ {u}) \right|. \\ \end{array}
$$
The first two inequalities are owing to the triangle inequality, and the third inequality is due to the given condition  $\mathcal{D}_{\mathrm{L}}(p\| q)\leq \left|\epsilon_u(h_u^*) - \epsilon_u(\widehat{h}_u)\right|$ . We complete the proof by applying Lemma A.2 to bound  $\left|\epsilon_{u}(h^{*}) - \epsilon_{u}(\widehat{h})\right|$  and Lemma A.1 to bound  $\left|\widehat{\epsilon}_{\widehat{p}}(\widehat{h}_u) - \epsilon_p(\widehat{h}_u)\right|$  and  $\left|\epsilon_q(\widehat{h}_u) - \widehat{\epsilon}_{\widehat{q}}(\widehat{h}_u)\right|$  in Eq.(7).
Proof. Based on the result on Proposition 4.3, for any  $\delta \in (0,1)$ , we know that
$$
\frac {4 L B (\sqrt {2 D \ln 2} + 1) \prod_ {i = 1} ^ {D} M _ {i}}{\sqrt {N}} + 1 2 c \sqrt {\frac {\ln (8 / \delta)}{N}} \rightarrow 0, \tag {11}
$$
when  $N\to \infty$  . We complete the proof by applying the triangle inequality.
B Algorithm Procedure
B.1 Training Procedure of R-Div
Algorithm 1 Estimate model-oriented distribution discrepancy by R-divergence
Input: two datasets  $\widehat{p}$  and  $\widehat{q}$  and a learning model  $\mathcal{T}$  with hypothesis space  $\mathcal{H}$  and loss function  $l$  Generate the merged dataset:  $\widehat{u} = \widehat{p} \cup \widehat{q}$
Learn the minimum hypothesis on the mixed data:
$$
\widehat{h}_{u}\in \operatorname *{arg  min}_{h\in \mathcal{H}}\widehat{\epsilon}_{\widetilde{u}}(h)
$$
Evaluate the empirical risks:  $\widehat{\epsilon}_{\widehat{p}}(\widehat{h}_{\widehat{u}})$  and  $\widehat{\epsilon}_{\widehat{q}}(\widehat{h}_{\widehat{u}})$
Estimate the R-divergence as the discrepancy:
$$
\widehat {\mathcal {D}} _ {\mathrm {R}} (\widehat {p} \| \widehat {q}) = | \widehat {\epsilon_ {\widehat {p}}} (\widehat {h} _ {\widehat {u}}) - \widehat {\epsilon_ {\widehat {q}}} (\widehat {h} _ {\widehat {u}}) |
$$
Output: empirical estimator  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}\| \widehat{q})$
B.2 Calculation Procedure of the Average Test Power
Algorithm 2 Calculate the average test power of R-divergence
Input: datasets  $\widehat{p}$  and  $\widehat{q}$ , empirical estimator  $\widehat{\mathcal{D}}_{\mathrm{R}}(\cdot \| \cdot)$ ,  $\alpha$ ,  $K$ ,  $Z$   
for  $k = 1$  to  $K$  do  
    for  $z = 1$  to  $Z$  do  
        if  $z == 1$  then  
             $(\widehat{p}^{1}, \widehat{q}^{1}) = (\widehat{p}, \widehat{q})$   
        else  
            Generate  $(\widehat{p}^{z}, \widehat{q}^{z})$  by uniformly randomly swapping samples between  $\widehat{p}$  and  $\widehat{q}$   
    end if  
    Calculate  $\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}^{z} \| \widehat{q}^{z})$   
end for  
Obtain  $\mathcal{G} = \{\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p}^{z} \| \widehat{q}^{z})\}_{z=1}^{Z}$ $r_k = \left(\widehat{\mathcal{D}}_{\mathrm{R}}(\widehat{p} \| \widehat{q})\right)$  is in the top  $\alpha$ -quantile among  $\mathcal{G}$   
end for  
Output: average test power  $\frac{\sum_{k=1}^{K} r_k}{K}$
Table 7: Method Comparison.
Methods	Discrepancy	Remarks
ME	√1/J∑j=1J(μp(Tj)−μq(Tj))2	I:μp(t) = ∫k(x,t)dp(x)
II:k(x,t) = exp(−||x−y||2/γ2)
III:T is drawn from an absolutely continuous distribution.
SCF	√1/2J∑j=12(zpsin(Tj)−zpsin(Tj))2+(zpcos(Tj)−zpcos(Tj))2	I:zs sin(t) = ∫κ(x) sin(xTt) dp(x)
II:zpcos(t) = ∫κ(x) cos(xTt) dp(x)
III:κ(x) is the Fourier transform of a kernel.
C2STS-S	1/|pte|∑x∈hatle fw(x) − 1/|qte|∑x′∈qte fw(x')	I:hattr, hatle ~ p, qtr, qte ~ q
II:fw is a binary classifier to separate hattr and qtr
C2ST-L	1/|hatle|+|qte|∑x∈hatle gw(x,0) + ∑x′∈qte gw(x',1)	III: Samples from p and q are labeled with 0 and 1, respectively.
IV: gw(x,y) = I [Π (fw(x) > 1/2) = y]
MMD-O	√E [k(x,x') + k(y,y') - 2k(x,y)], x, x ~ p, y, y' ~ q	I:k(x,y) is a simple kernel.
MMD-D	√E [kw(x,x') + kw(y,y') - 2kw(x,y)], x, x ~ p, y, y' ~ q	I:kw(x,y) = [(1-ε)k1(φw(x), φw(y)) + ε]k2(x,y)
II:φw is deep network with parameters w
III:kl(x,y) = exp(−||x−y||2/γ12)
IV:kl(x,y) = exp(−||x−y||2/γ2)
H-Div	φ(εu(hu*)−εp(hp*), εu(hu*)−εq(hq*))	I:h* ∈ arg minh∈H εu(h)
II:h* ∈ arg minh∈H εq(h)
III:εu(h) = E×~u l(h(x),a(x))
R-Div	|εp(hu*)−εq(hu*)|	IV:εq(h) = E×~ql(h(x),a(x))
V:φ(θ,λ) = θ+λ/2 or max(θ,λ)
C Compared Methods
Mean embedding (ME) [11] and smooth characteristic functions (SCF) [30] are the state-of-the-art methods using differences in Gaussian mean embeddings at a set of optimized points and frequencies, respectively. Classifier two-sample tests, including C2STS-S [43] and C2ST-L [10], apply the classification accuracy of a binary classifier to distinguish between the two distributions. The binary classifier treats samples from one dataset as positive and the other dataset as negative. These two methods assume that the binary classifier cannot distinguish these two kinds of samples if their distributions are identical. Differently, C2STS-S and C2ST-L apply the test error and the test error gap to evaluate the discrepancy, respectively. MMD-O [20] measures the maximum mean discrepancy (MMD) with a Gaussian Kernel [19], and MMD-D [41] improves the performance of MMD-O by replacing the Gaussian Kernel with a learnable deep kernel. H-Divergence (H-Div) [62] learns optimal hypotheses for the mixture distribution and each individual distribution for the specific model, assuming that the expected risk of training data on the mixture distribution is higher than that on each individual distribution if the two distributions are identical. The equations of these compared methods are presented in Figure 7.
D.1 Benchmark Dataset
Figure 3: The average test power on HDGM with the significant level  $\alpha = 0.05$ . Left panel: results with the same sample size (4,000) and different feature dimensions. Right panel: results with the same feature dimensions (10) and different sample sizes.
Figure 4: The average test power on Blob at the significant levels  $\alpha = 0.05$  and  $\alpha = 0.01$ .
Table 8: The average test power  $\pm$  standard error with the significant level  $\alpha  = {0.05}$  on MNIST.  $N$  represents the number of samples in each given dataset, and boldface values represent the relatively better discrepancy estimation.
N	200	400	600	800	1000	Avg.
ME	0.414±0.050	0.921±0.032	1.000±0.000	1.000±0.000	1.000±0.000	0.867
SCF	0.107±0.018	0.152±0.021	0.294±0.008	0.317±0.017	0.346±0.019	0.243
C2STS-S	0.193±0.037	0.646±0.039	1.000±0.000	1.000±0.000	1.000±0.000	0.768
C2ST-L	0.234±0.031	0.706±0.047	0.977±0.012	1.000±0.000	1.000±0.000	0.783
MMD-O	0.188±0.010	0.363±0.017	0.619±0.021	0.797±0.015	0.894±0.016	0.572
MMD-D	0.555±0.044	0.996±0.004	1.000±0.000	1.000±0.000	1.000±0.000	0.910
H-Div	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000
R-Div	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000±0.000	1.000
D.2 PACS Dataset
Figure 5: The average test power at the significant level  $\alpha = 0.05$  on the art painting and cartoon domains of PACS.
D.3 Learning with Noisy Labels
(b) Detection
Figure 6: Results on CIFAR10 with pair flipping. All values are averaged over five trials. Left: Classification accuracy of pretrained and retrained networks. Middle: Precision and recall rates of detecting clean and noisy samples. Right: Discrepancy between predicted clean and noisy samples.