SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For future work, it is interesting to explore how model valuation can help address the practical considerations encountered in existing data valuation methods [70, 78, 80] and to apply this technique to existing collaborative (learning) frameworks which require a valuation of models, both non-parametric ones [2, 62, 72, 74, 81] and parameterized ones [18, 49, 79]. Moreover, a more detailed investigation into satisfying the equitability of Shapley value [59] and its trade-off with the computational cost [82] is of practical interest, such as by applying more sophisticated analyses [47] or methods [9, 31, 32, 52] for our proposed Gaussian process regression learning approach.
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-018). Xinyi Xu is supported by the Institute for Infocomm Research of Agency for Science, Technology and Research (A*STAR).
[1] A. Agarwal, M. Dahleh, and T. Sarkar. A marketplace for data: An algorithmic solution. In Proc. ACM-EC, 2019.
[2] L. Agussurja, X. Xu, and B. K. H. Low. On the convergence of the shapley value in parametric Bayesian learning games. In Proc. ICML, pages 180-196, 2022.
[3] S. Arora, R. Ge, Y. Halpern, D. Mimno, A. Moitra, D. Sontag, Y. Wu, and M. Zhu. A practical algorithm for topic modeling with provable guarantees. In Proc. ICML, pages 280-288, 2013.
[4] M. Basseville. Distance measures for signal processing and pattern recognition. Signal Processing, 18(4):349-369, 1989.
[5] J. A. Blackard and D. J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3):131-151, 1999.
[6] S. Boyd, S. P. Boyd, and L. Vandenberghe. Convex optimization. Cambridge Univ. Press, 2004.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Proc. NeurIPS, 2020.
[8] H. Chang, V. Shejwalkar, R. Shokri, and A. Houmansadr. Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. In Proceedings of the 1st NeurIPS Workshop on New Frontiers in Federated Learning (NFFL 2021), 2021.
[9] J. Chen, N. Cao, K. H. Low, R. Ouyang, C. K.-Y. Tan, and P. Jaillet. Parallel gaussian process regression with low-rank covariance matrix approximations. In Proc. UAI, page 152-161, 2013.
[10] L. Chen, P. Koutris, and A. Kumar. Model-based pricing: Do not pay for more than what you learn! In Proc. ACM SIGMOD Workshop on Data Management for End-to-end Machine Learning, 2017.
[11] L. Chen, P. Koutris, and A. Kumar. Towards model-based pricing for machine learning in a data marketplace. In Proc. ACM SIGMOD, page 1535-1552, 2019.
[12] H. Chernoff. A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations. The Annals of Mathematical Statistics, 23(4):493-507, 1952.
[13] Z. Cong, X. Luo, J. Pei, F. Zhu, and Y. Zhang. Data pricing in machine learning pipelines. Knowledge and Information Systems, 64:1417-1455, 2022.
[14] A. S. Das, M. Datar, A. Garg, and S. Rajaram. Google news personalization: Scalable online collaborative filtering. In Proc. WWW, 2007.
[15] P. Devijver and J. Kittler. Pattern Recognition: A Statistical Approach. Prentice-Hall, London, 1982.
[16] J. M. Drazen, S. Morrissey, D. Malina, M. B. Hamel, and E. W. Campion. The importance — and the complexities — of data sharing. New England Journal of Medicine, 375(12):1182-1183, 2016.
A.1 Results Related to Maximum Likelihood Estimation (MLE)
Log-likelihood function  $F(1)$ . For completeness, we provide the full log-likelihood function [36, 57, 65] using the predictions as follows: To avoid notational overload, let  $y_{j} \coloneqq \mathbf{M}_{i}(x_{j})$  denote the predictive probability vector of  $\mathbf{M}_i$  on  $x_{j}$  and its dependence on  $i$  is suppressed. Similarly, we suppress the dependence of  $\alpha$  on  $i$  since the context is clear. Then,
$$
\begin{array}{l} F (\boldsymbol {\alpha}, \{y _ {j} \} _ {j = 1, \dots , D}) = \log p (\{y _ {j} \} _ {j = 1, \dots , D} | \boldsymbol {\alpha}) = \log \prod_ {j = 1} ^ {D} p (y _ {j} | \boldsymbol {\alpha}) \\ = \log \prod_ {j = 1} ^ {D} \frac {\Gamma (\sum_ {k = 1} ^ {C} \alpha_ {k})}{\prod_ {k = 1} ^ {C} \Gamma (\alpha_ {k})} \prod_ {k = 1} ^ {C} y _ {j, k} ^ {\alpha_ {k} - 1} \\ = D \left[ \log \Gamma \left(\sum_ {k = 1} ^ {C} \alpha_ {k}\right) - \sum_ {k = 1} ^ {C} \log \Gamma (\alpha_ {k}) + \sum_ {k = 1} ^ {C} (\alpha_ {k} - 1) \log \bar {h} _ {i, k} \right] \\ \end{array}
$$
where  $\log \bar{h}_{i,k}$  is the  $k$ -th component of  $\log \bar{h}_i$ . Since the final expression of the above log-likelihood function  $F$  only depends on the log-predictions  $\{\log y_j\}_{j = 1,\dots ,D}$  through the observed sufficient statistics  $\log \bar{h}_i$  (i.e., with an element-wise log operation), we can directly use  $\bar{h}_i$  in (1) instead of  $\{y_{j}\}_{j = 1,\dots ,D}$ .
Similar predictions between models imply similar MLE approximations of their Dirichlet abstractions. Intuitively, two models produce similar predictions if and only if their Dirichlet abstractions are similar. The result below precisely formalizes this intuition by exploiting the analytic tractability of the log-likelihood function  $F(1)$  which is concave with a unique maximizer:
Proposition 3. Suppose that the observed sufficient statistics  $\log \bar{h}_i$  ( $\log \bar{h}_{i'}$ ) based on the predictions of  $\mathbf{M}_i(\mathbf{M}_{i'})$  on  $\mathcal{D}$  and the MLE approximation of its Dirichlet abstraction  $\alpha_i(\alpha_{i'})$  are given. Then,
$$
\left| \bar {h} _ {i} - \bar {h} _ {i ^ {\prime}} \right| _ {1} = 0 \Longleftrightarrow \boldsymbol {\alpha} _ {i} = \boldsymbol {\alpha} _ {i ^ {\prime}}.
$$
Proof. From (1),
$$
\begin{array}{l} F (\boldsymbol {\alpha}, \bar {h} _ {i}) = D \times \left[ G (\boldsymbol {\alpha}) + \log \bar {h} _ {i} \boldsymbol {\alpha} ^ {\top} - \log \bar {h} _ {i} \mathbf {1} ^ {\top} \right] \\ \operatorname * {a r g m a x} _ {\boldsymbol {\alpha}} F (\boldsymbol {\alpha}, \bar {h} _ {i}) = \operatorname * {a r g m a x} _ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha}) + \log \bar {h} _ {i} \boldsymbol {\alpha} ^ {\top} \\ = \operatorname * {a r g m a x} _ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha}) + \log \bar {h} _ {i ^ {\prime}} \boldsymbol {\alpha} ^ {\top} - \log \bar {h} _ {i ^ {\prime}} \boldsymbol {\alpha} ^ {\top} + \log \bar {h} _ {i} \boldsymbol {\alpha} ^ {\top} \\ = \underset {\boldsymbol {\alpha}} {\operatorname {a r g m a x}} F (\boldsymbol {\alpha}, \bar {h} _ {i ^ {\prime}}) + D \left(\log \bar {h} _ {i} - \log \bar {h} _ {i ^ {\prime}}\right) \boldsymbol {\alpha} ^ {\top}. \tag {4} \\ \end{array}
$$
Note that the RHS expression of (4) to be maximized remains concave since  $F(\alpha, \bar{h}_{i'})$  is concave in  $\alpha$  [36] and the  $D\left(\log \bar{h}_i - \log \bar{h}_{i'}\right)\alpha^\top$  term is linear in  $\alpha$ . If  $|\bar{h}_i - \bar{h}_{i'}|_1 = 0$ , then the  $D\left(\log \bar{h}_i - \log \bar{h}_{i'}\right)\alpha^\top$  term in (4) becomes 0. So,  $\alpha_i = \operatorname{argmax}_{\alpha} F(\alpha, \bar{h}_i) = \operatorname{argmax}_{\alpha} F(\alpha, \bar{h}_{i'}) = \alpha_{i'}$  since  $F$  is concave with a unique maximizer. Therefore,  $|\bar{h}_i - \bar{h}_{i'}|_1 = 0 \Rightarrow \alpha_i = \alpha_{i'}$ .
From Lemma 1 below,  $\log \bar{h}_{i,k} = \psi (\alpha_{i,k}) - \psi (|\pmb{\alpha}_i|_1)$  and  $\log \bar{h}_{i',k} = \psi (\alpha_{i',k}) - \psi (|\pmb{\alpha}_{i'}|_1)$  where  $\alpha_{i,k}(\alpha_{i',k})$  is the  $k$ -th component of  $\pmb{\alpha}_i(\pmb{\alpha}_{i'})$  and  $\psi$  is the digamma function. It follows immediately that
$$
\bar {h} _ {i, k} - \bar {h} _ {i ^ {\prime}, k} = \exp \left[ \psi \left(\alpha_ {i, k}\right) - \psi \left(\left| \boldsymbol {\alpha} _ {i} \right| _ {1}\right) \right] - \exp \left[ \psi \left(\alpha_ {i ^ {\prime}, k}\right) - \psi \left(\left| \boldsymbol {\alpha} _ {i ^ {\prime}} \right| _ {1}\right) \right],
$$
$$
\mathrm {s o} \alpha_ {i} = \alpha_ {i ^ {\prime}} \Rightarrow | \bar {h} _ {i} - \bar {h} _ {i ^ {\prime}} | _ {1} = 0.
$$
Importantly, both abstractions are able to preserve the similarity between  $\mathbf{M}_i$  and  $\mathbf{M}_{i'}$ , via either the Hellinger distance  $d_{\mathrm{H}}(\pmb{\alpha}_i,\pmb{\alpha}_{i'})$  (further discussed in Sec. 4.1) or the  $\ell_1$  distance  $|\bar{h}_i - \bar{h}_{i'}|_1$
The digamma function  $\psi (x)\coloneqq \mathrm{d} / \mathrm{d}x(\ln \Gamma (x))$  is a monotonically increasing function that converges to  $\ln x - 1 / 2x$  for any  $x > 0$ .
(Proposition 3 above) where a lower  $\ell_{1}$  or  $d_{\mathrm{H}}$  distance is equivalent to a higher similarity. We also empirically compare  $\alpha_{i},\bar{h}_{i}$  in Sec. 4.2 and find that in general  $\alpha_{i}$  is better.
Lemma 1. Suppose that the observed sufficient statistics  $\log \bar{h}_i$  based on the predictions of  $\mathbf{M}_i$  on  $\mathcal{D}$  and the MLE approximation of its Dirichlet abstraction  $\alpha_{i}$  are given. Then,
$$
\log \bar {h} _ {i, k} = \psi \left(\alpha_ {i, k}\right) - \psi \left(\left| \boldsymbol {\alpha} _ {i} \right| _ {1}\right).
$$
Proof. The partial derivative of  $F(\alpha, \bar{h}_i)$  w.r.t.  $\alpha_k$  can be explicitly derived as follows:
$$
\frac {\partial F (\boldsymbol {\alpha} , \bar {h} _ {i})}{\partial \alpha_ {k}} = D \left(\psi \left(\sum_ {k = 1} ^ {C} \alpha_ {k}\right) - \psi (\alpha_ {k}) + \log \bar {h} _ {i, k}\right).
$$
For a concave optimization problem (i.e., maximizing the log-likelihood) subject to a non-negative orthant constraint (i.e.,  $\alpha \succeq 0$ ), the work of [6] has shown that the optimality conditions can be expressed as
$$
\nabla F (\boldsymbol {\alpha}; \bar {h} _ {i}) \succeq 0 \quad \text {a n d} \quad \alpha_ {k} \times \frac {\partial F (\boldsymbol {\alpha} , \bar {h} _ {i})}{\partial \alpha_ {k}} = 0 \quad \text {f o r} k = 1, \dots , C.
$$
The last condition is known as the complementarity. Focusing on the complementarity condition w.r.t. the  $k$ -th components of  $\alpha_{i}$  and  $\bar{h}_i$  gives
$$
\alpha_ {i, k} = 0 \quad \vee \quad \frac {\partial F (\boldsymbol {\alpha} , \bar {h} _ {i})}{\partial \alpha_ {k}} = 0.
$$
Since  $\alpha_{i,k} > 0$  (Definition 1),
$$
\frac {\partial F (\boldsymbol {\alpha} , \bar {h} _ {i})}{\partial \alpha_ {k}} = D \left(\psi (| \boldsymbol {\alpha} _ {i} | _ {1}) - \psi (\alpha_ {i, k}) + \log \bar {h} _ {i, k}\right) = 0
$$
which is simplified to
$$
\psi \left(\left| \boldsymbol {\alpha} _ {i} \right| _ {1}\right) - \psi \left(\alpha_ {i, k}\right) + \log \bar {h} _ {i, k} = 0
$$
A.2 Results Related to Dirichlet Distribution and Distributional Distance
Suitable measures of distance between Dirichlet distributions. We highlight the challenges in applying the distance measures beside the Hellinger distance in App. A. Specifically, the Hellinger distance provides unique and desirable theoretical properties (e.g., satisfying the triangle inequality) which we exploit (e.g., to derive Theorem 1)
In our context, we seek a distance measure that is well-defined between two Dirichlet distributions, can be evaluated in closed form, and has analytic properties. The work of [63] has painstakingly compared several probabilistic distance measures including the Kullback-Leibler (KL) and symmetric KL divergences [38, 42], Patrick-Fischer distance [60], generalized Matusita distance [4], Lissack-Fu distance [50], Kolmogorov [15] distance, Chernoff distance [12], and Hellinger distance [27], and made the following observations: The KL and symmetric KL divergences and the Patrick-Fischer distance can encounter cases where the distance becomes undefined. The generalized Matusita distance, Lissack-Fu distance, and Kolmogorov distance all lack an anti-derivative. So, we consider and compare the remaining two options: Chernoff and Hellinger distances, which are connected as follows.
Chernoff vs. Hellinger distances. As mentioned earlier, the Chernoff distance  $d_{\mathrm{H}}$  is the other theoretically appealing choice for Dirichlet distributions [63], so we discuss its connection with the Hellinger distance as follows. We start by recalling the definition for Chernoff distance and derive an analytic connection between them.
Definition 4 (Chernoff distance [12]). The Chernoff distance between two distributions  $p$  and  $q$  is  $d_{\mathrm{C}}(p,q;\lambda)\coloneqq -\ln \int p^{\lambda}(x)q^{1 - \lambda}(x)\mathrm{d}x$  where  $\lambda \in (0,1)$ . Note that  $d_{\mathrm{C}}(p,q;\lambda = 1 / 2)$  is symmetric in  $p$  and  $q$ .
For two continuous distributions  $p, q$ , their Hellinger distance  $d_{\mathrm{H}}(p, q)$  and their Chernoff distance (with  $\lambda = 1/2$ )  $d_{\mathbb{C}}(p, q; \lambda = 1/2)$  are connected via the Bhattacharyya coefficient  $\operatorname{BC}(p, q) := \int \sqrt{p(x)q(x)} dx \in [0, 1]$  as follows,
$$
1 - d _ {\mathrm {H}} ^ {2} (p, q) = \exp (- d _ {\mathrm {C}} (p, q; \lambda = 1 / 2))
$$
by substituting the equalities
$$
d _ {\mathrm {H}} (p, q) = \sqrt {1 - \mathrm {B C} (p , q)} ,
$$
$$
d _ {\mathrm {C}} (p, q; \lambda = 1 / 2) = - \ln (\mathbf {B C} (p, q)).
$$
$$
d _ {\mathrm {H}} (p, q) = \sqrt {1 - \exp \left(- d _ {\mathrm {C}} (p , q ; \lambda = 1 / 2)\right)}, \tag {5}
$$
$$
d _ {\mathrm {C}} (p, q; \lambda = 1 / 2) = - \ln \left(1 - d _ {\mathrm {H}} ^ {2} (p, q)\right). \tag {6}
$$
From Equ.(6),  $d_{\mathrm{C}}$  is monotonic w.r.t.  $d_{\mathrm{H}}$ , but  $d_{\mathrm{C}}$  has a logarithmic dependence which results in it not being a proper metric (i.e., does not satisfy the triangle inequality) unlike  $d_{\mathrm{H}}$ . We specifically leverage the triangle inequality to prove Theorem 1, which seems difficult to do for the Chernoff distance.
Interestingly, this logarithmic dependence turns out to be a practical advantage where  $d_{\mathrm{H}}$  can run the risk of numerical overflow [63]. Our clustering experiment of the models (App. C) runs into the issue of numerical overflow of the Hellinger distance and we resort to the Chernoff distance. Our empirical results (App. C) also support this that using  $d_{\mathrm{C}}$  can obtain better results than using  $d_{\mathrm{H}}$ .
Proof of Proposition 2. Continuing from our discussion in Sec. 3,  $\nu_{\mathrm{CE}}(\mathcal{C}) \leq \nu_{\mathrm{H}}^{2}(\mathcal{C})$  follows from Lemma 2 below by requiring a sufficient condition on the differential entropy of  $\mathbb{Q}_{\mathcal{C}}$ , as stated in Proposition 2. $^{10}$  On the other hand, constructing a lower bound of  $\nu_{\mathrm{CE}}(\mathcal{C})$  via  $\nu_{\mathrm{H}}^{2}(\mathcal{C})$  is less direct, as can be seen from the upper bound on CE using  $d_{\mathrm{H}}^{2}$  in Lemma 2. Deriving such a lower bound (Proposition 2) involves exploiting a property specific to Dirichlet distributions that the differential entropy is non-positive and bounded from above.
Proof of Proposition 2. By substituting  $p = \mathbb{Q}_{\mathcal{C}}$  and  $q = \mathbb{Q}^*$  into the first inequality of Lemma 2,  $2d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})\leq \mathrm{CE}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*}) - H(\mathbb{Q}_{\mathcal{C}})$ . Using  $H(\mathbb{Q}_{\mathcal{C}}) + d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})\geq 0$ , it follows that  $d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})\leq \mathrm{CE}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})$ , so  $\nu_{\mathrm{CE}}(\mathcal{C})\leq \nu_{\mathrm{H}}^{2}(\mathcal{C})$  by plugging in the definitions of  $\nu_{\mathrm{CE}}(\mathcal{C}):= -\mathrm{CE}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})$  and  $\nu_{\mathrm{H}}^{2}(\mathcal{C}):= -d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}},\mathbb{Q}^{*})$ .
Next, the differential entropy of a Dirichlet distribution  $p$  parameterized by  $\alpha$  [48, Table 2.1] is
$$
H (p) = - G (\boldsymbol {\alpha}) + \left(\sum_ {k = 1} ^ {C} \alpha_ {k} - C\right) \psi \left(\sum_ {k = 1} ^ {C} \alpha_ {k}\right) - \sum_ {k = 1} ^ {C} \left(\alpha_ {k} - 1\right) \psi \left(\alpha_ {k}\right)
$$
which is maximized at  $\alpha = 1_{C}$  [55, 54], and its maximum value is therefore  $-\log \Gamma (C)$ . Substituting  $H(p)\leq -\log \Gamma (C)$  into the second inequality of Lemma 2 gives
$$
\mathrm {C E} (p, q) \leq \operatorname {c o n s t} _ {q} \times \left(1 - \left[ 1 - d _ {\mathrm {H}} ^ {2} (p, q) \right] ^ {2}\right) - \log \Gamma (C).
$$
By substituting  $p = \mathbb{Q}_{\mathcal{C}}$  and  $q = \mathbb{Q}^*$  into the above and plugging in the definitions of  $\nu_{\mathrm{CE}}(\mathcal{C}) \coloneqq -\mathrm{CE}(\mathbb{Q}_{\mathcal{C}}, \mathbb{Q}^{*})$  and  $\nu_{\mathrm{H}}^{2}(\mathcal{C}) \coloneqq -d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}}, \mathbb{Q}^{*})$ ,
$$
- \nu_ {\mathrm {C E}} (\mathcal {C}) \leq \operatorname {c o n s t} _ {\mathbb {Q} ^ {*}} \times \left(1 - \left[ 1 + \nu_ {\mathrm {H}} ^ {2} (\mathcal {C}) \right] ^ {2}\right) - \log \Gamma (C),
$$
which can be rearranged to complete the proof.
Lemma 2. For any two continuous distributions  $p$  and  $q$ ,
$$
2 d _ {\mathrm {H}} ^ {2} (p, q) \leq \operatorname {C E} (p, q) - H (p) \leq \operatorname {c o n s t} _ {q} \times \left(1 - \left[ 1 - d _ {\mathrm {H}} ^ {2} (p, q) \right] ^ {2}\right)
$$
where  $H(p) \coloneqq -\int p(x) \log p(x) \, \mathrm{d}x$  is the differential entropy of  $p$ ,  $d_{\mathrm{H}}(p,q) \in [0,1]$  is the Hellinger distance, and  $\mathrm{const}_q \coloneqq [\log (1 / q_{\min} - 1)] / (1 - 2q_{\min})$  with  $q_{\min} \coloneqq \min_z q(z)$ .
$^{10}$ Note  $\nu_{\mathrm{H}}^{2}(\mathcal{C}) \coloneqq -d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}}, \mathbb{Q}^{*})$ .
Proof. Firstly,
$$
\begin{array}{l} \operatorname {C E} (p, q) := - \int p (x) \log q (x) d x \\ = d _ {\mathrm {K L}} (p, q) + H (p) \tag {7} \\ \end{array}
$$
where  $d_{\mathrm{KL}}(p,q) \coloneqq \int p(x)\log (p(x) / q(x))\mathrm{d}x$  is the Kullback-Leibler distance. Next, we derive a lower bound of  $d_{\mathrm{KL}}(p,q)$  in terms of  $d_{\mathrm{H}}(p,q)$ :
$$
d _ {\mathrm {K L}} (p, q) \geq 2 d _ {\mathrm {H}} ^ {2} (p, q). \tag {8}
$$
To prove (8),
$$
\begin{array}{l} d _ {\mathrm {K L}} (p, q) = \int p (x) \log \frac {p (x)}{q (x)} d x \\ = 2 \int p (x) \log \frac {\sqrt {p (x)}}{\sqrt {q (x)}} d x \\ = 2 \int p (x) \left(- \log \frac {\sqrt {q (x)}}{\sqrt {p (x)}}\right) d x \\ \geq 2 \int p (x) \left(1 - \frac {\sqrt {q (x)}}{\sqrt {p (x)}}\right) d x \\ = \int \left\{p (x) + p (x) - 2 \sqrt {p (x)} \sqrt {q (x)} \right\} d x \\ = 1 + \int \left\{p (x) - 2 \sqrt {p (x)} \sqrt {q (x)} \right\} d x \\ = \int q (x) d x + \int \left\{p (x) - 2 \sqrt {p (x)} \sqrt {q (x)} \right\} d x \\ = \int \left\{q (x) + p (x) - 2 \sqrt {p (x)} \sqrt {q (x)} \right\} d x \\ = \int \left\{\sqrt {p (x)} - \sqrt {q (x)} \right\} ^ {2} d x \\ = 2 d _ {\mathrm {H}} ^ {2} (p, q) \\ \end{array}
$$
where the inequality is due to  $-\log z \geq 1 - z$  for all  $z \geq 0$  by setting  $z = \sqrt{q(x)} / \sqrt{p(x)} \geq 0$ . Moreover, we have an upper bound of  $d_{\mathrm{KL}}(p,q)$  from [61, Equation 7.27]:
$$
d _ {\mathrm {K L}} (p, q) \leq \operatorname {c o n s t} _ {q} \times \left(1 - \left[ 1 - d _ {\mathrm {H} (p, q)} ^ {2} \right] ^ {2}\right). \tag {9}
$$
Lastly, substituting  $d_{\mathrm{KL}}(p,q) = \mathrm{CE}(p,q) - H(p)$  from (7) into (8) and (9) completes the proof.
An example on how cross-entropy loss encodes the predictive accuracy and certainty. The CE loss is used to construct upper and lower bounds for our proposed method (i.e.,  $\nu$  in Eq. (3)). Hence, our proposed method also encodes the predictive accuracy and certainty, as exemplified below.
Recall that the CE loss of a  $C$ -dimensional predicted probability vector  $\hat{y}$  w.r.t. the one-hot encoded true label  $y$ :
$$
- \sum_ {k = 1} ^ {C} y _ {k} \times \ln (\hat {y} _ {k}).
$$
W.l.o.g., assume that  $y_{1} = 1$  (i.e., the correct class is the first class).
1. For two predictions  $[0.9, 0.1, 0, \dots, 0]$  vs.  $[0.1, 0.9, 0, \dots, 0]$ . The CE losses are 0.105 and 2.30, respectively. Note that the first prediction is correct while the second in incorrect and that both predictions are "equally certain". Hence, a higher predictive accuracy implies a lower CE.
2. For two predictions  $[0.9, 0.1, 0, \dots, 0]$  vs.  $[0.6, 0.4, 0, \dots, 0]$ . The CE losses are 0.105 and 0.511, respectively. Note that both predictions are correct while the first prediction is "more certain". Hence, a higher predictive certainty implies a lower CE, if the prediction is correct.
Extension to other model evaluation criteria. In addition to predictive accuracy and certainty, there are other possible desirable criteria for model valuation such as adversarial robustness [43], distributional robustness [67], and algorithmic fairness (i.e., by removing prediction bias) in ML [56]. Put differently, it is possible to devise other model valuations based on adversarial robustness, distributional robustness, or the algorithmic fairness of a model. Interestingly, since these more sophisticated criteria all utilize the same building blocks (i.e., the predictive accuracy and certainty of models on carefully selected query sets), our proposed approach can subsume these different criteria, as described below and summarized in Table 1. To be a little technical, the high-level idea is that, since these criteria leverage the CE loss  $\ell_{\mathrm{CE}}$  in very specific ways, and we have derived the relationship between  $d_{\mathrm{H}}$  and CE above (Lemma 2), our approach can be extended to incorporate these criteria through careful choices of  $\nu$ .
For instance, the work of [43] explicitly defines "adversarial" training examples  $\mathcal{D}_{\mathrm{adv}}$  to be distinguished from "clean" training examples  $\mathcal{D}_{\mathrm{clean}}$  and evaluates a model's performance (i.e., a linear combination of the CE losses w.r.t. the adversarial and clean training examples separately) as follows:
$$
\operatorname {o b j e c t i v e} _ {\mathrm {a d v}} := \ell_ {\mathrm {C E}} (\mathbf {M}; \mathcal {D} _ {\mathrm {a d v}}) + v \ell_ {\mathrm {C E}} (\mathbf {M}; \mathcal {D} _ {\mathrm {c l e a n}})
$$
for some weight  $\upsilon > 0$  where we suppress the constants that linearly depend on the sizes of  $\mathcal{D}_{\mathrm{adv}}$  and  $\mathcal{D}_{\mathrm{clean}}$  for simplicity and  $\ell_{\mathrm{CE}}(\mathbf{M};\mathcal{D})$  is the common CE loss incurred by the predictions of  $\mathbf{M}$  on query set  $\mathcal{D}$ .
The work of [67, Equation 5] gives the group-adjusted distributionally robust optimization (DRO) estimator where each "group"  $g \in \mathcal{G}$  contains training examples from a possibly different data distribution. Effectively, the optimizer minimizes the maximum CE loss over different groups/query sets s.t. each query set is a dataset from a possibly different data distribution:
$$
\text {o b j e c t i v e} _ {\mathrm {D R O}} := \max  _ {g \in \mathcal {G}} \ell_ {\mathrm {C E}} (\mathbf {M}; \mathcal {D} _ {\mathrm {g}})
$$
where, for simplicity, a group size-dependent constant and a model capacity-dependent constant are ignored.