SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Its proof follows directly from [20, Proposition 2.1]. Since (2) coincides with the Shapley value [69], we refer to  $\Phi$  as model Shapley and  $\phi_{i}$  as the model Shapley value (MSV). Note that P3 requires two distinct tasks  $\mathcal{D}$  and  $\mathcal{D}'$  to distinguish between the MSVs of  $\mathbf{M}_i$  w.r.t. to these two tasks. P4 additionally requires  $\nu$  to be conditionally redundant $^7$  (Proposition 5 in App. A): The benefit of a redundant copy  $\mathbf{M}_{i_c}$  (conditioned on model  $\mathbf{M}_i$  already being added) is not more than the initial benefit of adding  $\mathbf{M}_i$ . Intuitively, as adding  $\mathbf{M}_i$  is already sufficient for the desired task, subsequently adding  $\mathbf{M}_{i_c}$  does not yield (as much) extra benefit [24]. In contrast, [51] also adopts (2) but assumes  $\nu$  already exists while we explicitly design  $\nu$  to encode predictive accuracy and certainty:
$$
\nu (\mathcal {C}) := - d \left(\mathbb {Q} _ {\mathcal {C}}, \mathbb {Q} ^ {*}\right) \tag {3}
$$
where  $\mathbb{Q}_{\mathcal{C}}$  is the precision-weighted fusion of the Dirichlet abstractions in  $\mathcal{C}$  and  $d(\cdot, \cdot)$  is a distributional distance measure between two Dirichlet abstractions. Recall from Sec. 2 that the predictive distribution of  $\mathbf{M}_i$  is represented in its Dirichlet abstraction (visualized in Fig. 1). In particular, the more accurate  $\mathbf{M}_i$  is, the closer (distributionally)  $\mathbb{Q}_i$  is to the Dirichlet abstraction  $\mathbb{Q}^*$  of an expert (i.e., optimal classifier). Hence, a (high) similarity between  $\mathbb{Q}_i$  and  $\mathbb{Q}^*$  can suggest  $\mathbf{M}_i$ 's (high) predictive accuracy. Specifically,  $\mathbb{Q}^*$  is implemented as follows: For an input data-label pair  $(x_j, y_j)$ , take the one-hot encoded vector  $e_{y_j} \in [0,1]^C$  of  $y_j$ , add an independent uniform noise  $\epsilon_j \in [0,0.01]^C$  (to avoid degeneracy during MLE), normalize it to sum to 1 to yield  $e_{y_j, \epsilon_j}$  as a
$98.22\%$  of the data are from only 3 out of the 23 classes.
$^7\forall \mathcal{C}\subseteq [N]\setminus \{i\}$ $\nu (\mathcal{C}\cup \{i\}) - \nu (\mathcal{C})\geq \nu (\mathcal{C}\cup \{i,i_{\mathrm{c}}\}) - \nu (\mathcal{C}\cup \{i\})$  given that  $i_{\mathrm{c}}\notin [N]$
'prediction' of the expert, and solve (1) using these 'predictions' over  $\mathcal{D}$ . The predictive certainty of  $\mathbf{M}_i$  is encoded in the precision  $|\alpha_i|_1$  of its Dirichlet abstraction, which is then used as a weight to fuse the individual Dirichlet abstractions  $\mathbb{Q}_i$  in  $\mathcal{C}$  to obtain  $\mathbb{Q}_{\mathcal{C}}$ :
Definition 3 (Precision-weighted Fusion). Let the random vector  $[\beta_1, \ldots, \beta_n] \sim \mathrm{Dir}([|\alpha_1|_1, \ldots, |\alpha_n|_1])$  be independent of  $\mathbf{M}_i(x)$  for all  $i \in \mathcal{C} \subseteq [N]$  where  $n := |\mathcal{C}|$ . Then, the precision-weighted fusion  $\mathbb{Q}_{\mathcal{C}}$  is the distribution of  $\sum_{i=1}^{n} \beta_i \mathbf{M}_i(x)$ .
In Definition 3, the weight  $\beta_{i}$  on  $\mathbf{M}_i(x)$  is large if  $|\alpha_i|_1$  is large, i.e.,  $\mathbb{Q}_i$  has a high precision/ $\mathbf{M}_i$  has a high predictive certainty. Definition 3 has an important implication:  $\mathbb{Q}_{\mathcal{C}}$  is a fully specified Dirichlet, i.e.,  $\mathbb{Q}_{\mathcal{C}} = \mathrm{Dir}([\sum_{i=1}^{n} \alpha_{i,1}, \ldots, \sum_{i=1}^{n} \alpha_{i,\mathcal{C}}])$  (Lemma 3 in App. A) with four advantages: (A)  $\mathbb{Q}_i$  and  $\mathbb{Q}_{\mathcal{C}}$  are both Dirichlets so that a single  $d$  (e.g.,  $d_{\mathrm{H}}$ ) in (3) can be used for both singleton and non-singleton  $\mathcal{C}$ 's. Interestingly, it gives a perspective that each  $\mathbf{M}_i$  lives as  $\mathbb{Q}_i$  in a metric space w.r.t.  $d_{\mathrm{H}}$  (Fig. 11 in App. C). (B) We can theoretically justify learning model Shapley (Theorem 1). (C) (3) using  $d_{\mathrm{H}}$  can be evaluated in closed form with  $\mathcal{O}(1)$  time, which is important since (2) requires  $\mathcal{O}(2^N)$  evaluations of  $\nu(\mathcal{C})$  for different  $\mathcal{C}$ 's. (D) An alternative to specify  $\nu(\mathcal{C})$  is a linear combination of the performance of  $\mathbf{M}_i$ ,  $\forall i \in \mathcal{C}$  as  $\nu(\mathcal{C})$ . However, it is unclear what the weights in this linear combination should be. In contrast, Definition 3 'automatically' resolves this issue by fusing each  $\mathbb{Q}_i$  according to its precision  $|\alpha_i|_1$  into  $\mathbb{Q}_{\mathcal{C}}$ .
Interpreting MSV. Note that (2) is an average (over all possible  $\mathcal{C} \subseteq [N] \setminus \{i\}$ ) of how much  $\mathbf{M}_i$  (or, more precisely,  $\mathbb{Q}_i$ ) improves the distributional similarity between  $\mathbb{Q}_{\mathcal{C}}$  and  $\mathbb{Q}^*$  (i.e., the expert) after  $i$  joins  $\mathcal{C}$ . Both the predictive accuracy and certainty of  $\mathbf{M}_i$  can affect (2). To see this, a high predictive accuracy of  $\mathbf{M}_i$  implies that  $\mathbb{Q}_i$  is (distributionally) close to  $\mathbb{Q}^*$ ; a high predictive certainty of  $\mathbf{M}_i$  ensures its weight  $\beta_i$  is large when fused into  $\mathbb{Q}_{\mathcal{C}}$ , so the predictive certainty can amplify  $\mathbf{M}_i$ 's effect in bringing  $\mathbb{Q}_{\mathcal{C}}$  close to  $\mathbb{Q}^*$  (if  $\mathbb{Q}_i$  is close to  $\mathbb{Q}^*$ ). Hence, a model  $\mathbf{M}_i$  with a high predictive accuracy and certainty is likely to have a high MSV  $\phi_i$ . When considering several models jointly [66], we can use  $\phi_i$  to indicate how well (on average)  $\mathbf{M}_i$  combines with other models (i.e., whether  $\mathbf{M}_i$  joining  $\mathcal{C}$  leads to a performance improvement), as verified in Sec. 5. In contrast to the work of [66], which supports up to  $N = 32$  simpler binary classifiers, our approach –more scalable and general– supports up to  $N = 150$  C-way classifiers (Sec. 4.2).
3.2 Connection to Cross-entropy and Other Model Evaluation Criteria
We first formalize the connection between cross-entropy (CE) and our approach that uses the Hellinger distance, and then use this connection to extend our approach to other evaluation criteria such as adversarial robustness [43], distributional robustness [67], and algorithmic fairness in ML [56].
As a distributional distance measure, CE can be used specify (3) because the CE loss is used to evaluate the performance of a model. Then, specifying (3) with CE or the Hellinger distance (as proposed in Sec. 4) can be connected in how they evaluate the model performance.
Proposition 2. Let  $\nu_{\mathrm{CE}}(\mathcal{C}) \coloneqq -\mathrm{CE}(\mathbb{Q}_{\mathcal{C}}, \mathbb{Q}^{*}),^8$ $\nu_{\mathrm{H}}^{2}(\mathcal{C}) \coloneqq -d_{\mathrm{H}}^{2}(\mathbb{Q}_{\mathcal{C}}, \mathbb{Q}^{*})$  and  $H(\cdot)$  be differential entropy. Then,
$$
\nu_ {\mathrm {C E}} (\mathcal {C}) \leq \nu_ {\mathrm {H}} ^ {2} (\mathcal {C}) \text {i f} H (\mathbb {Q} _ {\mathcal {C}}) + d _ {\mathrm {H}} ^ {2} (\mathbb {Q} _ {\mathcal {C}}, \mathbb {Q} ^ {*}) \geq 0, \text {a n d} \nu_ {\mathrm {C E}} (\mathcal {C}) \geq \operatorname {c o n s t} _ {\mathbb {Q} ^ {*}} \times ([ 1 + \nu_ {\mathrm {H}} ^ {2} (\mathcal {C}) ] ^ {2} - 1) + \log \Gamma (C)
$$
where  $\mathrm{const}_q\coloneqq [\log (1 / q_{\mathrm{min}} - 1)] / (1 - 2q_{\mathrm{min}})$  with  $q_{\mathrm{min}}\coloneqq \min_zq(z)$  for a density  $q$
Proposition 2 shows that  $\nu_{\mathrm{CE}}(\mathcal{C})$  has upper and lower bounds that are monotonic in  $\nu_{\mathrm{H}}^{2}$ , providing some justification for using  $\nu_{\mathrm{H}}$  (instead of  $\nu_{\mathrm{CE}}$ ). Note that while Proposition 2 utilizes  $\nu_{\mathrm{H}}^{2}$  due to a key Lemma 2 (in App. A), we adopt  $\nu_{\mathrm{H}} \coloneqq -d_{\mathrm{H}}$  (Sec. 4) as  $d_{\mathrm{H}}$  satisfies the triangle inequality (for Theorem 1). Moreover, Proposition 2 confirms that  $\nu_{\mathrm{H}}$  encodes the predictive accuracy and certainty of a model since  $\nu_{\mathrm{CE}}$  encodes the predictive accuracy and certainty (see the example in App. A).
Interestingly, this connection to CE enables the extension to adversarial robustness, distributional robustness and algorithmic fairness, with different practical motivations. For instance, adversarial robustness (in model valuation) is important to application scenarios where the model can encounter adversarial attacks in deployment [43]. Formally, the respective objective functions  $\mathrm{objective}_{\mathrm{adv}}$  [43],  $\mathrm{objective}_{\mathrm{DRO}}$  [67, Equation 5] and  $\mathrm{objective}_{\mathrm{EO}}$  [56, Definition 2] can be achieved from a suitable definition of  $\nu$  using  $d_{\mathrm{H}}$  (precise definitions and full deviations are deferred to App. A), as summarized
Note that  $\operatorname{CE}(p,q) \coloneqq -\int p(x)\log q(x)\mathrm{d}x$  for two distributions with densities  $p, q$ .
in Table 1. We highlight that this illustrates the potential generality of MSVs using the Hellinger distance w.r.t. Dirichlet abstractions (i.e., using  $\nu_{\mathrm{H}}(\mathcal{C})$  in (2)), and defer the formal treatment of such theoretical connections to future work. A question one might ask is that: (How) can multiple such evaluation criteria be combined? The answer is yes, by leveraging (P3) to linearly combine selected evaluation criteria, as discussed in App. A.
Table 1: Extension to other evaluation criteria for model valuation. The notation dependence on the query set is made explicit.  $\mathcal{D}_{\mathrm{adv}}$ ,  $\mathcal{D}_{\mathrm{clean}}$  denote the query sets containing adversarial and non-adversarial (clean) training examples.  $\{\mathcal{D}_g\}_{g \in \mathcal{G}}$  is a collection of query sets where  $\mathcal{D}_g$  contains training examples from a particular "group"/data distribution.  $\mathcal{D}_{\mathrm{prot}}^{+}$ ,  $\mathcal{D}_{\mathrm{unprot}}^{+}$  contain positive training examples under the protected and unprotected groups, respectively.
Criteria	Query sets	Choices ofν
objectiveadv	Dadv, Dclean	-(dH(∅, Q*; Dadv) + v dH(∅, Q*; Dclean))
objectivedRO	{Dg}g∈G	- maxg∈G dH(∅, Q*; Dg)
objectiveEO	D+/prot, D+/unprot	-|dH(∅, Q*; D+/prot) - dH(∅, Q*; D+/unprot)|
4 Learning Model Shapley
To address challenge (3) in Sec. 1, we propose a learning approach to train a model appraiser (i.e., a regression learner) from the MSVs of a small subset of models for predicting MSVs of the remaining models (further elaborated in App. C). If we can learn a good appraiser with only  $20\%$  of all models (empirically verified), then the marketplace size can (theoretically) quintuple. To justify this learning approach, we derive a Lipschitz continuity of model Shapley, which is also empirically verified using 5 real-world datasets and various model types. Next, we implement this learning approach by training a Gaussian process regression (as the model appraiser) on a subset (from  $5\%$  to  $50\%$  in size) of 150 model-MSV pairs and examine its predictive performance on the rest. Our implementation is available at https://github.com/XinyiYS/ModelShapley.
4.1 Lipschitz Continuity of Model Shapley
We derive a Lipschitz continuity of the model Shapley function  $\Phi :[N]\mapsto \mathbb{R}$ : The difference between the MSVs of two models  $\mathbf{M}_i,\mathbf{M}_{i'}$  (i.e., inputs to  $\Phi$ ) is bounded by the distance  $d_{\mathrm{H}}(\mathbb{Q}_i,\mathbb{Q}_{i'})$  between them, multiplied by a constant factor.
$$
\begin{array}{l} \text {T h e o r e m 1 (L i p s c h i t z C o n t i n u i t y) . L e t d : = d _ {\mathrm {H}} i n (3) . T h e n ,   \forall i , i ^ {\prime} \in [ N ] (\forall \mathcal {C} \subseteq [ N ] \setminus} \\ \{i, i ^ {\prime} \} d _ {\mathrm {H}} (\mathbb {Q} _ {\mathcal {C} \cup i}, \mathbb {Q} _ {\mathcal {C} \cup i ^ {\prime}}) \leq d _ {\mathrm {H}} (\mathbb {Q} _ {i}, \mathbb {Q} _ {i ^ {\prime}})) \Longrightarrow | \phi_ {i} - \phi_ {i ^ {\prime}} | \leq Z d _ {\mathrm {H}} (\mathbb {Q} _ {i}, \mathbb {Q} _ {i ^ {\prime}}). \end{array}
$$
Its proof is in App. A. Theorem 1 states that the difference in MSVs of two models is bounded by the Hellinger distance between their Dirichlet abstractions, and the constant  $Z$  from (2) is the Lipschitz constant. This is based on a simple fusion-increases-similarity condition: When  $\mathbb{Q}_i$  and  $\mathbb{Q}_{i'}$  are each fused with a common  $\mathbb{Q}_{\mathcal{C}}$ , the resulting similarity is higher (i.e., smaller  $d_{\mathrm{H}}$ ) since  $\mathbb{Q}_{\mathcal{C} \cup i}$  and  $\mathbb{Q}_{\mathcal{C} \cup i'}$  have  $\mathbb{Q}_{\mathcal{C}}$  in common (see Proposition 4 in App. A). Moreover, Table 3 and Fig. 6 (in App. A) empirically verify Theorem 1. Then, Theorem 1 provides a theoretical justification for the learning approach because it guarantees that similar inputs (i.e., small  $d_{\mathrm{H}}(\mathbb{Q}_i, \mathbb{Q}_{i'})$ ) imply similar outputs (i.e., small  $|\phi_i - \phi_{i'}|$ ). Namely, the model Shapley function is well-behaved w.r.t. its inputs, and hence learnable. This reasoning is applied to justify learning the data Shapley value [21].
4.2 Empirical Learning Performance via Gaussian Process Regression (GPR)
To exploit the Lipschitz continuity (i.e., Theorem 1), we adopt the Gaussian process regression (GPR) due to a uniform error bound of GPR on Lipschitz continuous functions [47]. Our implementation trains a GPR (as the model appraiser) on the MSVs of a subset of  $N = 150$  models and examines its predictive performance on the remaining ones.
Regression setting. We train  $N = 150$  independent models on MNIST (CIFAR-10): 50 of logistic regression (LR), multi-layer perceptron (MLP), and convolutional neural network (CNN) each (ResNet-18, SqueezeNet, and DenseNet-121 each). For simplicity, we use the test set without
partitioning as the query set  $\mathcal{D}$ . For each  $\mathbf{M}_i$ , we obtain  $\bar{h}_i$  via its predictions on  $\mathcal{D}$  and solve (1) to obtain  $\alpha_{i}$  as input features for separate regressions. For the regression labels, as calculating  $\phi_{i}$  exactly incurs  $\mathcal{O}(2^{150})$  time, we use the  $(\epsilon = 0.1,\delta = 0.1)$ -approximation [53]  $\hat{\phi}_i$  as the average of 3745 Monte-Carlo samples. This results in two sets  $\{\pmb {\alpha}_i,\hat{\pmb{\phi}}_i\}$  and  $\{\bar{h}_i,\hat{\phi}_i\}$  of model-MSV pairs of size 150 each. We train a GPR on a random subset of 150 model-MSV pairs to learn to predict the MSV on the remaining pairs. In GPR, we use the squared exponential kernel  $\exp (-d(i,i^{\prime}) / (2\sigma^{2}))$  (the lengthscale  $\sigma$  is learnt) where  $d(i,i^{\prime})\coloneqq d_{\mathrm{H}}(\mathbb{Q}_i,\mathbb{Q}_{i'})$  for  $\{\pmb {\alpha}_i,\hat{\pmb{\phi}}_i\}$ , and  $d(i,i^{\prime})\coloneqq |\bar{h}_{i^{\prime}} - \bar{h}_i|_1$  for  $\{\bar{h}_i,\hat{\phi}_i\}$ .
High regression performance verifies learnability. We examine the test performance using two error metrics: mean-squared error (MSE) and maximum error (MaxE) w.r.t. varied training ratios from  $5\%$  to  $50\%$ , in Fig. 2. In particular, results for training ratio of  $20\%$  are in Table 2. We observe that even using only  $20\%$  of model-MSV pairs for training, the learning is effective (i.e., low test errors), which shows its feasibility in a large-scale model marketplace. This can be attributed to the learnability justified by Theorem 1 and the uniform error bound of GPR [47]. In addition, learning on  $\alpha_{i}$  is more effective than learning on  $\bar{h}_{i}$  (as Table 2 and Fig. 2 show higher errors for the latter), since the average operation to get  $\bar{h}_{i}$  loses some model information.
Table 2: Top (bottom) are results on MNIST (CIFAR-10) for the training ratio of  $20\%$ . Average (std. error) over 10 random train-test splits.
MSE	MaxE
αi	1.59e-6(6.9e-8)	3.53e-3(9.7e-5)
hi	8.36e-5(3.1e-6)	1.59e-2(2.6e-4)
αi	1.79e-5(5.2e-6)	9.05e-3(3.9e-4)
hi	3.05e-4(4.5e-5)	3.23e-2(2.4e-3)
Figure 2: Average (std. errors) of test performance vs. training ratios for MNIST & CIFAR-10 over 10 random train-test splits for each training ratio. Dashed (solid) lines follow the left (right) axis. Colors indicate  $\alpha_{i}$  (blue) or  $\bar{h}_i$  (orange). To elaborate, at a training ratio of  $5\%$ , the GPR is trained on a random subset of  $5\%$  of the total 150 model-MSV pairs and its test performance on the remaining  $95\%$  of the model-MSV pairs is reported.
5 MSV vs. Common Evaluation Criteria
This work is motivated by the lack of a standardized model valuation, but there are different evaluation criteria useful in different scenarios (e.g., accuracy, F1 score, predictive certainty). Interestingly, we find that MSV can produce consistent model values with these criteria. Additionally, we evaluate the utility of MSV directly in a use case where a buyer wishes to purchase multiple models with black-box access to build a larger learner (e.g., random forest or voting classifier) [33, 66] and show that MSV can be used to effectively identify the most 'valuable' models for this purpose.
MSV vs. predictive performance. Fig. 3 (left) compares the MSVs of different model types (independently trained on the same data) for MNIST. Here  $\mathcal{D}$  consists of misclassified data (from the original test set) of all the models to highlight the difference in their predictive accuracies. Without
needing to partition  $\mathcal{D}$ , we observe that CNNs significantly outperform both MLPs and LRs (in terms of accuracy) and have the highest MSVs. This is expected, since CNNs are more capable of performing well in image-based tasks, and hence the MSVs for CNNs are correspondingly higher. Then, we examine predictive certainty. We use the same CNN model type independently trained on the same MNIST data (i.e., their accuracies are essentially equal), but artificially increase the predictive certainty for some: We multiply the highest probability of  $\mathbf{M}_i(x_j)$  by a factor of [1, 5, 10] and then normalize the resulting vector to sum to 1 without affecting the predicted class/accuracy. Fig. 3 (right) shows that models with higher predictive certainty have higher MSVs, confirmed by additional results on CIFAR-10, MedNIST, and DrugRe in App. C. These results confirm our intuition that a model with high predictive accuracy and certainty is likely to have a high MSV.
Figure 3: More suitable model types (left)/higher predictive certainty (right) lead to higher MSVs.
Next, we train 3 groups of 3 LR models independently with varying sizes of training data from 0.001 to 0.01 and 0.1 of the entire KDD99 dataset containing highly imbalanced data; the top 3 classes out of 23 constitute  $98.22\%$  of total data. Intuitively, the models trained on more data should perform better and thus be more valuable, but due to the class imbalance, Fig. 4 (left) shows difficulties in differentiating these models based on their accuracies (or MSVs w.r.t. the entire/unpartitioned query set). Intuitively, to differentiate them, we need a lower level (i.e., more refined) Dirichlet abstraction, namely the class-specific Dirichlet abstractions: Partition the entire query set according to the  $C = 23$  classes with  $\gamma_{k} := |\mathcal{D}_{k}|$ . Define  $d_{\mathrm{H}}(\mathbb{Q}_i,\mathbb{Q}_{i'};\{\mathcal{D}_k\}_{k = 1,\dots,C}) := \sum_{k = 1}^{C}\gamma_k d_{\mathrm{H}}(\mathbb{Q}_{i,\mathcal{D}_k},\mathbb{Q}_{i'},\mathcal{D}_k)$  to leverage P3 to compute  $\phi_i(\{\mathcal{D}_k\}_{k = 1,\dots,C}) = \sum_{k = 1}^{C}\gamma_k\phi_i(\mathcal{D}_k)$  (right of Fig. 4). Then we can see that MSVs are indeed consistent with F1 score (a criterion especially suited for imbalanced data) without explicitly using F1 score in the computation. In addition, for KDD, due to the high class imbalance, there are classes with extremely small data size (i.e.,  $\leq 5$ ) and our calculation of  $d_{\mathrm{H}}(\mathbb{Q}_i,\mathbb{Q}_{i'};\{\mathcal{D}_k\})$  naturally suppresses their effect (possibly inaccurate  $\mathbb{Q}_i$ ) via  $\gamma_{k} := |\mathcal{D}_{k}|$ . However, in practice, it should be noted that partitioning the query set to obtain a lower level of Dirichlet abstraction should be considered w.r.t. the size of available query set (Sec. 2). In other words, to obtain a lower level of Dirichlet abstraction (and thus a more refined representation), it incurs a higher cost from collecting a larger query set. In our experiments, we find that the size of each partitioned query set  $\mathcal{D}_k$  should contain at least  $10^2$  samples (e.g., for KDD most classes have at least or close to  $10^2$  samples).
Figure 4:  $\phi_i(\mathcal{D})$  (left) and  $\phi_i(\{\mathcal{D}_k\})$  (right) vs. the sizes of training data (as a proportion to the full dataset).
Identifying valuable models to purchase. For a more end-to-end use case (instead of a single evaluation criterion), we evaluate the MSVs for up to 50 models and the performance of a larger
learner by including a subset of these models based on their MSVs in a highest/lowest-first sequence in Fig. 5. The larger learner is random forests (voting classifier) and models are decision trees (LeNets [46]) for Breast Cancer (CIFAR-10). As the test accuracy of highest MSV-first increases more quickly (orange line), it verifies our previous comment on models that perform well when combined with other models are likely to have high MSVs. This characteristic offers some practical utility. If a buyer is looking to purchase models from a marketplace [66], then following the highest MSV sequence, the buyer only needs to purchase a subset of 15 (left of Fig. 5) or 25 (right of Fig. 5) out all 50 available models, thus saving cost. More results on CIFAR-100 with ResNet-18 are in App. C.
Figure 5: Test accuracy vs. number of models on Breast Cancer dataset (left) and CIFAR-10 (right).
The work of [66] investigates the binary classification setting and is not suitable for empirical comparison as we consider problems with multiple classes. The works of [10, 11, 13, 51] approach the design of a model marketplace from an economics perspective by addressing issues like arbitrage (e.g., via horizontal/vertical pricing). In contrast, we formalize the value of a model via what it has learned w.r.t. a task. The black-box access setting is appealing in a model marketplace as it accommodates different model types. Some existing methods [8, 29, 30, 33, 44] focus on how to learn a fused model from several trained models (possibly with black-box access) instead of how to value these models. We highlight that we design the fusion (Definition 3) to leverage its analytic properties in Theorem 1. The approach of learning the Shapley value arises in data valuation problems but has not been considered in model valuation. Interestingly, we can draw parallels between Theorem 1 and [19, Theorem 2.8]. For brevity, we include a more extensive contrasting comparison with data valuation in App. B.
We exploit a Dirichlet abstraction of classification models with only black-box access for proposing a novel equitable model valuation called the model Shapley. We discuss that choosing a suitable level of the Dirichlet abstraction can improve how accurately MSV reflects a model's predictive performance and empirically show that using the partitioned query sets (according to the classes) can provide a suitable trade-off between the level of abstraction and the size of the available query set. MSV behaves consistently (in our experiments) with some common model evaluation criteria (i.e., predictive accuracy and certainty, F1 score) and can be extended to more sophisticated criteria. This implies MSV can potentially help unify existing evaluation criteria to provide a simplified model valuation in practice, without needing to explicitly perform separate evaluations.