SlowGuess/NeurIPS_2023_OCR huggingface dataset:
I. Training Data Preparation. For data distance-based performance prediction, the first step of the pipeline is to prepare "training data" (introduced in Section 4.2) to fit parameters of Equations 1 and 2. The data consists of different data source compositions for some given data scales  $N_0$ ,  $N_1$ , where for each composition  $\mathbf{p}$ , we compute the OT distance of the composed dataset  $\mathcal{D}(N_1, \mathbf{p})$  to the validation data and then train a model on  $\mathcal{D}(N_1, \mathbf{p})$  to get the actual model performance. For simplicity, we select compositions through grid search. This step is represented in Algorithm 1 from lines 4-9 and is the first (I) step in the pipeline Figure 2.
II. Fitting Predictor Function. Once the training data is prepared, we proceed to fit our function in Eqs. 1 and 2. This step is shown in lines 10-11 in Algorithm 1 and is the second (II) step of the pipeline Figure 2. Then, with the performance predictors fitted for data scales  $N_{0}$  and  $N_{1}$ , we move on to the inference stage, where we can perform 2 tasks: performance projection and data source selection.
III. Two-Stage Performance Projection. For performance projection, we project the performance to any data size  $N$  for any mixing ratio  $\mathbf{p}$  in two stages. (1) We predict the performance given the mixing ratio  $\mathbf{p}$  at data scales  $N_0$  and  $N_{1}$ . (2) We use Eq. 5 to project performance prediction to any data size  $N$ . This process is represented in line 12 of Algorithm 1 and is the third (III) step of the pipeline Figure 2.
IV. Optimal Data Source Selection. For optimal data source selection, we solve an optimization problem through gradient descent, which is provided in Eq. B.3.1. The gradient computation uses parameters of the fitted functions from step II and the process terminates when the mixing ratio converges. This process is represented in line 12 of Algorithm 2 and is the fourth (IV) step of the pipeline Figure 2.
Algorithm 1: projector performance predictor.  
In :Pilot Datasets  $D_{1}^{pi}$ ,  $D_{2}^{pi}$ , ...,  $D_{m}^{pi}$ ; Query Data Budget  $N$ ; Query Mixing Ratio  $\mathbf{p}$ ; 0-Data Scale Size  $N_{0}$ ; 1-Data Scale Size  $N_{1}$ ; Learning Algorithm  $\mathcal{A}$ ; Performance Metric Function  $\mathcal{L}(\cdot, D^{val})$ ; OT Distance Function  $OT(\cdot, D^{val})$ .  
Out: Projected Model Performance  $\rightarrow$  [0, 1].  
P  $\leftarrow$  Generate mixing ratios  
DT0, DT1  $\leftarrow$  Initialize empty lists to store OT distances  
L0, L1  $\leftarrow$  Initialize empty lists to store performance values  
for Mixing Ratio  $\mathbf{p_i}$  in P do  
S0, S1 = D(N0, p), D(N1, p) newly composed datasets of size N0, N1  
DT0  $\leftarrow$  append OT(S0, Dval) Optimal Transport distance between S0 and Dval  
DT1  $\leftarrow$  append OT(S1, Dval) Optimal Transport distance between S1 and Dval  
L0  $\leftarrow$  append L(A(S0), Dval) Performance of a model trained on S0  
L1  $\leftarrow$  append L(A(S1), Dval) Performance of a model trained on S1  
 $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_0, \cdot)), D^{val}) \leftarrow$  Fit the function from Eq. 2 with ((P, DT0), L0)  
 $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_1, \cdot)), D^{val}) \leftarrow$  Fit the function from Eq. 2 with ((P, DT1), L1)  
 $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N, \mathbf{p})); D^{val}) \leftarrow$  Project performance by substituting  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_0, \mathbf{p})), D^{val})$  and  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_1, \mathbf{p})), D^{val})$  into Eq. 5  
return  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N, \mathbf{p}); D^{val})$
Algorithm 2: Optimal data source composition  $\mathbf{p}^*$  Update.
Out:Optimal data source composition  $\mathbf{p}^*$
In : Pilot Datasets  $D_1^{pi}$ ,  $D_2^{pi}$ , ...,  $D_m^{pi}$ ; Query Data Budget  $N$ ; Query Mixing Ratio  $\mathbf{p}$ ; 0-Data Scale Size  $N_0$ ; 1-Data Scale Size  $N_1$ ; Trained projector Models with  $N_0$  and  $N_1$  Data Budgets:  $f_0, f_1$ ; Enquired Data Budget:  $N$ ; OT Distance Function  $OT(\cdot, D^{val})$ ; Validation Set:  $D^{val}$ .
2 while p not converged do
1  $\mathbf{p} \gets$  Initialize Random Data Source Composition
4 gradient  $\leftarrow$  Compute gradient wrt.  $\mathbf{p}$  using Eqs. 6, 7, 8.
3  $S_0, S_1 = \mathcal{D}(N_0, \mathbf{p_i}), \mathcal{D}(N_1, \mathbf{p_i})$  newly composed datasets of size  $N_0, N_1$
6 return p
5  $\mathbf{p} \gets$  Update composition  $\mathbf{p}$  with the gradient update according to Eq. B.3.1
Appendix B Proofs and Optimization Details
B.1 Proof for Theorem 1
Theorem 1 (Data Composition Dependent Performance Projection (restated)). Consider log-linear performance scaling relationship depending on both data size  $N$  and data composition  $\mathbf{p}$  given as
$$
\mathbb {E} _ {V} \left[ \mathcal {L} \left(\mathcal {A} \left(\mathcal {D} (N, \mathbf {p})\right); D ^ {\text {v a l}}\right) \right] = - \alpha (\mathbf {p}) \log (N) + C (\mathbf {p}) \tag {4}
$$
Assume one has completed the fitting of the performance predictor on two different scales  $N_0 < N_1$ , which gives  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_0, \mathbf{p})); D^{val})$  and  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N_1, \mathbf{p})); D^{val})$  for all data mixtures  $\mathbf{p}$ . Then, the model performance  $\hat{\mathcal{L}}(\mathcal{A}(\mathcal{D}(N, \mathbf{p})); D^{val})$  for any data mixture  $\mathbf{p}$  at any data scale  $N$  can be predicted as
$$
\hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N, \mathbf {p}); D ^ {v a l}) = \left(\log \frac {N _ {1}}{N _ {0}}\right) ^ {- 1} \left[ \log \frac {N}{N _ {0}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1}, \mathbf {p})); D ^ {v a l}) - \log \frac {N}{N _ {1}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0}, \mathbf {p}); D ^ {v a l}) \right] \tag {5}
$$
Proof. From Eq. (4), for any data mixture  $\mathbf{p}_{\mathbf{z}}$ , we have
$$
\mathbb {E} _ {V} \left[ \mathcal {L} \left(\mathcal {A} \left(\mathcal {D} (N, \mathbf {p} _ {\mathbf {z}})\right); D ^ {\text {v a l}}\right) \right] = - \alpha \left(\mathbf {p} _ {\mathbf {z}}\right) \log (N) + C \left(\mathbf {p} _ {\mathbf {z}}\right)
$$
Then, for data scales  $N_0$  and  $N_1$  where one has completed fitting the performance predictors, we have
$$
\mathbb {E} _ {V} \left[ \mathcal {L} \left(\mathcal {A} \left(\mathcal {D} \left(N _ {0}, \mathbf {p} _ {\mathbf {z}}\right)\right); D ^ {\text {v a l}}\right) \right] = - \alpha (\mathbf {p} _ {\mathbf {z}}) \log \left(N _ {0}\right) + C \left(\mathbf {p} _ {\mathbf {z}}\right)
$$
$$
\mathbb {E} _ {V} \left[ \mathcal {L} \left(\mathcal {A} \left(\mathcal {D} \left(N _ {1}, \mathbf {p} _ {\mathbf {z}}\right)\right); D ^ {\text {v a l}}\right) \right] = - \alpha \left(\mathbf {p} _ {\mathbf {z}}\right) \log \left(N _ {1}\right) + C \left(\mathbf {p} _ {\mathbf {z}}\right)
$$
$$
\hat {\alpha} (\mathbf {p _ {z}}) = \frac {\hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0} , \mathbf {p _ {z}})) ; D ^ {\mathrm {v a l}}) - \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1} , \mathbf {p _ {z}})) ; D ^ {\mathrm {v a l}})}{\log (N _ {1}) - \log (N _ {0})}
$$
where  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_0,\mathbf{p_z}));D^{\mathrm{val}})$  and  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_1,\mathbf{p_z}));D^{\mathrm{val}})$  are given by the fitted predictors.
For any data scale  $N_z$ , we have
$$
\mathbb {E} _ {V} \left[ \mathcal {L} \left(\mathcal {A} \left(\mathcal {D} \left(N _ {z}, \mathbf {p} _ {\mathbf {z}}\right)\right); D ^ {\text {v a l}}\right) \right] = - \alpha \left(\mathbf {p} _ {\mathbf {z}}\right) \log \left(N _ {z}\right) + C \left(\mathbf {p} _ {\mathbf {z}}\right)
$$
Plugging in the above equations, the performance prediction can be given by
$$
\begin{array}{l} \hat {\mathcal {L}} \left(\mathcal {A} \left(\mathcal {D} \left(N _ {z}, \mathbf {p} _ {\mathbf {z}}\right)\right); D ^ {\text {v a l}}\right) = - \hat {\alpha} (\mathbf {p} _ {\mathbf {z}}) \cdot \left[ \log \left(N _ {z}\right) - \log \left(N _ {1}\right) \right] + \hat {\mathcal {L}} \left(\mathcal {A} \left(\mathcal {D} \left(N _ {1}, \mathbf {p}\right)\right); D ^ {\text {v a l}}\right) \\ = - \left[ \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0}, \mathbf {p _ {z}})); D ^ {\mathrm {v a l}}) - \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1}, \mathbf {p _ {z}})); D ^ {\mathrm {v a l}}) \right] \frac {\log (N _ {z}) - \log (N _ {1})}{\log (N _ {1}) - \log (N _ {0})} \\ + \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1}, \mathbf {p})); D ^ {\mathrm {v a l}}) \\ = \left(\log \frac {N _ {1}}{N _ {0}}\right) ^ {- 1} \left[ \log \frac {N _ {z}}{N _ {0}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1}, \mathbf {p} _ {\mathbf {z}})); D ^ {\text {v a l}} - \log \frac {N _ {z}}{N _ {1}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0}, \mathbf {p} _ {\mathbf {z}}); D ^ {\text {v a l}}) \right] \\ \end{array}
$$
B.2 Objectives for Data Selection
Problem formulation is provided in Section 3, where the following two objectives are introduced. Given a selection budget of  $N$  samples, a mixing ratio of data sources  $\mathbf{p} = \{p_1,\dots ,p_m\}$  such that  $\forall i,0\leq p_i\leq 1$  and  $\sum_{i = 1}^{m}p_{i} = 1$ , and  $m$  datasets  $D_{1},\ldots ,D_{m}$  to be mixed, we denote the selected dataset by  $\mathcal{D}(N,\mathbf{p}) = S_1\cup \dots \cup S_m$ , where each  $S_{i}$  is a random subset of  $D_{i}^{\mathrm{all}}$  and  $|S_{i}| = p_{i}N$ . Using these notations, we now describe the typical acquisition goals that can be accommodated by our approach:
The primary objective, "fixed-budget selection for maximal performance", is formulated as a convex optimization and we solve it via gradient-based methods. The alternative objective can be treated as a direct extension of the primary, where one solves the "fixed-budget selection for maximal performance" problem for different data quantities  $N$  and performs a line search for minimal data quantity  $N$  that meets the performance requirement.
B.3 Optimization and Convexity
B.3.1 Primary
For our primary objective fixed-budget selection for maximal performance, with the proposed performance predictors with projection, we solve for
$$
\mathbf {p} ^ {*} = \arg \max  _ {\mathbf {p}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {s}, \mathbf {p})), D ^ {\mathrm {v a l}})
$$
We show this objective function is convex in data composition  $\mathbf{p}$ , where the proposed gradient-based method will guarantee to find its optimal solution  $\mathbf{p}^*$  efficiently.
Empirically, model performance  $\mathcal{L}$  always appears convex in data composition  $\mathbf{p}$ , which is also reported in [13]. This means the model trained on data combined from multiple sources always achieves no worse performance than the average performance of models trained separately on data from each source. Theoretically, as given in [38], the gap between training and validation performance can be tightly bounded by the OT distance between training and validation data, whereas the OT distance is always convex in data composition  $\mathbf{p}$ . We now show our proposed performance predictors as well as the optimization problem based on them are also convex in data composition  $\mathbf{p}$ .
For projector/CS in Eq. (1), consider data compositions  $\mathbf{p_0} \neq \mathbf{p_1}$  and a convex combination  $\mathbf{p_2} = \alpha \mathbf{p_0} + (1 - \alpha) \mathbf{p_1}$  for some constant  $\alpha \in (0, 1)$ . Then, we have
$$
\begin{array}{l} \hat {\mathcal {L}} \left(\mathcal {A} (\mathcal {D} (N, \mathbf {p _ {2}})), D ^ {\mathrm {v a l}}\right) - \alpha \hat {\mathcal {L}} \left(\mathcal {A} (\mathcal {D} (N, \mathbf {p _ {0}})), D ^ {\mathrm {v a l}}\right) - (1 - \alpha) \hat {\mathcal {L}} \left(\mathcal {A} (\mathcal {D} (N, \mathbf {p _ {1}})), D ^ {\mathrm {v a l}}\right) \\ = a _ {1} \cdot \left[ \operatorname {O T} \left(\mathcal {D} (N, \mathbf {p} _ {\mathbf {2}}), D ^ {\text {v a l}}\right) - \alpha \operatorname {O T} \left(\mathcal {D} (N, \mathbf {p} _ {\mathbf {0}}), D ^ {\text {v a l}}\right) - (1 - \alpha) \operatorname {O T} \left(\mathcal {D} (N, \mathbf {p} _ {\mathbf {1}}), D ^ {\text {v a l}}\right) \right] \\ \geq 0 \\ \end{array}
$$
where the inequality holds because  $a_1 > 0$  always holds and OT distance is always convex in data composition  $\mathbf{p}$  by definition [16]. Thus, projector/CS is convex in data composition  $\mathbf{p}$ . projector/PQ is constructed as projector/CS with pseudo quadratic terms and its convexity in  $\mathbf{p}$  can be shown similarly.
We solve the above optimization problem iteratively with the following procedure. First, we initialize the algorithm with  $\mathbf{p} = \mathbf{p}^0$  where  $\mathbf{p}^0$  can be chosen arbitrarily provided that  $\sum_{i} p_i = 1$ . Then, at each step, we perform the gradient update as
where  $d^t$  is the step size at iteration  $t$  and we obtain  $\mathbf{p}^{*} = \mathbf{p}^{\mathrm{T}}$  at convergence as the desired optimal solution.
B.3.2 Alternative
Then, for flexible-budget selection for reaching performance threshold with minimal budget:
$$
\min  _ {N, \mathbf {p}} N \text {s . t .} \max  _ {\mathbf {p}} \mathcal {L} (\mathcal {A} (\mathcal {D} (N, \mathbf {p}), D ^ {\mathrm {v a l}}) \geq u ^ {\mathrm {t a r}}
$$
We solve it through a bi-level optimization, where the lower level is the same as the primary objective and the upper level is a line search for optimal data quantity  $N^{*}$ . Note that the model performance  $\mathcal{L}(\mathcal{A}(\mathcal{D}(N,\mathbf{p}),D^{\mathrm{val}})$  is monotonically non-decreasing and concave in  $N$ . Thus, the optimal data quantity  $N^{*}$  can be found via a straightforward line search. Initialize at  $N_0 = 0$  and  $\mathbf{p} = \mathbf{p}^0$  where  $\mathbf{p}^0$  can be chosen arbitrarily provided that  $\sum_{i} p_{i} = 1$ . Then, at each step, we perform the gradient update as
$$
N ^ {t + 1} \leftarrow N ^ {t} + d ^ {t} \cdot \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N , \mathbf {p} ^ {\mathbf {t} *})), D ^ {\mathrm {v a l}})}{\partial N} \Bigg | _ {N = N ^ {t}}
$$
$$
\mathbf {p} ^ {\mathbf {t} *} = \arg \max _ {\mathbf {p}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {t}, \mathbf {p})), D ^ {\mathrm {v a l}})
$$
where  $d^{t}$  is the step size at iteration  $t$  and  $\mathbf{p}^{t*}$  is the optimal data mixture at  $N_{t}$ , respectively. Continue until  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_t,\mathbf{p}^{\mathbf{t}^*})),D^{\mathrm{val}})\geq u^{tar}$  is achieved and then the data acquisition strategy  $\mathcal{D}(N_t,\mathbf{p}^{\mathbf{t}^*})$  is accepted. Note that at each step,  $\mathbf{p}^t$  is initialized from  $\mathbf{p}^{(t - 1)*}$  and the optimization for  $\mathbf{p}^{t*}$  is completed fairly easily within a few steps.
B.4 Gradients Calculation, Stepsize Selection, and Convergence
For the primary problem optimizing over  $\mathbf{p}(N)$  where  $N$  is the target data quantity, the gradients can be calculated as
$$
\frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\text {v a l}})}{\partial \mathbf {p}} = \left(\log \frac {N _ {1}}{N _ {0}}\right) ^ {- 1} \left[ \log \frac {N}{N _ {0}} \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1} , \mathbf {p})) ; D ^ {\text {v a l}})}{\partial \mathbf {p}} - \log \frac {N}{N _ {1}} \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0} , \mathbf {p}) ; D ^ {\text {v a l}})}{\partial \mathbf {p}} \right] \tag {6}
$$
Optimal Transport naturally provides the gradient information of the OT distance w.r.t. the probability mass of datapoints on which it is computed in its dual solutions. [8] provides an approach that directly constructs the calibrated gradient from the output of the OT solver that informs how the OT distance changes as the probability mass of the datapoints changes, while ensuring the updated mixture  $\mathbf{p}$  remains within the simplex  $\sum_{i} p_{i} = 1$  at each step.
Specifically, for projector/CS, recalling Eq. (1), we have
$$
\frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial \mathbf {p}} = a _ {1} \cdot \frac {\partial \mathrm {O T} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial \mathbf {p}} = \left[ \frac {\partial \mathrm {O T} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial p _ {1}} \dots \frac {\partial \mathrm {O T} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial p _ {m}} \right],
$$
where  $p_1 + p_2 + \ldots + p_m = 1$ . Let  $\{r_1^1, r_1^2 \ldots r_1^{n_1} \ldots r_i^{n_i} \ldots r_m^{n_m}\}$  be the samples consisting  $R(N, \mathbf{p})$  where  $r_i$  represents samples from data source  $i$ . Then, the calibrated gradient can be given as
$$
\frac {\partial \mathrm {O T} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial p _ {i}} = \frac {1}{n _ {i}} \left(\sum_ {j = 1} ^ {n _ {i}} f _ {i} ^ {j} - \frac {n _ {i}}{N - n _ {i}} \sum_ {x = \{1 \dots m \} \backslash i} \sum_ {y = 1} ^ {n _ {x}} f _ {x} ^ {y}\right), \tag {7}
$$
where  $f_{i}^{j}$  is the dual solution of OT that corresponds to  $r_{i}^{j}$ . The calibrated gradient ensures the updated mixture  $\mathbf{p}$  remains within the simplex  $\sum_{i} p_{i} = 1$  at each step. Similarly, for projector/PQ in Eq. (2), the calibrated gradient is given as
$$
\begin{array}{l} \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial p _ {i}} = \left(b _ {2} ^ {i} \cdot p _ {i} ^ {2} + b _ {1} ^ {i} \cdot p _ {i} + b _ {0}\right) \cdot \frac {\partial \mathrm {O T} (\mathcal {A} (\mathcal {D} (N , \mathbf {p}) ; D ^ {\mathrm {v a l}})}{\partial p _ {i}} \tag {8} \\ + \left[ b _ {2} ^ {i} \cdot (2 p _ {i}) + b _ {1} ^ {i} \right] \cdot \mathrm {O T} (\mathcal {A} (\mathcal {D} (N, \mathbf {p}); D ^ {\mathrm {v a l}}) + \left[ c _ {2} ^ {i} \cdot (2 p _ {i}) + c _ {1} ^ {i} \right]. \\ \end{array}
$$
Then, to perform gradient-based optimization, first, we initialize the algorithm with  $\mathbf{p} = \mathbf{p}^0$  where  $\mathbf{p}^0$  can be chosen arbitrarily provided at  $\sum_{i} p_i = 1$ . Then, at each step, we perform the gradient update as
$$
\mathbf {p} ^ {\mathbf {t} + \mathbf {1}} \leftarrow \mathbf {p} ^ {\mathbf {t}} + d ^ {t} \cdot \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {s} , \mathbf {p})), D ^ {\mathrm {v a l}})}{\partial \mathbf {p}} \Bigg | _ {\mathbf {p} = \mathbf {p} ^ {t}}
$$
where  $d^t > 0$  is the step size at iteration  $t$ . In practice, we choose diminishing step sizes that satisfy Robbins—Monro conditions such that  $d^t < d^{t+1}$ ,  $\sum_t d^t = \infty$ , and  $\sum_t (d^t)^2 < \infty$ . Then the series  $\mathbf{p}^{\mathbf{t}}$  is guaranteed to converge to the optimal solution  $\mathbf{p}^*$  [44] given that objective function is convex and bounded. Gradients for projector/PQ can be obtained similarly and the solution procedure is the same. The proposed method is shown to achieve remarkable performance and fast convergence, yielding satisfactory results in a swift manner.
Appendix C Sampling Stochasticity and Market Practices
In the data selection problem formulated in this work, we aim to optimize predicted performance with objectives given in terms of finite samples from the pilot dataset. We note that this pilot dataset is considered a random sample from the whole dataset of each data provider and is inevitably affected by the stochasticity of the sampling process. Our performance predictions  $\hat{\mathcal{L}}$  are empirical estimates of the expectation for the variable based on the samples. Thus, it substantially depends on the sampling process for the estimates to be unbiased and precise. Data providers should adhere to certain guidelines when selecting the pilot datasets. The sampling process should be unbiased where each sample is selected with an equal chance. Examples include sampling with a Bernoulli process where each sample is selected with a fixed probability  $p$ ; or permutation sampling where one selects the first  $N$  samples from the random permutation.
There might be strategic providers that do not adhere to the guidelines. Multiple mechanisms are available to incentivize providers to provide true samples. Since the full dataset will be revealed after purchase, the data buyer can examine the posterior probability for the pilot dataset being sampled from the whole dataset according to the prescribed sampling protocol. The chance of the distribution of the pilot dataset having a large deviation from that of the whole dataset should be small. A threshold for hypothesis testing can be set to determine whether to accept or reject the pilot dataset as an unbiased sample from the whole dataset. If there are external supervisions (e.g., market regulators), they can conduct sequential hypothesis testing to check whether the samples provided by each seller converge to the whole dataset or comply to the prescribed sampling procedure.
Appendix D Experiment Details and Additional Results
D.1 Datasets and Models
For our experiments, we use the following vision and language datasets:
Dataset	Total Training Data	Total Test Data	Number of Classes
IMDB [45]	25,000	25,000	2
MNIST [46]	50,000	10,000	10
CIFAR-10 [47]	50,000	10,000	10
ImageNet100 [48]	130,000	5,000	100
BDD100K [15]	70,000	10,000	10
Table 2: Details on datasets used in experiments.
For IMDB dataset, we trained a Long Short-Term Memory (LSTM) network model [49] for 20 epochs; for MNIST dataset, we use Support Vector Machines (SVMs) with RBF kernel. For CIFAR-10 dataset, we trained on the pre-activation Resnet with identity mappings (PreActResNet-18 [50]) for 100 epochs. We trained ImageNet-100 on ResNet-50 [51] backbone for 200 epochs with cosine annealing as the learning rate scheduler. For the autonomous driving object detection dataset, BDD100K, we fine-tune an improved pre-trained faster region proposal-CNN network (Faster-RCNN ResNet50 [52]) on COCO dataset [53] for 30 epochs.
D.2 Details on Baseline Methods
For  $N$  samples (data quantity) from  $m$  data sources with a mixing ratio  $\mathbf{p} = \{p_1,\dots ,p_m\}$ , we consider the following baselines
Linear:  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N,\mathbf{p});D^{\mathrm{val}})\coloneqq \mathbf{a}'\mathbf{p} + b\log (N) + c$  where  $\mathbf{a} = \{a_0,a_1,\dots,a_m\}$ ,  $b$ , and  $c$  are coefficients to be fitted.
Leave-one-out (LOO) and Shapley can be considered special cases for Linear, where the coefficients are calculated as the marginal contribution of the data source (LOO) or its averaged contribution to different combinations of other data sources (Shapley) [18], as opposed to the least-square fitting as in Linear.
Pseudo-quadratic:  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N,\mathbf{p});D^{\mathrm{val}}):= \sum_{i = 1}^{m}(c_2^i\cdot p_i^2 +c_1^i\cdot p_i + c_0) + b\log (N)$
Quadratic:  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N,\mathbf{p});D^{\mathrm{val}}):= \sum_{i = 1}^{m}(c_2^i\cdot p_i^2 +c_1^i\cdot p_i + c_0) + \sum_{i = 1}^{m}\sum_{j = 1}^{i}(c_3^{ij}\cdot p_ip_j) + b\log (N)$
Rational:  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N,\mathbf{p});D^{\mathrm{val}})\coloneqq \sum_{i = 1}^{m}\left(\sum_{j = 1}^{m}c^{ij}\cdot p_{j}\right)^{-1} + b\log (N)$
We fit the Rational baseline according to the setup detailed in [13] and to our best effort. Originally, the method is intended for predicting log loss, whereas in our case, we aim to predict model accuracy. Thus, we replaced the log loss with  $\log (1 - \text{accuracy})$  for the prediction target.
D.3 Performance Prediction for Unseen Data Mixtures p: Ablation Study
Data Source	Case 1	Case 2	Case 3	Case 4	Case 5
1	{0,3,6,7}	{1,7}	{1,5,6,8,9}	{4,6,9}	{0,1,2}
2	{4,5,9}	{0,3,6,8,9}	{2,7}	{8}	{2,4,6}
3	{1,2,8}	{2,4,5}	{0,3,4}	{0,1,2,3,5,7}	{3,5,7,8}