SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the previous experiments, we presented results over a single data source class distribution. Here, we present a more comprehensive view of the projector's performance by running multiple times over random class distributions of data sources, according to Table 3. As shown in Table 4, for all cases, either projector/CS or projector/PQ achieves the highest performance, while baseline methods struggle to get close performance. Although, in many cases (1,2,3,5), Quadratic or Rational baseline obtains the lowest training data MAE, but in test performance, these methods have poor generalization scores, which indicate high overfitting to the training data. These results indicate that predicting performance from data source composition is insufficient for fitting and OT distance plays an important role in better alignment of data sources to the model accuracy.
Table 3: Class distributions of data sources for five different cases.
Method	Case 1	Case 2	Case 3	Case 4	Case 5
projektor/PQ	1.41 · 2.63 (Δ 0.00)	1.11 · 1.35 (Δ 0.00)	1.24 · 1.22 (Δ 0.15)	0.88 · 1.39 (Δ 0.00)	0.84 · 1.26 (Δ 0.00)
projektor/CS	1.45 · 7.07 (Δ 4.44)	1.12 · 1.49 (Δ 0.14)	1.24 · 1.07 (Δ 0.00)	0.91 · 2.31 (Δ 0.92)	0.97 · 1.95 (Δ 0.69)
Linear	3.31 · 10.67 (Δ 8.04)	5.51 · 7.70 (Δ 6.35)	4.35 · 7.31 (Δ 6.24)	5.38 · 8.92 (Δ 7.53)	4.44 · 6.04 (Δ 4.78)
Pseudo-Quadratic	1.35 · 10.18 (Δ 7.55)	1.04 · 5.21 (Δ 3.86)	1.82 · 4.94 (Δ 3.87)	1.34 · 3.54 (Δ 2.15)	1.82 · 2.90 (Δ 1.64)
Quadratic	1.05 · 9.42 (Δ 6.79)	1.02 · 4.94 (Δ 3.59)	0.97 · 3.48 (Δ 3.41)	0.93 · 3.28 (Δ 1.89)	0.88 · 5.22 (Δ 3.96)
Rational	1.06 · 15.99 (Δ 13.36)	1.09 · 6.24 (Δ 4.89)	1.04 · 4.89 (Δ 3.82)	1.04 · 3.02 (Δ 1.63)	0.83 · 2.95 (Δ 1.69)
Table 4: Multiple runs over random data source class distributions in CIFAR10. For each cell the left-column value denotes the fitting MAE value for fitting the training data and we marked in underlined green, whereas the right-column value denotes the MAE value for testing data and we marked in bold red.
D.4 Extended Application to Multiple Number of Data Sources
So far, our experiments are focused solely on the cases with three data sources. To show the practicality of our method, we extend projector to more challenging cases, where we have more than three data sources. Specifically, we explore the setting with 4, 5, and 6 data sources and illustrate
the results with baseline comparisons in Table 5. As we observe, our methods, both projector/CS and projector/PQ, achieve the best testing MAE scores as well as one of the lowest training MAE scores. While both Linear and Pseudo-Quadratic baselines are under-performing and receive poor testing MAE values. As observed in previous experiments, Quadratic baseline presents strong training data fitting but weak testing predictions. For the Rational baseline, we tried our best to fit this method, but we demonstrate that with the larger number of data sources, it is even harder to properly train this function which results in increasing errors, respectively. This experiment demonstrates the capability of our method to extend to more practical settings with multiple data sources.
Method	4 Sources	5 Sources	6 Sources
projektor/PQ	1.15 · 2.83(Δ 0.00)	0.99 · 3.40(Δ 0.00)	1.14 · 2.53(Δ 0.00)
projektor/CS	1.28 · 3.51 (Δ 0.68)	1.23 · 4.14 (Δ 0.74)	1.29 · 2.91 (Δ 0.38)
Linear	2.12 · 4.58 (Δ 1.75)	2.38 · 4.74 (Δ 1.34)	1.99 · 4.03 (Δ 1.50)
Psuedo-Quadratic	1.32 · 3.97 (Δ 1.14)	1.43 · 5.24 (Δ 1.84)	1.34 · 3.91 (Δ 1.38)
Quadratic	1.06 · 3.93 (Δ 1.10)	0.97 · 8.70 (Δ 5.30)	1.02 · 4.67 (Δ 2.14)
Rational	52.72 · 57.93 (Δ 55.1)	611.5 · 501 (Δ 497.6)	1034 · 1155 (Δ 1152)
Table 5: Performance on various number of data sources (4,5,6) in CIFAR-10. For each cell the left-column value denotes the fitting MAE value for fitting the training data and we marked in underlined green, whereas the right-column value denotes the MAE value for testing data and we marked in bold red.
D.5 Optimal Data Source Composition
Here, we showcase the optimal data source selection on CIFAR-10 dataset. Similarly as in Section 5.2 given a pilot dataset of size  $1.5K$  from each of the three data sources, we would like to find the mixing ratio that can maximize the model performance when trained on  $10K$  dataset. As illustrated in Fig. 9, projector/PQ and projector/CS select mixing ratios that achieve the highest model performance, gaining  $4\%$  and  $2\%$  in performance improvement over the best baseline method. Our methods select nontrivial mixing ratios which also outperform the uniform mixing ratio by  $5\%$  and  $3\%$ , respectively. Furthermore, we observe in Fig. 10 that our methods also perform well in performance prediction into larger data scales. Specifically projector/PQ and projector/CS predictions are within  $2.2\%$  discrepancy from the actual model performance for  $10K$  dataset, while the best baseline method prediction has over  $5\%$  error. To demonstrate projector capability, we additionally present results on the case where data sources contain some unlabeled data.
Figure 10: Performance projection of selected optimal mixture ratios in (Fig. 9) onto 10K CIFAR-10 from 1.5K samples. Comparison with the actual model performance.
Figure 9: Optimal data source composition selection for 10K CIFAR-10 from 1.5K samples and the actual model performance.
In this case, we assume data sources have noisy labels. In particular, data sources have  $20\%$ ,  $15\%$ ,  $25\%$  of mislabeled data, respectively. In Fig. 11, surprisingly, we observe that even though the first data source has  $20\%$  of mislabeled data, choosing more of that source improves the model performance. A possible explanation is that the first source dataset contains classes that are important for learning
and the second source has less important for model performance or requires less to be learned (especially since it has a lower mislabeled rate). Moreover, we notice that projector's mixing ratio can improve the model performance by over  $3\%$  from the performance of the best baseline. Results from Fig. 12 indicate that projector's performance projection onto  $10K$  dataset has a prediction error within  $1.7\%$ , while the best baseline has an error of over  $2.2\%$ . To sum up, we have shown possibilities of projector in mixing ratio selection and performance prediction. While baseline methods have the same mixing ratio over any data scales, our method chooses different optimal mixing ratios for different data scales, which has shown some advantages in improving model performance.
Figure 11: Optimal data source composition selection for 10K CIFAR-10 from 1.5K samples with mislabeled data sources, and the actual model performance.
Figure 12: Performance projection of selected optimal mixture ratios in (Fig. 11) onto 10K CIFAR0-10 from 1.5K samples. Comparison with the actual model performance.
Appendix E Additional Discussion and Broader Impacts
Limitations. Despite contributing a new perspective with performance and efficiency improvements, this work still has some limitations and opens up many new investigation venues: (1) How to quantify the influence or further lift the dependence on validation data? While a validation set representative of the downstream learning task is a common assumption in the ML literature, it may or may not be available during data exchange and its quality may vary. (2) Our design could be vulnerable to estimation errors in the scaling law for data sizes, which could lead to magnified prediction errors on larger scales and affect data acquisition decisions [54]. Especially, noises are inevitable due to the performance stochasticity of ML models as well as the sampling process to generate the pilot dataset. (3) Our current framework does not take into consideration of broader tasks that aim for goals beyond accuracy, e.g., fairness, variable data costs, as well as broader acquisition scenarios where data sources have misaligned feature space. Incorporating other objectives and extending to heterogeneous data sources is an exciting direction. (4) Our setup considers honest data providers and the requested samples are faithfully sampled from the actual data sources, leaving an in-depth study of potential security risks, such as malicious data manipulation [28] to future work.
Broader Impacts. This work will have significant impacts beyond advancing the research on data selection and data markets. The techniques developed in this work can be applied to a variety of other subfields of ML related to data acquisition, data valuation, interpretability, robustness, etc. The results of this paper will facilitate the automation of data selection and quality management in machine learning, which in turn, accelerates research and improves services based on ML. Data exchanges and data markets are also at the heart of the global data economy. The advancements in practical data exchanges in this work will substantially benefit the development of data markets and promote data sharing, contributing to the business and economy as well as society as a whole.