SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\alpha$  and  $C$  are some constants [26]. Recent work [14] shows that when data from different sources differ in "quality" (e.g., noise level), which is the most likely scenario in practice, the scaling parameters are often vastly different, rendering parameters  $\alpha$  and  $C$  functions of data composition  $\mathbf{p}$  and model performance for different data mixtures  $\mathbf{p}$  scales with different rates. [13] assumes the same constant  $\alpha$  for all data mixtures, leading to unsatisfactory scaling results. The difficulty underlying this dependence is that these scaling parameters differ for every data mixture and there is no closed-form expression available for this functional relationship. With the performance prediction tools proposed above, it is possible to directly predict model performance of any data mixture at the scale the tool has been fitted. That is, for any given data scale  $N_0$ , as long as one completes the one-off fitting process of the performance predictor, model performance for any data composition at data size  $N_0$  can be inferred directly using Eq. (1) or Eq. (2). Thus, by performing the fitting process at different small scales for once, for any desired data mixture, we can directly fit the neural scaling laws for this particular distribution and project it onto larger data scales, without needing to train any additional model or make any further approximations. We formalize it into the following theorem.
Theorem 1 (Data Composition Dependent Performance Projection). Consider log-linear performance scaling relationship depending on both data size  $N$  and data composition  $\mathbf{p}$  given as  $\mathbb{E}_V[\mathcal{L}(\mathcal{A}(\mathcal{D}(N,\mathbf{p}));D^{val})] = -\alpha (\mathbf{p})\log (N) + C(\mathbf{p})$ . Assume one has completed the fitting of the performance predictor on two different scales  $N_0 < N_1$ , which gives  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_0,\mathbf{p}));D^{val})$  and  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_1,\mathbf{p}));D^{val})$  for all data mixtures  $\mathbf{p}$ . Then, the model performance  $\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N,\mathbf{p}));D^{val})$  for any data mixture  $\mathbf{p}$  at any data scale  $N$  can be predicted as
$$
\hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N, \mathbf {p}); D ^ {\text {v a l}}) = \left(\log \frac {N _ {1}}{N _ {0}}\right) ^ {- 1} \left[ \log \frac {N}{N _ {0}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {1}, \mathbf {p})); D ^ {\text {v a l}}) - \log \frac {N}{N _ {1}} \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {0}, \mathbf {p}); D ^ {\text {v a l}}) \right] \tag {3}
$$
without requiring fitting any additional parameters. The proof and derivations are given in Appendix.B. We refer to this method as parameter-free projection for model performance. As this procedure does not rely on any additional assumption or parameterized surrogate, it requires minimal computational overhead while achieving considerably higher prediction accuracy compared to existing approaches such as [13]. Not exclusive to the performance predictor proposed in this work, this method can be plugged into other predictors seamlessly and provides benefits at large, marking a novel contribution to performance projection in data acquisition problems.
4.4 Performance-guided data selection
The intention of creating the proposed tools is not limited to providing predictions for model performance, rather, we expect the predictions to support determining the optimal data acquisition strategy. We show that these problems are convex and differentiable (Appendix.B) and thus can be solved effectively via gradient-based methods. Specifically, for our primary objective fixed-budget selection for maximal performance, with the proposed performance predictors with projection, we solve for  $\mathbf{p}^{*} = \arg \max_{\mathbf{p}}\hat{\mathcal{L}} (\mathcal{A}(\mathcal{D}(N_{s},\mathbf{p})),D^{\mathrm{val}})$ . We solve it iteratively with the following procedure. First, we initialize the algorithm with  $\mathbf{p} = \mathbf{p}^0$  where  $\mathbf{p}^0$  can be chosen arbitrarily provided that
$\sum_{i}p_{i} = 1$  . Then, at each step, we perform the gradient update as
$$
\mathbf {p} ^ {\mathbf {t} + \mathbf {1}} \leftarrow \mathbf {p} ^ {\mathbf {t}} + d ^ {t} \cdot \left. \frac {\partial \hat {\mathcal {L}} (\mathcal {A} (\mathcal {D} (N _ {s} , \mathbf {p})), D ^ {\mathrm {v a l}})}{\partial \mathbf {p}} \right| _ {\mathbf {p} = \mathbf {p} ^ {t}}
$$
where  $d^{t}$  is the step size at iteration  $t$  and we obtain  $\mathbf{p}^{*} = \mathbf{p}^{\mathrm{T}}$  at convergence as the desired solution. Optimal Transport naturally provides its gradient w.r.t. the probability mass of data points in its dual solutions [8], which directly gives the gradients w.r.t. data mixtures  $\mathbf{p}$ . This easy availability of gradients renders the optimization highly efficient in computation, resulting in remarkably fast solutions. We use the calibrated gradient of OT from [8] which ensures the updated mixture  $\mathbf{p}$  remains within the simplex  $\sum_{i} p_{i} = 1$  at each step. We provide technical details of the gradient computation in Appendix.B. The alternative objective can be treated as a direct extension of the primary and we defer to Appendix.B for its solution procedure. The pseudo-code for projector is provided in Appendix A.
Figure 2: Workflow chart for the two-stage performance inference process of projector.
In this section, we cover two main applications for projector. 1) Performance projection, where for any mixing ratio of data sources and any data scale, we want to predict the performance of the model trained on a given composed dataset. We also demonstrate efficiency and efficacy of our method for different scenarios of data sources, such as unlabeled or unlabeled data. 2) Optimal data source acquisition strategy, where for a given data budget, we find a mixing ratio of data sources that can maximize the performance of a model. We present a solution to select optimal data source composition for two learning paradigms scenarios: training from scratch and model fine-tuning.
We compare with six existing baseline methods, where the first four: (1) Linear [13], which assumes a linear relationship with the data compositions; (2) Pseudo-Quadratic assumes a simple non-linear relationship; (3) Quadratic [13] assumes a fully quadratic relationship; (4) Rational [13] models a relationship through the sum of a set of rational functions. For  $m$  data sources, each function has  $m$  parameters, and there are  $m$  such functions, totaling  $m^2$  parameters; (5) LOO [5] measures the importance of a data source by computing the performance difference after removing that source; (6) Shapley [18] is a game-theoretic method which computes the average marginal contribution of a data source to different subsets of other sources. Baselines (5) and (6) are suitable for informing the selection of data sources but are unable to predict model performance, so we only include them in data source selection experiments. Details on implemented baselines are described in Appendix D and further explained in [13]. For all experiments, we set up the problem with three data sources, where each source consists of different classes, and we refer the reader for additional information on the experimental setup, algorithm, datasets, models, implementations, code repository, and ablation studies on the number of data sources to Appendix D. We also showcase the runtime vs performance prediction trade-off and comparison with baseline methods.
Evaluation Metrics. We use mean absolute error (MAE) to measure the performance of our method by calculating the absolute difference between the predicted accuracy and the actual accuracy. For
Figure 3: Predicted model performance vs. actual model performance for extrapolation on CIFAR-10 with total of 1000 data points for 3 data sources. Comparison between projector and baselines.
Figure 4: MAE vs. number of model trainings used for fitting the method. Efficiency comparison between Projektor and baselines.
the object detection task, we adopt a commonly used metric, mean average precision (mAP), which measures the average precision of a model across multiple classes or categories, providing a single value that represents the overall accuracy. The average precision represents the area under the precision-recall curve for a single class.
Hyperparameters. For practical reasons, we set the data scale  $N_{1}$  in Eq. 5 to be the size of the smallest pilot dataset, i.e.  $N_{1} = \min_{i}|D_{i}^{\mathrm{pi}}|$ . Upon selecting  $N_{1}$ , we empirically choose  $N_{0}$  to be  $\frac{2}{3} N_{1}$ . For further investigations on selecting  $N_{0}$ , we provide sensitivity analysis on  $N_{0}$  in Appendix D.
5.1 Performance Prediction
Method	MNIST	CIFAR-10	ImageNet100	IMDB
Projektor/PQ (Ours)	1.63 · 7.27(Δ 3.01)	0.84 · 1.26(Δ 0.00)	0.53 · 0.38(Δ 0.00)	0.25 · 1.16(Δ 0.00)
Projektor/CS (Ours)	3.30 · 4.26(Δ 0.00)	0.97 · 1.95(Δ 0.69)	0.54 · 1.03(Δ 0.65)	0.90 · 1.87(Δ 0.71)
Linear	5.27 · 28.03(Δ 23.77)	4.44 · 6.04(Δ 4.78)	1.47 · 4.59(Δ 4.21)	1.88 · 7.48(Δ 6.32)
Pseudo-Quadratic	1.63 · 7.27(Δ 3.01)	1.82 · 2.90(Δ 1.64)	0.77 · 2.72(Δ 2.34)	0.43 · 2.74(Δ 1.58)
Quadratic	1.63 · 7.27(Δ 3.01)	0.88 · 5.22(Δ 3.96)	0.50 · 2.34(Δ 1.96)	0.43 · 2.74(Δ 1.58)
Rational	5.28 · 27.29(Δ 23.03)	0.83 · 2.95(Δ 1.69)	0.44 · 2.27(Δ 1.89)	1.91 · 7.11(Δ 5.95)
Table 1: Training and test MAE values for extrapolation of data source compositions for each method. For each cell the left/right column value denotes the MAE value for predicting the training/test data, respectively. Underlined green/bold red denotes lowest training/test MAE value, respectively.
A. Predicting Performance for Unseen Data Mixtures p
In this experiment, we fit the parameters on limited compositions and extrapolate the prediction to unseen compositions. Specifically, we choose one data source and limit its maximum composition to  $< 55\%$  of contribution in the training set of the performance predictor, then we predict accuracy on the compositions consisting of  $\geq 55\%$  of contribution. As we observe in Figure 3, Linear and Pseudo-Quadratic methods cannot fit well the training data, which indicates that these methods do not have a strong representation power. While Quadratic and Rational baselines can fit the training data, they suffer from overfitting and do not generalize to unseen compositions. On the other hand, as seen in Table 1, our method projector/PQ achieves the best training and extrapolation performance.该项目/CS achieves second best extrapolation performance. Furthermore, we analyze the efficiency of our method compared to other baselines. As observed in Figure 4, projector not only achieves the lowest MAE score but also converges with around 15 training data for projector/CS and 25 for projector/PQ, which demonstrates low computational requirement of our method. With shown strong predictive power of projector, we now proceed to practical applications in performance projections onto larger data scales.
B. Performance Projection to Larger Data Scales
Mislabeled Data Sources. In this experiment, we project performance onto larger data scales and also assume a more practical setting where data sources might not be of high quality and contain
Figure 6: Optimal data source composition selection for 50K ImageNet-100 from 10K samples and the actual model performance.
Figure 7: Performance projection of selected optimal mixture ratios in (Fig. 6) onto 50K ImageNet-100 from 10K samples. Comparison with the actual model performance.
noisy labels [43]. It is then critical to factor such irregularity for the performance prediction into our method. Given three mislabeled data sources formed by sampling CIFAR-10, each of which releases a pilot dataset of size 1K, we project performance for various mixing ratios onto larger data sizes, i.e. 2K, 5K, 7K, and 10K. We then measure the MAE value across all data scales. We observe in Fig. 5 that projector achieves the best projection performance compared to all baseline methods. projector/PQ achieves the lowest MAE score below  $2\%$  and projector/CS is slightly above  $2\%$ . The improved performance of our method can be attributed to the incorporation of actual data distance computation. This inclusion allows for a more accurate representation of mislabeling information in performance projection, unlike baseline methods that neglect this crucial information.
The promising results demonstrate the potential of our method to project performance of any composition to any data scale, which is important in the case of mistrabeled data sources in the partially-revealed setting.
Unlabeled Data Sources. As mentioned earlier, data sources often contain noisy labels, and the process of labeling data can be costly. On the other hand, it is not uncommon to encounter unlabeled data sources. Therefore, we would like to extend our method to accommodate the setting of data sources without labeled data, and we aim to project performance of unlabeled data compositions from pilot data source mixtures of 1K samples from CIFAR-10. The three data sources contain unlabeled data from different classes of CIFAR-10. In this case, we compute the optimal transport distance on the feature space only, and we assume access to the labels of the pilot datasets,
which enables us to train a model and obtain performance values. Consequently, we project performance across various mixing ratios onto larger data sizes (2K, 5K, 7K, and 10K). To visualize the performance of our method, we plot the difference between the projected performance and the actual performance. The closer the value approaches zero, the more optimal the projection becomes. Surprisingly, as we illustrate in Fig. 8, projetor can consistently maintain good projection performance for larger data sizes. Even at  $10K$ , the largest error is below  $10\%$ . These outcomes show that projetor can also be extended to unlabeled data sources, which demonstrates the flexibility and practicality of our method.
Figure 5: Performance projections from 1K CIFAR-10 samples across various mixing ratios and larger data scales: 2K, 5K, 7K, 10K. Comparison between projector and baselines.
5.2 Optimal Data Source Selection
Training a model requires a tremendous amount of resources to tune hyperparameters to achieve the highest performance. We demonstrate that by choosing data strategically, we can also improve model performance. We consider a setting, where we are facing the problem of choosing only  $50K$  to train ResNet-50 on ImageNet100 and would like to maximize the model's performance.
However, we are provided with only a pilot dataset of size  $10K$  from each data source. As we observe in Figure 6, our optimized mixing ratio based on (4.4) achieves the highest model performance compared to all baselines. Further, using the functions from the previous step, we project the performance of our selected mixing ratio, and we observe in Figure 7 that projector not only most closely predicts the accuracy to the actual accuracy but also attains the highest actual accuracy out of all methods. The improved selection of mixture ratios in our method can be attributed to our proposed selection approach (4.4). Unlike baseline methods that assume the same optimal composition for all data scales, our method finds optimal compositions specific to each data scale. For more experiments on CIFAR-10, we refer the reader to Appendix D.
5.3 Application to Fine-tuning
As powerful architectures has been introduced and computation power has improved, larger models and datasets has become increasingly prevalent in training visual and natural language tasks. However, retraining these large pretrained foundation models can be cost-prohibitive, which leads to widespread adoption for fine-tuning these models. However, these large pre-trained foundation models are expensive to retrain but are popular for fine-tuning for more customized tasks. In our case, we adopt a pretrained Faster R-CNN model trained on COCO dataset. Our task is to fine-tune the model on the autonomous driving dataset for object detection, BDD100K [15]. We assume each data source to specialize in taking pictures at a specific time of day, i.e. daytime, night, or dawn/dusk pictures. Similarly to the previous task, we select optimal data source composition and project the mean average precision (mAP) onto larger data scales. In Figure 1(b), we observe fine-tuning accuracy projection onto eight larger data scales from 1000 samples and observe that our predictions do not deviate from the actual accuracy by more than 0.4, which indicates projector's extended capability of performance projection for fine-tuning.
6 Discussion and Outlook
Figure 8: A direct visualization for the landscape of prediction errors in model performance for all data compositions as the scale of data to be predicted grows. We consider the setting of 3 data sources with unlabeled samples from CIFAR-10. We construct the proposed performance predictors within 1k samples that are considered accessible to the practitioner. We then predict model performance for all data compositions and on larger data scales (2-10k samples). X-Y axes represent the proportion of data from data sources #1 and #2 (consequently, the proportion of data from data source #3 will be  $100\% - \mathrm{X} - \mathrm{Y}$ ) and the prediction errors are visualized in Z-axis (horizontal plane represents zero error, red spikes blue dips represent deviations from over/under-prediction). As the data scale gets larger, the performance predicted from the initial 1k samples becomes slightly less accurate, but prediction error remains mostly within  $5\%$  even at the scale of 10k data (10 times larger than the pilot data), retaining its effectiveness and functionality.
This paper presents a novel framework to conduct data selection from partially revealed sources, addressing the practical challenge in emerging data market scenarios. In particular, in contrast to existing work that tries to directly fit non-informative parametric surrogates on the limited available samples to predict model performance at different data sizes and compositions of data sources, which suffers from pronounced computational burdens and often unsatisfactory results, our key technical contribution is an OT-based performance scaling method. The take-away from our empirical study is that despite being extensively adopted in the past, fitting non-informative parametric surrogates for predicting performance scaling is actually suboptimal-computationally inefficient, often impractical, and less accurate; utilizing data distance in the performance prediction provides immediate benefits and presents a better pathway to construct the predictors.
Contributing a new perspective with performance and efficiency improvements, this work still has some Limitations and opens up many new investigation venues, such as lifting the requirement on validation data, accounting for malicious data owners, and extending to data sources that are misaligned in feature space. Additional discussions and Broader Impacts are provided in Appendix.E.
RJ and the ReDS lab acknowledge support through grants from the Amazon-Virginia Tech Initiative for Efficient and Robust Machine Learning, the National Science Foundation under Grant No. IIS-2312794, NSF IIS-2313130, NSF OAC-2239622, and the Commonwealth Cyber Initiative.
[1] Burr Settles. Active learning: Synthesis lectures on artificial intelligence and machine learning. Long Island, NY: Morgan & Clay Pool, 10:S00429ED1V01Y201207AIM018, 2012. 2
[2] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I, pages 181-195. Springer, 2022. 2, 3
[3] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017. 2, 3
[4] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMLR, 2019. 3
[5] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019. 7
[6] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
[7] Tom Yan and Ariel D Procaccia. If you like shapley then you'll love the core. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5751-5759, 2021.
[8] Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. In 11th International Conference on Learning Representations, ICLR, page to appear, 2023. 2, 3, 5, 7, 19
[9] Datarade. https://datarade.ai/.2,4
[10] Innodata. https://innodata.com/ai-data-marketplace/.2
[11] Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476, 2017. 2
[12] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022. 2
[13] Tatsunori Hashimoto. Model performance scaling with multiple data sources. In International Conference on Machine Learning, pages 4107-4116. PMLR, 2021. 2, 6, 7, 18, 21
[14] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021. 2, 3, 6
[15] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020. 2, 10, 20
[16] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009. 3, 4, 18
[17] Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:2110.14049, 2021. 3
[53] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014. 20
[54] Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M Alvarez, Zhiding Yu, Sanja Fidler, and Marc T Law. How much more data do i need? estimating requirements for downstream tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 275-284, 2022. 23
A projector: Framework and Algorithms 16
A.1 projector Pipeline 16
B Proofs and Optimization Details 17
B.1 Proof for Theorem 1 17
B.2 Objectives for Data Selection 18
B.3 Optimization and Convexity 18
B.4 Gradients Calculation, Stepsize Selection, and Convergence 19
C Sampling Stochasticity and Market Practices 20
D Experiment Details and Additional Results 20
D.1 Datasets and Models 20
D.2 Details on Baseline Methods 21
D.3 Performance Prediction for Unseen Data Mixtures p: Ablation Study 21
D.4 Extended Application to Multiple Number of Data Sources 21
D.5 Optimal Data Source Composition 22
E Additional Discussion and Broader Impacts 23
Appendix A Projektor: Framework and Algorithms
A.1 Projektor Pipeline