Optimizing Data Collection for Machine Learning
Authors: Rafid Mahmood
James Lucas
Jose M. Alvarez
Sanja Fidler
Marc T. Law
Venue: NeurIPS 2022 Accept (2022)
Keywords: data collection
sequential decision-making

Abstract: Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.

Optimizing Data Collection for Machine Learning
Rafid Mahmood'
James Lucas' Jose M. Alvarez! 'NVIDIA UUniversity of Toronto {rmahmood, jalucas, josea, sfidler, marcl)onvidia.com com
Sanja Fidler 1.233
Mare T. Law!
Vector Institute
Abstract
Modern deep learning systems require huge data sets to achieve impressive per- formance, but there is little guidance how much what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose new paradigm for modeling the data collection workflow as a formal optimal data collection problem that lows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes tasks requiring multiple data sources, such as labeled and unlabeled data used in sunervupere learning. To solve our problem, we develop Gaarn-Optimize- Collect (LOC), which minimizes expected future collection costs. Finally, numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.
Introduction
When deploying a deep learning model in an industrial application, designers often mandate that the model must meet pre-determined baseline performance, such target metric over validation data set. For example, an object detector may require certain minimum mean average precision before being deployed
safety-critical setting. One of the most effective ways of meeting target performances is by collecting more training data for a given model.
Determining how much data is needed to meet performance targets impact costs and development delays. Overestimating the data requirement incurs excess costs from collection, cleaning, and annotation. For instance, annotating segmentation masks for driving data set takes between 15 to 40 seconds per object. For 100,000 images the annotation could require between 170 and 460 days-equivalent of time 2 On the other hand, collecting workflow delays from having to collect more later. For example, in medical imaging applications, this means further clinical data acquisition rounds that require expensive clinician time. the worst case, designers may even realize that project is infeasible only after collecting insufficient data.
little data may incur future costs and
The growing literature sample complexity in machine learning has identified neural scaling laws that scale model performance with data set sizes according to power laws 11 For instance, Rosen- feld et al. [6] fit power law functions the performance statistics of small data sets to extrapolate the learning curve with more data. contrast, Mahmood et al. [2] consider estimating data requirements and show that even small errors in a power law model of the learning curve can translate to massively over- or underestimating how much data is needed. Beyond this, different data sources have different costs and scale differently with performance [TH-14) For example, although unlabeled data may be easier to collect than labeled data, some cemi-supervised learning tasks may need an order of magnitude more unlabeled data to match the performance of small labeled set. Thus, collecting more data based only on estimation will fail to capture uncertainty and collection costs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Learn
Optimize
Collect
Sample bootstrap performance with subsets of data to learn data requirement distribution
Use estimated distribution to optimize collection cost plus failing to V
Pay olg, - q, to collect additional data
No
Have hit time limit T?
No
Have we achieved score V-?
Yes
Yes
Initialize with 9, data points target
and terminate
Terminate
Figure 1: In the optimal data collection problem, we iteratively determine the amount of data that we should have, pay to collect the additional data, and then re-evaluate our model. Our approach, Learn-Optinize-Collen optimizes for the minimum amount of data qf to collect.
In this paper, we propose new paradigm for modeling the data collection workflow as an optimal data collection problem. Here, designer must minimize the cost of collecting enough data obtain model capable of desired performance score. They have multiple collection rounds, where after each round, they re-evaluate the model and decide how much more data to order. The data has per-sample costs and moreover, the designer pays a penalty if they fail to meet the target score within a finite horizon. Using this formal framework, we develop an optimization approach for minimizing the expected future collection costs and show that this problem can be optimized in each collection round via gradient descent. Furthermore, our optimization problem immediately generalizes to decisions over multiple data sources (e.g., unlabeled, long-tail, cross-domain, synthetic) that have different costs and impacts on performance. Finally, we demonstrate the value of optimization over naïvely estimating data set requirements (c.g.. (2) for several machine learning tasks and data sets.
Our contributions are as follows. (1) We propose the optimal data collection problem in machine learning, which formalizes data collection workflows. (2) We introduce Learn-Optimine Optimize- Colllect (LOC), learning- wewninng-nddoooee and-oniini framework that minimizes future collection costs, can be solved via gradient descent, and has analytic solutions in some settings. (3) We generalize the data collection problem and LOC to multi-variate setting where different types of data have different costs. To the best of our knowledge, this is the first exploration of data collection with general multiple data sets in machine learning, covering for example, cupi-super and long-tail learning. (4) We perform experiments classification, segmentation, and detection tasks to show, average, approximately a 2 reduction in the chances of failing to meet performance targets, versus estimation baselines.
2 Related work
Neural Scaling Laws. According to the neural scaling law literature, the performance of model on a validation set scales with the size of the training data set via power law V x 00g [5] 6
10 15-19) Hestness et al. [5] observe this property over vision, language, and audio tasks, Bahri et al. 9 develop a theoretical relationship under assumptions on over- peee-parametrization and the Lipschitz continuity of the loss, model, and data, and Rosenfeld et al. [6] estimate power laws using smaller data sets and models to extrapolate future performance. Multi-variate scaling laws have also been considered for some specific tasks, for example in transfer learning from synthetic real data sets [I]]. Finally, Mahmood et al. [2] explore data collection by estimating the minimum amount of data needed to meet a given target performance over multiple rounds. Our paper extends these prior studies by developing an optimization problem to minimize the expected total cost of data collected. Specifically, we incorporate the uncertainty in any regression estimate of data requirements and further generalize to multiple data sources with different costs.
Active Learning. In active learning, model sequentially collects data by selecting new subsets of an unlabeled data pool to label under pre-determined labeling budget that replenishes after each round (20 24). In contrast, our work focuses on systematically determining optimal collection budget. After determining how much data to collect, we can use active learning techniques to collect the desired amount of data.
Statistical Learning Theory. Theoretical analysis of the sample complexity of machine learning models is typically only tight asymptotically, but some recent work have empirically analyzed these

---

Particularly, Bisla et al. [00 study generalization bounds for deep neural networks, provide empirical validation, and suggest using them to estimate data requirements. In contrast, our paper formally explores the consequences of collection costs on data requirements.
relationships (25] 26
Optimal Experiment Design. The topic of how collect data, select samples, and design scientific experiments or controlled trials is well-studied in econometrics [27/29]. For example, Bertsimas al. 30 optimize the assignment of samples into control and trial groups to minimize inter-group variances, Most recently, Carneiro et al. 31 optimize how many samples and covariates to collect in a statistical experiment by minimizing a treatment effect estimation error or maximizing f-test power. However, our focus on industrial machine learning applications differs from experiment design by having target performance metrics and continual rounds of collection and modeling.
3 Main Problem
In this section, we give motivating example before introducing the formal data collection problem. We include table of notation in Appendix A
Motivating Example. A startup developing an object detector for use in autonomous vehicles within the next T = 5 years. Their model must achieve mean Average Precision greater than V 95% on a pre-determined validation set or else they will lose an expected profit of P = 000, 000. Collecting training data requires employing drivers record videos and annotators to label the data, where the marginal cost of obtaining each image is approximately c = $1. In order to manage annual finances, the startup must plan how much data collect at the beginning of each year.
Let z ~ p(z) be data drawn from a distribution p. For instance, 2 == (xx, 3) may correspond to images and labels y. Consider prediction problem for which we train a model with data set D of points sampled from p(2). Let (D) be a score function evaluating the model trained on D.
Optimal Data Collection. We possess an initial data set D := {2 1 of gu points; we omit the subscript on D referring to its size when
obvious. Our problem defined by a target score V* > V ). cost- pest-per- e of collection, a horizon of T rounds, and a penalty P for failure. At the end of each round t E {1,..., T), let qe be the current amount of data collected. Our goal to minimize the total collection cost while building model that can achieve the target score:
min 4 O
c(TT 90) P1(V V
s.t. 40
S
QT
min Tr
> - - --1 P VV}
s.t. 90 S 41 S
S 4r
(1)
The collection cost is measured by the difference in data set size between the final and the 0-th round clyr go) e E'" (g: 1)- Because we collect data iteratively over multiple rounds (see
Figure D. we break into the sum of differences per round. Specifically in each round, we
1. Decide to grow the data set to qi 2 Qt-1 points by sampling ò = {$) ~ p(=). Pay
a cost c(g- 9)-1) and update D < DUD.
2. Train the model and evaluate the score. If V(D): 2 V*, then terminate.
3. If T' then pay the penalty and terminate. Otherwise, repeat for the next round.
The model score typically increases monotonically with data set size 5 6.. This means that the minimum cost strategy for (1) is to collect just enough data such that V( V*. We can estimate this minimum data requirement by modeling the score function as a stochastic process. Let Vg V(D₂) and let (Vg)gEz, be stochastic process whose indices represent training set sizes in different rounds. Then, collecting data in each round yields a sequence of subsampled data sets D Da. C Dr, and their performances V VD.,).. The minimum data requirement is the stopping time
D* = arg min {q V, 2 V
(2)
which is random variable giving the first time that we pass the target. Note that qी .... 97 D is a minimum cost solution to the optimal data collection problem, incurring total cost c(D" 90 P
We assume that c(D* 40) P. since otherwise the optimal strategy would be to collect no data.
3

---

Estimating D using past observations of the learning curve is difficult since we have only T rounds. Further, Mahmood al. [2] empirically show that small errors in fitting the learning curve can cause massive over- or under-collection. Thus, robust policies must capture the uncertainty of estimation.
4 Learn-Optimiz-Coolet (LOC)
Our solution approach, which we refer to as Lear-Optinize-Colles (LOC), minimizes the total collection cost while incorporating the uncertainty of estimating D Although D is discrete random variable, is realized typically on the order of thousands or greater. To simplify our problem and ensure differentiaailityy
assume that D* is continuous and has well-defined density.
Assumption 1. The random variable D is absolutely continuous and has a cumulative density function (CDF) F(q) and probability density function (PDF) f(()) == dF(())
In Section 4.1 we first develop optimization model when given access to the CDF (g) and PDF F(q). In Section 4.2 we estimate these distributions and combine them with the optimization model. Finally in Section we delineate our optimization approach from prior regression methods.
4.1 Optimization Model
We propose optimization problem that for any I, can simultaneously solve for the optimal amounts of data to collect ....... 9r in all future rounds. Consider t = 1 and to develop intuition, suppose we know a priori the exact stopping time D Then, problem I can be re-written
min Lign....... qT: D*)
8. t. 40 S ... 9T
(3)
where the objective function is defined recursively follows
L(gi..... "......,99:.. D*) c(q1 40) + +1g < D*) (c 41) + 1 < DJJ (c -92)...
( < D*) (dr 9T-1) + P1 p1 D')) )..)
T
[-1
14. DD) + 11 < D*}
T (q:- 11-1) D*} Pl < D
t=1
The objective differs slightly from I due to the indicator terms, which ensure that once we collect enough data, we terminate the problem. The second line follows from gathering the terms. The third line follows from observing that q1 S 92 S ... S Qr are constrained.
In practice, we do not know D a priori since it is an unobserved random variable. Instead, suppose we have access to the CDF Fq)). Then, we take the expectation over the objective [L (........ qr; D*)] to formulate a stochastic optimization problem for determining how much data to collect:
T
min Q1
an-- P(1--))) s. S S -.. gr-
(4)
1=1
Note that the collection variables should be discrete (1....; Cr E Z4, but similar to the modeling of D*, relax the integrality requirement, optimize
continuous variables, and round the final solutions. Furthermore, although problem (4) is constrained, we can re-formulate it with variables
d, = q: 4-q ; this consequently replaces the current constraints with only non-negativity constraints d, 2 0. Finally due to Assumption] I problem (4) can be optimized via gradient descent.
4.2 Learning and Optimizing the Data Requirement
Solving problem 4 requires access to the true distribution F(g), which we do not have in reality. In each round, given current training data set D,, of q: points, we must estimate these distribution functions F(g) and f(g) and then incorporate them into optimization problem.

---

Given current data set D we may sample an increasing sequence of R subsets Day Qq/R C D29./R C C gg., fit our model to each subset, and compute the scores to obtain a data set of the learning curve R (rgg R, V (Drag rr.///))))))))))))))))))) [n))) In order to model the distribution of D we can take B bootstrap resamples of R to fit a series of regression functions and obtain corresponding estimates {D,) Given set of estimates of the data requirement, we estimate the PDF via Kernel Density Estimation (KDE). Finally to fit the CDE, we numerically integrate the PDF.
In our complete framework, LOC, we first estimate (q) and f(g). We then use these models to solve problem 4.. Note that in the t-th round collection, we fix the prior decision variables q1.... qt-1 constant. Finally, we collect data as determined by the optimal solution qi to problem (4). Full details of the learning and optimization steps, including the complete Algorithm, are in Appendix B
4.3 Comparison to Mahmood et al. 22
Our prediction model extends the previous approach of Mahmood et al. [2]. who consider only point estimation of D They (i) build the set R, (ii) fit parametric function (g; 0) to R via least-squares minimization, and (iii) solve for D arg min, 1/ Ö(g; 0) N V*J. They use several parametric functions from the neural scaling law literature, including the power law function (i.e., Ö(g: 0) := 0ug + 0 2 g where f := {fu. 81. 123),, and use an ad hoc correction factor obtained by trial and error on past tasks to help decrease the failure rate. Instead, we take bootstrap samples of RR. to fit multiple regression functions, estimate a distribution for D, and incorporate them into our novel optimization model. Finally, we show in the next two sections that our optimization problem has analytic solutions and extends to multiple sources.
5 Analytic Solutions for the T Setting
In this section, we explore analytic solutions for problem The unobservable D* and sequential decision-making nature suggest this problem can be formulated as a Partially Observable Markov Decision Process (POMDP) with an infinite state and action space (see Appendix [C.]), but such problems rarely permit exact solution methods [32). Nonetheless, we can derive exact solutions for the simple case a single T round, re-stated below
min cg1 40) + P(1 - F(()))
S. go S 9
(5)
Theorem 1. Assume F(g) is strictly increasing and continuous. If there exists 2 20. &
0 where
c P 91 4o
F(g)) F(go)
1-1- P P= 2))
then there exists an E < 1 F-Fo()) that satisfies P P PPccf((F-((L E)) and an optimal solution to the corresponding problem 55 qi = F- c). Otherwise, the optimal solution is qं qo-
When the penalty P is specified via a failure risk e, the optimal solution to problem (5) is equal to a quantile of the distribution of D We defer the proof and some auxiliary results to Appendix C.2
Theorem further provides guidelines on choosing values for the cost and penalty parameters. While e is the dollar-value cost per-sample, which includes acquisition, cleaning, and annotation, / can reflect their inherent regret or opportunity cost of failing to meet their target score. A designer can accept risk of failing to collect enough data Pr[q" < D*} c. From Theorem [ their optimal strategy should be to collect F-1(1- - points, which also the optimal solution to problem (5).
The Multi-variate LOC: Collecting Data from Multiple Sources
So far, we have assumed that a designer only chooses how much data to collect and must pay fixed per-sample collection cost. We now explore the multi-variate extension of the data collection problem where there are different types of data with different costs. For example, consider long-tail learning where samples for some rare classes are harder to obtain and thus, more expensive (33 semi-supervised learning where labeling data may cost more than collecting unlabeled data [34], or
domain adaptation where a source data set is easier to obtain than target set (35). In this section, we highlight our main formulation and defer the complete multi-variate LOC to Appendix
5

---

Consider K E I data sources (e.g., K = 2 with labeled and unlabeled) and for each k E {1..... .......KK let 2t ~ p=(==*) ) he data drawn from their distribution. We train a model with a data set DD == U DD where each D. contains points of the k-th source. The performance or score function of our model is V(D), ,..,, D.)., For each k, we initialize with o points. Let qo (qb: ... g denote the vector of data set sizes and let (c², ...... c²*) denote costs (i.e., c is the cost of collecting data from p=(=*)) Given target V*, penalty P, and T rounds, we want to minimize the total cost of collection
<' ( P1 (V o (..)))......><<>> 91 AT
min
8. L. q0 Sq S q2
qT
We follow the same steps shown in Section for this problem. First, the learning curve is now stochastic process {Va)qezz indexed in dimensions. Next, the multi-variate analogue of the minimum data requirement in (2) is the minimum cost amount of data needed to meet the target:
D arg min {c'q Vg 2 v
We randomly pick a unique solution to break ties. From Assumption | D is a random vector with a PDF f(g) and CDF F(g)= := == f(q)dq. Finally, the multi varti-variate analogue of problem is
T >>
min 91 9T
T
q. 1) .) (1 - F(g- + P F( s.t. qo S sq S ... S qT
The Multi-variate LOC requires multi-variate PDFs, which we can fit in the same way as discussed in Section 4.2 However, we now need multi-variate regression functions that can accommodate different types of data. In Appendix [D] we propose an additive family of power law regression functions that can handle an arbitrary number of K sources. In our experiments, we also generalize the estimation approach Mahmood et al. to the multi-source setting for comparison.
Empirical Results
We explore the data collection problem over sets of experiments covering single-variate K (Section 4 and multi-variate K 2 (Section (6) problems. We consider image classification, segmentation, and object detection tasks. For every data set and task, LOC significantly reduces the number of instances where we fail to meet a data requirement V*, while incurring a competitive cost with respect to the conventional baseline of naïvely estimating the data requirement
In this section, we summarize the main results. We detail our data collection and experiment setup in Appendix E We expand our full results and experiments with additional baselines in Appendix
7.1 Data and Methods
When K = 1, the designer decides how much data to sample without controlling the type of data. We explore classification CIFAR-10 36), CIFAR- 100 (36), and ImageNet (37), where train ResNets (38] to meet a target validation accuracy. We explore semantic segmentation using Deeplabv3 39 BDD100K [4). which is large-scale driving data
well as Bird's --ye View (BEV) segmentation on nuScenes [41 using the 'Lift Splat' architecture [42]; for both tasks, we desire a target mean intersection-over-unioo (loU). We explore 2-D object detection PASCAL VOC
using SSD300 45 where we evaluate mean average precision (mAP).
When K = 2, the designer collects two types of data with different costs. We first divide CIFAR-100 into two subsets containing data from the first and last 50 classes, respectively. Here, we assume that the first 50 classes are more expensive to collect than the last; this mimics real-world scenario where collecting data for some classes (e.g., long-tail) is more expensive than others. We then explore semi-supervised learning on BDD100K where the labeled subset of this data is more expensive than the unlabeled data; the cost difference between these two types is equal to the cost of data annotation.
We use simulation model of the deep learning workflow following the procedure of Mahmood al. (2) to approximate the true problem while simplifying the experiments (see Appendix [F] for full details). To avoid repeatedly sampling data, re-training model, and evaluating the score, each

---

VI
10D1ON (T-2)
CIAA-D (T-3)
CEAM-CED (T-5)
Irogengk (T-51
Figure 2: Mean + standard deviation of 5 seeds the ratio of data collected oj/ D for different V The rows correspond to T 1,3,5 5 and the columns to different data sets. The black line corresponds to collecting exactly the minimum data requirement. LOC almost always remains slightly above the black line, meaning we rarely fail to meet the target.
simulation uses a piecewise-linear approximation of 'ground truth' learning curve that returns model performance as a function of data set size. In our problems, we initialize with go = 10% of the full data set (we use 20% for VOC). Then in each round, solve for the amount of data to collect and then call the piecewise-linear learning curve to obtain the current score.
We compare LOC against the conventional estimation approach of Mahmood et al. 2 who fit regression model to the learning curve statistics, extrapolate the learning curve for larger data sets, and then solve for the minimum data requirement under this extrapolation. There are many different regression models that can be used to fit learning curves 15 5 g Since power laws are the most commonly studied approach in the neural scaling law literature, we focus on these. In Appendix F.4 we show that our optimization approach can be incorporated with other regression models.
7.2 Main Results
[V 1, V(D) targets, where 'D is the entire data set. We evaluate all methods on (i) the failure rate, which is how often the method fails to achieve the given V within T rounds, and (ii) the cost ratio, which is the suboptimality of an algorithm for solving problem i... c (qr - qn)/c" (D' - qo) - Note that the suboptimality does not count the penalty for failure since this would distort the average metrics. For K 1, we also measure the ratio of points collected / (under-collecting) and failure rate (over-collecting), we emphasize that our goal is to have low cost but with zero chance of failure.
We consider T = 1,3. 3. 5 rounds and V
/ D Although there natural trade-off between low cost ratio
The Value of Optimization over Estimation when K = 1. Figure 2 compares LOC versus the corresponding power law regression baseline when e = and P TO (P = 10 for VOC and P P= 100 for ImageNet). If curve is below the black line, then it failed to collect enough data to meet the target. LOC consistently remains above this black line for most settings. In contrast, even with up to T = 5 rounds, collecting data based only regression estimates leads to failure.
Table I aggregates the failure rates and cost ratios for each setting. To summarize, LOC fails less than 10% of instances for 12/18 settings, whereas regression fails over 30% for 15/18 /88 settings. In particular, regression nearly always under-collects data when given single T round. Here, LOC reduces the risk of under-collecting by 40% to 90% over the baseline. While this leads to marginal increase in costs, our cost ratios are consistently less than 0.5 for 12/ 18 settings, meaning that we spend at most 50% more than the true minimum cost.
We remark that previously, Mahmood et al. 2 observed that incorrect regression estimates necessi- tated real machine learning workflows to collect data over multiple rounds. Instead, with LOC, we can make significantly improved data collection decisions even with single round.
Robustness to Cost and Penalty Parameters (see Appendix F.2 for details). Figure evaluates the ratio of points collected for T = 5 when the cost and the penalty of the optimization problem are
7

---

Table 1: Average cost ratio c' (q7 qo)/c (D*-qul qu) and failure rate measured over range of V for each T and data set. We fix c 1 and P 10" (P 10 for VOC and P 10 for ImageNet). The best performing failure rate for each setting is bolded. The cost ratio is measured only for in- stances that achieve V*. LOC consistently reduces the aver- age failure rate, often down to 0%, while keeping the average Data set
T
Power Law Regression
LOC
Failure rate
Cost ratio
Failure rate
Cost ratio
100%
60%
0.19
CIFAR-I0
95%
0.00
32%
0.05
86%
0.00
29%
0.03
Class.
56%
0.12
4%
0.99
CIFAR-100
48%
0.10
3%
0.31
48%
0.10
2%
0.19
99%
0.00
37%
0.49
Imagenet
75%
0.01
5%
0.16
50%
0.01
2%
0.10
77%
0.03
12%
2.03
BDD100K
31%
0.00
0%
0.72
Seg.
23%
0.01
0%
0.35
95%
0.00
52%
0.16
nuSnnnes
71%
0.01
0%
0.09
cost ratio almost always below (i.e., spending at most 2 the optimal amount).
62%
0.00
0%
0.04
36%
1.24
25%
0.56
Det.
VOC
8%
0.88
0%
1.10
6%
0.86
0%
0.84
CIFAR-10 (T=5)
CIFAR-100 (T=5)
Imagenet (T=5)
1.3
of points di
Ratio of points q 10
1.0
1.4
c-1e
1.2
c-1e
1.0
c-16:02
c=1e02
1.
c=1o-02
0.9
c=1e-02
0.6
80
c-l--0
85
90
0.8
40
c-1e-00
50
60
70
0.8
45
c=1e+00
50
60
65
CIFAR-10 (T
CIFAR-100 (T=5)
Imagenet (T=5)
1.3
Ratio of points q 1D
Rabio of points q 10"
of points 4
P-ler 06
P-le+06
1.0
P=1e-07
1
Pwle+07
P=le+08
1.0
P-10108
P=lc107
0.
P-1e-00
P=Ie-08
P=le+09
0.8 40
P=le+09
50
60
70
0.8
45
P=1e-10
50
60
65
vOC (T=5)
BDD100K (T=5)
nuScenes (T=5)
8-
Ratio of points o 1D
Ratio of peints q 'D
points
2
Power Law
Power Law
Power Law
03
c=1e:03
2:
c=1e-02
c=1e-02
c-le-01
c=16-01
c-1e-01
c-le+00
C=1e100
c=les:0
66
63
70
72
74
20
22
24
26
vOC (T=5)
V
BDD100K 5T=5)
nuScenes (T=5)
4000
points 100
Rabio of points F/D
2000
1000
P=le+06
Ie+
Pale407
P=10107
P-Ic+00 07
P-1c105
1000
P=Ic109
P=le+09
66
68
70
72
74
22
24
26
28
30
Figure 3: Mean + standard deviation of 5 seeds of the ratio of data collected o D for different V and fixed
5. Rows / & 3: We sweep the cost parameter from 0.001 to 1 and fix P = 107. Rows 2 & 4: We sweep the penalty parameter from 10 to 10 and fix c = 1. The dashed black line corresponds to collecting exactly the minimum data requirement. See Appendix all T.

---

CITAR-100 12 Typest [T
CITAR-100 12 Typest (T=5)
CIFAR-1O0 12 Typesi (T
CITAR-100 (2 Typeet (T=))
400
16
Col.
Colluu
Col Coxc
10
13
33
B00100K (Semi supervised
B0D100K (Semi superviadd IT=S1
BOOLOOK (Sen supervisdd
B00LOCK (Semi oupervsedi
Case tadts
Coa tats
100
Con.
20-1
20-1
-2-
33
33
Naw 5a
Sllm Sat.
Palur Talk
Falure Lake
Figure 4: Mean standard deviation over seeds of the cost ratio c q //)
qo
and failure rate for different V*, after removing 99-th percentile outliers. The columns correspond to scenarios where the first set c² costs increasingly more than the second c². See Appendix for all T.
varied. Our algorithm is robust to variations these parameters, as LOC retains the same shape and scale for almost every parameter setting and data set. Further, LOC consistently remains above the horizontal 1 line, showing that even after varying c and P. we do not fail as frequently as the baseline. Finally, validating Theorem | the penalty parameter P provides natural control over the amount of data collected. As we increase P, the ratio of data collected increases consistently.
The Value of Optimization over Estimation when K 2 (Appendix F.3 Figure 4 compares LOC versus regression at T with different costs, showing that we maintain similar cost ratio to the regression alternative, but with lower failure rates. Table 2 aggregates failure rates and cost ratios for all settings, showing I.OC consistently achieves lower failure rates for nearly all settings of T. When 7 = 5, LOC also achieves lower cost ratios versus regression on CIFAR-100 100, meaning that with multiple rounds of collection, we can ensure meeting performance requirements while paying nearly the optimal amount of data. However, solving the optimization problem is generally more difficult as K increases, and we sometimes over-collect data by large margins. In practice, these outliers can be identified from common sense (e.g.. if policy suggests collecting more data than we can reasonably afford, then we would not use the policy suggestion). Consequently, we report these results after removing the 99-th percentile outliers with respect total cost for both methods. Nonetheless, this challenge remains when T = 1, particularly for CIFAR-100.
Discussion
We develop rigorous framework for optimizing data collection workflows in machine learning applications, by introducing an optimal data collection problem that captures the uncertainty in estimating data requirements. We generalize this problem to more realistic settings where multiple data sources incur different collection costs. We validate our solution algorithm. LOC, on six data sets covering classification, segmentation, and detection tasks to show that consistently meet pre-determined performance metries regardless of costs and time horizons.
Our approach relies on estimating the CDF and PDF of the minimum data requirement, which is a challenging problem, especially with multiple data sources. Nonetheless, LOC can be deployed on top of future advances in estimating neural scaling laws. Further, we allow practitioners to input problem-specific costs and penalties, but these quantities may not always be readily available. We provide some theoretical insight into parameter selection and show that LOC is robust to these parameters. Finally, our empirical analysis focuses on computer vision, but we expect our approach to be viable in other domains governed by scaling laws.

---

Data set
Cost
Power Law Regression
LOC
Failure rate
Cost ratio
Failure rate
Cost ratio
(0.01, 0.0005)
0.89
41.80
Table 2: Average cost ratio
(0.01, 0.001)
58%
1.19
46%
9.85
c c'(q7 (077-////' qoo -0) and failure rate over different V for each I and c. after re- moving 99-th percentile out- liers. fix P 10 for CIFAR-100 and P = 10 for BDDIOOK. The best perform- ing failure rate for each set- ting is bolded. The cost ra- tio is measured over instances that achieve V*. LOC con- sistently reduces the average failure rate, and for T > 1, preserves the ratio. Fur- ther, LOC is more robust to uneven costs than regression.
CIFAR- CIFAR-LO0 100 (2 Types)
(0.01, 0.002)
56%
1.55
54%
6.98
(0.01, 0.005)
54%
1.65
33%
(0.01, 0.0005)
3.47
30%
10.01,0.000
45%
1.22
43%
1.31
(0.01, 0.002)
45%
1.47
44%
1.21
(.001,0.005)
10.01,0.0005
38%
3.31
24%
0.001)
35%
1.22
24%
0.79
(0.01, 0.002)
37%
1.33
38%
0.90
0.005)
1,30
(1.00005)
80%
0.11
44%
7.02
BDD100K (Semi-superviee)
01,0,01)
79%
0.15
30%
13.47
(1.0.05)
72%
0.19
49%
1.02
(1,0.1)
65%
0.40
(1,0.005)
23%
0,18
7%
1.20
0,,0)
21%
0,15
7%
2.57
(1.0.05)
26%
0.18
28%
0.50
(1,0.1)
0,21
30%
0.15
(1,0.005)
16%
0.22
2%
1.91
(1,0.01)
21%
0.15
2%
0.86
(L. 0.05)
16%
0.17
9%
(1.0.))
16%
0.20
7%
0.32
Improving data collection practices yields potentially positive and negative societal impacts. LOC reduces the collection of extraneous data, which can, in turn, reduce the environmental costs of training models. On the other hand, equitable data collection should also be considered in real-world data collection practices that involve humans. We envision potential future work to incorporate privacy and fairness constraints prevent over- or sampligg of protected groups. Finally, our method is guided by a score function on held-out validation set. Biases in this set may be exacerbated when optimizing data collection to meet target performance.
There is a folklore observation that over 80% of industry machine learning projects fail to reach production, often due to insufficient, noisy, inappropriate data [46] 47). Our experiments verify this by showing that naïvely estimating data requirements will often yield failures to meet target performances. We believe that robust data collection policies obtained via LOC can reduce failures while further guiding practitioners on how to manage both costs and time.
Acknowledgments and Disclosure of Funding
We thank Jonathan Lorraine, Daiqing Li, Jonah Philion, David Acuna, and Zhiding Yu, as well as the anonymous reviewers and meta-reviewers for valuable feedback earlier versions of this paper.
References
[1] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmentation datasets with polygon- polygon-rnn+++. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), June 2018.
[2] Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M. Alvarez, Zhiding Yu, Sanja Fidler, and Marc T. Law. How much more data do we need? estimating requirements for downstream tasks. In 2022 JEEE Conference Computer Vision and Pattern Recognition. leee, 2022.
[3] Lewis J Frey and Douglas H Fisher. Modeling decision tree performance with the power law. In Seventh International Workshop Artificial Intelligence and Statistics. PMLR, 1999.
[4] Baohua Gu, Feifang Hu, and Huan Liu. Modelling classification performance for large data sets. In International Conference on Web-Age Information Management, pages 317-328. Springer, 2001.
[5] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
10

---

[6] Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference Learning Reprexentations, 2020.
[7] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeftrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 2020.
[8] Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shiapentokh-Rothman. Learning curves for analysis of deep networks. In International Conference on Machine Learning, pages 4287-428844 4296. PMLR, 2021.
191 Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lec, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.005 06701, 2021.
[10] Devansh Bisla, Apoorva Nandini Saridena, and Anna Choromanska. A theoretical-empirical approach to estimating sample complexity of dans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3270-3280, 2021.
[11] Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. scaling law for synthetic-to-red transfer: How much is your pre-training effective? arXiv preprint arXiv:2 1018, 2021.
[12] David Acuna, Guojun Zhang, Mare T Law, and Sanja Fidler. f-domain adversarial learning: Theory and algorithms. In International Conference on Machine Leurning, pages 66-75. PMLR, 2021.
[13] Aayush Prakash, Shoubhik Debmath, Can-Francois Lafleche, Eric Cameracci, Gavriel State, Stan Birchfield, and Mare T. Law. sulf-supe real-to-sim scene generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16044, October 2021.
[14] David Acuna, Marc T Law, Guojun Zhang, and Sanja Fidler. Domain adversarial training: A game perspective. In International Conference on Learning Representations, 2022.
[15] S Jones, S Carley, and M Harrison. An introduction to power and sample size estimation. Emergency Medicine Journal: EMJ, 20(5))453, 2003.
[16] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE International Conference on Computer Vision, pages 843-852, 2017.
[17] Rosa L Figueroa. Qing Zeng-Treitled Sasikiran Kandula, and Long H Ngo. Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making, 12(1):1-10, 2012.
[18] Tom Viering and Marco Loog. The shape of learning curves: a review. arXiv preprint arXiv:2103.10945 2021.
[19] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lueas Beyer. Sealing vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[20] Bur Settles. Active learning literature survey. 2009.
[21] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018
[22] Donggeun Yoo and In So Kweon. Learning loss for active learning. Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition, pages 93-102, 2019.
[23] Samarth Sinha, Sayna Ebrahimi. and Trevor Darrell. Variational adversarial active learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
[24] Randd Mahmood, Sanja Fidler, and Mare T. Law, Low-budget active learning via wasserstein distance; An
integer programming approach. In International Conference on Learning Representations, 2022.
[25] Yiding Jiang, Pierre Foret, Scott Yak, Daniel M Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, and Behnam Neyshabur. Neurips 2020 competition: Predicting generalization in deep learning. arXiv preprint or wxXiv:202..09975 2020.
[26] Yiding Jiang, Parth Natekar, Manik Sharma, Sumukh K Aithal, Dhruva Kashyap, Natarajan Subramanyan amm. Carlos Lassance, Daniel M Roy, Gintare Karolina Dziugaite, Suriya Gunasekar, et al. Methods and analysis of the first competition in predicting generalization of deep lcarning. In NeurlPS 2020 Competition and Demonstration Track, pages 170-190. PMLR, 2021.
[27] Kirstine Smith. On the standard deviations of adjusted and interpolated values of an observed polynomial function and its constants and the guidance they give towards a proper choice of the distribution of observations. Biometrika, 12(1/2):1-85, 1918.
[28] David Cohn. Neural network exploration using optimal experiment design. Advances in neural information processing systems, 6, 1993.
11

---

[29] Ashley F Emery and Aleksey V Nenarokomov. Optimal experiment design. Measurement Science and Technology, 96))864, 1998.
[30] Dimitris Bertsimas, Mac Johnson, and Nathan Kallus. The power of optimization over randomization in designing experiments involving small samples. Operations Research, 63(4):868 876, 2015.
[31]
Pedro Cameiro, Sokbae Lee, and Daniel Wilhelm. Optimal data collection for randomized control trials. The Econometrics Journal, 23(1):1-31, 2020.
[32] Hao Zhang. Dynamic learning and decision making via basis weight vectors. Operations Research, 2022.
[33] Guo Haixiang, Li Yijing. Jennifer Shang. Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from class-imbalanced data: Review of methods and applications. Expert systems with applications, 73:220-220 239,
2017.
[34] Jesper E Van Engelen and Holger H Hoos. A survey on sumi-supereised learning. Machine Learning, 109 (2):373-440, 2020.
[35] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning. 79(1):151-175, 2010.
[36] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[37] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
Icce, 2009.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[39] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 170.00:700005887, 2017.
[40] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages (636-2645, 2020.
[41] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Veniee Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11621- -
1631, 2020.
[42] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.
[43] Mark Everingham, Luc Van Gool. Christopher KI Williams, John Winn, and Andrew Zisser- man. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal http://aallaggs////ooo//////////
[44] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://ww..pscal. 2twgg//caallleee////////22000000
[45] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scout Reed, Cheng-Yane Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision, pages 21 37, Springer, 2016.
[46] Rob van der Meulen and Thomas McCall. Gartner says nearly half of cios are planning to deploy artificial intelligence. Feb 2018. URL U htttps://www. gartner. com / wwwtwesrwoooo////////eeeeessooee peress-releases 2018 02- 13 gartner- says-nearly hww-hhhlfffffff- cio--ccos-ae are- planennganinn----- toooigg-to--pppe deploy artificial inteelllitteeeeeenee.
[47] Why do duction?.
87% of data science
projects never
make it into pro-
Jul 2019.
URL https://venturebeatt cttettccomm21110 com/2099/0/1999/
why-do- 87 of data- science- pcience-projects never- make- it- into protooorroductoonooo
[48] Jorge Moré. The lavenner algorithm: implementation and theory. In Numerical analysis, pages 105-116. Springer, 1978.
[49] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapcau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman. Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 2721, 2020. doi: 10. 10.1038/s41592-00 019-066--2
12

---

[50] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[51] Dimitri Bertsekas. Dynamic programming and optimal control: Volume 1, volume 1. Athena scientific, 2012.
[52] David Easley and Nicholas M Kiefer. Controlling a stochastic process with unknown parameters. Econo- metrica: Journal of the Econometric Society, pages 1045-1064, 1988.
[53] Richard Smallwood and Edward Sondik. The optimal control of partially observable markov processes over finite horizon. Operations research, 21(5):1071-1088 1973.
[54] Eric Zhao, Angi Liu, Animashree Anandkumar, and Yisong Yue. Active learning under label shift. In International Conference Artificial Intelligence and Statistics, pages 3412 3420. PMLR, 2021.
[55] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019.
[56] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020.
[57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. International Conference on Learning Representations, 2015.
[58] Ismail Elezi, Zhiding Yu, Anima Anandkumar, Laura Leal-Taixe, and Jose M Alvarez. Not all labels are equal: Rationalizing the labeling costs for training object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Conclusion.
(C) Did you discuss any potential negative societal impacts of your work? [Yes] See the Conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See Assump- tion and the theorem statements.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix C
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URLY? [No] The code is proprictary. The data is publicly available. See Algorithm [ and Appendix F for implementation details.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D and F for details.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] All experiments were over five seeds. See the for standard deviation ranges.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix H.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all data sets and model architectures in Appendix [
(b) Did you mention the license of the assets? [N/A] All data sets and models are publicly available.
(c) Did you include any assets either in the supplemental material URL? [No]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "UEfc2acX6V", "reviewer": "Reviewer_MHTm", "summary": "This paper provides an extension to a recently proposed problem: how many data to collect to meet the requirement in a specific task when using a specfic model. The authors add one more key step to the orginal algortihm, making its performance better according to the results reported in the paper, and also capable of dealing with multi-variate tasks. The main contributions are:\n1. Propose a new algorithm to solve the data collection problem based on an existed approach;\n2. Extend the problem to a multi-variate setting;\n3. Theoretically and practically prove the method has better performance compared to the previous algorithm;\n", "questions": "See the previous section.", "limitations": "The authors has discussed the limitation of this paper, and it is well-organized. That is, the cost c and the penalty P need to be given by the user, which is not always available. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths are:\n1. The problem itself is interesting and of growing interest;\n2. The proposed method is technically sound;\n3. According to the experiments results, the proposed method outperformes its baseline method;\n4. Mostly the paper is well-organized.\n\nThe main weaknesses are:\n1. The 'Related work' section seems overlaps heavily with the baseline paper -- not sure whether that's legal for Neurips, checking with the ACs;\n2. The motivation for the new algorithm doesn't look strong. The key step of the proposed algorithm is to add a 'bootstrap sampling' after getting the support set V, to further estimate the distribution of the real optimal D by training separate regression models. This step can be time-consuming, considering one has to get a certain amount of data points to meaningfully simulate a distribution. Also, the data used for multiple regressions are different subsets of the overall support set V, which makes it unreasonable since for regression models, one generally needs more data to train a more precise model, spliting the dataset and train serveral sub-regression models and use them all for predictions does not make sense;\n3. The experiments are not convincing enough. As reported in the original paper, they tried four types of regression and the results shown indicate different tasks correspond to different types of regression function in order to get the best performance. However, in this paper the authors only discussed one of those four functions. Also for the choice of K. only K=1, 2 are discussed, which is too few;\n4. Minor issue. For line 139, the third line of the equation, did you mean qT rather than qs for the second term?\n5. This may not be a weakness, but why would one wants to use these kind of framework at all if he/she already has a support set and validation set to roughly predict the number of samples needed by training a regression model on it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PbUYBCx-NpO", "reviewer": "Reviewer_kQGA", "summary": "Labeling data is an important task in machine learning. However, it can also be expensive, making it unproductive to \"over-label\" data. Conversely, collecting too little data can lead to poor model accuracy. This paper presents an approach to determine the \"right\" amount of data to label for a particular model and task in order to achieve a provided target accuracy. The proposed framework can extend to a multi-dimensional setup, where some particular data items might be more expensive to label.", "questions": "- How accurately can $F(q)$ be estimated in practice? Also, does this procedure make sense if multiple versions of the model are being trained on various subsets of the full training dataset (including the full dataset itself)? Why do you need multiple rounds of data collection given the CDF $F(q)$?\n- What does this mean: \"In order to model the distribution of $D*$ , we can take $B$ bootstrap resamples of $R$\"?\n- \"To avoid repeatedly sampling data, re-training a model, and evaluating the score, each 241 simulation uses a piecewise-linear approximation of a ‘ground truth’ learning curve that returns 242 model performance as a function of data set size.\" --> is the model trained at the end of this simulation with all of the collected data $q_0 + q_1 + … + q_T$ to check if the final accuracy of the model is greater than $V*$?\n- Figure 2: is $D*$ the same across different values of $T$ for a given $V*$ value and task?\n- What do the cost ratios with respect to labeling the entire dataset look like? I ask because if it's not substantially more expensive to just label the entire dataset every time, then going through this entire process is perhaps not necessary? In general, the cost ratio metric is a bit hard to understand.\n- Why are failure rates so different across tasks for a given $T$ value (e.g., $T=1$)?\n- \"after removing 99-th percentile outliers\" --> what are the different trials here? Different $V*$ values? Something more than that?", "limitations": "The authors did not discuss technical limitations of their work. Some discussion of problem settings where the approach doesn't work would be useful. The authors do discuss both the positive and negative societal implications of their work, which is helpful.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- The paper addresses an important problem that I think is understudied.\n- The paper is well written and clear.\n- The approach is clean, and seems to work well compared to relevant baselines like directly using a power law distribution.\n\n#### Weaknesses\n- I would have liked to see more exposition on the takeaways from the results. For example, why do the cost ratios not monotonically decrease as $T$ increases? In general, the main metrics presented are a bit confusing to understand.\n- I would also have liked to see how effective estimates of the cumulative density function $F(q)$ are in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2fDMNRIBmEv", "reviewer": "Reviewer_buMu", "summary": "This paper proposes a framework (LOC) for optimizing data collection workflows in machine learning applications with capturing the uncertainty in estimating data requirements. Besides, the paper further extends the proposed framework to multiple data resources cases.  The results show that LOC greatly reduces the chances of failing to meet the target performance.", "questions": "1. Report the baselines as stated in Weakness 1.\n\n2. In your contribution statements (line 50), \"We propose the optimal data collection problem in machine learning\",   what is the difference between \"the optimal data collection problem\" and the data collection problem in Mahmood et al.[2]. It seems that both of your problems are to find the minimum datasets required to meet the target performance V^{*}. \n\n3. How to effectively choose an optimal cost c. In Table 2, it seems that the cost ratio is very sensitive to the cost c. e.g. When T=5 on CIFAR100 and c is changing from (0.01,0.0005) to (0.01, 0.005), the cost ratio decreases to 0.82 from 5.19. ", "limitations": "The authors have discussed about the limitation and social impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper is well written and easy to follow and the motivation is clear. \n\n2. The idea is novel and the proposed LOC can be applied to multiple data resources case.\n\nWeakness\n1. Important baseline is missing. Mahmood et al.[2] shows that the correction factor can boost the performance of power-law regression. It is necessary to report the results of power-law regression with a correction factor. Besides, the conventional estimation approach with other regression functions performs better than Power-law on some datasets (Mahmood et al.[2]), e.g., Algebraic Root consistently performs better than Power Law. It would be better to add the results of the convention estimation approach with Algebraic Root, Arctan. \n\n2. The proposed LOC tends to overestimate the data requirements in some cases. e.g. In Figure 2., LOC has a much higher Ratio of  q_{T}^{*}/D^{*} on CIFAR-100 and BDD100K.  \n\n3. Some descriptions might be inaccurate/mistakes. e.g. \n\n(1) In the caption of Figure 2., \"LOC consistently remains slightly above the black line\" might not be accurate since there exists cases where the ratio of q_{T}^{*}/D^{*} is below 1 (in high V^{*}) for LOC.   \n\n(2) In line 149, \"problem (6) can be optimized via gradient descent\" ---> \"problem (4) can be....\"?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept