Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper provides an extension to a recently proposed problem: how many data to collect to meet the requirement in a specific task when using a specfic model. The authors add one more key step to the orginal algortihm, making its performance better according to the results reported in the paper, and also capable of dealing with multi-variate tasks. The main contributions are:
1. Propose a new algorithm to solve the data collection problem based on an existed approach;
2. Extend the problem to a multi-variate setting;
3. Theoretically and practically prove the method has better performance compared to the previous algorithm;

strengths_and_weaknesses: The main strengths are:
1. The problem itself is interesting and of growing interest;
2. The proposed method is technically sound;
3. According to the experiments results, the proposed method outperformes its baseline method;
4. Mostly the paper is well-organized.

The main weaknesses are:
1. The 'Related work' section seems overlaps heavily with the baseline paper -- not sure whether that's legal for Neurips, checking with the ACs;
2. The motivation for the new algorithm doesn't look strong. The key step of the proposed algorithm is to add a 'bootstrap sampling' after getting the support set V, to further estimate the distribution of the real optimal D by training separate regression models. This step can be time-consuming, considering one has to get a certain amount of data points to meaningfully simulate a distribution. Also, the data used for multiple regressions are different subsets of the overall support set V, which makes it unreasonable since for regression models, one generally needs more data to train a more precise model, spliting the dataset and train serveral sub-regression models and use them all for predictions does not make sense;
3. The experiments are not convincing enough. As reported in the original paper, they tried four types of regression and the results shown indicate different tasks correspond to different types of regression function in order to get the best performance. However, in this paper the authors only discussed one of those four functions. Also for the choice of K. only K=1, 2 are discussed, which is too few;
4. Minor issue. For line 139, the third line of the equation, did you mean qT rather than qs for the second term?
5. This may not be a weakness, but why would one wants to use these kind of framework at all if he/she already has a support set and validation set to roughly predict the number of samples needed by training a regression model on it?
questions: See the previous section.
limitations: The authors has discussed the limitation of this paper, and it is well-organized. That is, the cost c and the penalty P need to be given by the user, which is not always available.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Labeling data is an important task in machine learning. However, it can also be expensive, making it unproductive to "over-label" data. Conversely, collecting too little data can lead to poor model accuracy. This paper presents an approach to determine the "right" amount of data to label for a particular model and task in order to achieve a provided target accuracy. The proposed framework can extend to a multi-dimensional setup, where some particular data items might be more expensive to label.
strengths_and_weaknesses: #### Strengths
- The paper addresses an important problem that I think is understudied.
- The paper is well written and clear.
- The approach is clean, and seems to work well compared to relevant baselines like directly using a power law distribution.

#### Weaknesses
- I would have liked to see more exposition on the takeaways from the results. For example, why do the cost ratios not monotonically decrease as $T$ increases? In general, the main metrics presented are a bit confusing to understand.
- I would also have liked to see how effective estimates of the cumulative density function $F(q)$ are in practice.
questions: - How accurately can $F(q)$ be estimated in practice? Also, does this procedure make sense if multiple versions of the model are being trained on various subsets of the full training dataset (including the full dataset itself)? Why do you need multiple rounds of data collection given the CDF $F(q)$?
- What does this mean: "In order to model the distribution of $D*$ , we can take $B$ bootstrap resamples of $R$"?
- "To avoid repeatedly sampling data, re-training a model, and evaluating the score, each 241 simulation uses a piecewise-linear approximation of a ‘ground truth’ learning curve that returns 242 model performance as a function of data set size." --> is the model trained at the end of this simulation with all of the collected data $q_0 + q_1 + … + q_T$ to check if the final accuracy of the model is greater than $V*$?
- Figure 2: is $D*$ the same across different values of $T$ for a given $V*$ value and task?
- What do the cost ratios with respect to labeling the entire dataset look like? I ask because if it's not substantially more expensive to just label the entire dataset every time, then going through this entire process is perhaps not necessary? In general, the cost ratio metric is a bit hard to understand.
- Why are failure rates so different across tasks for a given $T$ value (e.g., $T=1$)?
- "after removing 99-th percentile outliers" --> what are the different trials here? Different $V*$ values? Something more than that?
limitations: The authors did not discuss technical limitations of their work. Some discussion of problem settings where the approach doesn't work would be useful. The authors do discuss both the positive and negative societal implications of their work, which is helpful.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a framework (LOC) for optimizing data collection workflows in machine learning applications with capturing the uncertainty in estimating data requirements. Besides, the paper further extends the proposed framework to multiple data resources cases.  The results show that LOC greatly reduces the chances of failing to meet the target performance.
strengths_and_weaknesses: Strengths
1. This paper is well written and easy to follow and the motivation is clear. 

2. The idea is novel and the proposed LOC can be applied to multiple data resources case.

Weakness
1. Important baseline is missing. Mahmood et al.[2] shows that the correction factor can boost the performance of power-law regression. It is necessary to report the results of power-law regression with a correction factor. Besides, the conventional estimation approach with other regression functions performs better than Power-law on some datasets (Mahmood et al.[2]), e.g., Algebraic Root consistently performs better than Power Law. It would be better to add the results of the convention estimation approach with Algebraic Root, Arctan. 

2. The proposed LOC tends to overestimate the data requirements in some cases. e.g. In Figure 2., LOC has a much higher Ratio of  q_{T}^{*}/D^{*} on CIFAR-100 and BDD100K.  

3. Some descriptions might be inaccurate/mistakes. e.g. 

(1) In the caption of Figure 2., "LOC consistently remains slightly above the black line" might not be accurate since there exists cases where the ratio of q_{T}^{*}/D^{*} is below 1 (in high V^{*}) for LOC.   

(2) In line 149, "problem (6) can be optimized via gradient descent" ---> "problem (4) can be...."?
questions: 1. Report the baselines as stated in Weakness 1.

2. In your contribution statements (line 50), "We propose the optimal data collection problem in machine learning",   what is the difference between "the optimal data collection problem" and the data collection problem in Mahmood et al.[2]. It seems that both of your problems are to find the minimum datasets required to meet the target performance V^{*}. 

3. How to effectively choose an optimal cost c. In Table 2, it seems that the cost ratio is very sensitive to the cost c. e.g. When T=5 on CIFAR100 and c is changing from (0.01,0.0005) to (0.01, 0.005), the cost ratio decreases to 0.82 from 5.19. 
limitations: The authors have discussed about the limitation and social impacts.
review_title: null

Meta-review: metareview: The paper addresses a critical problem in the era of massive-data-set ML, which is how to estimate the size of data that needs to be collected to train a model of a given performance level.  This is not the first paper to propose the problem, but it does give improvements and extensions over previous work.

The reviewers reached clear consensus that the problem is important, that the paper is extremely clear, and that the methods appear to work well.  However, there was concern that the current paper was perhaps not a significant step forward over a previous recent paper "How Much More Data Do I Need? Estimating Requirements for Downstream Tasks".  In order to help resolve this question, I read both the current submission (and revision), but also went back to read the previous paper -- in addition, of course, to fully going through all reviews and author responses.  

In the end, I think that there is enough new material here to justify publication.  The use of the bootstrap seems to give useful improvement and is computationally tractable, and the extension to the multivariate / multi-source case is interesting and important.  But equally importantly, in my opinion, is the fact that the empirical work is so thoroughly done and adds additional empirical grounding to a nascent-but-critical area of the field.  For these reasons, I am discounting to some degree the arguments from the "borderline reject" review and am recommending acceptance.

Lastly, note that an earlier version of the paper included text from the related work that was disturbingly similar to that of a previously published paper.  The authors have since revised the paper and removed the similar / near overlapping text, which is appropriate, but even so should consider this a strong warning to avoid submitting overlapping text in the future.  
confidence: Certain
Decision: Accept