Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an ‘active labeling’ (inspired from active learning) based approach to automatically label parts of a dataset given some error bounds by jointly optimizing for human labeling and machine labeling costs (eg. training costs) together. It formulates the problem as a minimum-cost labeling problem in an optimization framework that minimizes total cost by jointly selecting which samples to human label and which to machine label. A major assumption of this work is that it considers the truncated power-law distribution to mimic how model error behaves with training set size. 
strength_and_weaknesses: Strength: 
- The paper presents approach that allows control over accuracy and cost with the cost minimization objective to launch data labeling task for developing a high accuracy model training.
- The approach scales well with complexity of the data and also suggests (with a ‘exploration tax’) if it’s feasible to automatically label the data or not (given the size and complexity)
- The approach can use any active learning metric for selecting samples for human-labeling. Also they found empirically that some popular metrics (like coreset) don’t work as well as others with this objective. 
- The approach is extensible to allow multiple model architectures 

Weakness:
- It is not clear if the approach would work with a dataset containing imbalanced distributions and/or open classes to decide which ones to human-level vs machine-label. Also not clear if the power-law distribution would follow in the situation. 
- Fig. 4 needs more explanation - the role of batch size for model errors is low which is unintuitive. 
- The human-labeling is assumed to be oracle in this work - how realistic is this assumption - if the model can accommodate imperfect oracles in human labels, the method would become more realistic. 
- Fig. 6 needs more explanation 
- It is not clear how does ‘fast search’ work in the method. 

clarity,_quality,_novelty_and_reproducibility: The paper presents very interesting approach but there is no comparison with other existing cost sensitive active labeling methods (expect with active learning method).  A comparison to other weak labeling methods with active learning would have made a good comparative study in this work. 

summary_of_the_review: This paper presents some practical ideas for cost reduction for data labeling which is growing need today. The work presents a solid extensible framework where one can set various thresholds for performing the labeling tasks. The methods have been testing on multiple datasets and understanding the role of various parameters have been explored and discussed well in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors presents a novel active learning setup using a "hybrid human-machine labeling" framework to reduce the cost of labeling/annotating large datasets. They compared their approach on a number of public datasets and claims to provide a 6x overall reduction in cost to entire human labeling of the dataset.
strength_and_weaknesses: Some of the key strengths of the paper are as follows:

- The authors presented a general solution to a very important problem. With the impressive performance of AI/DL models for various tasks, the proposed method could be very important in operationalizing AI for real-world problems. Further, the MCAL Algorithm is not dependent on the underlying DL architecture / task and thus could be applied as a meta learner over multiple problems
- The authors have presented evaluations across multiple datasets/publicly available benchmarks to support the claim about the effectiveness of the solution
- Further, the detailed analysis about the reduction in labeling cost from multiple view points provides better insights for the method to be operationalized

The paper could improve upon the following aspects

- The presentation of the paper can be improved upon. The two components of MCAL algorithm is hard to follow. Symbols are sometime introduced before they are properly defined (e.g. delta in section 3.1, 3rd line from end of the section; first introduced in 3.2) and intuitions behind certain selections not presented well
- While the authors provide interesting insights about the drivers of the reduced cost, the discussion is lacking depth when considering the problem in a holistic manner. For example, in addition to labeling cost, other metrics of interest can include turnover time which can be affected as part of this multi-loop process
- In a similar manner, more details/analysis on which samples are selected during the AL phase would be useful. The authors presented a comparative analysis of the sample selection candidates - however, the intuition and effect of the selected samples may need further analysis. This could also be useful for downstream tasks in identifying "good enough" solutions under budget constraints
clarity,_quality,_novelty_and_reproducibility: The proposed solution is simple but intuitive - including the model training objective under the AL task can be intuitively thought of being more "intelligent" about the selected samples while adhering to the realities of increasingly costly DL models. While the results are impressive, the clarity of presentation could be improved upon. This will also help with reproducing the results across domains.


summary_of_the_review: Overall, the paper discusses an intuitive and simple solution to gather ground truths for large datasets and reduce the overall labeling costs for the process. While there are avenues for improving the paper, the results are impressive and general enough to have a significant impact.

Apart from the previously highlighted strengths and weakness, the authors may also want to address the following:

- Section 3.1, the intuition for the upper-truncated power-law could be motivated better. For example, a plot showing the effects of various values of the parameters from equation (3) could be useful to understand the desired shaping of the rewards
- MCAL Algorithm -provide better details about the fitting procedure. Discuss about the cost for the optimization function
- Provide intuitions behind the sample selection - e.g. how does the ranking of samples selected change with reduced labeling cost etc. Investigate possible issues with fairness / group fairness in selection of samples
- In recent years, a number of papers have identified labeling/data issues in the original benchmarks. The authors may want to compare if their proposed solution is able to circumvent those issues
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a system for training of machine learning classification models. The main focus
of the paper is to reduce the overal cost of training. The total cost considered in the paper is a
combination of the cost due to human labeling and the cost due to computation (e.g. renting a GPU on
Amazon Cloud).

In order to reduce the training cost the paper proposes to rely on combination of active learning and
automatic labeling.

The training in the proposed MCAL approach proceeds iteratively. On each iteration the approach
selects which examples should be labeled by human annotators, and which examples are assigned labels
automatically using a classifier trained on previously labeled data (using automatic + human labels).
strength_and_weaknesses: I generally have a negative impression about the paper. I can see that the approach is trying to
address a valid practical problem. The overall approach sounds very much like a common sense, i.e.
let's label some data points with already pre-trained model, and let human's label "interesting"
examples. The paper relies on standard metrics to assess what to give humans for labeling and how to
choose examples for automatic labels (i.e. examples where classifier is confident about the label).

The main technical contribution of the paper appears to be the use of the power law to predict how
many examples to label automatically (Sec. 3.1) and the overall system (Sec. 4).

After going through the paper several times, what I am left with is the following: The paper
presents an idea that sounds like a common sense, which is followed by hard to comprehend notation
and cost model (Sec. 2,3), followed by description of standard mechanisms of sample selection (Sec.
3.3), followed by hard to understand Sec. 4 that describes the algorithm.

The text does not communicate the ideas well unfortunately, and at least in my case it leaves the
reader unexcited about the approach. Perhaps using clear real-world examples would help here.

For example, I don't understand why power law in Sec. 4 even applies to the approach presented in
the paper. I can see how such power law works for increasingly larger training sets labeled by
humans. However, why does this law apply to automatically labeled examples, where adding larger
number of examples would result in adding more of the incorrectly labeled examples.

I addition I am not taking it for granted that "auto-labeling" generally works. Why is automatically
labeling examples where classifier is already confident going to help with classifying examples
where classifier is not confident? I understand that it might be the case, but I disagree that this
can be taken for granted.

Prior to reading this paper I believed that the main questions for the type of approaches proposed
in the paper is how to choose what to label manually, and how to choose what to label automatically.
If we knew how to do that well we would be in good shape to construct such systems. The paper does
not contribute to addressing these questions, and the paper didn't convince me that it makes other
valuable contributions.

I found some statements in the paper which I would like to have clarified:
1. "it chooses to label CIFAR-100 mostly using humans",
I am very curious why this is the case, and why similar outcome does not happen for ImageNet.

2. "MCAL’s goal is to select samples to train a classifier that can machine label the largest number
of samples possible."
That is certainly a good goal, but I do not understand how the approach achieves that. All I see in
the text is some application of the power law, whereas the problem above appears to be too complex to be
addressed by a power law.

3. "When MCAL uses margin or least confidence for L(.), the samples selected have high accuracy (close to 100%, Fig. 5)
. The samples selected by a core-set based algorithm such as k-centers is poorly correlated
with accuracy (Fig. 5) and margin (Fig. 6)."
I din't quite understand what is meant by these sentences. Why is accuracy relevant here if we are
selecting examples to be labeled and to be used for training of the next-iteration classifiers?
Also, if we select examples that have high confidence already why would such examples improve the
classifier?

Finally, I also doubt that the topic of the paper is within the scope of ICLR. For me ICLR stands
for interesting new representations (i.e. neural network models), or perhaps interesting techniques
for training neural networks. This paper sounds like a system paper that would perhaps be a good fit
for WACV (Winter Conference on Applications of Computer Vision) or some other more applied
conference (e.g. International Conference on Computer Vision Systems).

Positive:
The overall observation that in addition to human labeling cost there is also a computational cost is
valuable.

clarity,_quality,_novelty_and_reproducibility: As I mentioned above I didn't find the paper clearly written.
summary_of_the_review: I refer to my text in the "Strength And Weaknesses" section. I would like to avoid repeating myself here.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: The paper studies the problem of active labeling of image datasets. The authors propose a minimum cost optimization framework that jointly models the cost of human labeling and machine labeling. The framework is evaluated on a number of image classification datasets.


*Strengths*: (1) Active labeling is an important topic for current ML systems, given that they are trained on massive amounts of supervised and self-supervised data. (2) The framework seems agnostic to the underlying model and the labeling task. This makes it more broadly useful beyond the settings used in the paper. (3) The paper has good empirical evaluations, and includes findings about how current metrics may not work well for active labeling. 

*Weaknesses*: (1) The presentation in this work is quite unclear. This point was raised by all the reviewers as well. (2) The empirical evaluations in this paper are limited: the unlabeled data is "close" world, and not heavy tailed. This makes the overall applicability of the paper weaker.
justification_for_why_not_higher_score: Presentation lacks clarity and the experimental settings are not realistic to judge the immediate impact of this work.
justification_for_why_not_lower_score: The problem setting studied in this paper is important, and the solution is technically correct and validated by experiments.
Decision: Accept: poster