Machine Learning Algorithms for Data Labeling: An Empirical Evaluation
Authors: Teodor Anders Fredriksson
David Issa Mattos
Jan Bosch
Helena Holmström Olsson
Venue:  (2021)
Keywords: Data Labeling
Empirical Evaluation
Active Machine Learning
Semi-Supervised Learning

Abstract: The lack of labeled data is a major problem in both research and industrial settings since obtaining labels is often an expensive and time-consuming activity. In the past years, several machine learning algorithms were developed to assist and perform automated labeling in partially labeled datasets. While many of these algorithms are available in open-source packages, there is no research that investigates how these algorithms compare to each other in different types of datasets and with different percentages of available labels. To address this problem, this paper empirically evaluates and compares seven algorithms for automated labeling in terms of accuracy. We investigate how these algorithms perform in six different and well-known datasets with three different types of data, images, texts, and numerical values. We evaluate these algorithms under two different experimental conditions, with 10\% and 50\% labels of available labels in the dataset. Each algorithm, in each dataset for each experimental condition, is evaluated independently ten times with different random seeds. The results are analyzed and the algorithms are compared utilizing a Bayesian Bradley-Terry model. The results indicate that while the algorithms label spreading with K-nearest neighbors perform better in the aggregated results, the active learning algorithms query by instance QBC and query instance uncertainty sample perform better when there is only 10\% of labels available. These results can help machine learning practitioners in choosing optimal machine learning algorithms to label their data.

Under review as a conference paper at ICLR 2021
MACHINE LEARNING ALGORITHMS FOR DATA LA- BELING: AN EMPIRICAL EVALUATION
Anonymous authors Paper under double-blind review
ABSTRACT
The lack of labeled data is a major problem in both research and industrial set- tings since obtaining labels is often an expensive and timecconsuming activity. In the past years, several machine learning algorithms were developed to assist and perform automated labeling in partially labeled datasets. While many of these algorithms are available in open-source packages, there is no research that inves- tigates how these algorithms compare to each other in different types of datasets and with different percentages of available labels. To address this problem, this paper empirically evaluates and compares seven algorithms for automated labeling in terms of accuracy. We investigate how these algorithms perform in six differ- ent and nell-nnown datasets with three different types of data, images, texts, and numerical values. We evaluate these algorithms under two different experimental conditions, with 10% and 50% labels of available labels in the dataset. Each algo- rithm, in each dataset for each experimental condition, is evaluated independently ten times with different random seeds. The results are analyzed and the algorithms are compared utilizing Bayesian Bradley-Terry model. The results indicate that while the algorithms label spreading with K-nearest neighbors perform better in the aggregated results, the active learning algorithms query by instance QBC and query instance uncertainty sample perform better when there is only 10% of la- bels available. These results can help machine learning practitioners in choosing optimal machine learning algorithms label their data.
INTRODUCTION
Supervised learning is the most commonly used machine learning paradigms. There are problems with supervised learning and machine learning in general. The first problem is that machine learning requires huge amounts of data. Secondly, supervised learning needs labels in the data. In a case study performed with industry, several labeling issues were found (Anonymous) (020a).
A recent systematic literature review was conducted to see what type of machine learning algorithms exist make the labeling easier. recent systematic literature review investigated the of Semi- supervised learning and Active learning for automatic labeling of data (Anonymous) 2020b). From those results the authors concluded which active and semi-supervised learning algorithms were the most popular and which datatypes they can be used on. However, even if there has been work done on active and supeevipe learning, these learning paradigms are still very new for many companies and consequentially seldomly used.
Utilizing simulation study we evaluated seven semi-supervised and active learning algorithms on six datasets of different types, numerical, text and image data. Implementing Bayesian Bradley Terry model we ranked the algorithms according to accuracy and effort.
The contribution of this paper is to provide taxonomy of automatic labeling algorithms and an empirical evaluation of algorithms in the taxonomy evaluated across two dimensions: Performance, how accurate the algorithm is, and Effort, how much manual work has to be done from the data scientist.
The remainder of this paper is organized as follows. In the upcoming section provide the an overview about semi-supervised and active learning algorithms and how they work. In 3 will describe our study, how preformed the simulations, what datasets and source code we used,

---

Under review as a conference paper at ICLR 2021
and what kind of metries used to evaluate performance, effort and applicability. In section 4 we provide the results from the simulation study and finally, we will interpret the results and conclude the paper in section 5
2 BACKGROUND
2.1 ACTIVE LEARNING
Suppose a large unlabeled dataset to be used for training classification algorithm. Active Learn- ing (AL).poses query strategies on the data and selects points to be labeled according to measure of informativeness called a Query Strategy. After the instances has been labeled with the help of the oracle, the machine learning algorithm is trained with this newly labeled data. If the learner thinks that the accuracy of the algorithm is too low and that the accuracy can be improved, the learner will request new and or replace some of the old labels. The algorithm will then be re-trained and evaluated once again. This procedure will continue iterative until some other stopping criteria has been reached. As reference AL, the reader is recommended to look other sources such as Settles 2012). We shall now present the query strategies that we used in this text.
Uncertainty Sampling according to (Anonymous) 2020b) the most commonly used active learn- ing strategy. The idca of this approach is query the instances that we are the least certain about and then label these. Uncertainty sampling strategies are very commonly used and work especially well for probabilistic algorithms such as logistic regression according to (Lewis & Catlett 1994). (Lewis) & Catlett 1994) concluded that uncertainty sampling has the ability to outperform random sampling by evaluating and comparing it to a text classification dataset and (Joshi et al., 2009) concluded the same on image data by comparing accuracy scores on two uncertainty-samplii based methods and random sampling.
Moy-Committee((BCC means that we train a committee of classifiers and then query the instance which the committee disagrees. Add the newly labeled instance to the labeled training data and retrain the algorithm on the new training set and repeat this procedure. What is important here is the way we measure disagreement. Some way to measure disagreement is through entropy, 2012). QBC is relatively straightforward to implement and
vote-entropy and KL divergence applicable to any basic machine learning mode. (Seung al., 1992) and (Freund et al., 1997) were the first to formulate QBC. In SSeun et al. (1992) they use Monte Carlo simulation to show that QBC can outperform random sampling.
Random sampling is when the learner chooses to query the instances randomly and not according to any strategy. If learner does not choose his query strategy carefully with respect to his data and machine learning algorithm, then active learning might not outperform choosing your instances randomly.
2.2 SEMI-SUPERVISED LEARNING
Semi-supervised machine learning is class of machine learning algorithms that utilizes both labeled and unlabeled data. Semi-supervised algorithms are then trained on both the unlabeled and the labeled data and in some cases it even outperforms supervised classifiers. For more information on semi-supervised learning we refer the reader to (Zhu) 2005)..
According to (Anonymous 2020bb the second most popular cupervupe learning algorithms are the graph-based algorithms. The idea of these algorithms is to build a graph from the training data. These graphs contains both labeled and unlabeled instances. Let each pair (x,, and (x,, Vj represent each vertex and its corresponding label. Let the edge weight wij represent the weight of the edge between vertex and vertex j. The larger Wij becomes the more similar are the labels of both vertices. The question is then how to compute the weight Wiy. Two examples of graph-based methods are Label Propagation and Label Spreading (Zha al. 2009).
Label propagation was first introduced in (Zhu & Ghahramani
2002) and presented as follows.
Given labeled and unlabeled data, define the weight matrix Wij- The probabilistic transition matrix T is defined as the probability of jumping from vertex i to vertex
Ty = P(j +
Wij
Σ
k== Wkj
I+u
2

---

Under review as a conference paper at ICLR 2021
The matrix Y is called the label matrix and it's ith row represents the label probability distribution of vertex 2; The label propagation algorithm consists of the following steps
1. All nodes propagate for one step: Y t TY
2. Row normalize Y
3. Clamp the labeled data.
Repeat Step 1-2 until Y converges. (Zhu & Ghahramani 2002) evaluates the label propagation algorithm on both synthetic data and real-world classification data (Hull 1994). by comparing it's enror rates to that of kNN with k 1. The results show that label propagation can outperform KNN when the number for labeled instances is greater than 40. Label propagation algorithms have
used and evaluated in image annotation (Tang al. 2011 Chua et al. 2009) and text classification
(Pawar et al. 2016)
Label Spreading was first introduced in (Zhou et al. 2004). Given partially labeled dataset with different labels. Let F be the set of all matrices of size n x c with non-negative entries. Let F E F. Each entry Fiy in F depends on how we label 24. We have that
Y: arg max Fij 85c
Define matrix Y E F such that
Y1J = 1 if y/ =j
Y
Yij 0 otherwise
The label spreading algorithm is:
1. Define the affinity matrix
W wi if i W = 0
W
2. Define the matrix S = D where D is diagonal matrix Du: = Ex Wak
3. Iterate F(t + 1) = (tF + (1 a until convergence, E (0, 1).
4. Label xy as Vi = arg max Se Fi where F* is the limit of the sequence (F(t)).
Zhou et al. 2004) evaluates the label spreading algorithm on toy dataset, images in the form of handwritten digits and text classification and concludes that it outperforms baseline models ANN with & = 1, SVM with RBF kernel.
2.3
THE BRADLEY TERRY MODEL
The Bradley-Terry model (Brddeey& Terry 1952 Cattelan 2012)
one of the most commonly used models when it comes to analysis of paired comparison data between two objects i and j for i,j =-j=1 1, ..., n. The comparison can be by done several subjects possible paired comparisons is equal to ntnn 1)/2. Let ys (Js,1,2; ..., 1s.a-1,1) be the vector of outcomes of all paired comparisons, we will assume that we outcomes are independent.
.... and the total number of
.... n denote a latent "strength" of the algorithm being compared. If the paired comparison can have only two outcomes and ties are randomly resolved, the probability of beating j can be represented by:
Let μ4 E R, i = 1, 2,......
e
Pî beats j]
efft + esly
Reducing the expression to a logistic regression (Bradley & Terry 1952):
P(i over j) = loggit 114 (g)
By estimating the strength latent variable μ4, we can infer the probability one algorithm to beat the other and this information to rank the algorithms
3

---

Under review as a conference paper at ICLR 2021
3 RESEARCH METHOD
In this section we present the details about the datasets that we used for our simulations, the experi- mental conditions and the used algorithms.
The goal of this study is to show in detail how machine learning algorithms can be used to help with data labeling and to provide an in-depth comparison on how these different algorithms perform on different types of data. To achieve this we performed an empirical evaluation of seven different active learning and semi-supervised learning algorithms and evaluated them on six datasets under different conditions.
The main research questions that we use to evaluate the machine learning algorithms are the follow- ing.
RQI: How can we rank different active learning and semi-supervised learning algorithms terms of accuracy?
How do the rank of these algorithms with changes in the amount of manual labeling effort prior to applying these methods?
RQ2:
3.1 SIMULATIONS
As recognized in (Anonymous, 2020b) Co-training/multi-vinn learning are the most popular al- gorithms but based on the assumption than can watch an instance from multiple views. Graph-based algorithms are the second most common type of semi-supervised learning algorithm. Uncertainty sampling methods are very popular active learning query strategies followed by QBC.
Furthermore we have included two different graph-based algorithms Label Spreading and Label Propagation. Both methods are easy to implement using python an
Label Spreading using k-NN is implemented with wi KNN, k = 7, 0.2 Pyt b).
Label Spreading using RBF implemented with way = exp(-x) 1x,1--11222000 23 12,,) 20,
0.2
Label Propagation using h-NN is implemented with way = ANN, h: = 7 (Pyt
Label Propagation using RBF is implemented with wij exp expp(>/lzz x,, 20
Radnom Sampling, Uncertainty Sampling and QBC: Each dataset was randomly split into training and test set, unlabeled and labeled set. 80% of the data was allocated for training and 20% was allocated for testing. As stopping criterion choose to stop after 50 instances had been queried.
We choose six benchmarked datasets to be used our experiments. Two numerical datasets, two text datasets and two image datasets. Due to the size of some datasets and to limited time and computational resources required we had to reduce the number of images used our experiments. However, we made sure we used the same ration for the classes to get fair estimated.
Image data:
This dataset originally contains 60000 32x32 colored images that can be divided into ten classes, airplane, automobile, bird, car, deer, dog, frog, horse, ship and truck (cif).
Cifar-10:
Digits: This dataset contains 1797 samples of 8x8 images containing one digit each.
There ten classes that represent which digits is contained in each image. (dig)
Text data:
Fake and true news: This is a dataset containing 44594 instances and 5 features. The features are, "title", the title of the news article. "text", the text of the article, "subject" the article subject and column representing the label classes, "False" or "Truthful". From this dataset we only extracted the "text" column and used it as a features to predict the labels. The dataset can be download from Kaggle (fak).
4

---

Under review as a conference paper at ICLR 2021
20news: This dataset contains 18846 instances divided into 20 classes that describes the 20 different types of news. (20n).
Numerical data
Iris: This dataset classic example for multi-class classification. It contains 150 instances across three classes. (iri).
Wine: The wine dataset also classic example of multi-class classification. It contains 178 instances across three (win).
For each dataset we ran each iteration times with different random seeds. Furthermore, the only parameter that we change is number of labeled instances. To answer RQ2 we have to vary the amount instances in dataset that are already labeled. In our experiments we choose 10% to represent small amount of manual effort required and 50% for large amount of effort required. From each iteration we logged the Fi -scre to measure the accuracy of our predictive labels.
RESULTS
From the simulations a dataset of 840 instances was collected. analyze this data, we first rank each algorithm in each of the ten iterations of each dataset in each experimental condition. This data them expanded into paired comparisons for the use the Bradlcy-Terry model (Turner al. 2020 Cattelan 2012). In this model, y is binary variable that indicates which algorithm beats the other:
Bernouli(pp):
logit "ggit-'(urnn (Haaloll Malgot);
14 Normal(0,5).
The same model is used to analyze both research questions. The model is written in Stan (Carpenter ct al. 2017), which implements the No U-turn Hamiltonian Monte Carlo sampler (Hoffman Gel-
man 2014). We utilize the following configurations: chains, warm-up of 200 iterations and total 2000 iterations. The data transformation, tables, plots and assessing the convergence the chains conducted in R together with package rst and the collection of packages t tidyverse.
The prior distributions of the μi parameters are adjusted to be weakly-informative distributions. The presented model estimates the posterior distribution of the latent strength parameters μ4, In turn, sampling and ranking over the posterior distribution of the strength parameters allows us to obtain a posterior distribution of the ranks.
4.1 AGGREGATED RESULTS
The descriptive statistics of these seven datasets are summarized in Table 1 Table T contains the mean, standard deviation, median as well as 5% and 95% quantiles for each method. Figure provides descriptive statistics in the form of boxplot.
Table 1: Summary statistics for the accuracy of the aggregated data. 5% and 95% represents the quantiles.
Model
Mean
Median
5%
95%
LabelPropagationK.N
0.328
0.288
0.209
0.018
0.967
LabelPropagationRBBF
0.367
0.370
0.165
0.018
0.987
LabeISpreadingKNN
0.750
0.286
0.829
0.099
0.987
LabeISpreadingRBF
0.578
0.316
0.595
0.128
0.993
QBC
0.775
0.281
0,910
0.150
0.974
RandomSampling
0.758
0.277
0.899
0.146
0.973
UncertaintySamplin
0.783
0.284
0.923
0.157
1.000
5

---

Under review as a conference paper at ICLR 2021
Accuracy (all datasets)
1.00
Model
Labelf LabelP PropagationKNi
0.75
Label PropagattonnR
Value
Label5prea
LabelSpre BF
QBC
0.25
RandomS ampling
UncertaintS ampling
0.00
Model
Figure 1: Boxplot of all algorithms
Based on the Bradley-Terry model described above, the parameter strength computed for each algorithm. Figure 2 illustrates the distribution of the parameter strengths along with their High Posterior Density interval. To rank the algorithms we sample over the posterior distribution of the strength parameters 1000 times. The median ranks and their corresponding variances is displayed Table
HPDI of the strength of the model
UncertaintySampiing
RandomSampling
QBC
Model
LabelS preadigge
Labels PreadingKNN
LabelP ropagationRBF
LabelP roaeaarrpgga
Estimate
Figure 2
Table 2: Ranking of algorithms
Models
Median Rank
Variance of the Rank
LabelSpreadingKNN
0.377
UncertaintySampling
2
0.416
QBC
0.229
RandomSampling
4
0.006
I Prradngre
0.000
LabelProgagationRBF
0.000
LabelPropagationKNNN
0.000
4.2
MANUAL EFFORT
The descriptive statistics of the seven datasets located in Table 3 Table 3 contains the mean, standard deviation, median as well as 5% and 95% quantiles for each method. Figure 3 provides descriptive statistics in the form of two boxplots, one for 10% and one for 50% labels.
Based on the Bradley-Tere model described above, the parameter strength computed for each algorithm. Figure 4a and Figure 4b illustrate the distribution of the parameter strengths along with their High Posterior Density interval for 10% and 50% labels respectively. To rank the algorithms sample over the posterior distribution of the strength parameters 1000 times. The median ranks

---

Under review as a conference paper at ICLR 2021
Accuracy (all datasets)
10%
50%
Model
1.00
Labe PoopagationKNN
Label ropaggationn
Value
LS KNeadingKKN
Label5 preadggre
0.25
QBC
LandomSampiin
0.00
uncertaintySamping
Model
Figure 3: Boxplots of algorithms, The left boxplot is for 10% available lables, the right boxplot is for 50% available labels.
Table 3: Summary statistics for the accuracy aggregated data
Model
Mean
SD
Median
5%
95%
10% labels available
LabelPropagationkKNN
0.135
0.111
0.128
0.018
0.333
LabelPropagattonRRFF
0.301
0.332
0.165
0.018
0.961
LabelSpreadingKNN)
0.718
0.304
0.796
0.090
0.981
LaahelSpreadnngRBFF
0.518
0.316
0.444
0.128
0.982
QBC
0.768
0.280
0.900
0.174
0.974
RandomSaapling
0.740
0.272
0.864
0.161
0.967
UncertaintySampling
0.781
0.278
0.917
0.170
1.000
50% labels available
LabelPropaggationKNNN
0.521
0.281
0,482
0.136
0.973
LabelProoaggationRRBB
0.433
0.396
0.346
0.018
0.987
LabelSpreading
0.782
0.265
0.873
0.218
0.988
LabelSpreadingRBBF
0.638
0.306
0.706
0.132
0.994
QBC
0.782
0.284
0.918
0.146
0.974
RandomSampling
0.777
0.282
0.902
0.146
1.000
Uncertainty
0.784
0.292
0.933
0.156
1.000
and their corresponding variances displayed in Table and Table for 10% and 50% labels respectively.
HPDI of the strength of model with 10% availabee labels
HPDI of the strength of the model with 50% of availlable labels
Lusrtant
Sumplig
030
030
LedeSpeedi
LabeSgead goo
Estiaaee
(a) The HPDI interval of the estimated strength parameters of the algorithms with 10% available labels parameters of the algorithms with 50% available labels
(b) The HPDI interval of the estimated strength

---

Under review as a conference paper at ICLR 2021
Table Ranking of the algorithms with 10% of available labels
Table Ranking of the algorithms with 50% of available labels
Models
Median Rank
Variance of the Rank
Models
Median Rank
Variance of the Rank
UncertaintySmmling
LabelSpreadnng
QBC LobelSprea
2
0.157
Uncertannt-Smmping
2
0.353
QBC
RandonSamping
4
0.299
RundomSampling
4
0.161
Spreadggne
5
LabeeSpred
5
I Label LahelPPrpagtionRRFF
6
0.000
I Lahel Prepappppgt
6
0.234
LabelProaagtionNNN
LahelPreaagtionRRf
CONCLUSION
According Table 2 Label Spreading using kNN is the highest ranking algorithm followed by uncertainty sampling. QBC and then random sampling. The uncertainty intervals of the posterior distribution are shown in Figure 2 The large overlap between the top three algorithms strength parameters indicates the uncertainty in rank between them (which can be observed in the large variance of each rank).
According to Table the highest ranking algorithm when having access to 10% available labels is uncertainty sampling, followed by QBC, label spreading using &NN and random sampling. When having access to 50% labels the highest ranking algorithm is label spreading using kNN, followed by uncertainty sampling, QBC, and random sampling according to Table The uncertainty intervals of the posterior distribution are shown in Figures 4a and 4b) for 10% and 50% respectively. The overlap between the top algorithms strength parameters indicates the uncertainty in their estimates, this can also be observed in their variance.
The goal of this study is to provide a detailed overview of what machine learning algorithm should be used for automatic labeling of data in industrial contexts. Based the results,the top four algo- rithms are label spreading using kNN, uncertainty sampling, QBC and random sampling. For the aggregated results well when having access 50% labeled data,the highest ranking algorithm is label spreading using kNN. However, when 10% labels are available, Uncertainty Sampling ranks highest followed by QBC Thus this paper contributes in assisting machine learning practitioners choose the optimal machine learning algorithm for automatic labeling. In future work, simulations will include more datasets provide better understanding of how well algorithms perform on different types of data.
ACKNOWLEDGMENT
This work was partially supported by the Wallenberg AI Autonomous Systems and Software Pro- gram (WASP) funded by Knut and Alice Wallenberg Fundation.
REFERENCES
Adecmm.dttsctssfcccch.000 20negrrgo - scikit-learn 0,23,2 documentation.
URL https:
//tpattt..eaaannn.....//ooeeeren datasets.
discikit-eeann
Eetchi_
akleam.semi.superviseee
scikit-learn 0.23.2 documentation,
URL
Gelearn.semt
semi {_ [_
keee supervied..abeeSSpeeddig - scikit-learn 0.23.2 documentation,
URL.
semi ( t "emn[L/oopperrieeedlaaaaeeeeaaap
CIFAR-10 and CIFAR-100 /tt.:///////.......oooooooe/eee. ci far. h
8

---

Under review as a conference paper at ICLR 2021
UCI Machine Learning Repository: Optical Recognition of Handwritten Digits Data Set.
URL
of+ 2
Fake and
real
news dataset
Kaggle.
URL
com/
1lmentbisa sai on/ fake-ankennaa real news daw.sattaaaaaaaaaaa
Ileeam.dttasets.eee iris scikit-leam 0.23.2 documentation.
URL
https:/
scikit maakit-learn..o
httpaa/wwww....aeee///////////ee htt.:eeddd..leaaaaaaaaateeeeee datanets.
load (_ iir.srri...... html
Kkeammdataaeersloddd.
scikit-learn 0.23.2 documentation.
URL
https:
datasets.
load wine skleakk www.lleannddtttstssss.. load wwine
Anonymous. Undisclosed due to double blind. Blind2, 2020a.
Anonymous. Undisclosed due double blind. 2020b.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betan- court, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic program- ming language. Journal of statistical software, 76(1), 2017.
Manuela Cattelan. Models for paired comparison data: review with emphasis on dependent data. Statistical Science, pp. 412-433, 2012.
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: real-world web image database from national university of singapore. In Proceedings of the ACM international conference on image and video retrieval, pp. 1-9, 2009.
Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby, Selective sampling using the query by committee algorithm. Machine learning, 28(2-)):133-168, 1997.
Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. Mach. Learn. Res., 15(1):1593-162 2014.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 1665):550-554, 1994.
Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 2372- 2379. IEEE, 2009.
David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994, pp. 148-156. Elsevier, 1994.
Sachin Pawar, Nitin Ramrakhiyani, Swapnil Hingmire, and Girish K Palshikar. Topics and label propagation: Best of both worlds for weakly supervised text classification. In International Con- ference Intelligent Text Processing and Computational Linguistics, pp. 446-459. Springer, 2016.
Burr Settles. Active learning. morgan claypool. Synthesis Lectures on Al and MI., 2012.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294, 1992.
Jinhui Tang, Richang Hong, Shuicheng Yan, Tat-Seng Chua, Guo-Jun Qi, and Ramesh Jain. Image annotation by k nn-sparse graph-base based label propagation over noisily tagged web images. ACM Transactions Intelligent Systems and Technology (TIST), 2 2011.

---

Under review as a conference paper at ICLR 2021
Heather L Turner, Jacob van Etten, David Firth, and loannis Kosmidis. Modelling rankings in The plackettluce package. Computational Statistics, pp. 1-31, 2020.
Zheng-Jun Zha, Tao Mei, Jingdong Wang, Zengfu Wang, and Xian-Sheng Hua. Graph-based semi- supervised learning with multiple labels. Journal of Visual Communication and Image Represen- tation, 20(2):97-103, 2009.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learn- ing with local and global consistency. Advances in neural information processing systems, pp- 321-328, 2004.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propa- gation. 2002.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical report, University of Sosconsin-Madise Department of Computer Sciences, 2005.
10


Reviews:
[{"review_id": "xUHwg1oV2Jy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n## Detailed Comments\n- \"There are problems with supervised learning and machine learning in general.\" - This statement is so general, it is essentially vacuous.\n- \"machine learning requires huge amounts of data\" - unclear what this means. and whatever it means it's not true in general.\n- \"severe labeling issues were found\" - what issues?\n- \"we provide the an overview\"\n- \"(AL),\" - missing space. this happens at several places in the text\n- \"continue iterative\" - iteratively\n- \"other stopping criteria\" -> \"criterion\". also, why \"other\"? there was no stopping criterion mentioned so far. \n- \"If a learner does not choose his strategy\" -> their strategy\n- eq at bottom of page 2: what is \"u\"?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iTGQtmUBFBA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an empirical comparison of different approaches for data\nlabeling. The authors describe their experimental setup and findings, making\nrecommendations for when to use what approach in practice.\n\nThe authors reference their own anonymous work throughout the paper as\njustification for the presented investigation and its parameters. This is\nproblematic as the reviewers are now unable to confirm that the presented\ninvestigation is well-grounded.\n\nThe authors evaluate their approaches on only six datasets. It is unclear to\nwhat extent the results generalize, in particular as no detailed results per\ndataset are given. There could be significant differences between the different\ntypes of datasets, but not enough data is presented to judge. This matters in\nparticular with respect to the recommendations the authors make at the end of\nthe paper.\n\nSome details of the experimental setup are unclear. The authors say that they\nmeasure F1 score, but then refer to accuracy (e.g. in Figure 1). Which measure\nwas used? The experimental setup describes six datasets, but the results text\nrefers to seven. The results presented in Table 1 and Figure 1 seem to disagree\nwith Table 2 -- LabelSpreadingKNN is the highest-ranked algorithm, but\nUncertaintySampling performs better in terms of all the statistics presented in\nTable 1. The same is true for the second set of experiments (Tables 3 and 4).\nFor the first set of experiments it is unclear what fraction of labels were\nmissing.\n\nIt is unclear why the Bradley-Terry model was used here to compare outcomes.\nThere are multiple other methods to judge how and whether paired distributions\ndiffer. It appears that only ranks were used for this comparison and not the\nactual performance numbers.\n\nFinally, all methods evaluated by the authors have hyperparameters that need to\nbe set. It is unclear how the authors chose the particular values they used in\nthe experiments, and tuning them for best performance may have a major impact on\ntheir performance and the rankings. Conclusions from untuned methods are\nunlikely to generalize.\n\nThere are numerous typos and grammatical mistakes throughout the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KRL6cnsc2nA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present an empirical analysis of seven machine learning algorithms based on six benchmark datasets: Four graph-based semi-supervised learning algorithms and three active learning algorithms have been evaluated on image data sets (CIFAR10, Digits), texts (Fake and true news, 20news), and other data types (Iris and Wine). Based on the empirical performances of these algorithms, the authors provided a ranked list of the studied algorithms.\n\nEmpirical comparisons of multiple machine learning algorithms on real-world datasets are important as they help understand the strengths and weaknesses of different algorithms when they are deployed in real-world environments. However, I think this paper will benefit significantly from a major revision addressing the concerns listed below:\n1) Algorithm choices: all algorithms considered in the current paper were already extensively studied. Focusing on state of the art approaches (including recent deep learning-based approach) could significantly strengthen the practical relevance and impact of experiments and conclusions reported here.\n2) Data set choices: more challenging datasets can be considered: The datasets used in the current paper (including classical Iris and Wine) have up to only 20 categories.\n3) Presentation: I found it challenging to comprehend the exact experimental settings.\n- Section 3.1 states that 80% of data are allocated for training while the remaining 20% are used in testing. How does this setting apply for active learning algorithms? Did they actively select data points to label until when 80% are labeled or use only `50 instances’?\n- How is 80/20% decomposition applied to semi-supervised learning algorithms? Did they use the entire dataset with 80% of the entities labeled? In typical application scenarios of semi-supervised learning, only small portions of data instances are labeled. I was not sure if the experimental setting prepared in the current paper reflects well the real-world application scenarios of semi-supervised learning.\n- Also, in general, semi-supervised learning and active learning are different problems. I was not sure how the results reported in the current paper including Table 1 should be interpreted: I guess this does not suggest that semi-supervised learning algorithms are better than active learning approaches.\n4) Without having access to two `anonymous’ papers cited in this submission, it is hard to properly assess the contributions of this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hs2Nn8mjM3f", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to evaluate the performance of seven automated labeling algorithms in terms of accuracy. The authors conducted a set of experiments on six datasets from different domains under two typical settings where 10% and 50%of labels in the datasets are available. Experimental results show that the algorithms label spreading with KNN perform better in the aggregated results, the active learning algorithms  QBC and query instance uncertainty sample perform better when 10% of labels available.\n\nOverall, this paper cannot meet the high-quality requirements of ICLR.  First, active learning algorithms such as QBC and uncertainty sampling is not automated labeling algorithms. They are only strategies for the selection of unlabeled instances. The selected instance either can be labeled by human experts or automated labeling algorithms. Second, when evaluating an automated labeling algorithm, merely using accuracy is not enough. For example, when the underlying class distributions are imbalanced, the accuracy is not sufficient to characterize the generalization performance of a learning algorithm. Third, two settings of 10% and 50% of labels available are also insufficient. Many papers of the empirical study investigated the performance under more complicated settings. Finally, the number of investigated methods is two small and the paper should cover more state-of-the-art algorithms.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper evaluates several different strategies for labeling of missing data, and recommend the best strategy in practice based on the empirical results on six data sets.

The reviewers agree that empirical evaluation is important for providing a good guideline for this practical problem. The concerns of the reviewers include the lack of motivation on the chosen strategies, the lack of novelty (the tools in the strategies are all pretty standard in the literature), the lack of reproducibility (by referring to the authors' own anonymous work for parameters), and the lack of breadth (e.g #data sets) and depth (e.g. metrics explored) in the experiments.

Decision: Reject