document: Efficiently learning the accuracy of labeling sources for selective sampling Many scalable data mining tasks rely on active learning to provide the most useful accurately labeled instances . However , what if there are multiple labeling sources ( ` oracles ' or ` experts ' ) with different but unknown reliabilities ? With the recent advent of inexpensive and scalable online annotation tools , such as Amazon 's Mechanical Turk , the labeling process has become more vulnerable to noise - and without prior knowledge of the accuracy of each individual labeler . This paper addresses exactly such a challenge : how to jointly learn the accuracy of labeling sources and obtain the most informative labels for the active learning task at hand minimizing total labeling effort . More specifically , we present IEThresh ( Interval Estimate Threshold ) as a strategy to intelligently select the expert ( s ) with the highest estimated labeling accuracy . IEThresh estimates a confidence interval for the reliability of each expert and filters out the one ( s ) whose estimated upper-bound confidence interval is below a threshold - which jointly optimizes expected accuracy ( mean ) and need to better estimate the expert 's accuracy ( variance ) . Our framework is flexible enough to work with a wide range of different noise levels and outperforms baselines such as asking all available experts and random expert selection . In particular , IEThresh achieves a given level of accuracy with less than half the queries issued by all-experts labeling and less than a third the queries required by random expert selection on datasets such as the UCI mushroom one . The results show that our method naturally balances exploration and exploitation as it gains knowledge of which experts to rely upon , and selects them with increasing frequency .
doc_bio_tags: O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O