Practical methods of Active Learning

In many machine learning applications, labeled data is scarce but unlabeled data plentiful. However, labeling them requires tedious human supervision. The goal of active learning [2, 6, 12, 3, 1, 5] is to seek out from the unlabeled pool those instances which when labeled will help strengthen the classifier at the fastest possible rate. An active learner starts with a limited labeled and a large unlabeled pool of instances. The labeled set forms the training data for an initial preliminary classifier. The active learner then carefully selects a few examples from the unlabeled set and seeks its labels from a user. The newly labeled set is added to the training set and the classifier retrained. The new knowledge is used to select another set of unlabeled instance and this continues in a loop until the user is happy with the learnt classifier. The number of examples required to be manually labeled can reduced significantly using active learning. This has been both observed in several empirical studies and proved theoretically. [? , ? ]. Hence, there is increasing interest in applying this method of labeling data in several practical applications, including, text classification [6, 12, 5], information extraction [11, 1], duplicate detection [8, ? , ? ], image labeling and querying [] and speech recognition []. Starting with some of the early work in [? , ? , ? ], a number of methods have been proposed for active learning. Some of these have been theoretically analyzed, others are specific to particular kinds of classifiers like Support Vector Machines, and developed and tested specifically in particular application settings. We present a taxonomy of the various methods of active learning in Section ? In this paper our goal is to consolidate all these myriad approaches, present a detailed empirical evaluation of the various methods of doing active learning, on different classification methods and against multiple different datasets. We hope that this study will provide a practical perspective on the alternatives for active learning. We will therefore only consider those methods that do not require excessive amount of computation in coming up with the instance to be labeled next. 1 Conceptual overview of active learning We will first give a conceptual overview of the criteria used for active learning without bothering about the efficiency of evaluation. Later, in Section ? we will present practical alternatives. The main challenge in active learning is estimating an instance’s impact in improving a classifier without knowing its label. We need to explicitly model the informativeness of an instance by measuring how much including that instance in the training set results in the improvement of the classifier. we will need to addresss two issues: • How do we quantify the quality C of a classifier for a given training set T? • Without knowing the label of an instance x how do we estimate the reduction in C possible by augmenting the training set T with x correctly labeled? The best we can do is to measure expected impact as follows. We can calculate expected value of Ey(Q(L+ < x, y >) with the addition of an unlabeled instance x ∈ U . We use the current classifier C to find the probability Pr(y|x) that an instance x will get assigned class y. We then estimate expected impact U(x) of an instance x as