Patent ID: 8498950

Claim:
A method for training classifiers in multiple categories through an active learning system, the method executable by a computer having a processor and memory, the method comprising: training, by the processor, an initial set of m binary one-versus-all classifiers, one for each category in a taxonomy, on a labeled dataset of examples stored in a database coupled with the computer; uniformly sampling, by the processor, up to a predetermined number of examples from a second, larger dataset of unlabeled examples stored in a database coupled with the computer; ordering, by the processor, the sampled unlabeled examples in order of informativeness for each classifier; determining, by the processor, a minimum subset of the unlabeled examples that are most informative for a maximum number of the classifiers to form an active set for learning; where for each iteration of active learning, the processor selecting a pool, q, of potentially most-informative examples from which to select examples to label for each of the m binary classifiers; providing a first constraint that at most / examples be chosen for to form the active set, where / is derived from editor bandwidth; providing as a second constraint that at least k examples be sampled for each binary classifier; providing an objective function that selects as the active set the examples that improve classification accuracy for the maximum number of the classifiers; generating an integer optimization problem from the objective function and the pool within the first and second constraints; and solving the integer optimization problem with an integer programming solver; and using, by the processor, editorially-labeled versions of the examples of the active set to re-train the classifiers, thereby improving the accuracy of at least some of the classifiers.