Methods, systems, and media for selecting candidates for annotation for use in training classifiers

Methods, systems, and media for selecting candidates for annotation for use in training classifiers are provided. In some embodiments, the method comprises: identifying, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample includes a plurality of patches; for each patch of the plurality of patches, determining a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels; identifying a subset of the patches in the plurality of patches; for each patch in the subset of the patches, calculating a metric that indicates a variance of the probabilities assigned to each patch; selecting a subset of the candidate training samples based on the metric; labeling candidate training samples in the subset of the candidate training samples by querying an external source; and re-training the CNN using the labeled candidate training samples.

TECHNICAL FIELD

The disclosed subject matter relates to methods, systems, and media for selecting candidates for annotation for use in training classifiers.

BACKGROUND

Intense interest in applying classifiers (such as convolutional neural networks (CNNs)) in biomedical image analysis is widespread. For example, CNNs can be used to suggest to an authorized professional whether one or more biomedical images are likely to have one or more given characteristics (which can be represented by one or more of |Y| possible labels) so that the professional can diagnose a medical condition of a patient.

In order for a CNN to perform this function, the CNN needs to be trained using annotated biomedical training images that indicate whether the training images have one or more of the |Y| possible labels. For example, for the CNN to be able to spot a condition in an image, many training images annotated as showing the condition and many training images annotated as not showing the condition can be used to train the CNN. The better trained the CNN is, the less likely the CNN is to misclassify an image.

The success of CNNs for this purpose, however, is impeded by the lack of large annotated datasets in biomedical imaging. Annotating biomedical images is not only tedious and time consuming, but also demanding of costly, specialty-oriented knowledge and skills, which are not easily accessible.

Accordingly, it is desirable to provide new methods, systems, and media for selecting candidates for annotation for use in training classifiers.

SUMMARY

Methods, systems, and media for selecting candidates for annotation for use in training classifiers are provided. In accordance with some embodiments of the disclosed subject matter, a method for selecting candidates for annotation for use in training classifiers is provided, the method comprising: identifying, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image; for each candidate training sample in the group of candidate training samples: for each patch of the plurality of patches associated with the candidate training sample, determining a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN; identifying a subset of the patches in the plurality of patches; and for each patch in the subset of the patches, calculating a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches; selecting a subset of the candidate training samples from the group of candidate training samples based on the metric, wherein the subset does not include all of the candidate training samples; labeling candidate training samples in the subset of the candidate training samples by querying an external source; and re-training the CNN using the labeled candidate training samples.

In accordance with some embodiments of the disclosed subject matter, a system for selecting candidates for annotation for use in training classifiers is provided, the system comprising: a memory; and a hardware processor that, when executing computer-executable instructions stored in the memory, is configured to: identify, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image; for each candidate training sample in the group of candidate training samples: for each patch of the plurality of patches associated with the candidate training sample, determine a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN; identify a subset of the patches in the plurality of patches; and for each patch in the subset of the patches, calculate a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches; select a subset of the candidate training samples from the group of candidate training samples based on the metric, wherein the subset does not include all of the candidate training samples; label candidate training samples in the subset of the candidate training samples by querying an external source; and re-train the CNN using the labeled candidate training samples.

In accordance with some embodiments of the disclosed subject matter, non-transitory computer-readable media containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for selecting candidates for annotation for use in training classifiers. The method comprises: identifying, for a trained Convolutional Neural Network (CNN), a group of candidate training samples, wherein each candidate training sample is a portion of an image, and wherein each candidate training sample includes a plurality of patches of the portion of the image; for each candidate training sample in the group of candidate training samples: for each patch of the plurality of patches associated with the candidate training sample, determining a plurality of probabilities, each probability being a probability that the patch corresponds to a label of a plurality of labels, wherein the plurality of probabilities are determined using the trained CNN; identifying a subset of the patches in the plurality of patches; and for each patch in the subset of the patches, calculating a metric that indicates at least a variance of the probabilities assigned to each patch in the subset of the patches; selecting a subset of the candidate training samples from the group of candidate training samples based on the metric, wherein the subset does not include all of the candidate training samples; labeling candidate training samples in the subset of the candidate training samples by querying an external source; and re-training the CNN using the labeled candidate training samples.

DETAILED DESCRIPTION

In accordance with various embodiments, mechanisms (which can include methods, systems, and media) for selecting candidates for annotation for use in training classifiers are provided.

In some embodiments, the mechanisms described herein can identify candidate samples to fine-tune training, or boost performance, of a Convolutional Neural Network (CNN). In some embodiments, the mechanisms described herein can begin with a pre-trained CNN and can use the techniques described herein to identify particularly salient samples that have not yet been annotated. The mechanisms can then transmit identified salient samples for manual annotation (e.g., by a qualified human annotator), and can use the manually annotated samples to update training of the CNN.

In some embodiments, the mechanisms described herein can identify salient candidate samples for manual annotation using any suitable technique or combination of techniques. For example, in some embodiments, the mechanisms can generate multiple patches for a particular candidate sample (e.g., generate multiple image patches by cropping, scaling, etc. portions of a candidate sample image). Note that patches generated from the same candidate image are expected to have similar predicted labels by a pre-trained CNN. Therefore, the entropy and diversity of the predictions for the patches, where entropy indicates a classification uncertainty and where diversity indicates a prediction consistency, can be used to determine a “power” of a candidate associated with the patches for improving the performance of the currently trained CNN. In some embodiments, the mechanisms can then select candidate samples that are identified as being particularly useful for fine-tuning the training, and can then transmit the identified candidate samples for manual annotation. In some embodiments, as described below in connection withFIG. 1, fine-tuning of training of the CNN can be performed with both the identified candidate samples as well as with samples that have been previously misclassified by the CNN. Note that, as described below in connection withFIG. 1, in some embodiments, the mechanisms can incorporate randomness in selection of the particularly useful candidate samples to strike a balance between exploration and exploitation. Furthermore, combination of selected candidate samples with previously misclassified candidates can prevent catastrophic forgetting.

In some embodiments, the mechanisms described herein can provide many advantages for training of a CNN. For example, in some embodiments, the mechanisms can use an empty labeled dataset, and do not require seed-labeled candidates. As another example, in some embodiments, the mechanisms described herein can improve a classifier through continuous fine-tuning rather than through repeated re-training of the classifier. As yet another example, in some embodiments, the mechanisms can be used to select candidate samples that are likely to be the most informative by naturally exploiting consistency among patches associated with a candidate sample. As still another example, in some embodiments, the mechanisms can compute selection criteria locally on a small number of patches associated with a candidate sample, thereby saving considerable computation time. As still another example, in some embodiments, the mechanisms can handle noisy labels via majority selection. As still another example, in some embodiments, the mechanisms can autonomously balance training samples among different classes. As still another example, in some embodiments, by incorporating fine-tuning training using hard samples (e.g., previously misclassified samples), the mechanisms can prevent catastrophic forgetting. As still another example, in some embodiments, the mechanisms can balance exploration and exploitation by incorporating randomness into active selection.

Note that, in some embodiments, a CNN can be used to classify samples related to any suitable topic or genre, such as Computer Aided Diagnoses (CAD), and/or any other suitable type of genre. For example, in some embodiments, the mechanisms described herein can be particularly useful for CAD using biomedical images (e.g., MRI images, CT images, images captured from a camera during a medical procedure, and/or any other suitable type of biomedical images). As a more particular example, because current regulations require that CAD systems be deployed in a “closed” environment in which all CAD results are reviewed, and errors, if any, must be corrected by radiologists, the continuous, fine-tuning techniques described herein can be used for substantial improvement of CAD systems.

Turning toFIG. 1, an example100of a process for selecting candidates for annotation for use in training classifiers is shown in accordance with some embodiments of the disclosed subject matter. In some embodiments, blocks of process100can be executed on any suitable device, such as a server, a desktop computer, a laptop computer, and/or any other suitable type of device. Note that, in some embodiments, block of process100can be executed on multiple devices, for example, in parallel, and/or in any other suitable manner.

Process100can begin at102by identifying, for a trained CNN, a group of candidate samples, where each candidate sample is associated with multiple patches of an image. In some embodiments, each candidate sample in the group of candidate samples can correspond to any suitable type of content. For example, in some embodiments, each candidate sample can be an image. As a more particular example, in some embodiments, each candidate sample can be a medical image (e.g., an MM image, a CT image, an image captured from a camera used during a medical procedure, and/or any other suitable type of medical image). As another more particular example, in some embodiments, a candidate sample can be a frame from a video captured during a medical procedure. Note that, in some embodiments, each sample in the group of candidate samples can be a sample that has not yet been labeled or annotated. In some embodiments, process100can identify the group of candidate samples in any suitable manner. For example, in some embodiments, process100can identify the group of candidate samples from any suitable dataset or database. Note that, in some embodiments, as shown in process200as shown inFIG. 2, the group of candidate samples can be referred to herein as set U, where U has n candidates where i∈[1, n].

In some embodiments, each candidate sample in the group of candidate samples can be associated with a group of patches, where each patch is itself an image. For example, in some embodiments, each candidate sample Cican have m patches xij, where j∈[1, m]. Note that, in some embodiments, m can be any suitable number (e.g., five, ten, twenty, one hundred, and/or any other suitable number). Additionally, note that, in some embodiments, each candidate sample Cican be associated with a different number of patches.

In some embodiments, process100can generate the patches associated with each candidate sample in any suitable manner. For example, in some embodiments, the patches can be generated automatically for each candidate sample through data augmentation. In some such embodiments, data augmentation can be performed in any suitable manner, and any suitable amount of data augmentation can be performed in some embodiments. For example, in some embodiments, an image that is a single frame of a colonoscopy video and that has a size of 712 pixels by 480 pixels can be received and used to form a candidate. The whole image can be labeled as informative or non-informative. The image can then be cropped into 21 patches (e.g., images that are 50 pixels by 50 pixels) from the image by translating the image by ten (or any other suitable numbers, such as twenty) percent of a resized bounding box in vertical and horizontal directions. Each resulting patch can be rotated eight times by mirroring and flipping. All 21 patches can then be associated with one candidate sample. Note that, in some embodiments, a factor f (e.g., where factor f∈{1.0, 1.2, 1.5}) can be used to enlarge a patch (e.g., to realize an augmented data set of the original size, 1.2 times larger, and 1.5 times larger) and then crop it back to the original size. For example, if one patch is sized at 10 pixels by 10 pixels, it can be enlarged by a factor f equal to 1.2 to produce a patch of 12 pixels by 12 pixels, and then the patch can be cropped to the center 10 pixels by 10 pixels as a new patch after data augmentation. Additionally, note that, a manner of performing data augmentation can be based on the application. For example, for colonoscopy frame classification, translation data augmentation can be applied by ten percent of a resized bounding box in vertical and horizontal directions. As another example, for polyp detection, rotation data augmentation can be applied at the center of a polyp location. As still another example, for pulmonary embolism detection, scale plus rotation data augmentation can be applied—e.g., by extracting three different physical sizes, e.g., 10 mm, 15 mm, 20 mm wide, by rotating the longitudinal and cross-sectional vessel planes around the vessel axis.

As described above, in some embodiments, process100can receive a pre-trained CNN. In some embodiments, the pre-trained CNN can be referred to herein as M0, and the model at step T can be referred to as MT. In some embodiments, any suitable pre-trained CNN can be used, and the pre-trained CNN can be received from any suitable source. For example, in some embodiments, the pre-trained CNN be a pre-trained AlexNet. As other examples, in some embodiments, VGG, GoogleNet, or the ResNet can be used instead of an AlexNet.

In some embodiments, process100can iterate through104-108as shown inFIG. 1for each candidate sample in the group of candidate samples. Note that104-108are described below in more detail. Additionally, note that pseudo-code for performing the calculations associated with104-108are shown in the for loop shown in lines3-8ofFIG. 2.

Referring back toFIG. 1, at104, process100can, for a candidate sample in the group of candidate samples and using the trained CNN, predict labels for each patch associated with the candidate sample. In some embodiments, a label assigned to the patch can be any suitable label from a group of labels. For example, in some embodiments, the label can correspond to a classification for the patch. As a more particular example, in an instance in which the CNN is a classifier for classifying aspects of medical images (e.g., whether a particular medical image is an informative medical image or an uninformative medical image, whether a particular image includes healthy tissue or diseased issue, and/or any other suitable aspect), the label can correspond to a classification from a group of classifications. In some embodiments, the label can be any label y from a group of |Y| possible labels. For example, in some embodiments, the group of possible labels can include: {“healthy” and “diseased”}; {“informative” and “uninformative”}; {“true positive” and “false positive”}; and/or any other suitable group of possible labels. In some embodiments, the group of possible labels can include any suitable number of labels (e.g., two, three, five, and/or any other suitable number). An example of pseudo-code for predicting labels for each patch of a candidate sample is shown in line4ofFIG. 2. Note that, in some embodiments, the trained CNN can assign a probability that a particular patch is associated with each label in the group of labels. For example, in some embodiments, in an instance in which the group of labels includes three labels (e.g., “Label 1,” “Label 2,” and “Label 3”), process100can use the trained CNN to determine probabilities that a particular patch is associated with each of the three labels (e.g., Label 1—0.2, Label 2—0.3, Label 3—0.5, and/or any other suitable probabilities). In some such embodiments, the probabilities associated by the trained CNN, when summed across all of the labels in the group of labels for a particular patch, can be 1 or substantially close to 1. Note that, as shown inFIG. 2, process100can predict the labels for each patch using the trained CNN M at step T-1 (referred to as MT-1).

At106, process100can sort the patches associated with the candidate sample based on the labels for each patch. Turning toFIG. 5, an example of patches associated with candidate samples that are sorted based on predicted labels is shown in accordance with some embodiments of the disclosed subject matter.

FIG. 5shows two examples,502and510of candidate samples, where candidate sample502corresponds to an image of an office, and where candidate sample510corresponds to an image of a living room. Group of patches504shows twenty patches (e.g., such as individual patch506) constructed based on candidate sample502, and group of patches512shows twenty patches constructed based on candidate sample510. As described above in connection with104, each patch in group of patches504and group of patches512is assigned a probability that the patch is associated with a particular label in the group of labels. In the example shown inFIG. 5, the group of labels includes labels “k,” “l,” and “o,” which can correspond to, for example “kitchen,” “living room,” and “office,” respectively (e.g., an indication of a location depicted in the corresponding sample). For example, as shown inFIG. 5, patch506has been assigned probabilities of: k=4.52%, 1=4.05%, and o=91.43%.

In some embodiments, a dominant class or label can be identified based on the assigned probabilities for each patch in a group of patches associated with the candidate sample. In some embodiments, the dominant class or label can be identified in any suitable manner, for example, based on the mean probabilities associated with each label in the group of labels for each patch in the group of patches. For example, for candidate sample502shown inFIG. 5, a dominant class or label of “o” (e.g., corresponding to “office”) can be identified, because the mean percentages508for the labels is highest for label “o,” as shown inFIG. 5. As another example, for candidate sample510shown inFIG. 5, a dominant class or label of “o” (e.g., corresponding to “office”) can be identified, because the mean percentages514for the labels is highest for label “o,” as shown inFIG. 5. Note that,FIG. 5shows the patches in each of the group of patches (e.g., patches in group of patches504and patches in group of patches512) sorted based on the probabilities assigned to the identified dominant class or label. For example, patches within group of patches504are sorted in a descending order based on the probability assigned, for each patch in group of patches504, to label “o.”

An example of pseudo-code for sorting the patches associated with the candidate sample based on the labels for each patch is shown in line5ofFIG. 2. For example, as shown inFIG. 2, Ci, can be sorted in descending order according to the predicted dominant class, where the predicted dominant class is ŷ, and where ŷ is defined as:

Referring back toFIG. 1, process100can, at108, for the top a % of patches, compute a metric A that indicates a diversity and an entropy of the labels for the patches. In some embodiments, a can be any suitable percentage, such as 25%, 30%, and/or any other suitable percentage.

In some embodiments, A can be computed in any suitable manner. For example, in some embodiments, process100can compute, for the candidate sample, an entropy metric eiand a diversity metric diand can compute A as a combination of eiand di. An example of an equation that can be used to calculate eiis:

In some embodiments, an example of an equation that can be used to calculate diis:

di=∑k=1y⁢∑j=1m⁢∑l=jm⁢(pij,k-pil,k)⁢log⁢⁢pij,kpil,k,
where k is an index that iterates over the labels in the group of labels |Y| In some embodiments, A for a candidate sample Cican then be calculated as:
i=λ1ei+λ2di.
In some embodiments, λ1and λ2can correspond to weighting parameters for the entropy metric and the diversity metric, respectively. In some embodiments, λ1and λ2can have any suitable values, including 0. Note that, in some embodiments, Aican be a score matrix of size αm×αm for each candidate sample Ci.

Note that, in some embodiments, the entropy and the diversity calculated for a particular candidate sample can indicate any suitable information. For example, in some embodiments, entropy can indicate a classification certainty, where a higher entropy value indicates a higher uncertainty in the classification of the patches associated with the candidate sample. As another example, in some embodiments, diversity can indicate prediction consistency among the patches associated with the candidate sample, where a higher diversity value indicates a greater degree of prediction inconsistency.

Turning toFIG. 6, an example of prediction patterns for patches associated with a particular candidate sample as well as diversity and entropy metrics are shown in accordance with some embodiments of the disclosed subject matter. In some embodiments, for a prediction pijof a particular patch xijby a current trained version of a CNN (e.g., the model at a current time-step), the prediction pattern of a candidate Cican be a histogram Pi=pij, jϵ[1, m].FIG. 6shows seven examples of histograms for prediction patterns for a candidate Ci—patterns A, B, C, D, E, F, and G. In some embodiments, the prediction patterns shown inFIG. 6can be used to identify a candidate Cithat is particularly useful for fine-tuning training of the CNN. Description of the patterns A-G are given below. Note that entropy and diversity metrics for each pattern are shown inFIG. 6. As illustrated, diversity and entropy are calculated both for all of the patches associated with the candidate sample as well as for the top α patches (indicated as entropyαand diversityα).

Referring to pattern A ofFIG. 6, the histogram is mostly concentrated at 0.5, with a higher degree of uncertainty.

Referring to pattern B ofFIG. 6, the histogram of pattern B is flatter than that of pattern A, as the patches' predictions are spread widely from 0 to 1 with a higher degree of inconsistency among the patches' predictions. Since all of the patches associated with a particular candidate sample are generated via data augmentation, the patches are expected to generate similar predictions by the CNN, and therefore, the candidate associated with pattern B have the potential to significantly enhance the current CNN's performance.

Referring to pattern C, the histogram is clustered at both ends, with a higher degree of diversity. In some embodiments, candidates associated with this type of histogram pattern are most likely associated with noise labels at the patch level, and are therefore the least favorable for use in active selection because they may cause confusion when fine-tuning the CNN.

Referring to patterns D and E, the histograms are clustered at either end (i.e.,0or1), with a higher degree of certainty. In some embodiments, candidates associated with these types of histogram patterns should not be used for manual annotation and fine-tuning, because it is likely that the current CNN has correctly predicted these candidates, and that these candidates would therefore contribute little toward fine-tuning the current CNN.

Referring to patterns F and G, patches have a higher degree of certainty for some of the predictions, but there are some outliers in the predictions. In some embodiments, candidates associated with these types of histogram patterns can be valuable because they are capable of smoothly improving the CNN's performance. In some embodiments, while such candidates might not make dramatic contributions, they do not significantly degrade the CNN's performance either.

Note that, an example of pseudo-code for calculating A for the top a % of patches is shown in lines6and7ofFIG. 2.

Referring back toFIG. 1, in some embodiments, process100can loop back to104and iterate through104-108with another candidate sample from the group of candidate samples. In some embodiments, process100can iterate through104-108for each candidate sample in the group of candidate samples. Note that, in some embodiments, any suitable optimizer (e.g., Adam, and/or any other suitable optimizer) can be used to solve any of the objective functions described herein.

At110, process100can select a subset of the candidate samples from the group of candidate samples for manual annotation based on the metric A. In some embodiments, the subset can include any suitable number b (e.g., five, ten, twenty, and/or any other suitable number) of the candidate samples from the group of candidate samples. In some embodiments, the subset of the selected candidate samples can be referred to as Q.

In some embodiments, process100can select the subset of the candidate samples in any suitable manner. For example, in some embodiments, process100can sort the candidate samples in the group of candidate samples (e.g., the Ciin set U) based on the value of A associated with each candidate sample. In some embodiments, process100can then use any suitable randomization technique to select b candidates from the sorted group of candidate samples. For example, in some embodiments, process100can use a random extension parameter ω such that b samples are selected from the top cob samples in the sorted group of candidate samples. Note that, in some embodiments, ω can have any suitable value, such as two, five, ten, and/or any other suitable value. A more particular example for selecting b candidates is:

Ai′←(Ai′-Aω⁢⁢b′)/(A1′-Aω⁢⁢b′),⁢Ais←Ai′/∑i⁢⁢A1′,∀i∈[1,ω⁢⁢b],
where A′iis a sorted list of Aiin descending order of A, and where Aisis the sampling probability.

In some embodiments, process100can determine or identify manually annotated labels for each of the b samples in set Q in any suitable manner. For example, in some embodiments, process100can transmit information associated with each of the selected candidate samples in Q (e.g., an image that corresponds to the candidate sample, and/or any other suitable information) to a user device associated with a qualified annotater, and can receive a classification for each sample in Q from the user device associated with the qualified annotater. In some embodiments, process100can then associate the manually-annotated classification with each of the selected candidate samples in Q such that each sample in Q is then labeled with a correct classification.

Note that, an example of pseudo-code for selecting the subset of the candidate samples and assigning manually annotated labels to candidate samples for the subset of the candidate samples is shown in lines9-11ofFIG. 2.

Referring back toFIG. 1, at112, process100can identify a group of misclassified samples that were misclassified by the trained CNN. Note that, in some embodiments, process100can identify samples that were misclassified by the trained CNN at step T−1. In some embodiments, the group of misclassified samples can be training samples from a labeled training set L that were associated with ground-truth classifications and that were misclassified by the trained CNN at step T−1. In some embodiments, the group of misclassified samples can be referred to as set H. Note that, in some embodiments, the misclassified samples can be referred to as “hard” samples.

Note that, an example of pseudo-code for identifying the group of misclassified samples is shown in lines12-13ofFIG. 2.

Referring back toFIG. 1, at114, process100can re-train the CNN using both the selected candidate samples (that is, set Q) and the identified group of misclassified samples (that is, set II). More particularly, in some embodiments, process100can re-train the CNN with the union of H and Q. In some embodiments, re-training the CNN using H and Q can generate an updated model, that is, the model at step T (e.g., MT). Note that, in some embodiments, re-training the CNN using the selected candidate samples and the misclassified samples can be referred to as fine-tuning the CNN.

Note that, an example of pseudo-code for re-training the CNN is shown in line14ofFIG. 2.

Referring back toFIG. 1, at116, process100can update the group of unlabeled training samples U. For example, in some embodiments, process100can remove the selected candidate samples Q from set U. Note that, in some embodiments, process100can additionally update the group of labeled samples L to include the selected candidate samples Q (e.g., with the manual annotations associated with each sample in Q).

Note that, an example of pseudo-code for updating the unlabeled and labeled samples is shown in line15ofFIG. 2.

Referring back toFIG. 1, in some embodiments, process100can loop back to102and select a new group of candidate samples. In some embodiments, process100can iterate through the blocks of process100until any suitable classification performance is satisfied. In some embodiments, any suitable criteria can be used to determine if classification performance is satisfactory. For example, in some embodiments, the criteria can include whether more than a predetermined percentage of newly annotated samples are correctly predicted by the model at a current time-step (e.g., more than 70%, more than 80%, and/or any other suitable percentage).

Turning toFIG. 3, an example300of hardware for selecting candidates for annotation for use in training classifiers that can be used in accordance with some embodiments of the disclosed subject matter is shown. As illustrated, hardware300can include a server302, a communication network304, and/or one or more user devices306, such as user devices308and/or310.

Server302can be any suitable server(s) for storing information, datasets, programs, and/or any other suitable type of content. For example, in some embodiments, server302can store any suitable datasets used for training, validating, or testing a classifier. In some embodiments, server302can transmit any portion of any suitable dataset to user devices306, for example, in response to a request from user devices306. Note that, in some embodiments, server302can execute any suitable programs or algorithms for selecting candidates for annotation for use in training classifiers. For example, in some embodiments, server302can execute any of the blocks shown in and described above in connection withFIG. 1for selecting candidates for annotation for use in training classifiers. In some such embodiments, server302can receive any suitable data or parameters associated with an algorithm for selecting candidates, as described above in connection withFIGS. 1 and 2. In some embodiments, server302can be omitted.

Communication network304can be any suitable combination of one or more wired and/or wireless networks in some embodiments. For example, communication network304can include any one or more of the Internet, an intranet, a wide-area network (WAN), a local-area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and/or any other suitable communication network. User devices306can be connected by one or more communications links to communication network304that can be linked via one or more communications links to server302. The communications links can be any communications links suitable for communicating data among user devices306and server302such as network links, dial-up links, wireless links, hard-wired links, any other suitable communications links, or any suitable combination of such links.

User devices306can include any one or more user devices. In some embodiments, user devices306can perform any suitable function(s). For example, in some embodiments, user devices306can execute any suitable blocks shown in and described above in connection withFIGS. 1 and 2for selecting candidates for annotation for training classifiers. As another example, in some embodiments, user devices306can initiate execution of any suitable blocks of processes100or200on server302, for example, by transmitting instructions to server302in connection with any suitable parameters for selecting candidates for annotation for training classifiers. In some embodiments, user devices306can include any suitable types of user devices, such as a desktop computer, a laptop computer, a mobile phone, a tablet computer, and/or any other suitable type of user device.

Although server302is illustrated as one device, the functions performed by server302can be performed using any suitable number of devices in some embodiments. For example, in some embodiments, multiple devices can be used to implement the functions performed by server302.

Although two user devices308and310are shown inFIG. 3to avoid over-complicating the figure, any suitable number of user devices, and/or any suitable types of user devices, can be used in some embodiments.

Server302and user devices306can be implemented using any suitable hardware in some embodiments. For example, in some embodiments, devices302and306can be implemented using any suitable general-purpose computer or special-purpose computer. For example, a mobile phone may be implemented using a special-purpose computer. Any such general-purpose computer or special-purpose computer can include any suitable hardware. For example, as illustrated in example hardware400ofFIG. 4, such hardware can include hardware processor402, memory and/or storage404, an input device controller406, an input device408, display/audio drivers410, display and audio output circuitry412, communication interface(s)414, an antenna416, and a bus418.

Hardware processor402can include any suitable hardware processor, such as a microprocessor, a micro-controller, digital signal processor(s), dedicated logic, and/or any other suitable circuitry for controlling the functioning of a general-purpose computer or a special-purpose computer in some embodiments. In some embodiments, hardware processor402can be controlled by a server program stored in memory and/or storage of a server, such as server302. In some embodiments, hardware processor402can be controlled by a computer program stored in memory and/or storage404of user device306.

Memory and/or storage404can be any suitable memory and/or storage for storing programs, data, and/or any other suitable information in some embodiments. For example, memory and/or storage404can include random access memory, read-only memory, flash memory, hard disk storage, optical media, and/or any other suitable memory.

Input device controller406can be any suitable circuitry for controlling and receiving input from one or more input devices408in some embodiments. For example, input device controller406can be circuitry for receiving input from a touchscreen, from a keyboard, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near field sensor, from a pressure sensor, from an encoder, and/or any other type of input device.

Display/audio drivers410can be any suitable circuitry for controlling and driving output to one or more display/audio output devices412in some embodiments. For example, display/audio drivers410can be circuitry for driving a touchscreen, a flat-panel display, a cathode ray tube display, a projector, a speaker or speakers, and/or any other suitable display and/or presentation devices.

Communication interface(s)414can be any suitable circuitry for interfacing with one or more communication networks (e.g., computer network304). For example, interface(s)414can include network interface card circuitry, wireless communication circuitry, and/or any other suitable type of communication network circuitry.

Antenna416can be any suitable one or more antennas for wirelessly communicating with a communication network (e.g., communication network304) in some embodiments. In some embodiments, antenna416can be omitted.

Bus418can be any suitable mechanism for communicating between two or more components402,404,406,410, and414in some embodiments.

Any other suitable components can be included in hardware400in accordance with some embodiments.

In some embodiments, at least some of the above described blocks of the processes ofFIGS. 1 and 2can be executed or performed in any order or sequence not limited to the order and sequence shown in and described in connection with the figures. Also, some of the above blocks ofFIGS. 1 and 2can be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. Additionally or alternatively, some of the above described blocks of the processes ofFIGS. 1 and 2can be omitted.

Accordingly, methods, systems, and media for selecting candidates for annotation for use in training classifiers are provided.