Patent ID: 9704106
Date: 2017-07-11
CPC Classifications: G06F,G06K,G06N

Claim:
1. A method for determining labels for crowdsourced annotations, comprising: obtaining a set of source data using a distributed data annotation server system comprising a processor and a memory readable by the processor, where the source data comprises a set of unlabeled data; obtaining a set of training data using the distributed data annotation server system, where the set of training data comprises a subset of the source data representative of the set of source data; determining ground truth data describing the ground truth for each piece of training data in the set of training data using the distributed data annotation server system, where the ground truth data for a piece of training data describes the content of the piece of data; providing the sets of annotator data to a plurality of data annotation devices, wherein a data annotation device: obtaining the sets of annotation data from the plurality of data annotation devices using the distributed data annotation server system; calculating annotator accuracy data for each data annotation device for at least one piece of training data in the set of training data based on the ground truth for each piece of training source data and the set of training data annotations using the distributed data annotation server system, wherein the annotator accuracy data describes the accuracy of annotation data provided by a particular data annotation device based on the accuracy and confidence indicated for one or more pieces of training data provided to the particular data annotation device; and automatically generating a set of labels for each piece of unlabeled data in the set of source data based on the calculated annotator accuracy data and the set of annotator labels received from the plurality of data annotation devices using the distributed data annotation server system.