Patent Document ID: 20160275417
Application ID: 15166609
Patent Status: 0

Claim One:
1. A method for determining confidence labels for crowdsourced annotations, comprising: obtaining a set of source data using a distributed data annotation server system; obtaining a set of training data using the distributed data annotation server system, where the set of training data comprises a subset of the source data representative of the set of source data; determining the ground truth for each piece of training data in the set of training data using the distributed data annotation server system, where the ground truth for a piece of data describes the content of the piece of data; obtaining a set of training data annotations from a plurality of data annotation devices using the distributed data annotation server system, where a training data annotation comprises a confidence label selected from a set of confidence labels describing an estimation of the content of a piece of training data in the set of training data; measuring annotator accuracy data for at least one piece of training data in the set of training data based on the ground truth for each piece of training source data and the set of training data annotations using the distributed data annotation server system, where annotator accuracy data describes the difficulty of determining the ground truth for a piece of training data; and automatically generating a set of confidence labels for the set of unlabeled data based on the measured annotator accuracy data and the set of annotator labels used using the distributed data annotation server system.