Patent Document ID: 9767419
Application ID: 14162962
Patent Flag: 1

Claim One:
1. A computer-implemented method of labeling items comprising: issuing, over a communications network, a labeling task to a plurality of workers participating in a crowdsourcing system; receiving, over the communications network, labels assigned to the issued labeling task by two or more of the plurality of workers; sending the received labels and task data to inference components arranged to jointly infer both worker characteristics and characteristics of worker communities of which the two or more workers are members, by assuming relationships between the two or more workers; and receiving inference results from the inference components, the inference results comprising an aggregated label for the labeling task, the aggregated label taking into account the inferred worker characteristics and the inferred characteristics of the worker communities, wherein the inference components are configured to further carry out an inference to update probability distributions representing belief about latent variables representing the characteristics of the worker communities, the worker characteristics and the aggregated label.