Patent Document ID: 8914321
Application ID: 13757845

Base Claim:
1. A method for determining a plurality of inferred labels, for a corresponding plurality of items, wherein the inferred labels are generated from a plurality of judgments, wherein multiple judges select the judgments from a specified choice of labels for each of the items, the method comprising: determining a true label for each of the items based on the judgments; determining a characterization of labeling expertise for each judge based on the judgments; determining a labeling difficulty for each item based on the judgments; determining, using maximum entropy, a probability distribution over the specified choice of labels for each judge and item; selecting a plurality of improved labels for the items from the specified choice of labels such that the entropy over the probability distribution is reduced, wherein the improved labels represent an improvement from the judgments toward the true labels; and perform an iterative procedure wherein the true labels, the characterizations of labeling expertise, and the labeling difficulties are determined.

---

Claim 4:
4. The method of claim 1 , wherein the labeling difficulty for each of the items represents a relative difficulty in comparison to all other items.