Patent Document ID: 8868471
Application ID: 13239275
Patent Status: 1

Claim One:
1. A method for evaluating results of tasks performed by multiple judges, the method comprising: under control of a computer system comprising computer hardware: accessing from physical computer storage task result data, the task result data comprising judgments made by multiple judges for tasks having a plurality of outcomes, the task result data including overlapped data for pairs of judges in which each of the judges in the pair of judges has provided a judgment of the same task as provided by the other judge in the pair of judges; determining, from the overlapped data associated with each pair of judges, a number of overlapped judgments and a number of disagreements among the overlapped judgments for the pair of judges; calculating, for each of the judges, an individualized error rate based at least in part on the determined numbers of disagreements and the determined numbers of overlapped judgments, wherein calculating the individualized error is based at least in part on differences between: the determined number of disagreements between the pair of judges; and an estimated number of disagreements between the pair of judges, the estimated number of disagreements based at least in part on the determined number of overlapped judgments for the pair of judges and the individualized error rates for each of the judges in the pair of judges; generating, for each of the judges, a quality metric based at least in part on the individualized error rate calculated for the judge, the quality metric reflective of the quality of the judge's judgment of the tasks performed by the judge; and creating an electronic report in computer memory including the quality metric.