Patent ID: 8396718

Claim:
A method of evaluating a response that was inputted by a human participant, the method performed by a computing apparatus comprising storage and a processor, the method comprising: receiving and storing in the storage an assessment wherein the assessment comprises one selected from a group comprising a negative assessment and a positive assessment; creating, by the processor, and storing in the storage, a contingency table with four entries (n 1 , n 2 , n 3 , n 4 ) comparing: an authoritative positive assessment and an authoritative negative assessment to a positive assessment and a negative assessment; if the assessment is positive and an authoritative assessment is negative (n 2 ), determining, by the processor, a probability that the assessment is positive (p 1 ) by dividing a number of assessments that are positive (n 1 +n 2 ) by a total number of assessments (N); determining, by the processor, a probability that the authoritative assessment is negative (p 2 ) by dividing a number of authoritative entries that are negative (n 2 +n 4 ) by the total number of assessments (N); multiplying, by the processor, a probability that the assessment is positive (p 1 ) by the probability that the authoritative assessment is negative (p) by the total number of assessments (N) to determine an expected entry of a positive assessment and a negative authoritative assessment (pN); comparing, by the processor, the expected entry (pN) to the actual entry (n 2 ): if the expected entry (pN) and actual entry (n 2 ) are within a threshold of each other, determining that the assessment is independent; if the expected entry (pN) and actual entry (n 2 ) are different by more than the threshold, determining that the assessment is not independent; if the assessment is negative and the authoritative assessment is positive (n 3 ), determining, by the processor, a probability that the assessment is negative (r 1 ) by dividing the negative assessments (n 3 +n 4 ) by the total number of assessments (N); determining, by the processor, and storing in the storage, a probability that the authoritative assessment is positive (r 2 ) by dividing the number of authoritative entries that are positive (n 1 +n 3 ) by the total number of assessments (N); multiplying, by the processor, and storing in the storage, the probability that the assessment is negative (r 1 ) by a probability that the authoritative assessment is positive (r) by the total number of assessments (N) to determine an expected entry of a negative assessment and a positive authoritative assessment (rN); comparing the expected entry (rN) to the actual entry (n 3 ): if the expected entry (rN) and actual entry (n 3 ) are within the threshold of each other, determining that the assessment is fraudulent; and if the expected entry (rN) and actual entry (n 3 ) are different by more than the threshold, determining that the assessment is not fraudulent; and storing in the storage information indicating whether the assessment was determined to be fraudulent.