Patent ID: 7890362

Claim:
A computer implemented method for selecting a sub-population of items with the highest probabilities of meeting a criterion based on evaluations of items by respondents, comprising: (A) providing a population of I items i, (B) presenting to each of a plurality of respondents one or more items, i, of said population for evaluation, (C) receiving from said plurality of respondents said evaluations, (D) based on said evaluations selecting a sub-population of items with the highest probability of meeting a criterion S, wherein said sub-population meeting said criterion is identified by: (i) for each respondent, for each i in I, calculating an estimated probability of meeting criterion S based on the evaluations of previous respondents; (ii) establishing a threshold estimated probability to divide the items in I into a group of m items with the highest estimated probabilities of meeting the criterion and a group of I−m items with the lowest probabilities of meeting said criterion; (iii) determining the lowest probability estimate, P , among the m items with probabilities above the threshold, and the highest probability estimate, p , among the items with probabilities below the threshold; (iv) computing a misclassification probability, t i , wherein for items above the threshold t i =Prob(p i ≦ p ), and for items below the threshold t i =Prob(p i ≧ p ); and (v) selecting the k items with the highest misclassification probabilities for evaluation by the next respondent; and (vi) repeating steps (i) through (v) until said sub-population is identified, and wherein steps (i) through (vi) are performed by a computer.