Patent ID: 8346685

Claim:
A method for enhancing expert-based processes when receiving input from a plurality of experts operating a corresponding plurality of computerized expert-based processes on a body of data, said data including an agreement set including at least one point of agreement regarding which all of said plurality of experts agree, the method comprising: determining a discrepancy set including at least one point of discrepancy regarding which less than all of the plurality of experts agree, said determining including performing a computerized comparison of input received from said plurality of experts thereby to identify points of discrepancy; providing at least a portion of said discrepancy set to an oracle and receiving oracle input from the oracle resolving at least said point of discrepancy; and selecting a subset of better computerized expert-based processes, from among said plurality of computerized expert-based processes, based on said oracle input, wherein said oracle input resolves less than all of a total number of points of discrepancy in said discrepancy set by randomly sampling a subset of the total number of points of discrepancy, and wherein each of the expert-based processes has a binary output defining first and second output values and wherein said discrepancy set includes a first set of points of discrepancy in which first and second experts from among said plurality of experts output the first and second output values respectively and a second set of points of discrepancy in which first and second experts from among said plurality of experts output the second and first output values respectively.