Patent Document ID: 20110145156
Application ID: 12639915
Patent Flag: 0

Claim One:
1. A computing system comprising: a validation module comprising: a submit validation project sub module configured to submit validation projects to an online crowdsourcing system, wherein each validation project has one or more human intelligence tasks (HITs) and has two validation control parameters, wherein the two validation control parameters are: a payment per human intelligence task (HIT); and a number of solutions per HIT; a quality sub module configured to compute an agreement probability of each validation project, wherein the agreement probability is a percentage of solutions that match references; a cost sub module configured to compute a cost of each validation project; a time sub module configured to compute a time that each validation project takes to complete; a selection sub module configured to: identify an eligible validation project from the validation projects, wherein the eligible validation project meets a goal; and select two validation control parameters from the eligible validation project to be two final control parameters; and a submission module configured to submit a final project along with the two final control parameters to the online crowdsourcing system.