Patent Document ID: 8700418
Application ID: 12639915
Patent Flag: 1

Claim One:
1. A computing system comprising: one or more processors to execute instructions corresponding to one or more modules: one or more storage media coupled to the one or more processors to retain the instructions; and wherein the one or more modules comprises a validation module, the validation module comprising: a submit validation project sub module configured to submit validation projects to an online crowdsourcing system, wherein each validation project has one or more human intelligence tasks (HITs) and has two validation control parameters, wherein the two validation control parameters are: a payment per human intelligence task (HIT); and a number of solutions per HIT; a quality sub module configured to compute an agreement probability of each validation project, wherein the agreement probability is a percentage of solutions that match one or more references, and wherein the computing the agreement probability comprises: determining one or more sets of solutions that match one or more of the references; and comparing quantity information corresponding to the one or more sets of solutions to aggregate quantity information for solutions corresponding to the validation project; a cost sub module configured to compute a cost of each validation project; a time sub module configured to compute a time that each validation project takes to complete; a selection sub module configured to: identify an eligible validation project from the validation projects, wherein the eligible validation project meets a goal; and select two validation control parameters from the eligible validation project to be two final control parameters; and a submission module configured to submit a final project along with the two final control parameters to the online crowdsourcing system.