Patent ID: 9009675
Filing Date: 2015-04-14
Classification: G06Q

Abstract:
1. A method of implementing verification of a complex workflow, the complex workflow having a number of inputs in the form of one or more of data, signals and materials, and a set of processing steps that yield a number of desired outputs in the form of one or more of data, signals and materials, the method comprising: partitioning the workflow into modules, wherein the modules also have inputs, processing steps and outputs; selecting, from the workflow, one or more of the partitioned modules for independent verification by challenge thereof; wherein the challenge comprises a community-based challenge in which outputs from modules of external participants are compared with the outputs of the selected module and the reference outputs of the ideal performing module or well-established module; running, with a computing device, a challenge of the selected module, the challenge comprising comparing reference outputs to outputs of the selected module, wherein reference inputs are received by the selected module and the reference outputs are generated using the reference inputs and one of an ideal performing module or a well established module, wherein the challenge is implemented with at least one of a double blind approach, and crowd sourcing of a problem to a community; determining whether outputs of the selected module meet verification criteria with respect to the reference outputs, and based on the determining, implementing one of: declaring the selected module verified and in the event a best performer is determined to result from an external participant replacing the selected module in the workflow; subdividing the selected module into smaller modules and repeating the challenge on the smaller modules; or declaring the selected module not verified, wherein verification comprises a process of evaluating a system, component, product or research output to determine whether the module meets projected performance or accuracy goals.