AI Platform System and Method

A computer-implemented method, computer program product and computing system for defining a test truth set from a master truth set; processing the test truth set using an automated analysis process to generate an automated result set; determining a process efficacy for the automated analysis process based, at least in part, upon the test truth set and the automated result set; and rendering the process efficacy of the automated analysis process.

TECHNICAL FIELD

This disclosure relates to platform systems and methods and, more particularly, to platform systems and methods concerning artificial intelligence and machine learning functionality.

BACKGROUND

Recent advances in the fields of artificial intelligence and machine learning are showing promising outcomes in the analysis of clinical content, examples of which may include medical imagery. Accordingly, processes and algorithms are constantly being developed that may aid in the processing and analysis of such medical imagery. Unfortunately, the efficacy of such processes and algorithms may be less than clear and an interested party may wish to determine how effective a particular process/algorithm is prior to licensing/purchasing the same. Further, the interested party may wish to compare a plurality of processes/algorithms prior to licensing/purchasing the same and/or monitor the continued temporal accuracy of any purchased processes/algorithms.

SUMMARY OF DISCLOSURE

In one implementation, a computer-implemented method is executed on a computing device and includes: processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets; determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies; and comparatively rendering the plurality of process efficacies.

One or more of the following features may be included. The test truth set may be defined from a master truth set. Defining the test truth set from a master truth set may include: enabling a user to define narrowing criteria for the master truth set; and applying the narrowing criteria to the master truth set to generate the test truth set, wherein the test truth set is a subset of the master truth set. The narrowing criteria may concern one or more of: content type; patient type; and anomaly type. The test truth set may include a plurality of medical images and a plurality of related human-generated reports. The plurality of automated result sets may each include a plurality of machine-generated reports. Processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets may include: processing the plurality of medical images using each of the plurality of automated analysis processes to generate the plurality of machine-generated reports included in the plurality of automated result sets, based upon the plurality of medical images. Determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies may include: comparing the plurality of related human-generated reports to each of the plurality of machine-generated reports. Comparatively rendering the plurality of process efficacies may include: textually comparatively rendering the plurality of process efficacies. Comparatively rendering the plurality of process efficacies may include: graphically comparatively rendering the plurality of process efficacies.

In another implementation, a computer program product resides on a computer readable medium and has a plurality of instructions stored on it. When executed by a processor, the instructions cause the processor to perform operations including: processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets; determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies; and comparatively rendering the plurality of process efficacies.

One or more of the following features may be included. The test truth set may be defined from a master truth set. Defining the test truth set from a master truth set may include: enabling a user to define narrowing criteria for the master truth set; and applying the narrowing criteria to the master truth set to generate the test truth set, wherein the test truth set is a subset of the master truth set. The narrowing criteria may concern one or more of: content type; patient type; and anomaly type. The test truth set may include a plurality of medical images and a plurality of related human-generated reports. The plurality of automated result sets may each include a plurality of machine-generated reports. Processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets may include: processing the plurality of medical images using each of the plurality of automated analysis processes to generate the plurality of machine-generated reports included in the plurality of automated result sets, based upon the plurality of medical images. Determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies may include: comparing the plurality of related human-generated reports to each of the plurality of machine-generated reports. Comparatively rendering the plurality of process efficacies may include: textually comparatively rendering the plurality of process efficacies. Comparatively rendering the plurality of process efficacies may include: graphically comparatively rendering the plurality of process efficacies.

In another implementation, a computing system includes a processor and a memory system configured to perform operations including: processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets; determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies; and comparatively rendering the plurality of process efficacies.

One or more of the following features may be included. The test truth set may be defined from a master truth set. Defining the test truth set from a master truth set may include: enabling a user to define narrowing criteria for the master truth set; and applying the narrowing criteria to the master truth set to generate the test truth set, wherein the test truth set is a subset of the master truth set. The narrowing criteria may concern one or more of: content type; patient type; and anomaly type. The test truth set may include a plurality of medical images and a plurality of related human-generated reports. The plurality of automated result sets may each include a plurality of machine-generated reports. Processing a test truth set using a plurality of automated analysis processes to generate a plurality of automated result sets may include: processing the plurality of medical images using each of the plurality of automated analysis processes to generate the plurality of machine-generated reports included in the plurality of automated result sets, based upon the plurality of medical images. Determining a process efficacy for each of the plurality of automated analysis processes based, at least in part, upon the test truth set and each of the plurality of automated result sets, thus defining a plurality of process efficacies may include: comparing the plurality of related human-generated reports to each of the plurality of machine-generated reports. Comparatively rendering the plurality of process efficacies may include: textually comparatively rendering the plurality of process efficacies. Comparatively rendering the plurality of process efficacies may include: graphically comparatively rendering the plurality of process efficacies.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

System Overview

Referring toFIG. 1, there is shown online platform process10. Online platform process10may be implemented as a server-side process, a client-side process, or a hybrid server-side/client-side process. For example, online platform process10may be implemented as a purely server-side process via online platform process10s. Alternatively, online platform process10may be implemented as a purely client-side process via one or more of online platform process10c1, online platform process10c2, online platform process10c3, and online platform process10c4. Alternatively still, online platform process10may be implemented as a hybrid server-side/client-side process via online platform process10sin combination with one or more of online platform process10c1, online platform process10c2, online platform process10c3, and online platform process10c4. Accordingly, online platform process10as used in this disclosure may include any combination of online platform process10s, online platform process10c1, online platform process10c2, online platform process10c3, and online platform process10c4. Examples of online platform process10may include but are not limited to all or a portion of the PowerShare™ platform and/or the PowerScribe™ platform available from Nuance Communications™ of Burlington, Mass.

Online platform process10smay be a server application and may reside on and may be executed by computing device12, which may be connected to network14(e.g., the Internet or a local area network). Examples of computing device12may include, but are not limited to: a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, or a cloud-based computing platform.

The instruction sets and subroutines of online platform process10s, which may be stored on storage device16coupled to computing device12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within computing device12. Examples of storage device16may include but are not limited to: a hard disk drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices.

Examples of online platform processes10c1,10c2,10c3,10c4may include but are not limited to a web browser, a game console user interface, a mobile device user interface, or a specialized application (e.g., an application running on e.g., the Android™ platform, the iOS™ platform, the Windows™ platform, the Linux™ platform or the UNIX™ platform). The instruction sets and subroutines of online platform processes10c1,10c2,10c3,10c4, which may be stored on storage devices20,22,24,26(respectively) coupled to client electronic devices28,30,32,34(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices28,30,32,34(respectively). Examples of storage devices20,22,24,26may include but are not limited to: hard disk drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices.

Examples of client electronic devices28,30,32,34may include, but are not limited to, a smartphone (not shown), a personal digital assistant (not shown), a tablet computer (not shown), laptop computers28,30,32, personal computer34, a notebook computer (not shown), a server computer (not shown), a gaming console (not shown), and a dedicated network device (not shown). Client electronic devices28,30,32,34may each execute an operating system, examples of which may include but are not limited to Microsoft Windows™, Android™, iOS™, Linux™, or a custom operating system.

Users36,38,40,42may access online platform process10directly through network14or through secondary network18. Further, online platform process10may be connected to network14through secondary network18, as illustrated with link line43.

The various client electronic devices (e.g., client electronic devices28,30,32,34) may be directly or indirectly coupled to network14(or network18). For example, laptop computer28and laptop computer30are shown wirelessly coupled to network14via wireless communication channels44,46(respectively) established between laptop computers28,30(respectively) and cellular network/bridge48, which is shown directly coupled to network14. Further, laptop computer32is shown wirelessly coupled to network14via wireless communication channel50established between laptop computer32and wireless access point (i.e., WAP)52, which is shown directly coupled to network14. Additionally, personal computer34is shown directly coupled to network18via a hardwired network connection.

WAP52may be, for example, an IEEE 802.11a, 802.11b, 802.11g, 802.11n, Wi-Fi, and/or Bluetooth device that is capable of establishing wireless communication channel50between laptop computer32and WAP52. As is known in the art, IEEE 802.11x specifications may use Ethernet protocol and carrier sense multiple access with collision avoidance (i.e., CSMA/CA) for path sharing. As is known in the art, Bluetooth is a telecommunications industry specification that allows e.g., mobile phones, computers, and personal digital assistants to be interconnected using a short-range wireless connection.

While the following discussion concerns medical imagery, this is for illustrative purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible and are considered to be within the scope of this disclosure. For example, the following discussion may concern any type of clinical content (e.g., DNA sequences, EKG results, EEG results, blood panel results, lab results, etc.) and/or non-medical content.

Assume for the following example that users36,38are medical service providers (e.g., radiologists) in two different medical facilities (e.g., hospitals, labs, diagnostic imaging centers, etc.). Accordingly and during the normal operation of these medical facilities, medical imagery may be generated by e.g., x-ray systems (not shown), MRI systems (not shown), CAT systems (not shown), PET systems (not shown) and ultrasound systems (not shown). For example, assume that user36generates medical imagery54and user38generates medical imagery56; wherein medical imagery54may be stored locally on storage device20coupled to laptop computer28and medical imagery56may be stored locally on storage device22coupled to laptop computer30. When locally storing medical imagery54,56, this medical imagery may be stored within e.g., a PACS (i.e., Picture Archiving and Communication System). Additionally/alternatively, the medical imagery (e.g., medical imagery54,56) may be stored on a cloud-based storage system (e.g., a cloud-based storage system (not shown) included within online platform58).

Online platform process10may enable online platform58that may be configured to allow for the offering of various medical diagnostic services to users (e.g., users36,38) of online platform58. For the following example, assume that user40is a medical research facility (e.g., the ABC Center) that performs cancer research. Assume that user40produced a process (e.g., automated analysis process60) that analyzes medical imagery to identify anomalies that may be cancer. Examples of automated analysis process60may include but are not limited to an application or an algorithm that may process medical imagery (e.g., medical imagery54and medical imagery56), wherein this application/algorithm may utilize artificial intelligence, machine learning and/or probabilistic modeling when analyzing the medical imagery (e.g., medical imagery54and medical imagery56). Examples of such probabilistic modeling may include but are not limited to discriminative modeling (e.g., a probabilistic model for only the content of interest), generative modeling (e.g., a full probabilistic model of all content), or combinations thereof.

Further assume that user42is a medical research corporation (e.g., the XYZ Corporation) that produces applications/algorithms (e.g., automated analysis process62) that analyze medical imagery to identify anomalies that may be cancer. Examples of automated analysis process62may include but are not limited to an application or an algorithm that may process medical imagery (e.g., medical imagery54and medical imagery56), wherein this application/algorithm may utilize artificial intelligence, machine learning algorithms and/or probabilistic modeling when analyzing the medical imagery (e.g., medical imagery54and medical imagery56). Examples of such probabilistic modeling may include but are not limited to discriminative modeling (e.g., a probabilistic model for only the content of interest), generative modeling (e.g., a full probabilistic model of all content), or combinations thereof.

Assume for the following example that user40(i.e., the ABC Center) wishes to offer automated analysis process60to others (e.g., users36,38) so that users36,38may use automated analysis process60to process their medical imagery (e.g., medical imagery54and medical imagery56, respectively). Further assume that user42(i.e., the XYZ Corporation) wishes to offer automated analysis process62to others (e.g., users36,38) so that users36,38may use automated analysis process62to process their medical imagery (e.g., medical imagery54and medical imagery56, respectively).

Accordingly, online platform process10and online platform58may allow user40(i.e., the ABC Center) and/or user42(i.e., the XYZ Corporation) to offer automated analysis process60and/or automated analysis process62(respectively) for use by e.g., user36and/or user38. Therefore, online platform process10and online platform58may be configured to allow user40(i.e., the ABC Center) and/or user42(i.e., the XYZ Corporation) to upload a remote copy of automated analysis process60and/or automated analysis process62to online platform58, resulting in automated analysis process60and/or automated analysis process62(respectively) being available for use via online platform58.

Generally speaking, online platform process10may offer a plurality of computer-based medical diagnostic services (e.g., automated analysis process60,62) within the online platform (e.g., online platform58), wherein online platform process10may identify the computer-based medical diagnostic services (e.g., automated analysis process60,62) that are available via online platform58and users (e.g., user36,38) may utilize these computer-based medical diagnostic services (e.g., automated analysis process60,62) to process the medical imagery (e.g., medical imagery54and medical imagery56).

Referring also toFIG. 2, online platform process10may define100a test truth set (e.g., test truth set64) from a master truth set (e.g., master truth set66), wherein the test truth set (e.g., test truth set64) may include a plurality of medical images (e.g., plurality of medical images68) and a plurality of related human-generated reports (e.g., plurality of related human-generated reports70).

As will be discussed below in greater detail, this test truth set (e.g., test truth set64) may be used by user36and/or user38to research the available computer-based medical diagnostic services (e.g., automated analysis process60,62) to determine which (if any) of these services they would like to e.g., purchase/license/subscribe to.

Generally speaking, master truth set (e.g., master truth set66) may include/have access to a massive quantity of (in his example) medical images, wherein these medical images may have been reviewed by medical professionals (e.g., radiologists). Medical reports concerning the findings of these medical professionals (e.g., radiologists) with respect to these medical images may be generated, resulting in related human-generated reports. This combination of medical images and related human-generated reports may form the master truth set (e.g., master truth set66), from which test truth set64(which includes plurality of medical images68and plurality of related human-generated reports70) may be defined 100.

For example, plurality of medical images68may include an x-ray of the chest of a patient and plurality of related human-generated reports70may include a related report that discusses an anomaly within the x-ray that is identified as lung cancer. Additionally, plurality of medical images68may include a CT scan of the head of a patient and plurality of related human-generated reports70may include a related report that discusses an anomaly within the CT scan that is identified as an intercranial hemorrhage. Further, plurality of medical images68may include an MRI scan of the ankle of a patient and plurality of related human-generated reports70may include a related report that discusses an anomaly within the MRI scan that is identified as a broken fibula.

While the following discussion concerns the processing of medical imagery, this is for illustrative purposes only and is not intended to be a limitation of this disclosure, as other configurations are possible and are considered to be within the scope of this disclosure. For example, other types of medical information may be processed, such as DNA sequences, EKG results, EEG results, blood panel results, lab results, etc. Additionally, other types of information may be processed that need not be medical in nature. Accordingly and with respect to this disclosure, master truth set66may include any type of content for which automated processing may be applicable, such as medical data, financial records, personal records, and identification information.

When defining100the test truth set (e.g., test truth set64) from the master truth set (e.g., master truth set66), online platform process10may enable102a user (e.g., user36or user38) to define narrowing criteria (e.g., narrowing criteria72) for the master truth set (e.g., master truth set66), wherein the narrowing criteria (e.g., narrowing criteria72) may concern one or more of: content type; patient type; and anomaly type (as will be discussed below). Further and when defining100the test truth set (e.g., test truth set64) from the master truth set (e.g., master truth set66), online platform process10may apply104the narrowing criteria (e.g., narrowing criteria72) to the master truth set (e.g., master truth set66) to generate the test truth set (e.g., test truth set64), wherein the test truth set (e.g., test truth set64) may be a subset of the master truth set (e.g., master truth set66).

Referring also toFIG. 3, when defining100the test truth set (e.g., test truth set64) from the master truth set (e.g., master truth set66), online platform process10may render user interface200that may enable102a user (e.g., user36or user38) to define narrowing criteria (e.g., narrowing criteria72) and may apply104the narrowing criteria (e.g., narrowing criteria72) to the master truth set (e.g., master truth set66) to generate the test truth set (e.g., test truth set64). As will be discussed below, the test truth set (e.g., test truth set64) may be a subset of the master truth set (e.g., master truth set66).

For example, assume that the master truth set (e.g., master truth set66) includes 10,163,279 medical images and, therefore, 10,163,279 related human-generated reports. Further assume that e.g., user36works at a medical facility that specializes in pediatric neurological issues, wherein user36wishes to research the available computer-based medical diagnostic services (e.g., automated analysis process60,62) to determine which (if any) of these services is suitable for pediatric neurological issues. As the master truth set (e.g., master truth set66) includes 10,163,279 medical images/10,163,279 related human-generated reports that may (or may not) concern pediatric neurological issues, user36may start to enter narrowing criteria72that may whittle away at master truth set66to define a test truth set (e.g., test truth set64) that is related to/pertinent for pediatric neurological issues.

Accordingly, user36may enter narrowing criteria72that includes:“MRI Scan”, as the facility in which user36works is only interested in the processing of MRI images. This in turn reduces the 10,163,279 medical images/related human-generated reports to 5,623,123 medical images/related human-generated reports.“General Electric MRI System”, as the facility in which user36works uses a General Electric MRI machine. This in turn reduces the 5,623,123 medical images/related human-generated reports to 1,623,721 medical images/related human-generated reports.“Head”, as the facility in which user36works focuses on neurological issues. This in turn reduces the 1,623,721 medical images/related human-generated reports to 80,321 medical images/related human-generated reports.“Child (12 and Younger)”, as the facility in which user36works focuses on pediatric issues. This in turn reduces the 80,321 medical images/related human-generated reports to 3,279 medical images/related human-generated reports.“Cancer”, as the facility in which user36works focuses on cancerous tumors. This in turn reduces the 3,279 medical images/related human-generated reports to 362 medical images/related human-generated reports.

Accordingly and through narrowing criteria72, a master truth set (e.g., master truth set66) that includes 10,163,279 medical images/related human-generated reports may be whittled down to a test truth set (e.g., test truth set64) that is focused on pediatric neurological issues and includes 362 medical images/related human-generated reports (selected from the 10,163,279 medical images/related human-generated reports included within master truth set66). Accordingly and as stated above, test truth set64may be a subset of master truth set66.

Online platform process10may process106the test truth set (e.g., test truth set64) using an automated analysis process (e.g., automated analysis process60or automated analysis process62) to generate an automated result set (e.g., automated result set74). The automated result set (e.g., automated result set74) may include a plurality of machine-generated reports (e.g., machine-generated reports76).

Continuing with the above-stated example, assume that user36is interested in automated analysis process60offered by the ABC Center (i.e., a cancer research medical facility) . . . but is uncertain as to the manner in which it will perform with respect to pediatric neurological issues. Accordingly and through the use of test truth set64(which was curated toward e.g., MRI scans made on General Electric MRI machines that concern pediatric brain cancer), the performance (e.g., accuracy/efficacy) of automated analysis process60may be scrutinized.

Accordingly and when processing106the test truth set (e.g., test truth set64) using an automated analysis process (e.g., automated analysis process60) to generate an automated result set (e.g., automated result set74), online platform process10may process108the plurality of medical images (e.g., plurality of medical images68) using the automated analysis process (e.g., automated analysis process60) to generate the plurality of machine-generated reports (e.g., plurality of machine-generated reports76), based upon the plurality of medical images (e.g., plurality of medical images68).

Generally speaking and if automated analysis process60is 100% accurate, the plurality of machine-generated reports (e.g., plurality of machine-generated reports76) should reach the same conclusion(s) as the plurality of related human-generated reports (e.g., plurality of related human-generated reports70), as both sets of reports are based upon the same plurality of medical images (e.g., plurality of medical images68).

Therefore, online platform process10may determine110a process efficacy (e.g., process efficacy78) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the test truth set (e.g., test truth set64) and the automated result set (e.g., automated result set74).

For example and when determining110the process efficacy (e.g., process efficacy78) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the test truth set (e.g., test truth set64) and the automated result set (e.g., automated result set74), online platform process10may compare112the plurality of related human-generated reports (e.g., plurality of related human-generated reports70) to the plurality of machine-generated reports (e.g., plurality of machine-generated reports76). Specifically, the higher the level of correlation between plurality of related human-generated reports70and plurality of machine-generated reports76, the hire the level of efficacy of (in this example) automated analysis process60.

Accordingly and in this example, once user36defines narrowing criteria72, user36may select “Run Analysis” button202, resulting in online platform process10processing106test truth set64(which includes 362 medical images/related human-generated reports) using automated analysis process60to generate automated result set74; thus allowing online platform process10to determine110process efficacy78for automated analysis process60based, at least in part, upon test truth set64and automated result set74.

Online platform process10may render114the process efficacy (e.g., process efficacy78) of the automated analysis process (e.g., automated analysis process60).

For example and when rendering114the process efficacy (e.g., process efficacy78) of the automated analysis process (e.g., automated analysis process60), online platform process10may textually render116the process efficacy (e.g., process efficacy78) of the automated analysis process (e.g., automated analysis process60).

In this particular illustrative example and as shown within result window204of user interface200, efficacy78is shown to be 93.37%, in that automated analysis process60produced 338 machine-generated reports (out of a total of 362 machine-generated reports) that reached the same conclusion(s) as the corresponding human-generated report within truth set64.

Concerning the 338 accurate results, 173 of the 338 results (i.e., 51.18%) were deemed to be “True Positives”, wherein an anomaly was detected and was properly identified as being malignant; and 165 of the 338 results (i.e., 48.82%) were deemed to be “True Negatives”, wherein an anomaly was detected and was properly identified as being benign.

Concerning the 24 inaccurate results, 19 of the 24 results (i.e., 79.10%) were deemed to be “False Positives”, wherein an anomaly was detected and was improperly identified as being malignant; and 5 of the 24 results (i.e., 20.90%) were deemed to be “False Negatives”, wherein an anomaly was detected and was improperly identified as being benign.

Additionally/alternatively and when rendering114the process efficacy (e.g., process efficacy78) of the automated analysis process (e.g., automated analysis process60), online platform process10may graphically render118the process efficacy (e.g., process efficacy78) of the automated analysis process (e.g., automated analysis process60). For example, online platform process10may graphically render118a multi-axis spider plot (e.g., graph206) within user interface200that visually identifies True Positives, True Negatives, False Positives, and False Negatives with respect to process efficacy78of automated analysis process60.

As will be discussed below in greater detail, online platform process10may allow a user (e.g., user36) to compare the performance of multiple computer-based medical diagnostic services (e.g., automated analysis process60,62) in order to enable the user to determine which (if any) of these services they would like to e.g., purchase/license/subscribe to.

As discussed above and referring also toFIG. 4, online platform process10may define100the test truth set (e.g., test truth set64) from a master truth set (e.g., master truth set66), wherein the test truth set (e.g., test truth set64) may include a plurality of medical images (e.g., plurality of medical images68) and a plurality of related human-generated reports (e.g., plurality of related human-generated reports70).

As also discussed above, when defining100the test truth set (e.g., test truth set64) from a master truth set (e.g., master truth set66), online platform process10may enable102a user (e.g., user36or user38) to define narrowing criteria (e.g., narrowing criteria72) for the master truth set (e.g., master truth set66) and apply104the narrowing criteria (e.g., narrowing criteria72) to the master truth set (e.g., master truth set66) to generate the test truth set (e.g., test truth set64), wherein the test truth set (e.g., test truth set64) is a subset of the master truth set (e.g., master truth set66). The narrowing criteria (e.g., narrowing criteria72) may concern one or more of: content type; patient type; and anomaly type.

Suppose for this example that the user (e.g., user36) is interested in both computer-based medical diagnostic services (e.g., automated analysis process60,62) but does not know which (if any) of these services to e.g., purchase/license/subscribe to.

Accordingly, online platform process10may process300the test truth set (e.g., test truth set64) using a plurality of automated analysis processes (e.g., automated analysis processes60,62) to generate a plurality of automated result sets (e.g., plurality of automated result sets80), wherein the plurality of automated result sets (e.g., plurality of automated result sets80) may each include a plurality of machine-generated reports (an example of which is machine-generated reports76included within automated result set74).

When processing300a test truth set (e.g., test truth set64) using a plurality of automated analysis processes (e.g., automated analysis processes60,62) to generate a plurality of automated result sets (e.g., automated result sets78), online platform process10may process302the plurality of medical images (e.g., plurality of medical images68) using each of the plurality of automated analysis processes (e.g., automated analysis processes60,62) to generate the plurality of machine-generated reports (an example of which is machine-generated reports76included within automated result set74) included in the plurality of automated result sets (e.g., automated result sets80), based upon the plurality of medical images (e.g., plurality of medical images68).

In this situation, being two automated analysis processes (e.g., automated analysis processes60,62) are being evaluated by user36, the plurality of automated result sets (e.g., plurality of automated result sets80) may include two automated result sets, namely: automated result set74which includes machine-generated reports76that were generated using automated analysis process60; and automated result set82which includes machine-generated reports84that were generated using automated analysis processes62.

In a similar fashion as described above, online platform process10may determine304a process efficacy (e.g., process efficacy78) for each of the plurality of automated analysis processes (e.g., automated analysis processes60,62) based, at least in part, upon the test truth set (e.g., test truth set64) and each of the plurality of automated result sets (e.g., automated result set74for automated analysis process60and automated result set82for automated analysis processes62), thus defining a plurality of process efficacies (as will be discussed below).

When determining304a process efficacy (e.g., process efficacy78) for each of the plurality of automated analysis processes (e.g., automated analysis processes60,62) based, at least in part, upon the test truth set (e.g., test truth set64) and each of the plurality of automated result sets (e.g., automated result set74for automated analysis process60and automated result set82for automated analysis processes62), thus defining a plurality of process efficacies (as will be discussed below), online platform process10may compare306the plurality of related human-generated reports (e.g., plurality of related human-generated reports70) to each of the plurality of machine-generated reports.

Specifically and when determining304a process efficacy for automated analysis process60, online platform process10may compare306plurality of related human-generated reports70to machine-generated reports76that are included within automated result set74that was generated using automated analysis process60. Further and when determining304a process efficacy for automated analysis process62, online platform process10may compare306plurality of related human-generated reports70to machine-generated reports84that are included within automated result set82that was generated using automated analysis process62.

Referring also toFIG. 5, assume that user36enters the same narrowing criteria72, namely:“MRI Scan”, as the facility in which user36works is only interested in the processing of MRI images. This in turn reduces the 10,163,279 medical images/related human-generated reports to 5,623,123 medical images/related human-generated reports.“General Electric MRI System”, as the facility in which user36works uses a General Electric MRI machine. This in turn reduces the 5,623,123 medical images/related human-generated reports to 1,623,721 medical images/related human-generated reports.“Head”, as the facility in which user36works focuses on neurological issues. This in turn reduces the 1,623,721 medical images/related human-generated reports to 80,321 medical images/related human-generated reports.“Child (12 and Younger)”, as the facility in which user36works focuses on pediatric issues. This in turn reduces the 80,321 medical images/related human-generated reports to 3,279 medical images/related human-generated reports.“Cancer”, as the facility in which user36works focuses on cancerous tumors. This in turn reduces the 3,279 medical images/related human-generated reports to 362 medical images/related human-generated reports.

As discussed above and through narrowing criteria72, a master truth set (e.g., master truth set66) that includes 10,163,279 medical images/related human-generated reports may be whittled down to a test truth set (e.g., test truth set64) that is focused on pediatric neurological issues and includes 362 medical images/related human-generated reports (selected from the 10,163,279 medical images/related human-generated reports included within master truth set66).

Once user36defines narrowing criteria72, user36may select “Run Analysis” button202, resulting in online platform process10processing300test truth set64(which includes 362 medical images/related human-generated reports) using automated analysis process60and automated analysis process62to generate automated result set74that was generated using automated analysis process60and automated result set82that was generated using automated analysis process62, thus allowing online platform process10to determine304a process efficacy (e.g., process efficacies78,400) for each of the plurality of automated analysis processes (e.g., automated analysis processes60,62) based, at least in part, upon the test truth set (e.g., test truth set64) and each of the plurality of automated result sets (e.g., automated result set74for automated analysis process60and automated result set82for automated analysis processes62), thus defining a plurality of process efficacies (e.g., plurality of process efficiencies78,400).

Online platform process10may comparatively render308the plurality of process efficacies (e.g., plurality of process efficiencies78,400). For example and when comparatively rendering308the plurality of process efficacies (e.g., process efficacies78,400), online platform process10may textually comparatively render310the plurality of process efficacies (e.g., plurality of process efficiencies78,400).

In this particular illustrative example and as shown within result window204of user interface200and with respect to automated analysis process60, efficacy78is shown to be 93.37%, in that automated analysis process60produced 338 machine-generated reports (out of a total of 362 machine-generated reports) that reached the same conclusion(s) as the corresponding human-generated report within truth set64.

Concerning the 338 accurate results, 173 of the 338 results (i.e., 51.18%) were deemed to be “True Positives”, wherein an anomaly was detected and was properly identified as being malignant; and 165 of the 338 results (i.e., 48.82%) were deemed to be “True Negatives”, wherein an anomaly was detected and was properly identified as being benign.

Concerning the 24 inaccurate results, 19 of the 24 results (i.e., 79.10%) were deemed to be “False Positives”, wherein an anomaly was detected and was improperly identified as being malignant; and 5 of the 24 results (i.e., 20.90%) were deemed to be “False Negatives”, wherein an anomaly was detected and was improperly identified as being benign.

In this particular illustrative example and as shown within result window402of user interface200and with respect to automated analysis process62, efficacy400is shown to be 90.33%, in that automated analysis process60produced 327 machine-generated reports (out of a total of 362 machine-generated reports) that reached the same conclusion(s) as the corresponding human-generated report within truth set64.

Concerning the 327 accurate results, 170 of the 327 results (i.e., 51.98%) were deemed to be “True Positives”, wherein an anomaly was detected and was properly identified as being malignant; and 157 of the 327 results (i.e., 48.02%) were deemed to be “True Negatives”, wherein an anomaly was detected and was properly identified as being benign.

Concerning the 35 inaccurate results, 17 of the 35 results (i.e., 48.57%) were deemed to be “False Positives”, wherein an anomaly was detected and was improperly identified as being malignant; and 18 of the 35 results (i.e., 51.43%) were deemed to be “False Negatives”, wherein an anomaly was detected and was improperly identified as being benign.

When comparatively rendering308the plurality of process efficacies, online platform process10may graphically comparatively render312the plurality of process efficacies (e.g., plurality of process efficiencies78,400). For example, online platform process10may graphically comparatively render312a multi-axis spider plot (e.g., graph206) within user interface200that visually identifies True Positives, True Negatives, False Positives, and False Negatives with respect to process efficacy78of automated analysis process60. Further, online platform process10may graphically comparatively render312a multi-axis spider plot (e.g., graph404) within user interface200that visually identifies True Positives, True Negatives, False Positives, and False Negatives with respect to process efficacy400of automated analysis process62.

As will be discussed below in greater detail, online platform process10may allow a user (e.g., user36) to monitor the performance of a computer-based medical diagnostic service (e.g., automated analysis process60,62) over time to enable the user to determine how the efficacy of the computer-based medical diagnostic service (e.g., automated analysis process60,62) changes over time (if at all).

As discussed above and referring also toFIG. 6, online platform process10may define100the test truth set (e.g., test truth set64) from a master truth set (e.g., master truth set66). wherein the test truth set (e.g., test truth set64) may include a plurality of medical images (e.g., plurality of medical images68) and a plurality of related human-generated reports (e.g., plurality of related human-generated reports70).

As also discussed above, when defining100the test truth set (e.g., test truth set64) from a master truth set (e.g., master truth set66), online platform process10may enable102a user (e.g., user36or user38) to define narrowing criteria (e.g., narrowing criteria72) for the master truth set (e.g., master truth set66); and apply102the narrowing criteria (e.g., narrowing criteria72) to the master truth set (e.g., master truth set66) to generate the test truth set (e.g., test truth set64), wherein the test truth set (e.g., test truth set64) is a subset of the master truth set (e.g., master truth set66). The narrowing criteria (e.g., narrowing criteria72) may concern one or more of: content type; patient type; and anomaly type.

Suppose for this example that the user (e.g., user36) purchased/licensed/subscribed to automated analysis process60and would like to know if the efficacy of automated analysis process60“ages” well. As discussed above and with respect to automated analysis process60, efficacy78was initially determined to be 93.37%. However and as is known in the art, computer-based medical diagnostic services are continuously learning/evolving based upon additional data that is used to train the computer-based medical diagnostic services. Accordingly, it is foreseeable that the efficacy of a computer-based medical diagnostic service may degrade if bad data is used to train the computer-based medical diagnostic service.

Accordingly and in order to monitor such long-term efficacy and the evolvement of the same, online platform process10may iteratively process500a test truth set (e.g., test truth set64) using an automated analysis process (e.g., automated analysis process60) to generate a plurality of temporarily-spaced automated result sets (e.g., plurality of automated result sets80).

When iteratively processing500a test truth set (e.g., test truth set64) using an automated analysis process (e.g., automated analysis process60) to generate a plurality of temporarily-spaced automated result sets (e.g., plurality of automated result sets80), online platform process10may iteratively process502the plurality of medical images (e.g., plurality of medical images68) using the automated analysis process (e.g., automated analysis process60) to generate the plurality of temporarily-spaced machine-generated reports included in the plurality of temporarily-spaced automated result sets (e.g., plurality of automated result sets80), based upon the plurality of medical images (e.g., plurality of medical images68).

As discussed above, each of the automated result sets (e.g., automated result set74) includes a plurality of machine-generated reports (e.g., machine-generated reports76). Accordingly, the plurality of temporarily-spaced automated result sets (e.g., plurality of automated result sets80) may each include a plurality of temporarily-spaced machine-generated reports.

Online platform process10may iteratively determine504a process efficacy (e.g., process efficacy78) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the test truth set (e.g., test truth set64) and the plurality of temporarily-spaced automated result sets (e.g., plurality of automated result sets80), thus defining a plurality of temporarily-spaced process efficacies (as will be discussed below).

Accordingly, online platform process10may iteratively process502the plurality of medical images (e.g., plurality of medical images68) using the automated analysis process (e.g., automated analysis process60) at a period of e.g., once every three months, thus generating one temporarily-spaced automated result set every three months. Importantly, the same test truth set (e.g., test truth set64) is used by automated analysis process60to generate each of these temporarily-spaced automated result sets (e.g., plurality of automated result sets80).

Online platform process10may then iteratively determine504a process efficacy (e.g., process efficacy78) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the test truth set (e.g., test truth set64) and each of these temporarily-spaced automated result sets (e.g., plurality of automated result sets80), thus defining (in this example) a series of temporarily-spaced process efficacies that define the manner in which these efficacies changes with respect to time (i.e., in three month intervals in this example).

For this particular example and referring also toFIG. 7, online platform process10may iteratively determine504a process efficacy for automated analysis process60once every three months (from Q12020through Q42021), resulting in the generation of eight temporarily-spaced process efficacies (namely temporarily-spaced process efficacies600(for Q12020),602(for Q22020),604(for Q32020),606(for Q42020),608(for Q12021),610(for Q22021),612(for Q32021),614(for Q42021) rendered within result screen616of user interface200. Result screen616may also include a change/trend indicator for each of temporarily-spaced process efficacies (namely trend indicator618,620,622,624,626,628,630,632, respectively).

Additionally, such plurality of temporarily-spaced process efficacies (e.g., temporarily-spaced process efficacies600,602,604,606,608,610,612,614) may be displayed graphical in the form of time-based graph634for (in this example) user36.

Online platform process10may determine506a long-term efficacy (e.g. long term efficacy636) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the plurality of temporarily-spaced process efficacies (e.g., temporarily-spaced process efficacies600,602,604,606,608,610,612,614). In this particular example, the long-term efficacy (e.g. long term efficacy636) for the automated analysis process (e.g., automated analysis process60) is shown to be the percentage increase over the monitored period (e.g., Q12020through Q42021). However, online platform process10may monitor many different things and express long term efficacy636many different ways.

For example and when determining506a long-term efficacy (e.g. long term efficacy636) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the plurality of temporarily-spaced process efficacies600,602,604,606,608,610,612,614), online platform process10may monitor508the plurality of temporarily-spaced process efficacies600,602,604,606,608,610,612,614) to define an efficacy trend (e.g., upward, downward, stable) for the automated analysis process (e.g., automated analysis process60or automated analysis process62).

Further and when determining506a long-term efficacy (e.g. long term efficacy636) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the plurality of temporarily-spaced process efficacies600,602,604,606,608,610,612,614), online platform process10may confirm510that the efficacy trend is stable/trending upward (as shown inFIG. 7).

Additionally and when determining506a long-term efficacy (e.g. long term efficacy636) for the automated analysis process (e.g., automated analysis process60) based, at least in part, upon the plurality of temporarily-spaced process efficacies600,602,604,606,608,610,612,614), online platform process10may confirm512that the efficacy trend is not trending downward and, in the event of such a downward trend, user36(in this example) may be notified.

General