Patent Description:
Machine learning or artificial intelligence is being applied in the area of medical research, particularly in areas of research where large amounts of data need to be collected and analyzed. Using adaptive redundant algorithms, machine learning is able to analysis thousands to millions of individual data sets and detect patterns within these large data sets that normally would be impossible or impractical to analyze given the scope.

Age-related macular degeneration (AMD) is the leading cause of visual impairment for people over age <NUM> in the developed world. In the retina, the macula is where the eye processes detailed vision. As part of the complex anatomic features of AMD, deposits known as drusen form between the retinal pigment epithelium (RPE) and Bruch's membrane. However, this is only one of a myriad of anatomic pathologies that are associated with AMD. While there is no known cure for AMD, some treatments have shown promise in slowing disease progression. One of the major obstacles to the development of an effective treatment is the inability to detect objective criteria that would either herald further progression of the disease or identify aspects that suggest improvement in patients undergoing treatment.

Optical coherence tomography (OCT) is a key tool in the diagnosis and treatment evaluation of AMD. OCT is a non-invasive imaging method utilizing light to provide high-resolution cross-sections of the neurosensory retina. OCT measures light waves reflected and scattered by the retinal tissue. Analysis of the cross-sections allows researchers to identify drusen and other anatomic features of AMD and would potentially allow researchers to monitor the progression of AMD in response to treatment. <CIT> discloses a system and method for medical diagnostic peer review using discrete containerised automated image processing engines and statistical methods of quality assurance and iterative image processing engine training. <CIT> relates to a transformational imaging and simulation platform that extends photorealism to healthcare and other uses. , discloses a predictive model using optical coherence tomography images for the prediction age-related macular degeneration.

Embodiments of the present disclosure provide for analyzing an image using machine learning and human computation.

In one embodiment, a method for analyzing an image is provided, as claimed in claims <NUM> to <NUM>.

In another embodiment, a system for analyzing an image is provided, as claimed in claims <NUM> to <NUM>.

In yet another embodiment, a non-transitory, computer-readable medium for analyzing an image is provided, as claimed in claims <NUM> to <NUM>.

Various embodiments of the present disclosure recognize that automated image evaluation procedures exist and continue to improve but have yet to provide a comprehensive solution, and often require verification or modification by a trained evaluator. Embodiments of the present disclosure further recognize the technical problem that automated solutions frequently misidentify anatomic boundaries, particularly with deformations such as, for example, those created by drusen as well as anatomic features that represent degenerative features of the retina. While research continues into improving automated techniques, trained OCT evaluators are often employed to manually evaluate or correct the analysis. Embodiments of the present disclosure recognize that medical imaging, such as OCT, has become widespread, with millions of scans performed per year, and the number expected to continue to increase with the large aging population. Embodiments of the present disclosure recognize that a limited number of trained image evaluators exist and demand for new evaluation approaches is increasing.

Various embodiments of the present disclosure recognize that human computing has proven to be an effective way to crowdsource a variety of scientific problems, as well as leverage human pattern-recognition ability. Human computing can be used to collect data (e.g., gathering or generating samples), perform human calculation problems (such as solving challenges as a replacement for all or part of an algorithm), or provide human analysis (e.g., classification, data verification). Embodiments of the present disclosure recognize that video games allow users to interact with the scientific data while also leveraging the elements game developers require to maintain engagement.

Accordingly, various embodiments of the present disclosure provide image analysis solutions that utilize video games to gain human intuition and leverage that human intuition in connection with machine learning to correctly identify structures (or the absence thereof) in an image.

<FIG> illustrates an example networked system <NUM> in which various embodiments of the present disclosure may be implemented. The embodiment of the networked system <NUM> shown in <FIG> is for illustration only. Other embodiments of the networked system <NUM> could be used without departing from the scope of this disclosure.

As shown in <FIG>, the system <NUM> includes a network <NUM>, which facilitates communication between various components in the system <NUM>. For example, the network <NUM> may communicate Internet Protocol (IP) packets or other information between network addresses. The network <NUM> may include one or more local area networks (LANs); metropolitan area networks (MANs); wide area networks (WANs); a virtual private network (VPN); all or a portion of a global network, such as the Internet; or any other communication system or systems at one or more locations.

The network <NUM> facilitates communications among various computer systems <NUM>-<NUM> and various user computing devices <NUM>-<NUM>. Each of the computer systems <NUM>-<NUM> may be any suitable electronic computing or processing device(s) that can provide computing services including software for one or more user computing devices <NUM>-<NUM>. Each of the computer systems <NUM>-<NUM> could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network <NUM>. For example, as discussed in greater detail below, computer system <NUM> may be a computer system for a researcher or data processor that is used to run machine learning algorithms that leverage human intuition to correctly identify structures (or the absence thereof) in an image. Additionally, computer system <NUM> may be associated with a medical provider or researcher to supply images for analysis and receive the results thereof.

Each user computing device <NUM>-<NUM> represents any suitable electronic computing or processing device that interacts with at least one computer system or other computing device(s) over the network <NUM>. In this example, the user computing devices <NUM>-<NUM> include a desktop computer <NUM>, a mobile telephone or smartphone <NUM>, a laptop computer <NUM>, a video game console <NUM>; a set-top box and/or television, etc. However, any other or additional user computing devices could be used in the networked system <NUM>. For example, any Internet or network connectable device or Internet of Things (IoT) device (e.g., Smart TVs, refrigerators, Raspberry PIs, etc.) could be used for one of the user computing devices <NUM>-<NUM> in system <NUM>. As discussed below, in various embodiments, user computing devices <NUM>-<NUM>, may be employed to provide a video game that drives user engagement to provide human intuition to help identify structures in images.

Although <FIG> illustrates one example of a networked system <NUM>, various changes may be made to <FIG>. For example, the system <NUM> could include any number of each component in any suitable arrangement and each of computer systems <NUM>-<NUM> and user computing devices <NUM>-<NUM> may be representative of any number of computer systems and/or user computing devices that are part of system <NUM>. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> and <FIG> illustrate example computing devices in a networked system according to various embodiments of the present disclosure. In particular, <FIG> illustrates an example computer system <NUM>, and <FIG> illustrates an example user computing device <NUM>. In this illustrative example, the computer system <NUM> represents any one of the computer systems <NUM>-<NUM> in <FIG>, and the user computing device <NUM> could represent one or more of the user computing devices <NUM>-<NUM> in <FIG>.

As shown in <FIG>, the computer system <NUM> includes a bus system <NUM>, which supports communication between processor(s) <NUM>, storage devices <NUM>, communication interface (or circuit) <NUM>, and input/output (I/O) unit <NUM>. The processor(s) <NUM> executes instructions that may be loaded into a memory <NUM>. The processor(s) <NUM> may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s) <NUM> include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The memory <NUM> and a persistent storage <NUM> are examples of storage devices <NUM>, which represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory <NUM> may represent a random-access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage <NUM> may contain one or more components or devices supporting longer-term storage of data, such as a read-only memory, hard drive, Flash memory, or optical disc. For example, persistent storage <NUM> may store images for analysis, human computational data <NUM> (such as training data representing human intuition inputs), and/or a machine learning algorithm <NUM> to process the images.

The communication interface <NUM> supports communications with other systems or devices. For example, the communication interface <NUM> could include a network interface card or a wireless transceiver facilitating communications over the network <NUM>. The communication interface <NUM> may support communications through any suitable physical or wireless communication link(s). The I/O unit <NUM> allows for input and output of data. For example, the I/O unit <NUM> may provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input devices. The I/O unit <NUM> may also send output to a display, printer, or other suitable output devices.

Although <FIG> illustrates one example of a computer system <NUM>, various changes may be made to <FIG>. As a particular example, while depicted as one system, the computer system <NUM> may include multiple computer system systems that may be remotely located.

<FIG> illustrates an example user computing device <NUM> according to embodiments of the present disclosure. The embodiment of the user computing device <NUM> illustrated in <FIG> is for illustration only, and the user computing devices <NUM>-<NUM> of <FIG> could have the same or similar configuration. However, user computing devices come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular implementation of an electronic device. As shown in <FIG>, the user computing device <NUM> includes a communication interface (or circuit) <NUM>, processor(s) <NUM>, an input/output (I/O) interface <NUM>, an input <NUM>, a display <NUM>, and a memory <NUM>. The memory <NUM> includes an operating system (OS) <NUM> and one or more gaming applications <NUM>.

The communication interface or circuit <NUM> supports communications with other systems or devices. For example, the communication interface <NUM> could include a network interface card or a wireless transceiver facilitating communications over the network <NUM>. The communication interface <NUM> may support communications through any suitable physical or wireless communication link(s). For embodiments utilizing wireless communication, the communication interface <NUM> may receive an incoming RF signal via one or more antennas using a variety of wireless communication protocols, (e.g., Bluetooth, Wi-Fi, cellular, LTE communication protocols etc.).

The processor(s) <NUM> can include one or more processors or other processing devices and execute the OS <NUM> stored in the memory <NUM> in order to control the overall operation of the user computing device <NUM>. The processor(s) <NUM> is also capable of executing gaming application(s) <NUM> resident in the memory <NUM>, such as, program code for one or more gaming applications for performing human computing tasks, such as, for example, games that incentivize engagement in image analysis as discussed in greater detail below. For example, the gaming application may be downloaded to and run by the user computing device <NUM> or may be web-based and played by accessing a server, for example, computing system <NUM>.

The processor(s) <NUM>, communication interface <NUM> and/or memory <NUM> constitute the computing resources of the user computing device <NUM>. The processor(s) <NUM> may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s) <NUM> include microprocessors, microcontrollers, graphical processing units (GPUs), digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The processor(s) <NUM> can move data into or out of the memory <NUM> as required by an executing process. The processor(s) <NUM> is also coupled to the I/O interface <NUM>, which provides the user computing device <NUM> with the ability to connect to other devices, such as laptop computers and handheld computers. The I/O interface <NUM> provides a communication path between accessories and the processor(s) <NUM>.

The processor(s) <NUM> is also coupled to the input <NUM> and the display <NUM>. The operator of the user computing device <NUM> can use the input <NUM> to enter data and inputs into the user computing device <NUM>. For example, the input <NUM> may be a touchscreen, button, keyboard, trackball, mouse, stylus, electronic pen, video game controller, etc. The display <NUM> may be a liquid crystal display, light emitting diode display, or other display capable of rendering text and/or at least limited graphics, such as from websites. The memory <NUM> is coupled to the processor(s) <NUM>. Part of the memory <NUM> could include a random-access memory (RAM), and another part of the memory <NUM> could include a Flash memory or other read-only memory (ROM).

Although <FIG> illustrates one example of user computing device <NUM>, various changes may be made to <FIG>. As a particular example, the processor(s) <NUM> could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In another example, the display <NUM> may be externally connected to or not a part of the user computing device <NUM>, such as for example, with a video game console or desktop computer.

<FIG> illustrates an example OCT image <NUM> of a normal retina and includes labels indicating the various structures within the retina tissue which may be analyzed in accordance with embodiments of the present disclosure. OCT images are generated by an OCT unit that can, noninvasively and without risk, capture two-dimensional images of the retina and be reconstructed to provide processed three-dimensional information as well. Using the OCT image <NUM>, details of the retinal architecture can be captured and analyzed. For example, laser light reflected into the OCT machine to reveal surfaces and layers within the object being imaged and the retina can have a large number of cell types and many layers of material and anatomic interfaces.

OCT image analysis can help in the diagnosis and management of patients with degenerative AMD. However the challenges that researchers face in fully utilizing OCT to manage degenerative AMD patients are that <NUM>) at the initial stages of the disease the OCT changes can be very subtle; <NUM>) the progression of changes seen on OCT in patients with degenerative AMD can be difficult to detect and <NUM>) the extent of possible changes seen on the OCT in patients with degenerative AMD can be very complex. To address these challenges, various embodiments of the present disclosure provide image analysis solutions that utilize interactive applications (such as video games or other applications requiring or using human input) to gain human intuition and leverage that human intuition in connection with machine learning to correctly identify structures (or the absence thereof) in an image.

<FIG> are example OCT images <NUM> of a retina with AMD which may be analyzed in accordance with embodiments of the present disclosure. In particular, <FIG> is annotated with has asterisks (<NUM> and <NUM>) identifying bumps which represent drusen material from a degenerative AMD subject. <FIG> is the same image but showing upper and lower lines (<NUM> and <NUM>, respectively) generated as a result of line segmentation having performed for use in measuring the volume of the drusen material in the image.

The amount of the material, (e.g., number of bumps, volume of material, average size of bumps, reflective index of the material, etc.) may be medically significant (relevant to progression or regression of the disease). For example, each OCT image is generated at <NUM> microns or smaller in thickness and approximately <NUM> such OCT images, or more, are generated per eye. While some of the images can be processed by machine run algorithms, these image analysis techniques can be highly inaccurate. Moreover, often times the image may be out of focus, for example, if the patient is moving or low-resolution images (e.g., <NUM> pixels) that do not contain significant enough amounts of information for machine run algorithms to generate accurate results. In some instances, the layers to be quantitated may be missing or very small (e.g., due to atrophy). As a result, the machine run algorithms may not provide enough useful information, for example, by not identifying exactly where the segmentation between the layers is. Often, the lines <NUM> and 510are inserted above or below where the actual segment between the layers is present in the image. As a result, technicians manually fix the placement of the lines <NUM> and <NUM>, which is very labor intensive particularly given the scope and quantity of the images. Given the need for accurate image analysis to determine efficacy of certain medications, many opportunities to precisely qualify the medication efficacy are missed because of the cost or performed with only a high level of detail.

This volume-based analysis is just one example. For just macular degeneration, numerous additional analyses may need to be performed at various stages of the disease. For example, the image analysis performed in various embodiments can be to provide correct identification of the material, quantity of material, consistency of identification, thickness of the material in a layer of material, the absence of at least a portion of a layer of material, and any additional granularity in the identification and classification of the material of interest. As one example of the variety in images to be analyzed for just AMD, <FIG> illustrates another example OCT image showing subretinal drusen material (as denoted by the asterisks labeled <NUM> and <NUM>) having variability in amount and appearance which may be analyzed in accordance with embodiments of the present disclosure.

<FIG> illustrates a user interface (UI) <NUM> for a gaming application (such as gaming application <NUM>) that is used to generate the human computational data <NUM> in accordance with various embodiments of the present disclosure. In this illustrative example, the inverted color scans of relevant portions (i.e., retinal tissue layers) of the OCT image, such as illustrated in <FIG>, are displayed on the UI <NUM> as roads or trails that enemy robots will travel (e.g., as denoted by upper and lower lines).

In playing the game, the user engages in the macular degeneration research by encouraging the creation of the upper and lower lines <NUM> and <NUM> defining the drusen volume within a given scan. This provided a constrained playfield, modeled after tower defense games where players would defend predefined paths by placing towers. The users are able to place points as "towers", for defense. After defining the line/towers, the enemy robots travel the path and the closer the player was to the correct lines, the higher score the player receives. Additional decoration such as trees and buildings are added in the upper and lower regions of the intended playfield to provide more of a gaming experience.

For an image which a defined (e.g., computer generated or technician generated) solution exists, the enemy robots follow the optimal routes defined by the manual corrected OCT analysis. Players, unaware of the optimal analysis, place nodes to define an upper and lower line to mirror the "roads" the enemy robots will follow. Upon defining their lines, the players submit the information which is sent to an online database storing the image solutions. The players watch the robots walk along their respective paths, leaving a trail behind to visually train players on the types of features (e.g., segments and layers in the retina) to follow in future levels representing new OCT scans. As the robots walk the roads, targeting lasers damage the robots depending on the overall accuracy of the line. The closer players place their nodes to the optimal path, the more damage the lasers do, with perfect placement eliminating all enemies.

After completing an image, the players receive a final scoring report detailing their accuracy. Accuracy extends from keeping their defined lines within a given number of units, which translated to the user as direct hits, solid hits, partial hits, and misses. After receiving their score, players proceed to the next stage where they interact with a new image with this process repeating for several images.

<FIG> are graphs illustrating example mean error results for a set of players playing their first and seventh images, respectively in accordance with various embodiments of the present disclosure.

In various embodiments, the results of the gameplay are processed, for example, by the computing system to determine the accuracy of results and/or improve machine learning image analysis. For example, as depicted by the example illustrated by <FIG>, the same images are played in sequential order. For each image, the average results from the set of all (e.g., <NUM> players) for each image is calculated. Then, to determine the error between the known solution and this average user position is calculated. At the outset, the players are fairly poor at identifying the correct locations of the layers in the image as illustrated in <FIG>. As depicted, the average player position for the top and bottom lines (<NUM> and <NUM>, respectively) is quite different than the position for the top and bottom lines (<NUM> and <NUM>, respectively) for the known solution for the first image. The shaded portions between the lines represent the error or deviation. However, embodiments of the present disclosure recognize that this error decreases overtime, for example, by the seventh image the results are reasonably accurate, for example, about a pixel difference, which is consistent with the range of accuracy in professional image analyzers. As depicted in <FIG>, the average player position for the top and bottom lines (<NUM> and <NUM>, respectively) much more closely tracks the position for the top and bottom lines (<NUM> and <NUM>, respectively) for the known solution for the seventh image. While illustrated as a line, in other embodiments, the user inputs in the image may form any other pattern or shape (e.g., circle, rectangle, etc.) for other types of objects or aspects thereof in the image being analyzed.

As a result, after such a training phase were results are compared against the known solution, the data from subsequent image analyses can be used as a reliable potential solution for the image that does not have an accepted or professionally determined solution. For example, for each image, the results of several analyses are averaged to determine the final solution. As part of this averaging process, the results data set may be limited to relevant results by, for example, discarding proposed solutions that fall at least a predetermined or weighted threshold above or below from the mean results for each point along the lines (i.e., obviously incorrect solutions) and/or removing all results for a user ID associated with the unreliable results.

To evaluate the performance of the human-generated results, a subset of the images that have known solutions are reserved and not used for generating the training data. With enough known solutions, the subset known solutions can be compared against the human-generated solutions for the same images to determine whether the overall set of generated training data is reliable and/or ascertain a rate of error or standard deviation that can be extrapolated to the overall training dataset.

In one or more embodiments, these results are used by themselves as the proposed solution and/or form human computational data, such as training data, that can be used as inputs to improve a machine learning algorithm that performs additional image analysis. As such, the gamification provides a mechanism to insert human intuition or human computation into the image analysis. This gamification not only trains users to perform image analysis but also the processing of the result data increases the accuracy of the ultimate proposed solution. Going one step further, in some embodiments, gamification is used as a training methodology to train technicians (i.e., professional image analyzers) on how to analyze the images. In other words, these embodiments use the game as a tool to teach technicians in reading centers how to analyze the images.

In additional embodiments, looking at the line generation of the image analysis, the lower line is relatively straight and receives fewer inputs to denote the curve of the line. However, at the location of the bumps, which are more medically relevant, several additional points are placed on the upper line to approximate the curve around the bump. Thus, not only is the location and/or slope curve relevant, the number of points entered to approximate positions on the curve can be used to identify potential areas of interest in the image. With the potential areas of interest identified, additional image analyses can be run just on that area of the image, for example, using the game-based analysis approach, flagged for review or fine-tuning by medical professionals, or used as an input for machine learning. In so doing, embodiments of the present disclosure generate meta-level data for the image analysis. Presently, machine learning algorithms process the image data, for example, passing a filter, such as convolution filters or color filters, over the image data for image analysis, with little ability to identify the relevance of areas in the image. Using the inputs from the game-based analysis, embodiments of the present discourse generate a heat-map image showing the density of dots or a derivative of points on average and provide this meta-level data to a machine learning algorithm. Using this input, the machine learning algorithm can identify that particular areas of the image are more relevant and perform additional processing (e.g., iterative processing) thereon, rather than wasting processing resources on less relevant areas such as normal tissue layer interfaces, noise, or background pixels.

Various embodiments of the present disclosure utilize several filter steps in real-time image analysis. For example, steps of edge detection, grayscale, color, density filters, derivative filters, etc. can be performed by the machine learning algorithm in any order. Additionally, for these embodiments, the present disclosure includes a "human intuition" filter as one or more of the steps in the image analysis. Providing these human intuition inputs to the algorithm can provide suggestions or mutations that allow the solutions provided machine learning algorithm to jump up a level in accuracy. Accordingly, in addition to or instead of assisting in identifying areas of interest/removing areas of non-interest, the generated meta-level data provides an additional input into the machine learning algorithm that indicates where the crowdsourced or human intuition data shows the characteristics of a given line. As discussed in the example above, the average position data for points placed by user input is viewable as a heat-map image. This generates a new data set that provides another layer of information captured by user input. Thus, the meta-level data can not only provide indicators of the relevant areas of the image, but also can providing insight about any given pixel. This, for example, assists the machine learning algorithm in identifying how to change the slope of the line when placing a next point.

Additionally, some embodiments provide a methodology to train an algorithm to identify the human intuition input or filter. For example, in deep neural networks, each layer of the deep neural network adds additional complexity or robustness. Using the data about where the user inputs occur in their image analysis (e.g., as a set of training data), embodiments of the present disclosure generate an algorithm that can identify points or curves in a manner similar to the way the human would identify the points or curves. Then in these embodiments, this "human layer" generation is used to improve the overall machine learning algorithm to remove the need for human input in the image analysis.

<FIG> illustrates an example of a process for analyzing an image in accordance with various embodiments of the present disclosure. For example, the process depicted in <FIG> is described as implemented by the server <NUM> in <FIG>; the process may also be implemented by any of the server systems <NUM>-<NUM> in <FIG>, and different servers systems may perform any number or portion of the steps of the process, collectively referred to herein as the system.

The process begins with the system providing multiple instances of an image for analysis via an interactive application (operation <NUM>). In operation <NUM>, the system may provide the respective instances of the image via respective instances of an interactive application designed for analysis of the image. In various embodiments, the interactive application is a gaming application (e.g., such as gaming application <NUM>) and includes a user interface representative of the image such as user interface <NUM> to receive multiple sets of user inputs per image analyzed via the interactive application.

Thereafter, the system receives data from results of analyses of the image including multiple sets of user inputs (operation <NUM>). In operation <NUM>, the multiple sets of user inputs are from the analyses of the multiple instances of the image, respectively; the multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from the multiple users, respectively; and the multiple users associated with the multiple instances of the interactive application, respectively. For example, the system may receive these results from each of multiple of client devices <NUM> from each of multiple users in a collaborative computing or citizen science manner. Additionally, the system may repeat the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images. For example, the system may provide sequential images of a same or similar type of structure to be analyzed, for example, in successive levels of a game, as discussed, for example, in connection with <FIG> above.

The system then determines whether the results of analyses for images are within a predetermined accuracy range (operation <NUM>). For example, as discussed above, a first set or a subset of the different images provided for analysis may have a known solution and a second set or the remainder do not have a known solution. The system may use the successive providing of the images to train and or provide feedback (e.g., via the interactive application, for example, as a score in a game or as a direct results comparison) to players or technicians regarding those images which have known solutions. Further, as discussed above, the system may perform processing on the results data to filter out or weight user results that deviate from mean results (e.g., for images with or without a known solution) beyond the predetermined accuracy range. For example, the system may only use the received data from the results of analyses for images that are within the predetermined accuracy range or deviation for later processing. Alternatively, the system may apply a lower weight to results data outside of the predetermined accuracy range, which may, for example, be proportionate in some manner to the amount of deviation from the known solution. Additionally, as discussed above, the system may use additional techniques to determine accuracy. For example, the system may average results and discard or apply lower weight to those solutions outside of a standard or desired deviation therefrom.

Thereafter, system processes the received data to identify area(s) of interest in the image(s) (operation <NUM>). In operation <NUM>, the system identifies the area(s) of interest based on the multiple sets of user inputs. For example, the system may generate a filter for the image based on both the pattern of each of the user inputs input on the image and a quantity of the user inputs for the pattern in an area of the image. For example, both the pattern and quantity of inputs can be indicative of the area interest. For example, multiple inputs within a small area can be indicative of some aspect or feature of the image that is significant for processing of the image. This filter may be a first filter for a first layer of an aspect of the structure(s) present in the image and multiple such filters can be generated and combined for use in analyzing the image. For example, the system may generate a second filter for another aspect of the structures in the image based on both of another pattern of each of the user inputs input on the image and quantity of the user inputs for the other pattern in a second area of the image. In these embodiments, the structures could be any structures that are medically relevant such as layers of tissue in an organ, such as, for example, the human eye. The aspects associated with each of the filters could be, for example, lines representative of an amount of a substance (e.g., drusen) present or could be different aspects, such as different aspects of the image to be analyzed simultaneously, for example.

The system then analyzes the image(s) using a machine learning algorithm to identify structure(s) in the image(s) (operation <NUM>). In operation <NUM>, the system identifies the structure(s) based on the identified area(s) of interest present within the image using the processed user input results data. For example, the system may use the filter(s) for the image as an input for the machine learning algorithm to focus the machine learning algorithm on the areas of interest in the image. For example, in some embodiments, the system may encode the filter(s) into the image as a data point or channel in the image itself similar to, for example, the red green blue (RGB) channels of an image. In some embodiments, the system may segment the image using the filter to focus processing of the machine learning algorithm to the one or more areas of interest present within the image, for example, by removing areas outside of the areas of interest using the filter. This may be advantageous in embodiments where reduced image processing time or reduced processing resources (e.g., computing cycles or memory) are desirable. In other embodiments, the system may process the entire image via the machine learning algorithm and use the filter(s) to focus the machine learning algorithm for additional processing inputs on the areas of interest. This may be advantageous in embodiments where precision and accuracy are more important that reduced image processing time or processing resources.

In various embodiments, the system may use the processed user input results data with the machine learning algorithm to improve the machine learning algorithm. For example, the system may use the processed user input results data as training data to train the machine learning algorithm, and, with enough training data, train the machine learning algorithm to generate similar results data to use in various aspects of image analysis, for example, without requiring further the user image analysis. For example, the system may generate a machine learning model for identifying probable user inputs in the images using the processed user input results data as training data and applying the machine learning model to identify probable user inputs in other images to train the machine learning algorithm to process the other images without requiring the multiple sets of user inputs. In various embodiments, the system may, additionally or alternatively, use the processed user input results data in other portions of the machine learning algorithm to modify the training or development of and/or improve machine learning algorithm to perform the image processing. For example, the system may use the processed user input results data (e.g., the user identified solutions for structure(s) or area(s) of interest in the image) to weight or error correct solutions proposed by the machine learning algorithm. In other examples, the system may modify the training of the machine learning algorithm itself via the processed user input results data.

In various embodiments, the system may analyze images over time to generate important statistical data or results. For example, the image may be of a portion of a bodily organ and this portion of a bodily organ may be imaged over time. In this example, the system may analyze these images provided over time to track a status of the structure(s) in the image over time and generate information using the tracked status. For example, this information can advantageously be used to monitor treatment efficacy for a disease of the bodily organ over the time and track progression of a disease of the bodily organ over the time as discussed in greater detail below.

Although <FIG> illustrates an example of process for analyzing an image, various changes could be made to <FIG>. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

Thus, various embodiments of the present disclosure provide an algorithm for generating a solution for individual image analysis and others provide a manner for generating inputs to provide a portion of the solution. For example, in some embodiments, the algorithm is trained to provide the solution in one step, and in others, a meta-level data set is generated first (e.g., by using human intuition computation or by a machine learning algorithm using training data) and use the algorithm to provide the solution using the meta-level data set.

Based on these substantial improvements to the image analysis, including decreasing the cost and time in performing the image analysis, significant increases in both the number of images processed and accuracy and precision in image processing are obtained. These analyzed images are then used to be used to validate (i.e., provide objective quantitative measurements) results of medical studies, such as drug trials, to determine treatment efficacy and better track progression of the disease for improved medical research. Other outcomes of the image analysis include detection of novel aspects on the image that have not been previously identified that provide insight to disease pathology or progression.

It is important to note that while the use of OCT images to for AMD analysis is discussed herein, this is just for an example illustrating possible applications of the embodiments of the present disclosure. Both OCT images and AMD analysis are example implementations, applications, or use cases and not intended as a limitation on the many different number and type of implementations, applications, or use cases that may be utilized in accordance with the embodiments of the present disclosure. Any type of images may be analyzed for any medical or other structural identification reasons in accordance with the principles of the present disclosure.

For example, without limitation, any type of OCT image analysis such as retina, cornea, lens, heart, lungs, other organs, can be used etc. Other medical diagnoses or purposes such as diabetic retinopathy, other retinal diseases, or OCT angiography (e.g., using consecutive images to track blood flow in blood vessels and generate images of blood flow to identify abnormalities in blood flow) may be utilized in the embodiments of the present disclosure. Furthermore, other types of images may be utilized in the embodiments of the present disclosure including but not limited to retinal fundus images, fluorescein angiographic images, indocynanine green angiographic images and ultrasound images. In addition, other images related to medicine may be utilized in the embodiments of the present disclosure including computerized tomography scans, magnetic resonance imaging, positive emission tomography, or x-rays may be amenable to the described analysis.

In one embodiment, a method for analyzing an image is provided. The method includes providing, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receiving, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively. The multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively. The method further includes processing the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyzing the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.

In another embodiment, a system for analyzing an image is provided. The system includes a communication interface and a processor operably connected to the communication interface. The communication interface is configured to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively. The multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively. The processor is configured to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.

In yet another embodiment, a non-transitory, computer-readable medium for analyzing an image is provided. The computer-readable medium comprises program code that, when executed by a processor of a system, causes the system to provide, via multiple instances of an interactive application for analysis of the image, multiple instances, respectively, of the image and receive, via the interactive application, data from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively. The multiple sets of user inputs are input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs are from multiple users, respectively; and the multiple users are associated with the multiple instances of the interactive application, respectively. The computer-readable medium further comprises program code that, when executed by a processor of a system, causes the system to process the received data to identify one or more areas of interest present within the image based on the multiple sets of user inputs and analyze the image using a machine learning algorithm to identify one or more structures present in the image based on the identified one or more areas of interest present within the image.

In any of the above examples and embodiments, the method further includes repeating the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images including the one or more structures, wherein a first set of the multiple different images have a known solution and a second set do not have a known solution; determining, per user in the multiple users, whether the results of analyses for images from the first set are within a predetermined accuracy range; and generating feedback regarding whether the results of analyses for images from the first set are within the predetermined accuracy range.

In any of the above examples and embodiments, the processing of the received data to identify one or more areas of interest comprises generating a filter for the image based on both a pattern of the user inputs input on the image and quantity of the user inputs for the pattern in an area of the image; and the analyzing of the image using the machine learning algorithm comprises using the filter for the image as an input for the machine learning algorithm.

In any of the above examples and embodiments, the filter is a first filter for a first layer of an aspect of the one or more structures present in the image; processing the received data to identify one or more areas of interest comprises generating a second filter for the image based on both a second pattern of the user inputs input on the image and quantity of the user inputs for the second pattern in a second area of the image, the second filter being for a second layer of the aspect of the one or more structures present in the image; and analyzing the image using the machine learning algorithm further comprises combining the first and second filters to form a multiple layer filter and using the multiple layer filter for the image as an input for the machine learning algorithm.

In any of the above examples and embodiments, using the filter for the image as an input for the machine learning algorithm comprises segmenting the image using the filter to focus processing of the machine learning algorithm to the one or more areas of interest present within the image.

In any of the above examples and embodiments, the method further includes repeating the providing of the multiple instances of the image and the receiving of the data from results of analyses of the image for each of multiple different images including the one or more structures; the analyzing of the image using the machine learning algorithm includes generating a machine learning model for identifying probable user inputs in the multiple different images using the results from the multiple different images as training data; and applying the machine learning model to identify probable user inputs in other images to train the machine learning algorithm to process the other images without requiring the multiple sets of user inputs.

In any of the above examples and embodiments, the analyzing of the image using the machine learning algorithm includes training the machine learning algorithm based on the identified one or more areas of interest present within the image; and analyzing the image using the trained machine learning algorithm to identify the one or more structures present in the image.

In any of the above examples and embodiments, the method further includes the interactive application is a gaming application and includes a user interface representative of the image and the multiple sets of user inputs are input into the user interface representative of the image.

In any of the above examples and embodiments, the image is of a portion of a bodily organ, and the analyzing of the image using a machine learning algorithm to identify one or more structures present in the image includes tracking a status of the one or more structures over time by processing additional images of the one or more structures using the machine learning algorithm; and generating information using the tracked status for at least one of (i) monitoring treatment efficacy for a disease of the bodily organ over the time and (ii) tracking progression of a disease of the bodily organ over the time.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The phrase "such as," when used among terms, means that the latter recited term(s) is(are) example(s) and not limitation(s) of the earlier recited term.

Moreover, various functions described herein can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer-readable medium. The phrase "computer-readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer-readable medium" includes any type of medium capable of being accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A "non-transitory" computer-readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. A non-transitory, computer-readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

Claim 1:
A method for analyzing an image (<NUM>), the method comprising:
providing, via multiple instances of an interactive application (<NUM>) for analysis of the image, multiple instances, respectively, of the image;
receiving, via the interactive application, data (<NUM>) from results of analyses of the image including multiple sets of user inputs from the analyses of the multiple instances of the image, respectively; the multiple sets of user inputs input via the multiple instances of the interactive application, respectively; the multiple sets of user inputs being from multiple users, respectively; the multiple users being associated with the multiple instances of the interactive application, respectively;
identifying, based on the received data, points in the image associated with the user inputs, respectively, of the multiple sets of user inputs;
identifying one or more areas of interest in the image at least based on the identified points that are associated with at least a subset of the multiple sets of user inputs, each set of the subset of the multiple sets of user inputs including multiple inputs associated with one or more areas of interest;
generating a filter for a first of the one or more areas of interest in the image based on both a pattern of the points in the image associated with the user inputs and quantity of the points in the image associated with the user inputs for the pattern in the first area of the image; and
analyzing the image using a machine learning algorithm (<NUM>), using the filter for the image as an input for the machine learning algorithm, to identify one or more structures present in the image based on the identified one or more areas of interest in the image.