Patent ID: 12243644

DETAILED DESCRIPTION OF THE INVENTION

Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. Generally, this application is intended to cover any adaptations or variations of the specific embodiments discussed herein.

FIG.1schematically illustrates a medical diagnosis assistance system100according to an embodiment of the present invention.

The medical diagnosis assistance system100comprises an input interface110and an output interface190. As has been described in the foregoing, these interfaces110,190may be realized as hardware and/or software, cable-bound or wireless, between local devices and/or to the Internet and/or the like.

The input interface110is configured to receive medical image data1of a patient. In the present example, an application of the present invention from the field of Hematology will be described, wherein the medical image data1are magnified visual images of blood samples, such as images taken with a microscope after a blood sample has been taken from the patient and has been arranged in a suitable vessel.

FIG.2shows a magnified visual image of a blood sample as an example for (a part of) medical image data1.FIG.2shows dyed blood cells, i.e., a blood sample to which dye has been applied before its visual scan. However, as has been described in the foregoing, blood samples may alternatively or additionally also be scanned before they are dyed, if they are dyed at all. Grayscale images of dyed and/or undyed images are also possible.

InFIG.2, two white blood cells10are visible as well as a large number of red blood cells20and a few platelets30, not all of which are labelled. It is immediately evident that even obtaining all possible information from this one image involves substantial effort for a human physician. If an analysis or diagnosis involves cross-references or interactions between all objects (blood cells) shown inFIG.2, this scope already exceeds the physical limitations of a human.

By contrast,FIG.2may illustrate only a very small part of the actual visual image (medical image data1) which may comprise more than 50, more than 100, more than 200, more than 500, or even more than 1000 individual blood cells. In the present example, each blood cell is a part of interest, POI, or, more specifically, an object of interest, OOI.

The medical diagnosis assistance system100further comprises a computing device150(a PC, a server, a terminal, a cloud computing platform, and the like). The computing device150is configured to implement a classification module151and an analysis module152, e.g., by executing a software by a CPU of the computing device150.

As has been described in the foregoing, the computing device150may comprise at least one processing unit such as at least one central processing unit, CPU, and/or at least one graphics processing unit, GPU, and/or at least one field-programmable gate array, FPGA, and/or at least one application-specific integrated circuit, ASIC, and/or any combination of the foregoing. The computing device150may further comprise a working memory operatively connected to the at least one processing unit and/or a non-transitory memory operatively connected to the at least one processing unit and/or the working memory. The computing device150may be implemented partially and/or completely in a local apparatus and/or partially and/or completely in a remote system such as by a cloud computing platform.

The computing device150may optionally comprise a pre-processing module149configured to pre-process the medical image data1received from the input interface110. The pre-processing module149may be provided for the case wherein the medical image data1are not captured in a single data set/sequence which contains the full information needed for assessing the medical problem. For such cases, the pre-processing module149may be configured to add, merge, and/or stick individual data sets and/or sequences to form a larger image composed of a number of sub-images added, merged, and/or stitched seamlessly together, preferably without any overlaying texture from the pre-processing. Pre-processing by the pre-processing module149may also contain scaling or normalizing of images in pixel-intensity reading or color balancing or in the physical dimension (size, rotation, distortion).

Thus, the medical image data1may be provided to the classification module151directly by the input interface110, or the medical image data1may be provided to the classification module151after pre-processing by the pre-processing module149. In other words, the pre-processing module149may be bypassed, or may simply forward the received medical image data1, when the medical image data1received by the input interface110are in a form suitable for the classification module151such as a single image file, or a time series, or the like.

The classification module151is configured to classify parts of interest, POI, comprising objects of interest, OOI, and/or regions of interest, ROI, within the received medical image data1, and to assign a corresponding reliability metric to each of the classified POI. In the present example, the classification module151is configured to classify the blood components10,20,30.

In this embodiment, the classification module151is configured to implement a classifying artificial intelligence entity, CAIE, specifically an artificial neural network, ANN, which is trained and configured to receive at least a portion of the received medical image data1and to generate, based thereon, a classifying artificial intelligence entity, CAIE, output classifying the blood components and/or the corresponding reliability metric. The received portion may be significantly larger, i.e., comprise significantly more individual blood components, as shown inFIG.2.

Furthermore, the classifying artificial intelligence entity, CAIE, comprises a first-level classifying artificial intelligence sub-entity, FLCAISE, and a second-level artificial intelligence sub-entity, SLCAISE, which in this embodiment are both realized as artificial neural networks, ANN, as well.

The first-level classifying artificial intelligence sub-entity, FLCAISE, is trained and configured to receive, as its input, at least a portion (preferably the entirety) of the received medical image data1(here: a magnified visual image of the blood sample) and to generate, based thereon, a first-level classifying artificial intelligence sub-entity, FLCAISE, output classifying all of the blood components according to a broad classification scheme, i.e., into blood component types (e.g.: white blood cell; red blood cell; platelet; unknown/rest).

The classification module151further comprises a cropping sub-module, CSM, configured to receive the FLCAISE output and to generate excerpts from the received medical image data1, in particular, by generating equally sized images for each of the blood components10,20,30, each image comprising the corresponding blood component10,20,30at its center, and a surrounding area defined by the pre-set size of the images. These images, or excerpts, are then provided to the second-level artificial intelligence sub-entity, SLCAISE, as its input.

FIG.3shows five exemplary excerpts created by the cropping sub-module.

The second-level artificial intelligence sub-entity, SLCAISE, is trained and configured to receive, as its input, the excerpts from the cropping sub-module and the FLCAISE output itself, and to further classify the blood components according to a refined classification scheme. In this refined classification scheme, each blood component type is further classified along one blood component sub-type for each blood component type that comprises blood component sub-types. For blood component types that do not comprise blood component sub-types, the type may be considered its own (and single) sub-type.

In the examples shown inFIG.3, the central blood component (object of interest, OOI) in each excerpt, respectively, from left to right is classified into the following blood component sub-types: a neutrophile granulocyte11, a lymphocyte12, a monocyte13, an eosinophile granulocyte14, and a basophile granulocyte15.

The reliability metric generated by the classification module151may be a probability percentage. A single part of interest, POI, may be assigned several reliability metrics, each according to a probability percentage associated with a certain blood component type or blood component sub-type. Usually, the classification into the broad classification scheme, i.e., into blood component types, is so reliable that preferably the reliability metric indicates the probability percentage for at least one blood component sub-type.

The analysis module152is configured to determine, based on the part of interest, POI, (here: blood components10,20,30) and the assigned reliability metric, an analysis of the medical image data1. As has been described in the foregoing, the presence of an Auer rod as one blood component may be indicative for acute leukemia (ALM), so the analysis in that case may comprise a diagnosis of acute leukemia or an assistance signal indicating that acute leukemia may be present and/or the like.

The analysis module152may take the reliability metric into account by basing the analysis only on the top percentile (e.g., 10%, 20%, 30% or the like) of reliable classifications according to the reliability metric. In other words, only the most trusted classifications are used as a basis for the analysis.

For example, blood cell classification cannot always be performed with high accuracy for every single cell, because abnormal cells do not always follow the morphologic criteria of a respective class of blood cells. Therefore, another strength of the present approach lies in the fact that the analysis preferably will be done on the most trusted blood cell classifications of input data and by omitting less accurate classifications which would introduce some additional uncertainty into the analysis or diagnosis. This approach is quite comparable to what is done by humans in diagnosing samples.

The output of the classification module151is preferably scanned, structured, and/or sorted, by the analysis module152, according to the reliability metric of the classification and the classification itself. Based on all the classified parts of interest, POI, with a reliability metric exceeding a predefined value, an analysis, in particular a medical diagnosis, gets inferred from an artificial intelligence entity such as an AI network or a computer algorithm either trained from other datasets as training data by supervised, semi-supervised, and/or unsupervised computer learning or based on given rules according to medical procedures or classification, preferably according to standard of care procedures.

As has been described in the foregoing, the analysis module152may also provide an analysis reliability value (or: diagnosis reliability value in the case of a diagnosis) which may be determined based on rules or patterns stored in the analysis module152, e.g., in a database thereof.

Especially in the combination of the reliability metric, and an optionally known diagnosis reliability, the medical diagnosis assistance system100may provide a complete diagnosis as its analysis.

In the above-mentioned case of acute leukemia, AML, for example, the presence of Auer rods indicates the presence of AML with certainty (diagnosis reliability 100%, as an example). If the classification includes, or indicates, the presence of an Auer rod in the medical image data with a reliability metric higher than a predefined threshold (e.g., 95%) then a diagnosis of AML may be output by the analysis module152. The diagnosis (or indication) may also be accompanied by a reliability estimate. In case both percentages are used as the reliability metric and the diagnosis reliability value, a product of the two percentages may be used. For example, the reliability metric for the detection of at least one Auer rod may be determined to be 95%, and the diagnosis reliability from an Auer rod to the diagnosis of leukemia is 100%, then the diagnosis of AML may be output together with a reliability estimate of 100% times 95%=95%.

The output interface190is configured to output an output signal71indicating the analysis. As depicted inFIG.1, the output signal71may be output to the exterior of the medical diagnosis assistance system100, e.g., into a cloud storage, a cloud computing platform, or the like. The output signal71indicating the analysis may also be provided to a graphical user interface153which may be a part of the medical diagnosis assistance system100(as illustrated inFIG.1) or which may be external to it.

The graphical user interface153may provide a user, on demand, with the option to quickly review the excerpts, e.g., as shown in the individual parts ofFIG.3, which relate to a specific result of the analysis, specifically, based on which the result of the analysis has been concluded. For example, in the case of AML, the analysis result may be provided by the graphical user interface153to the user next to a button labeled “show basis.” Touching this button would prompt the graphical user interface153to show the user all of the excerpts which show an Auer rod.

The graphical user interface153may also provide the user, on demand, with all excerpts together with a corresponding classification (broad and/or refined, i.e., here: blood component type and/or blood component sub-type), and give the user the option to re-classify the blood component with which each excerpt is associated (i.e., which is shown in its center). For example, the graphical user interface153may, on demand, show a drop-down menu for each excerpt, the drop-down menu comprising a list of all possible classifications. The list may be sorted according to the reliability metric (i.e., percentage of confidence) and/or the classification chosen by the classification module151may be highlighted in some manner. The user may then be able to manually select another classification, thus changing the classification of the blood component in that excerpt.

Since some classes are more stable classified by the classification module151, the analysis or diagnosis might be accompanied by the statistical information on the relative population of all available classes. This reference is advantageous since the selection of samples in step2according to the trust level of classification in step1might most probably change the statistics in the distribution of the population of the classes after step1. This distribution might also be a valuable input to step2in addition to the most prominent/secured representative samples in each class.

For example, for healthy donors there is a medical norm for the distribution of parts of interest, POI, among the available classifications, and the POIs are usually not evenly distributed. A change in population distribution of the classifications (with respect to the norm) may therefore be an indication towards a specific kind of disease or syndrome. Thus, information of the medical norm for the distribution and the actual distribution according to the classification may be used for generating the diagnosis or analysis.

As has been described in the foregoing, some steps in the described process may be iteratively performed in a so-called refinement loop. Such a loop can be started automatically, or on a specific trigger by the medical diagnosis assistance system100(e.g., an ambiguous result of the analysis) and/or by a user. For example, in the above case in which the user re-classifies one or more parts of interest, POI, the user may be asked by the graphical user interface153to confirm all changes. After the confirmation is made by the user and at least one change has been made, a refinement loop may automatically commence. In this way, the knowledge expressed by the user in the changes is taken into account for an eventually improved analysis.

For a refinement loop, first a preliminary analysis is generated by the analysis module152; the classification module152re-classifies the POI and/or re-assigns the corresponding reliability metric based on the preliminary analysis, and determines, based on the re-classified POI and/or the re-assigned corresponding reliability metric, another preliminary analysis which in turn may then be used to start another refinement loop or which may, instead, be used after the last refinement loop as the analysis of the analyzing module152.

As an example for an automatically triggered refinement loop, in the classification step, one object of interest, OOI, may have been classified as “class X” with 40% and “class Y” with 60% such that a classification into “class Y” is performed (i.e., into the class with the higher reliability metric value). The analysis in this example then provides a reliable (i.e., with an analysis reliability above a pre-defined threshold) diagnosis that either disease A or disease B is found, both of which are incompatible with a finding of “class Y.”

Then, in the refinement loop, that information is transferred to the analysis module which may then exclude the incompatible class (“class Y” in this example). Thus, the same object of interest, OOI, is then re-classified into “class X,” and the new reliability metric may now equal, for example, 75%. The next preliminary analysis within the loop is then based on the new entirety of classifications (after the re-classifying) which may then have a clear preference of the likelihood of disease B over the likelihood of disease A. This iterative process in the refinement loop(s) simulates the thinking of human physicians which evaluate possible analysis results and constantly challenge and re-evaluate their initial assumptions and classifications.

The number n of refinement loops may be pre-set (for example n=1, n=5, etc.), or the performing of a refinement loop may be dependent on a particular analysis result (e.g., likelihood of two different diagnoses differing by less than z %, for example, z=20, or z=10, or z=5).

A refinement loop may also be triggered by a user in other ways. For example, the preliminary diagnosis may be generated by the analysis module152based on a user input. For instance, a first analysis generated automatically by the classification module may be displayed to the user in the graphical user interface153, offering at least one alternative analysis result (or diagnosis result), each with a corresponding analysis probability.

The user, a physician, biologist, or the like, may then assess the first analysis, review the corresponding medical image data1, and so on, and may be able to dismiss one or more of the alternatives offered in the first analysis, for example, by selecting a “dismiss this alternative” function in the graphical user interface. Based on this user input, the analysis module generates a second analysis, being the previously mentioned “preliminary analysis” that goes into the refinement loop. After the loop, the user may be provided with new alternatives via the graphical user interface153, and so on. A stopping condition for breaking out of the refinement loops may be when the user indicates satisfaction (or agreement) with one or more alternatives or when only one alternative is left, and/or when no condition for automatically commencing another refinement loop is triggered.

In some variants, the result of all refinement loops is presented to the user by the graphical user interface153, in other variants the user may be given the opportunity to select, or trigger, after each refinement loop anew whether another refinement loop is to be performed or not. In any case, the result of the final refinement loop may be displayed by the graphical user interface153and/or be coded into an output signal71by the output interface190. The output signal71can be directed and configured to start a workflow, control a medical device, deposit data in a database, create a human-readable output or printout, and/or the like.

FIG.4shows a flow diagram schematically illustrating a computer-implemented medical diagnosis assistance method according to an embodiment of the present invention. It shall be understood that the method according to the present invention may be performed by the medical diagnosis assistance system100according to the present invention. Therefore, any embodiment of the method according to the present invention may be adapted, modified, or refined according to any of the variants, options, modifications, or refinements of any of the embodiments of the medical diagnosis assistance system100according to the present invention, in particular as has been described with respect toFIG.1throughFIG.3. For this reason, the method steps will only be briefly described, and a reference to the detailed foregoing description, both in general and referring to the drawings, is implied.

One or more method steps may be performed at the same time, in an integrated way, or separately from one another. The numbering of the method steps does not necessarily imply a time order of steps.

In a step S10, medical image data1of a patient is received, e.g., as has been described in the foregoing with respect to the input interface110.

In an optional step S15, the medical image data1may be pre-processed as has been described with respect to the pre-processing module149in the foregoing.

In a step S20, parts of interest, POI, comprising objects of interest, OOI, and/or regions of interest, ROI, within the received medical image data1are classified, in particular as has been described in the foregoing with respect to the classification module151.

In a step S30, a corresponding reliability metric is assigned to each of the classified part of interest, POI, advantageously again as has been described in the foregoing with respect to the classification module151.

In a step S40, an analysis of the medical image data1is determined based on the POI and the assigned reliability metric, in particular, as has been described in the foregoing with respect to the analysis module152.

One or more refinement loops as described in the foregoing may be performed, in particular, by repeating step S20, step S30, and/or step S40.

In some advantageous variants, the method may comprise one or more of the steps which have already been implicitly described in more detail in connection with the medical diagnosis assistance system100.

Specifically, the method may comprise steps of:generating, using a trained first-level classifying artificial intelligence sub-entity, FLCAISE, a FLCAISE output classifying at least a part of the part of interest, POI, according to a broad classification scheme, based on at least a portion of the received medical image data1; andgenerating, using a trained second-level classifying artificial intelligence sub-entity, SLCAISE, a SLCAISE output classifying the at least part of the part of interest, POI, according to a refined classification scheme, based on at least a portion of the received medical image data1and on the first-level classifying artificial intelligence sub-entity, FLCAISE, output.

Similarly, the method may comprise steps of:generating excerpts5from the received medical image data1, each excerpt5associated with a single POI and comprising at least the corresponding associated POI, andproviding the generated excerpts to the second-level artificial intelligence sub-entity, SLCAISE, as the input of the second-level artificial intelligence sub-entity, SLCAISE.

FIG.5shows a flow diagram illustrating a training method for training a classifying artificial intelligence entity, CAIE, according to another embodiment of the present invention.

In a step S01, first labelled samples of medical image data1are provided. For example, each sample may consist of, or comprise, an image as shown inFIG.2, wherein for each blood component, or part of interest, POI, a classification into a blood component type is provided as a label. The label may comprise additional data such as data defining a bounding box about each part of interest, POI. The labelled samples may be produced by a physician or biologist labelling the part of interest, POI, shown in the image. As has been described in the foregoing, sample sets with dyed and/or undyed images may be used, either simultaneously or sequentially.

In a step S02, a first-level classifying artificial intelligence sub-entity, FLCAISE, is provided in a preliminary state, e.g., with randomized initial values for an artificial neural network, ANN, especially a convolutional neural network, CNN.

In a step S03, supervised learning is performed, using the first labelled samples, in order to train the first-level classifying artificial intelligence sub-entity, FLCAISE, to classify parts of interest, POI, within an input of the first-level classifying artificial intelligence sub-entity, FLCAISE, according to a broad classification scheme.

In a step S04, second labelled samples of medical image data1are provided. For example, each sample may consist of, or comprise, an image as shown in any of the excerpts5ofFIG.3, wherein for each blood component, or part of interest, POI, in the center of the excerpt5a classification into a blood component sub-type is provided as a label. The label may comprise additional data such as data defining a bounding box about each part of interest, POI. The labelled samples may be produced by a physician or biologist labelling the part of interest, POI, shown in the image. As has been described in the foregoing, sample sets with dyed and/or undyed images may be used, either simultaneously or sequentially.

In a step S05, a second-level artificial intelligence sub-entity, SLCAISE, is provided in a preliminary state, e.g., with randomized initial values for an artificial neural network, ANN, especially a convolutional neural network, CNN.

In a step S06, supervised learning is performed, using the second labelled samples, in order to train second-level artificial intelligence sub-entity, SLCAISE, to classify parts of interest, POI, within an input of the second-level classifying artificial intelligence sub-entity, SLCAISE, according to a refined classification scheme.

In a step S07, a trained classifying artificial intelligence entity, CAIE, is provided which comprises both the trained first-level classifying artificial intelligence sub-entity, FLCAISE, and the trained second-level artificial intelligence sub-entity, SLCAISE.

FIG.6shows a schematic block diagram illustrating a computer program200product according to another embodiment of the present invention. The computer program product200comprises executable program code250configured to, when executed, perform the method according to any embodiment of the present invention, in particular the method as explained with respect toFIG.4and/or the method as explained with respect toFIG.5.

FIG.7shows a schematic block diagram illustrating a computer-readable, non-transitory data storage medium300product according to another embodiment of the present invention. The data storage medium300comprises executable program code350configured to, when executed, perform the method according to any embodiment of the present invention, in particular the method as explained with respect toFIG.4and/or the method as explained with respect toFIG.5.

In the foregoing detailed description, various features are grouped together in one or more examples or examples with the purpose of streamlining the disclosure. It is to be understood that the above description is intended to be illustrative, and not restrictive. It is intended to cover all alternatives, modifications and equivalents. Many other examples will be apparent to one skilled in the art upon reviewing the above specification.

The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated.

LIST OF REFERENCE SIGNS

1medical image data5excerpt10white blood cell11-15white blood cell sub-types20red blood cells30platelets71output signal100medical diagnosis assistance system110input interface149pre-processing module150computing device151classifying module152analysing module153graphical user interface190output interface200computer program product250program code300data storage medium350program codeS01-S06method stepsS10-S40method steps