METHODS FOR OPERATING AN EVALUATION SYSTEM FOR MEDICAL IMAGE DATA SETS, EVALUATION SYSTEMS, COMPUTER PROGRAMS AND ELECTRONICALLY READABLE STORAGE MEDIUMS

At least one example embodiment provides an improved handling of image data sets of patients.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application claims priority under 35 U.S.C. § 119 to European Patent Application No. 21171603.0, filed Apr. 30, 2021, the entire contents of which are incorporated herein by reference.

FIELD

At least some example embodiments relate to computer-implemented methods for operating an evaluation system for medical image data sets, wherein at least one evaluation algorithm is provided for evaluating a medical image data set to determine at least one evaluation information. At least some example embodiments also relate evaluation systems, computer programs and electronically readable storage mediums.

BACKGROUND

Imaging techniques are nowadays often used for diagnosis, monitoring and aftercare in medicine. An image data set of an imaging region of a patient is acquired and is then evaluated regarding medical concepts, in particular pathological and other anatomical and/or physiological anomalies and features. While, in particular in the past, evaluation was often done purely manually by so-called reading, in particular by a radiologist, with the rise of the number of imaging exams and different specialized imaging techniques, the workload and complexity of the evaluation tasks increase.

Hence, the development of evaluation algorithms working on a computing device to aid medical staff in reading medical image data sets, in particular regarding diagnosis, is an active field of research. Such evaluation algorithms use input data comprising at least one image data set to generate output data describing evaluation results, in particular physical features depicted in and/or derivable from the image data set. Recently, more evaluation algorithms based on artificial intelligence have been proposed, for example comprising a neural network. Such artificial intelligence evaluation algorithms work on training data comprising image data sets with associated ground truths regarding the output data, for example respective annotations made in a manual evaluation by a physician.

SUMMARY

Many evaluation algorithms require large amounts of computational resources and their use is time-consuming. In some cases, it has even been proposed to provide the algorithms as an, in particular paid, service on a server or cloud device, such that the image data need to be sent over a communication line, further producing traffic. However, evaluation algorithms may also fail to produce correct or usable evaluation results. For example, if an evaluation algorithm has been trained regarding a certain type of image data set, it may not function well when image data set properties differ. Furthermore, the image quality may not be sufficient and/or the relevant features to be evaluated may not be present. In such cases, the image evaluation using the evaluation algorithm yields no valuable information and can be considered wasted effort.

At least one example embodiment provides an improved handling of image data sets of patients, on particular providing consistent processing and/or avoiding unnecessary effort and cost.

This object is achieved by providing a computer-implemented method, an evaluation system, a computer program, and an electronically readable storage medium according to the independent claims. Advantageous embodiments are described in the dependent claims.

At least one example embodiments provides a computer-implemented method for operating an evaluation system for medical image data sets, the method comprising receiving at least one input set comprising at least one image data set of a patient and at least one evaluation algorithm; evaluating, using at least on evaluation algorithm, a medical image data set to determine at least one evaluation information, wherein for each evaluation algorithm, requirement information comprising requirements regarding at least one of a quality or a content of image data to be evaluated is provided or determined, the evaluating including, analyzing each image data set of each input set to determine a suitability information regarding each evaluation algorithm of the input set, the suitability information indicating at least fulfilment or non-fulfilment of the requirements of the corresponding requirement information, at least one of (i) forwarding for evaluation to at least one corresponding evaluation algorithm only at least one image data set, whose suitability information indicates fulfilment of the requirements of the requirement information of the at least one corresponding evaluation algorithm, or (ii) executing at least one of at least one user information action or image processing action for at least one evaluation algorithm of each input set whose associated suitability information all indicate non-fulfilment of the requirements of the requirement information.

According to at least one example embodiment, at least one image data set of at least one input set comprises meta information, and the meta information is, in addition to the image data, used to determine the suitability information.

According to at least one example embodiment, at least one of the medical image data sets of an input set or the evaluation algorithms to be applied are automatically determined from at least one of an electronic health record or a picture archiving and communications system (PACS) entry of the respective patient, wherein, for each evaluation algorithm not manually chosen to the input set and for which the requirements for at least one of the medical image data sets of the input set are fulfilled, a proposal information proposing use of this evaluation algorithm is output and the image data set is forwarded only if approving user input is received.

According to at least one example embodiment, the suitability information includes at least one suitability measure describing a degree of suitability of the image data set as input data for the evaluation algorithm.

According to at least one example embodiment, at least one of if multiple medical image data sets fulfil the requirements of an evaluation algorithm according to the suitability information, only the at least one image data set associated with the highest suitability measure is forwarded to the evaluation algorithm, or if an image data set fulfils the requirements of multiple evaluation algorithms all yielding a required evaluation information, the image data set is forwarded to the evaluation algorithm for which the suitability measure is highest.

According to at least one example embodiment, for each image data set, image information describing the at least one of the quality or the content of the image data set is determined by analysis of the image data set and, for each input set, the image information of each image data set is compared to the requirement information of each evaluation algorithm of the input set to determine the suitability information.

According to at least one example embodiment, at least one image property is determined as a quality parameter of the image information.

According to at least one example embodiment, during the analyzing, at least one of a presence or a strength of at least one type of artifacts is determined by at least one artifact detection algorithm as a part of the image information.

According to at least one example embodiment, a presence of at least one feature to be further analyzed by at least one evaluation algorithm is detected during the analyzing, and at least one requirement of the requirement information of this evaluation algorithm is based on the presence of the feature.

According to at least one example embodiment, at least one additional requirement information regarding the at least one of the quality or the content of image data to be evaluated manually to achieve the evaluation purpose of at least one of the at least one evaluation algorithm is provided, wherein, if, for at least one evaluation algorithm of each input set having an additional requirement information associated therewith, the associated suitability information all indicate non-fulfilment of the requirements of the requirement information, the image information is compared to the additional requirement information to determine additional suitability information for each image data set, wherein, if the additional suitability information of at least one of the at least one image data set indicates fulfilment of the requirements of the additional requirement information, a notification triggering manual evaluation of this image data set is output as the action, and else a notification triggering acquisition of a new image data set is output as the action.

According to at least one example embodiment, at least one image processing action comprises, postprocessing, to increase at least one quality parameter, of the at least one image data set, or re-reconstruction of the at least one image data set from raw data using modified reconstruction parameters to determine at least one updated image data set, for which an updated suitability information regarding the at least one evaluation algorithm is determined.

According to at least one example embodiment, the at least one evaluation algorithm is provided on a remote server, wherein image data sets to be evaluated are sent to the remote server.

According to at least one example embodiment, an evaluation system for medical image data sets, wherein at least one evaluation algorithm is provided in an evaluation unit for evaluating a medical image data set to determine at least one evaluation information, the system comprising a first interface configured to receive, for each evaluation algorithm, requirement information comprising requirements regarding at least one of a quality or a content of image data to be evaluated; a second interface configured to receive at least one input set comprising at least one image data set of a patient and at least one of the at least one evaluation algorithm to be applied; an analysis unit configured to analyze each image data set of each input set to determine a suitability information regarding each evaluation algorithm of the input set, the suitability information indicating at least fulfilment or non-fulfilment of the requirements of the corresponding requirement information; and a decision unit configured to at least one of forward at least one image data set, whose suitability information indicates fulfilment of the requirements of the requirement information, for evaluation by the corresponding evaluation algorithm to the evaluation unit, or execute at least one user information action and/or image processing action for at least one evaluation algorithm of each input set whose associated suitability information all indicate non-fulfilment of the requirements of the requirement information.

According to at least one example embodiment, a computer program, when executed, causes a computing device of an evaluation system to perform a method according example embodiments.

According to at least one example embodiment, an electronically readable storage medium stores the computer program.

According to at least one example embodiment, an evaluation system for medical image data sets comprises processing circuitry configured to execute computer-readable instructions to cause the evaluation system to receive, for each of at least one evaluation algorithm, requirement information comprising requirements regarding at least one of a quality or a content of image data to be evaluated, receive at least one input set comprising at least one image data set of a patient and at least one of the at least one evaluation algorithm to be applied, analyze each image data set of each input set to determine a suitability information regarding each evaluation algorithm of the input set, the suitability information indicating at least fulfilment or non-fulfilment of the requirements of the corresponding requirement information, and at least one of forward at least one image data set, whose suitability information indicates fulfilment of the requirements of the requirement information, for evaluation by the corresponding evaluation algorithm, or execute at least one user information action and/or image processing action for at least one evaluation algorithm of each input set whose associated suitability information all indicate non-fulfilment of the requirements of the requirement information.

DETAILED DESCRIPTION

In a computer-implemented operating method as initially described, according to at least one example embodiment,

for each evaluation algorithm, requirement information comprising requirements regarding the quality and/or the content of image data to be evaluated is provided or determined,

at least one input set comprising at least one image data set of a patient and at least one of the at least one evaluation algorithm to be applied is received,

each image data set of each input set is analyzed to determine a suitability information regarding each evaluation algorithm of the input set, the suitability information indicating at least fulfilment or non-fulfilment of the requirements of the corresponding requirement information,

only at least one image data set, whose suitability information indicates fulfilment of the requirements of the requirement information of at least one corresponding evaluation algorithm, is forwarded for evaluation to the corresponding evaluation algorithm and/or

at least one user information action and/or image processing action is executed for at least one evaluation algorithm of each input set whose associated suitability information all indicate non-fulfilment of the requirements of the requirement information.

At least one example embodiment provides an evaluation system for medical image data sets, wherein at least one evaluation algorithm is provided in an evaluation unit for evaluating a medical image data set to determine at least one evaluation information, the evaluation system comprising:

a first interface for receiving, for each evaluation algorithm, requirement information comprising requirements regarding the quality and/or the content of image data to be evaluated,
a second interface for receiving at least one input set comprising at least one image data set of a patient and at least one of the at least one evaluation algorithm to be applied,
an analysis unit analyzing each image data set of each input set to determine a suitability information regarding each evaluation algorithm of the input set, the suitability information indicating at least fulfillment or non-fulfillment of the requirements of the corresponding requirement information,
a decision unit for

forwarding at least one image data set, whose suitability information indicates fulfillment of the requirements of the requirement information, for evaluation by the corresponding evaluation algorithm to the evaluation unit and/or

executing at least one user information action and/or image processing action for at least one algorithm of each input set whose associated suitability information all indicate non-fulfillment of the requirements of the requirement information.

The methods and systems described here may be applied to single provided evaluation algorithms, but are preferably used in a system where multiple different evaluation algorithms are provided, for example relating to a certain imaging region, a certain anatomical region, a certain imaging modality, a certain imaging technique and/or a certain class of medical concepts. For example, all evaluation algorithms may relate to coronary imaging, for example to both coronary computed tomography angiography (CCTA) and calcium scoring imaging (CaSc). In other examples, liver imaging or diffusion magnetic resonance imaging (MRI) may be additional or alternative fields of application. It is, of course, also conceivable to provide evaluation algorithms regarding the whole patient and/or different imaging modalities.

Generally, an evaluation algorithm analyzes the image content of an image data set to derive evaluation information regarding at least one medical concept from imaged, that is physically existent, features of the patient in the imaging region covered by the image data set. Medical concepts may, for example, comprise pathological and other anatomical and/or physiological anomalies and features. For example, an evaluation algorithm may be a lesion detection algorithm, for example regarding lesions in a coronary vessel tree of a patient. In another example, a substance in the imaging region may be quantified, for example a contrast agent, sodium, or calcium. The evaluation algorithm may or may not comprise a trained function, for example at least one neural network, which has been trained by machine learning. For example, at least one evaluation algorithm may be or comprise a deep neural network, in particular a convolutional neural network (CNN).

The input set usually refers to a patient and may comprise one or more image data sets of the patient, for example multiple image data sets acquired in the course of an examination or correlated examination series. For example, the image data sets may comprise computed tomography data sets acquired during at least one coronary examination. Further, the input set comprises at least one identification of an evaluation algorithm to be applied, in particular associated to a subgroup of the image data sets.

At least some example embodiments do not simply apply certain, in particular manually chosen or automatically associated, evaluation algorithms to image data sets, but add a further, upstream analysis step which determines and judges the technical content of the image data sets with regard to at least one evaluation algorithm. In particular, it is at least checked whether an evaluation algorithm and an image data set are suitable for each other, and preferably further advantageous action may be taken. The additional analysis can be understood as an, in particular multi-step, decision support.

Basically, the suitability information allows to identify image data sets not suitable for evaluation by a certain evaluation algorithm, such that additional effort, in particular computational cost and/or data traffic, may be reduced by preventing the unsuitable image data sets from being evaluated by the corresponding evaluation algorithm. This can be done automatically, in particular involving no physician effort. In this manner, dependence on skill and/or availability of a physician, in particular a radiologist, can be prevented. Unnecessary cost and effort for performing unnecessary advanced image analysis, which does not lead to additional evaluation information of clinical relevance, in particular when using external analysis services, can be avoided, shortening the time to adequate treatment.

Example embodiments allow consistent processing, such that certain standards regarding quality and content can be expected if certain evaluation algorithms are to be applied.

It is, again, noted that the method according to at least some example embodiments is not limited to certain, defined cases in which it is already manually planned to perform a certain advanced analysis on image data. The decision support introduced by example embodiments may also provide for automatic decision support regarding which advanced analysis is to be applied to which image data sets and/or what the most appropriate advanced analysis method for a patient would be. In other words, as will be further described below, the methods and systems according to example embodiments may be employed to identify patients for whom an advanced analysis could reveal additional evaluation information of clinical relevance, to identify the most appropriate advanced analysis method, that is evaluation algorithm, to determine additional evaluation information of clinical relevance, and to identify appropriate image data sets for performing the advanced analysis, preferably taking into account image acquisition and reconstruction as well as image quality.

In preferred embodiments, not only the image data itself may be analyzed, but also additional available information. In particular, at least one image data set of at least one input set may comprise meta information, in particular in a DICOM format, wherein the meta information is, in addition to the image data, used to determine the suitability information. The DICOM standard is well-known and widely used in medical image data processing, archiving and storing. It allows to add meta information regarding the image data set, for example acquisition parameters, patient information and the like, into the same data structure, that is, the image data set itself. Preferably, this additional meta information is also used during analysis, for example to determine which imaging technique has been used, which contrasts result, which imaging region is shown, which resolution is used and the like.

In preferred embodiments, the image data sets of an input set and/or the evaluation algorithms to be applied are automatically determined from an electronic health record and/or a PACS entry of the respective patient, wherein, in particular, for each evaluation algorithm not manually chosen to the input set and for which the requirement for at least one of the image data sets of the input set are fulfilled, a proposal information proposing use of this evaluation algorithm is output and the image data set is forwarded only if proving user input is received.

Electronic health records and/or picture archiving and communication systems (PACS) serve to provide a data storage for associated information on a patient, for example all resulting image data sets of a certain examination or series of examinations. Other information, in particular patient information, for example suspected diagnoses, may be associated with the image data sets in such data basis, hence allowing to prepare input sets fully automatically. If, for example, a patient had a coronary examination using computer tomography, it can be deduced that algorithms regarding blood flow, coronary lesions and the like may be useful. Different image data sets may also be associated with different evaluation algorithms in advance, such that, for example, a non-contrast computer tomography scan of the coronary region of a patient may be associated with a calcium scoring algorithm and the like. Fully automatically, it is checked whether the respective evaluation algorithms are suitable, that is, whether clinically evaluation information can be expected from advanced analysis. While it is conceivable to automatically initiate application of the corresponding evaluation algorithms, it may, however, be preferred to output respective proposal information to a user first. In this manner, decision support is provided and the user can simply confirm the proposal to have the evaluation algorithm applied to the corresponding image data set.

In especially preferred embodiments, the suitability information may be determined comprising at least suitability measure describing a degree of suitability of the image data set as input data for the evaluation algorithm. Such a suitability measure, for example, a metric describing the relevant image quality regarding the evaluation algorithm, may serve as a basis for further automatical decisions leading to improved quality of the evaluation information.

In particular, if multiple image data sets fulfill the requirements of an evaluation algorithm according to the suitability information, only the at least one image data set associated with the highest suitability measure is forwarded to the evaluation algorithm, and/or, if an image data set fulfills the requirements of multiple evaluation algorithms all yielding a required evaluation information, the image data set is forwarded to the evaluation algorithm for which the suitability measure is highest. In other words, image data sets best suited as a basis for advanced analysis by the evaluation algorithm may be chosen such that the highest quality evaluation information can be determined. Furthermore, if multiple evaluation algorithms yield comparable evaluation information, the most suitable and promising evaluation algorithm for a certain image data set may be chosen. In combination, the best pair of image data set and evaluation algorithm for the most promising results of advanced analysis can be chosen. Of course, it is also possible to choose multiple combinations, for example forward the two or three best image data sets to an evaluation algorithm or the like.

In concrete embodiments, for each image data set, image information describing quality and/or content of the image data set is determined by analysis of the image data set and, for each input set, the image information of each image data set is compared to the requirement information of each evaluation algorithm of the input set to determine the suitability information. While the requirement information, for example, may define criteria, for example thresholds and the like, for certain parameters and information items, image information comprising these parameters and/or information items may be determined by technically analyzing the image data set. This allows evaluation of the criteria, such that the suitability information may be determined. The parameters and information items are image properties that are determined by applying respective processes/analysis algorithms.

For example, at least one image property may be determined as a quality parameter of the image information, in particular a signal-to-noise ratio and/or a contrast value and/or a dynamic range and/or an image sharpness.

In an exemplary embodiment, at least one requirement of at least one requirement information may be a minimum quality required for robust evaluation, in particular regarding a certain quality parameter. Of course, other quality parameters may also be used, in particular also quality metrics combining different quality aspects.

During analysis of the at least one image data set, the presence and/or strength of at least one type of artifact may be determined by at least one artifact detection algorithm as a part of the image information. In some evaluation algorithms, some artifacts should not be present or strong when automatically evaluating the image data set. Examples for artifacts are movement artifacts caused by patient motion, for example if the imaging region is subject to heart motion and/or breathing motion. Another source for artifacts may be a patient having a higher heart rate and/or a non-regular heart rate. Movement artifacts may, however, also occur in the imaging device used to acquire the image data set. For example, in an X-ray device, the detector and/or the X-ray source are often moved. In the state of the art, artifact detection algorithms have been proposed that are able to detect the presence of such artifacts, which may be relevant for evaluation algorithms. Respective image artifact information may be added to the image information.

It is noted that image quality may also be influenced by events happening during acquisition. If, for example, a contrast agent is used and the timing is off, not enough contrast agent may be in the imaging region, such that the contrast may be too low.

Problems regarding image data sets may also arise during reconstruction from raw data. For example, if computer tomography data sets are reconstructed from projection data, reconstruction kernels may be used which influence image properties, for example image sharpness. Finally, problems meeting the requirements may result from user choices, for example regarding resolution and/or slice thickness.

However, not only image quality may influence suitability for advanced analysis by an evaluation algorithm, but also image content. For example, if the evaluation algorithm analyzes certain, in particular medical, features, these features should be sufficiently distinguishable in the image data set. Hence, in embodiments, the presence of at least one, in particular medical, feature to be further analyzed by at least one evaluation algorithm is detected during analysis of the at least one image data set, wherein at least one requirement of the requirement information of this evaluation algorithm demands presence of the feature. In particular, such an analysis may also relate to the complete depiction of the feature by the image data set. Such a check is, in particular, important, if the feature cannot be easily discerned by the human eye, for example, in functional imaging. In embodiments, the feature may also relate to a pathology. In many cases, the pure presence of some pathology can be detected with a low expenditure, while the evaluation algorithm detailedly analyzes the pathology, for example, a lesion. If no relevant pathology is contained in an image data set, advanced analysis regarding this pathology also makes no sense, such that the operating methods according to example embodiments may prevent such image data sets from being forwarded to the respective evaluation algorithms.

In an especially preferred embodiment, at least one additional requirement information regarding the quality and/or the content of image data to be evaluated manually to achieve the evaluation purpose of at least one of the at least one evaluation algorithm may be provided, wherein, if, for at least one algorithm of each input set having an additional requirement information associated therewith, the associated suitability informations all indicate non-fulfillment of the requirements of the requirement information, the image information is compared to the additional requirement information to determine additional suitability information for each image data set, wherein, if the additional suitability information of at least one of the at least one image data set indicates fulfillment of the requirements of the additional requirement information, a notification triggering manual evaluation of this image data set is output as the action, and else a notification triggering acquisition of a new image data set is output as the action.

In other words, in this embodiment, there may be three outcomes regarding further analysis and evaluation of acquired image data sets. If the suitability information regarding an evaluation algorithm to be used indicates that the image quality and content are good enough to use the algorithm, advanced analysis may be automatically performed by the evaluation algorithm. However, even if the image quality and/or content may not suffice for the automatic advanced analysis by the evaluation algorithm, manual reading of the image data set may still provide relevant clinical results. Hence, if the suitability information regarding the evaluation algorithm indicates non-suitability, a second check is performed using additional requirement information regarding manual evaluation, such that an additional suitability information results. If the additional suitability information indicates suitability for manual reading of the image data set, the image data set may be forwarded to manual evaluation, for example queued and output to a radiologist. If, however, regarding a certain evaluation purpose, no image data set is found suitable for either an evaluation algorithm or manual reading, re-acquisition may be triggered since the image data are clinically useless. However, the results of the analysis may, also in this case, be put to use, since control information regarding the re-acquisition may be generated depending on the suitability information and used to control the corresponding imaging device. In this manner, the re-acquisition can be adapted to generate an image data set better suiting the requirements. Generally, in such an embodiment, by technically analyzing and judging image data sets, decision support is provided which allows the best use of any acquired image data set.

As already described, the action may be a user information action, for example triggering manual reading or re-acquisition or simply marking an image data set as not suitable or advanced analysis (and/or manual reading). Additionally, if re-acquisition is to be performed, control information may additionally be sent to the imaging device. However, in embodiments, also image processing actions may be triggered, in particular in addition to user information actions, for example asking for a confirmation, wherein such image processing actions may try to modify and/or improve the image data set to fulfill the requirements of at least one requirement information. In a concrete embodiment, at least one image processing action comprises post processing, in particular to increase at least one quality parameter, of at least one of the at least one image data set and/or re-reconstruction of at least one of the at least one image data set from raw data using modified reconstruction parameters to determine at least one updated image data set, for which an updated suitability information regarding the at least one evaluation algorithm is determined. Such image processing actions may, for example, comprise denoising, edge sharpening, artifact reduction and the like. If raw data are still available, advantageously, re-reconstruction may be triggered, if, for example, a different reconstruction kernel and/or different reconstruction parameters may lead to more suitable results regarding image quality and/or content.

In preferred embodiments, the at least one evaluation algorithm may be provided on a remote server, wherein image data sets to be evaluated are sent to the remote server. In such a configuration, by re-sorting image data sets as suitable or not suitable, not only computational effort and, optionally, monetary cost, can be reduced, but also communication traffic.

In an example, the at least one evaluation algorithm may comprise a coronary computed tomography angiography algorithm and/or a calcium scoring algorithm, wherein the image data sets are computed tomography data sets of an imaging region comprising the heart of the patient. In this context, for example, multiple image data sets may be acquired showing no contrast agent and/or different image data sets may comprise data acquired at different heart phases. For example, if multiple image data sets corresponding to different heart phases, that is, different intervals during the heart cycle, are provided in the input set, the image data set and thus heart phase best suited for a certain evaluation algorithm may be determined and chosen for advanced analysis.

All features and comments regarding the methods according to example embodiments apply to the evaluation systems according to example embodiments. The evaluation system may comprise at least one processor and/or at least one storage. The functional units may be implemented in software and/or hardware, in particular software running on the at least one processor. Further functional units may, of course, be provided to realize advantageous embodiments of the method.

A computer program according to at least one example embodiment can be directly loaded into an evaluation system and performs, if executed on a computing device of the evaluation system, the steps of a method according to example embodiments. The computer program may be stored on an electronically readable storage medium according to example embodiments, which hence comprises control information comprising at least one computer program according to example embodiments, such that, when the electronically readable storage medium is used in an evaluation system, the evaluation system performs the steps of a method according to example embodiments. The electronically readable storage medium may be a non-transitory medium, for example a CD-ROM.

FIG. 1is a flow chart of an embodiment of a method for operating an evaluation system. In the evaluation system, multiple evaluation algorithms are provided on a remote server to derive evaluation information from medical image data sets. The evaluation algorithms may or may not comprise trained functions of artificial intelligence, for example neural networks. In a concrete example, the evaluation algorithms may be configured to provide evaluation information regarding the coronary blood vessel tree of a patient, for example evaluation information regarding lesions, a calcium score and the like. For these evaluation algorithms to robustly yield high-quality evaluation information that is of clinical relevance, the image data set to be evaluated must fulfil certain requirements regarding quality and/or content.

InFIG. 1, the evaluation algorithms1a,1b, . . . are schematically indicated. For each evaluation algorithm provided in the evaluation system, requirement information2a,2b, . . . is provided, wherein the requirement information2a,2b, . . . describes the above-mentioned requirements regarding quality and/or content of image data sets to be evaluated by the corresponding evaluation algorithm1a,1b, . . . . In this embodiment, for each evaluation algorithm1a,1b, . . . or at least each group of evaluation algorithms1a,1b, . . . yielding the same evaluation information, that is, serving the same evaluation purpose, additional requirement information3a,3b, . . . is also provided. The additional requirement information3a,3b, . . . defines requirements regarding image quality and/or content required for a successful manual evaluation regarding the same evaluation purpose, i.e. determining the same evaluation information. Requirement information may comprise criteria evaluating certain parameters describing image quality and/or content, for example thresholds for quality parameters and the like.

If the evaluation system, in particular at least one of the evaluation algorithms1a,1b, . . . is to be used, an input set4is provided at an interface of the evaluation system. The input set4comprises at least one image data set5, wherein, in the example ofFIG. 1, for reasons of simplicity two image data sets5are shown. The input set4further comprises an identification of at least one of the evaluation algorithms1a,1b, . . . to be applied to at least one of the image data sets5. For reasons of a simpler explanation, in this case, only the evaluation algorithm1bis shown as a part of the exemplary input set4.

The input set4may be manually compiled, but can, at least partly, also be automatically derived. For example, the input set may be derived from a database like a PACS or electronic health records, as well as from DICOM studies. Such data bases may comprise image data sets5which have been acquired during an examination of a certain patient and/or a correlated series of examinations of a certain patient. While the image data sets5themselves, in particular if they are stored in the DICOM format, may already comprise meta information regarding the image data set5and the patient, additional patient information and/or meta information may be available in the data base, such that the purpose of the examination, suspected diagnosis and the like can be derived as well as information like imaging region, patient age, gender and more. Using such additional meta and/or patient information, evaluation algorithms1a,1b, . . . yielding useful, clinically relevant information when applied to the corresponding image data set5may be automatically chosen into the input set4. Regarding the image data sets5, for example, all image data sets5of a certain examination and/or series of examinations may be automatically selected.

If the input set4is at least partially compiled automatically, the decision support steps further discussed below may not only prevent unsuitable image data sets5from being evaluated by evaluation algorithms1a,1b, . . . , but is also becomes possible to identify patients for whom an advanced analysis by evaluation algorithms1a,1b, . . . could yield additional information of clinical relevance, to identify the most appropriate advanced analysis method to determine evaluation information of clinical relevance, and/or to identify appropriate image data set for performing the advanced analysis.

Every time an input set4is received, for each pair of image data set5and evaluation algorithm1bto be applied to it, in step S1a suitability information is determined. In the embodiment shown inFIG. 1, in a first sub-step S1a, the image data set5is analysed and image information describing image quality and/or content is derived. For example, the image information may comprise quality parameters like image sharpness, contrast, signal-to-noise ratio and/or information items like binary indicators of shown anatomical or generally medical features. Preferably, in step S1a, as part of the image information, it is also determined whether certain, in particular medical, features are included in the image data set, for example, whether a lesion and/or other medical concept is shown.

In step S1b, the image information is compared to the requirement information2bof the respective evaluation algorithm1bof the pair. For example, criteria can be evaluated. In this embodiment, the suitability information does not only include a binary indicator whether the requirements of the requirement information are fulfilled for the image data set, but also a suitability measure is determined. The suitability measure quantifies the evaluation algorithm-related quality and content of the image data set5. For example, it can be quantified by how much a threshold for a quality parameter is surpassed and/or different criteria can be weighted to deduce a single value for the suitability measure.

Once the suitability information for all pairs of image data sets5and evaluation algorithms1bof the input set4has been determined, in a step S2, it is checked whether suitability informations indicate fulfilment of all requirements of the respective requirement information. If the requirements are fulfilled regarding multiple image data sets5and one certain evaluation algorithm1b, only the image data set5having the highest suitability measure is forwarded for evaluation by the evaluation algorithm1bin a step S3. If, on the other hand, multiple evaluation algorithms1bhaving the same evaluation purpose, i.e. yielding the same evaluation information, were included in the input set4, the pair having the highest suitability measure is selected and the image data set5of the pair is forwarded to the evaluation algorithm1bof the pair. If, of course, the suitability information only indicates fulfilment for one pair of image data set5and a certain evaluation algorithm1b/group of comparable evaluation algorithms1b, the image data set5may simply be forwarded in step S3without further considering the suitability measure.

If, however, it is determined in step S2that for at least evaluation algorithm1bor group of comparable evaluation algorithms1bserving the same evaluation purpose, no suitability information indicates fulfilment of the requirements, it is proceeded to step S4, wherein respective additional requirement information3bis used to determine additional suitability information regarding manual reading of image data sets5. In this embodiment, the image information simply needs to be additionally compared to the additional requirement information3b, wherein, preferably, an additional suitability measure is determined as part of the additional suitability information.

In a step S5, at least the image data sets5for which an additional suitability information has been determined, which indicates fulfilling the requirements for manual reading, are marked for manual reading and respective user information is output as a user information action in step S6.

If, however, all additional suitability informations indicate non-fulfilment of the requirements of the additional requirement information3b, in step S7, corresponding user information may be output as a user information action. However, in embodiments, also image processing actions may be initiated in step S7, in particular for improving the image quality of at least one image data set5, which may then be checked again for suitability according to dashed arrow6. However, it is also possible to trigger re-reconstruction of an image data set5and/or even re-acquisition such that a completely new medical image data set5results and can be checked for suitability regarding advanced analysis. In this context, control information comprising acquisition parameters contributing to suitability may be generated and sent to an imaging device, where it may be used to control the re-acquisition process, increasing the probability to generate an image data set5suitable for advanced analysis by one of the evaluation algorithms1a,1b, . . . .

It is noted that, in case an updated image data set5is to be generated by an image processing action, confirmation from a user may be requested. Such confirmation may also be requested if, for example, an evaluation algorithm1a,1b, . . . and/or an image data set5not explicitly manually chosen is to be used in step S3. Hence, before forwarding the image data set5to the respective evaluation algorithm1a,1b, proposal information may be output and approving or disapproving user input may be received.

FIG. 2shows a principle, functional drawing of an evaluation system7according to at least one example embodiment. The evaluation system7comprises a local computing device8which may be positioned at a medical facility. The computing device8, which comprises at least one processor and at least one storage medium, is connected to a remote server9, which may be a server device and/or a cloud device, via a communication link10, in particular via the internet11. The remote server9comprises an evaluation unit12where the evaluation algorithms1a,1b, . . . , which are stored in a respective storage medium13, may be applied.

A local computing device8comprises a first interface14, where requirement information2a,2b, . . . and additional requirement information3a,3b, . . . may be provided, for example from the remote server9. The requirement information,2a,2b, . . . and3a,3b, . . . may be stored in a storage medium15of the computing device8. Via a second interface16, input sets4may be received, for example from a PACS17and/or directly from an imaging device18. In an analysis unit19, suitability information may be determined according to step S1, while in a decision unit20, the decision process of steps S2to S7may be executed, in particular including the respective actions in steps S3, S6and S7.

If, in step S3, an image data set5is forwarded to the evaluation unit12for respective processing by the respective evaluation algorithm1a,1b, . . . , the resulting evaluation information may be sent via the communication link10to the computing device8and output via a third interface21. It is noted that the third interface21may be at least partly integrated into the second interface16such that the evaluation information may be sent to the PACS17and/or the imaging device18for display and/or storing.

The computing device8may be part of an image reading work station also having an input and an output, such that, preferably, evaluation information can be displayed, manual reading may be alternatively or additionally performed, user information can be output and/or user input can be received.

Although example embodiments have been described in detail with reference to the preferred embodiments, example embodiments are not limited by the disclosed examples from which the skilled person is able to derive other variations without departing from the scope of example embodiments.