Patent Description:
Medical interventions in vessels to treat pathological vessel conditions, such as obstructions, typically rely on a catheter or some other means. Depending on the specific pathological vessel to be treated, the catheter is inserted into the vessels, e.g. starting at an arm or in the groin area and advanced via major vessels to reach the general vicinity of the pathological vessel. Since vessels tend to branch extensively, at least starting in the general vicinity of the pathological vessel a medical specialist or operator relies on imaging methods based on contrast media and radiation, such as fluoroscopy, to determine the path through the vessel to reach the pathological vessel. Depending on the position of the pathological vessel in the vessel tree and the skill of the medical specialist, a patient may be exposed to high doses of both radiation and contrast media during the medical intervention. Given the negative effects of such doses on the human body, shortening the time needed to reach the pathological vessel and thereby reducing the doses of radiation and of contrast medium would be beneficial. Therefore, it is an objective of the present invention to enable guidance to a pathological vessel.

<CIT> may be considered to disclose a computer-implemented pathological vessel guidance method, comprising the steps of: generating a vessel roadmap library , the vessel roadmap library including a plurality of vessel roadmaps , wherein each vessel roadmap comprises a vessel roadmap image and first and second alignment data ; wherein the method is used by the user to see, within the vessel roadmap images of the vessel roadmap library, a stenosis in a vessel; obtaining a real-time fluoroscopy image and corresponding real-time first and second fluoroscopy information; selecting a vessel roadmap from the roadmap library based on comparing the first real-time fluoroscopy information with the first alignment data of each vessel roadmap of the roadmap library; overlaying the real-time fluoroscopy image with the selected vessel roadmap image of the vessel roadmap library; aligning the vessel roadmap image of the selected vessel roadmap and the real-time fluoroscopy image.

To achieve this objective the present invention provides a computer-implemented pathological vessel guidance according to claim <NUM>.

Embodiments of the present invention will be described with reference to the following appended drawings, in which like reference signs refer to like elements.

It should be understood that the above-identified drawings are in no way meant to limit the disclosure of the present invention. Rather, these drawings are provided to assist in understanding the invention. The person skilled in the art will readily understand that aspects of the present invention shown in one drawing may be combined with aspects in another drawing or may be omitted without departing from the scope of the present invention.

The invention generally provides a computer-implemented pathological vessel guidance method. The method first generates a vessel roadmap library based on vessel images obtained from a patient using an imaging metho, such as angiography. The vessel images are then processed using image segmentation and to generate the vessel roadmap library. The processing of the vessel images additionally includes detecting one or more pathological vessels. During a subsequent fluoroscopy, the method selects a vessel roadmap from the vessel roadmap library and overlays the corresponding vessel roadmap image over the fluoroscopy image. To guide a fluoroscopy object inserted into the vessels during the fluoroscopy, the method limits the vessels visible in the vessel roadmap library to vessel leading to the detected one or more pathological vessels.

This general concept will be explained with reference to the appended drawings. <FIG> and <FIG> illustrate the various aspects of the pathological vessel guidance workflow based on the pathological vessel guidance method while <FIG> and <FIG> illustrate the various step of the pathological vessel guidance method. <FIG> are used throughout the discussion of <FIG> as well as 2A and 2B to illustrate certain aspects of the workflow and the method, respectively.

<FIG> and <FIG> shows a schematic diagram of a pathological vessel guidance workflow according to embodiments of the invention. <FIG> shows the orientation of <FIG> and <FIG> with regard to one another. The lines shown in <FIG> and <FIG> indicate where various data, such as vessel image <NUM> or the output of processing entities, such as vessel segmentation <NUM>, is provided to. The blocks shown in <FIG> indicate both processing entities, such as cardiac cycle detection <NUM>, and stored data elements, such as vessel roadmap <NUM>.

The workflow is separated into two phases, i.e., a roadmap generation phase <NUM> and a roadmap deployment phase <NUM>. Roadmap generation phase <NUM> is shown in <FIG>. Roadmap deployment phase <NUM> is shown in <FIG>. During roadmap generation phase <NUM>, a vessel image sequence <NUM> is processed to generate a vessel roadmap library <NUM>. Vessel roadmap library <NUM> is then deployed in roadmap deployment phase <NUM> to provide a dynamic vessel roadmap for guidance of a fluoroscopy object during fluoroscopy. First, roadmap generation phase <NUM> will be described.

Vessel image sequence <NUM> may include a plurality of vessel images <NUM>. Vessel images <NUM> are images of vessels or a vessel tree, such as the coronary arteries, the iliac veins or the lumbar lymph trunk. Generally, in the context of the present application, vessels may therefore refer to arteries, veins or lymphatic vessels. Vessel images <NUM> and more generally vessel image sequence <NUM> may be obtained using any medical imaging method capable of rendering vessels visible, such as angiography. In particular, medical imaging methods used to obtain vessel image sequence <NUM> may be based on injecting a radio-opaque contrast medium into vessels via a contrast application object and rendering the radio-opaque contrast medium visible via e.g., an x-ray. In some embodiments, vessel images <NUM> may be stored as or may be Digital Imaging and Communications in Medicine (DICOM) images.

Vessel image sequence <NUM> may include imaging physiological information <NUM>. Imaging physiological information <NUM> may be any physiological information of a patient recorded while the imaging method used to obtain vessel images <NUM> is performed and which may be used to later overlay and align vessel roadmaps with a real-time fluoroscopy image. In particular, imaging physiological information <NUM> may be recorded at approximately the same point in time as the corresponding vessel image <NUM>. In some embodiments, imaging physiological information <NUM> may include or may be an electrocardiogram (ECG) recorded while the imaging method used to obtain vessel images <NUM> is performed. In some embodiments, imaging physiological information may be stored as or may be a DICOM data tag included in the DICOM image file of the corresponding vessel image <NUM>. Imaging physiological information <NUM> may be included in, e.g. stored as part of, first alignment data <NUM> of a vessel roadmap <NUM>.

Vessel image sequence <NUM> may include imaging information <NUM>. Imaging information <NUM> may indicate one or more parameters associated with an imaging method used to obtain vessel images <NUM>. As discussed above, the imaging method used to obtain vessel images <NUM> may for example be angiography. Accordingly, imaging information <NUM> may include at least one of an angiography angle and a contrast medium dosage.

Briefly referring to <FIG>, both show a medical imaging system <NUM> including a rotatable C arm <NUM>. X-ray emission means <NUM> and X-ray detection means <NUM> may be mounted on C arm <NUM>. In <FIG>, C arm <NUM> is in a neutral position P<NUM>, i.e. X-ray emission means <NUM> are located directly above a patient surface <NUM>. In <FIG>, C arm <NUM> and thereby X-ray emission means <NUM> are rotated counterclockwise with respect to neutral position P<NUM> of C-arm <NUM> in <FIG> to a position P<NUM>. The angle between position P<NUM> and position P<NUM>, as indicated in <FIG>, is referred to as the angiography angle or the fluoroscopy angle, depending on the medical imaging process. It will be understood that the neutral position may be used as an imaging position. In such a case, the angiography angle is <NUM>°. Further, in case of a single axis angiography system, the neutral position is typically defined as shown in <FIG>. In multiple axis angiography systems, additional C arms may be present, such as a ceiling mounted C arm. In such a case, the neutral position may be defined as the position in which X-ray emission means <NUM> and X-ray detection means <NUM> are at the same level as a patient on patient surface <NUM>.

While the above definition of the angiography angle is based on the position of X-ray emission means <NUM>, the angiography angle may analogously be defined based on the position of X-ray detection means <NUM>.

The contrast medium dosage may indicate the dosage of radio-opaque contrast medium administered to a patient in order to render the vessels of the patient visible during the imaging method. The contrast medium dosage may be measured in milliliters per kilogram of body weight. In the case of contrast media including iodine, the contrast medium dosage may also be measured in milligram iodine per kilogram of body weight.

Vessel images <NUM> may be processed by contrast detection <NUM> to detect contrasted vessel images among the plurality of vessel images <NUM>. Contrast detection <NUM> may detect contrasted vessel images by detecting the presence of contrast medium in vessel images <NUM>. Contrast detection <NUM> may in some embodiments analyze the pixels of each vessel image <NUM> to detect, based on a property of the pixels, the presence of contrast medium in the respective vessel image <NUM>. The property of the pixels may e.g., a brightness value or a color value. In some embodiments, a vessel image <NUM> may be identified as a contrasted vessel image if a single pixel indicative of contrast medium is identified. In some embodiments, a vessel image <NUM> may be identified as a contrasted vessel image if a number of pixels indicative of contrast medium which exceeds a contrast medium detection threshold is identified. In some embodiments, contrast detection <NUM> may detect contrasted vessel images among the plurality of vessel images using a deep learning model, such as DeepNet, which identifies contrast medium in the vessel images <NUM> and provides a frame-by-frame classification of the vessel image sequence <NUM>. In some embodiments, contrast detection <NUM> may also take into account the contrast medium dosage indicated by imaging parameters <NUM>.

The contrasted vessel images detected by contrast detection <NUM> are provided to vessel segmentation <NUM> and contrast application segmentation <NUM>. Furthermore, each contrasted vessel image may be included in, e.g., stored as part of, a vessel roadmap image <NUM> of vessel roadmap <NUM>.

Vessel segmentation <NUM> may perform vessel segmentation on the contrasted vessel images to generate vessel segmentation data. Accordingly, vessel segmentation <NUM> may generate data indicative of the position and/or the course of the vessels within the contrasted vessel images. Vessel segmentation data may be generated by vessel generation <NUM> based on a variety of image segmentation approaches, such as based on convolutional neural networks (CNN), e.g. U-Net, densely connected neural networks, deep-learning methods, graph-partitioning methods, e.g. Markoff random fields (MRF), or region-growing methods, e.g. split-and-merge segmentation. The vessel segmentation data may then be included in, e.g. stored as part of, a vessel roadmap image <NUM> of a vessel roadmap <NUM>.

It should be noted that, while vessel segmentation <NUM> is shown in <FIG> as processing contrasted vessel images, vessel segmentation may also directly process vessel images <NUM> to generate vessel segmentation data prior to detecting contrasted vessel images. In some embodiments, vessel segmentation <NUM> may also be incorporated into or be part of contrast detection <NUM>. In such embodiments, detecting pixels indicative of contrast mediums may at the same time be used to generate vessel segmentation data.

Contrast application object segmentation <NUM> may perform contrast application object segmentation on the contrasted vessel images to generate contrast application object segmentation data. Accordingly, contrast application object segmentation data may identify a position and/or the course of the contrast application object in the contrasted vessel images. More precisely, the contrast application object segmentation data may in some embodiments be a catheter, which is used to apply the contrast medium. In such embodiments, the contrast application object segmentation data may indicate the position and/or the course of the tip of the catheter, the body of the catheter or the guidewire of the catheter. Contrast application object segmentation data may, like vessel segmentation data, be generated based on a variety of image segmentation approaches, including, but not limited to, U-Net or MRF. The contrast application object segmentation data may then be included in, e.g. stored as part of, second alignment data <NUM> of vessel roadmap <NUM>.

While contrast application object segmentation <NUM> is shown in <FIG> as processing contrasted vessel images, contrast application object segmentation <NUM> may also directly process vessel images <NUM>.

Both vessel segmentation <NUM> and contrast application object segmentation <NUM> perform image segmentation on vessel images <NUM> or contrasted vessel images, respectively. To better illustrate possible segmentations performed on vessel images <NUM> or contrasted vessel images, respectively, <FIG> shows three example segmentations of a vessel image <NUM>. From left to right, <FIG> shows an example of vessel segmentation data <NUM> and two examples of contrast application object segmentation data <NUM>. As can be seen, vessel segmentation data <NUM> indicate the position and the course of the vessels in vessel image <NUM>. The left vessel segmentation data <NUM> indicate the position and the course of a catheter body 161a. The right vessel segmentation data <NUM> indicate the position of a catheter tip 161b.

Pathological vessel detection <NUM> detects pathological vessels in the contrasted vessel images. To this end, pathological vessel detection <NUM> performs image segmentation on the contrasted vessel images to identify areas within the contrasted vessel images, which include pathological vessel segments. In one example, pathological vessel detection <NUM> may detect one or more pathological vessels based on any one of the image segmentation approaches trained on vessel images with annotated pathological vessels. The pathological vessels detected by pathological vessel detection <NUM> may then be included as pathological vessel data <NUM> in vessel roadmap <NUM>.

Pathological conditions in vessels often affect the lumina of the vessel. Accordingly, pathological vessel detection <NUM> may in a further example detect pathological vessels based on their lumina. Accordingly, pathological vessel detection <NUM> may first determine centerlines of the vessels included in the contrasted vessel images. Then, pathological vessel detection <NUM> may determine, based on the centerlines, lumina of the vessels included in each vessel roadmap image of the vessel roadmap library. Finally, pathological vessel detection <NUM> may detect one or more pathological vessels based on the lumina of the vessels included in the contrasted vessel images.

A pathological vessel in the context of the present application may be any vessel experiencing a pathological condition, such as a stenosis, a lesion, a vasoconstriction or a vasodilation.

Pathological vessel detection <NUM> may further determine, e.g., based on the determined lumen of the pathological vessel, the grade of a pathological condition, e.g. a stenosis in a pathological vessel, as e.g., defined by the Society of Cardiovascular Computed Tomography (SCCT). Accordingly, pathological vessel detection <NUM> may, based on a comparison of the total lumen with the unobstructed lumen, determine the grade of the stenosis. It will be understood that pathological vessel detection <NUM> may be able to also grade stenoses in vessels other than the coronary arteries based on the grading system applicable to such vessels.

In some cases, pathological vessel detection <NUM> may detect more than one pathological vessel. In such cases, pathological vessel detection <NUM> may further be able to determine in which order the pathological vessels may be treated during a medical intervention. Such a determination may for example be based on a starting position of a medical intervention and the position of the pathological vessel relative to the starting position or may be based on the grading of the severity of the pathological conditions of the pathological vessels discussed above.

It will be understood that pathological vessel detection <NUM> may also perform the detection based on vessel images <NUM> directly or may be integrated with any one of the other processing entities of roadmap generation phase <NUM> performing image segmentation, i.e., contrast detection <NUM>, vessel segmentation <NUM> and contrast application object segmentation <NUM>.

Imaging physiological information <NUM> may be processed by cardiac cycle detection <NUM>. Cardiac cycle detection <NUM> may identify one or more cardiac cycles within imaging physiological information <NUM>. Cardiac cycle detection <NUM> may identify at least one cardiac cycle within imaging physiological information <NUM> by detecting a first R peak and a second R peak within imaging physiological information <NUM>. The first and the second R peak may indicate the start and the end, respectively, of a cardiac cycle. It should be understood that any other graphical deflection of an ECG may be used to indicate the start and the end of a cardiac cycle. Accordingly, cardiac cycle detection may e.g. detect a first and a second P wave.

In some embodiments, detecting the first R peak and the second R peak is based on a combined adaptive electrical activity threshold, which may be the sum of a steep-slope threshold, an integration threshold and an expected beat threshold. In such embodiments, an R peak may be detected if the amplitude of a complex lead is equal to or above an amplitude value of a complex lead, e.g. derived from a <NUM>-lead. Of course, any type of ECG lead may be used to obtain an amplitude of a complex lead, such as a Wilson or a Goldberger ECG.

The steep-slope threshold may initially be set at <NUM>% of the maximum of the amplitude value of the complex lead measured during an initialization period, e.g. <NUM>. Once the steep slope threshold as well as the other two thresholds have been initialized, a first QRS complex is detected. During some recalculation interval after detection of the first QRS complex, e.g. <NUM>, the steep-slope threshold may be set at <NUM>% of the maximum of the amplitude value of the complex lead measured during the recalculation interval. Afterwards, the steep-slope threshold may be decreased during a decrease interval, e.g. <NUM>, i.e. <NUM> after detection of the first QRS complex, to <NUM>% of its value, i.e. <NUM>% of the measured amplitude of the complex lead. Afterwards, the steep-slope threshold remains at <NUM>%. The decrease may be stopped once a new QRS complex is detected, assuming that the decrease interval has not expired, and the steep-slope threshold may be recalculated as described above. In some embodiments, the steep slope threshold may be set to an average of the currently calculated <NUM>% of the maximum of the amplitude value of the complex lead measured during the recalculation interval and the preceding four calculated values. In some embodiments, the recalculated value of the steep-slope threshold may further be limited to <NUM>% of the previous value of the steep-slope threshold if the recalculated value exceeds <NUM>% of the previous value.

The integration threshold may initially be set to the mean value of the first derivative of the amplitude of the complex lead with regard to time during an integration threshold initialization interval, which may e.g., be <NUM>. Afterwards, the integration threshold is updated based on an update time interval, which may be <NUM>. For every update time interval, the integration threshold may be calculated as the sum of the previous integration threshold and an update value. The update value may be calculated by subtracting from the maximum value of the amplitude of the complex lead during an initial period of the update time interval, e.g., <NUM>, the maximum value of the amplitude of the complex lead during a final period of the update time interval, e.g., <NUM>. The update value may further be divided by a weighting factor, which may e.g., be <NUM>.

The expected beat threshold may initially be set to zero. To determine the expected beat threshold, the average time difference between R peaks may be calculated. In some embodiments, the average time difference may be based on the previous five time differences between R peaks. After a fraction, such as two-thirds, of the average time difference sine the last R peak has expired, the expected beat threshold may be reduced at a fraction of the rate at which the steep-slope threshold may be decreased during the decrease time interval, e.g. at <NUM>% percent of the decrease of the steep-slope threshold. Once the average time difference has expired, the expected beast threshold remains constant. Upon detection of a new R peak, the expected beat threshold may be re-set to zero and be decreased during the next cardiac cycle based on a re-calculated average time difference between R peaks as discussed above. Accordingly, the expected beat threshold may decrease the combined adaptive electrical activity threshold toward the expected end of a cardiac cycle.

It should be noted that the above discussion of how a cardiac cycle may be detected based on combined adaptive electrical activity threshold is merely provided as an example. Cardiac cycle detection <NUM> may detect cardiac cycles using any suitable analysis of an ECG. For example, cardiac cycle detection <NUM> may detect cardiac cycles based on performing different types of analyses based transforms, such as short-time Fourier-transform or based on deriving event vectors from the ECG and providing decision rules, which determine cardiac cycles based on the derived event vectors.

The one or more cardiac cycles detected by cardiac cycle detection <NUM> may be included in, e.g. stored as part of, first alignment data <NUM>. In addition, the one or more cardiac cycles detected by cardiac cycle detection <NUM> may be provided to EDR detection <NUM>.

EDR detection <NUM> may derive, based on the one or more cardiac cycles, an electrocardiogram derived respiratory (EDR) signal. The EDR signal may be derived by observing fluctuations between detected different cardiac cycles. Generally speaking, inhaling typically increases and exhaling typically decreases the heart rate. More precisely, such fluctuations can in some embodiments be used to derive the EDR signal by computing for the one or more detected cardiac cycles, the R-to-S peak. The R-to-S peak corresponds to the amplitude of the EDR signal. The R-to-S-peaks are then interpolated using cubic splines to obtain the EDR signal. It should be understood that in some embodiments, other approaches may be used to obtain an EDR signal from the imaging physiological information <NUM>. The EDR signal may be included in, e.g. stored as part of, second alignment data <NUM>.

It should be noted that in some embodiments, EDR detection <NUM> may be omitted. In such embodiments, second alignment data <NUM> may only include contrast application object segmentation data. Further, in some embodiments EDR detection <NUM> may be included in cardiac cycle detection <NUM>. For example, the approach used by cardiac cycle detection <NUM> to identify one or more cardiac cycles may also be able to directly provide an EDR signal or be able to provide an EDR signal based on an intermediate step of the approach.

As mentioned above, the output of processing entities <NUM> to <NUM> forms part of vessel roadmaps <NUM>, which are part of vessel roadmap library <NUM>. To illustrate the data, which may be included in a vessel roadmap <NUM>, <FIG> shows an example vessel roadmap <NUM>, including a vessel roadmap image <NUM>, first alignment data <NUM>, second alignment data <NUM>, imaging parameters <NUM> and pathological vessel information <NUM>.

Vessel roadmap image <NUM> may include a vessel image 211a. Vessel roadmap image 211a corresponds to a vessel image <NUM> of the vessel image sequence <NUM>, which has been identified by contrast detection <NUM> as including contrast. Further, vessel roadmap image <NUM> may include vessel segmentation data 211b generated by vessel segmentation <NUM>. Both vessel roadmap 211a and vessel segmentation data 211b may be used as the roadmap to be laid over a fluoroscopy image during the roadmap deployment phase <NUM> by vessel overlay roadmap <NUM> discussed later. Accordingly, vessel roadmap image <NUM> is the part of vessel roadmap <NUM> providing the actual vessel roadmap. In some embodiments, vessel roadmap image <NUM> may correspond to a vessel image <NUM> recorded as part of the vessel image sequence <NUM> as well as an image indicating the vessel segmentation data generated by vessel segmentation <NUM>. In some embodiments, vessel roadmap image <NUM> may only include an image indicating the vessel segmentation data generated by vessel segmentation <NUM>, as illustrated by vessel segmentation data 211b in <FIG>. In some embodiments, vessel roadmap image <NUM> may be data indicating the position of vessel pixels, which may subsequently be used to highlight corresponding pixels in a fluoroscopy image as vessel pixels.

First alignment data <NUM> may include imaging physiological information <NUM> as recorded as part of the vessel image sequence <NUM>. Since imaging physiological information <NUM> may in some embodiments be an ECG, imaging physiological information <NUM> is shown in <FIG> as an alignment ECG 212a. Further, first alignment data <NUM> may include one or more cardiac cycles as detected by cardiac cycle detection <NUM>. In <FIG>, a cardiac cycle 212b is illustrated as indicted by adjacent R peaks in alignment ECG 212a. While first alignment data <NUM> of vessel roadmap <NUM> is shown here as based on an ECG curve, it should be understood that alignment ECG 212a and cardiac cycle 212b may typically be stored within second alignment data <NUM> as an indication of the ECG value, i.e., e.g. the amplitude of the complex lead discussed above, and the position of the ECG value within a cardiac cycle.

Accordingly, the first alignment data may in some embodiments not include the entire alignment ECG 212a. Instead, first alignment data <NUM> may e.g., be a tuple with the first value indicating the amplitude and the second value indicating the position within a cardiac cycle. In some embodiments, if more than one cardiac cycle has been identified. Fist alignment data <NUM> may be a triple with the third value identifying the respective cardiac cycle. In some embodiments, in which cardiac cycle detection <NUM> is omitted, the cardiac cycle information may be replaced by temporal information indicating an associated point in time of the ECG value relative to other vessel roadmaps <NUM>.

More generally, first alignment data <NUM> provide data to enable aligning vessel roadmap image <NUM> with a fluoroscopy image. Vessels shift for a variety of factors, including, but not limited to, cardiac muscle contraction and relaxation, i.e., cardiac motion, as well as breathing. First alignment data <NUM> enable compensation of vessel shift caused by cardiac motion. To this end, first alignment data <NUM> provide physiological information relating to the heart. It should therefore be understood that first alignment data <NUM> generally provide data enabling an alignment of a vessel roadmap with a fluoroscopy image necessitated due to vessel shifts caused by cardiac motion. First alignment data <NUM> may thus include any cardiac information necessary to compensate such vessel shifts.

Second alignment data <NUM> may include an EDR signal 213a generated by EDR detection <NUM>, and contrast application object segmentation data 213b generated by contrast application object segmentation <NUM>. EDR signal 213a is shown as a curve in <FIG> in order to illustrate EDR signal 213a. However, similarly to the discussion of alignment ECG 212a, EDR signal 213a may in some embodiments be a data set indicating the value of EDR signal 213a as well as the temporal position of the value within the EDR signal curve. Further, as shown in <FIG>, contrast application object segmentation data 213b may in some embodiments be an image indicating the vessel segmentation data. In some embodiments, contrast application object segmentation data 213b may be data indicating the position of contrast application object pixels.

More generally, second alignment data <NUM> provide data to enable aligning vessel roadmap image <NUM> with a fluoroscopy image. While first alignment data <NUM> are described above as compensating vessel shift caused by cardiac motion, second alignment data <NUM> may compensate for vessel shift caused by breathing motion. To this end, second alignment data <NUM> provide both physiological information relating to the breathing and information relating to the position of the contrast application object, which may be shifted due to breathing. It should therefore be understood that second alignment data <NUM> generally provide data enabling an alignment of a vessel roadmap with a fluoroscopy image necessitated due to vessel shifts caused by breathing motion. Second alignment data <NUM> may thus include any breathing-related information necessary to compensate such vessel shifts. For example, in some embodiments, second alignment data <NUM> may include only one of EDR signal 213a and contrast application object segmentation data 213b since in some embodiments only one of the two may be sufficient to compensate vessel shifts cause by breathing motion. For example, in some embodiments EDR signal 213a may be omitted.

As discussed above, contrast application object segmentation data 213b is generated by contrast application object segmentation <NUM>. Accordingly, second alignment data <NUM> is in some embodiments at least derived from vessel roadmap image <NUM>. In embodiments, in which EDR signal 213a is also present, second alignment data <NUM> may further be derived from first alignment data <NUM> in addition to being derived from vessel image <NUM>.

Imaging parameters <NUM> may include imaging parameters <NUM> associated with the imaging method used to obtain vessel image <NUM> included as the vessel roadmap image <NUM>. As shown in <FIG>, imaging parameters <NUM> may thus include angiography angle 214a and contrast medium dosage 214b.

Pathological vessel information <NUM> is generated by pathological vessel detection <NUM>. Accordingly, pathological vessel information <NUM> indicates pathological vessels in vessel roadmap <NUM>. As illustrated in <FIG>, pathological vessel information <NUM> may e.g. correspond to vessel roadmap <NUM> with a highlighted region <NUM> indicating a pathological vessel. In addition to or instead of, pathological vessel information <NUM> may in some embodiments include data identifying pixels in vessel roadmap <NUM>, which are part of a pathological vessel segment. Generally speaking, pathological vessel information <NUM> may be any kind of information indicating which of the vessel visible in vessel roadmap <NUM> is a pathological vessel.

Vessel roadmap library <NUM> is the output generated by roadmap generation phase <NUM>. This output may subsequently be deployed during roadmap deployment phase <NUM>. In some embodiments, roadmap deployment phase <NUM> may be a medical intervention, such as a percutaneous coronary intervention (PCI). PCI is performed using fluoroscopy. Accordingly, vessel roadmap library <NUM> can be laid over the real-time fluoroscopy images during the PCI in order to guide a medical specialist through the coronary arteries to e.g., a stenosis without having to use a contrast medium. In such embodiments, roadmap generation phase <NUM> may include coronary angiography, which is used to obtain a coronary angiogram. The coronary angiogram in such embodiments corresponds to vessel image sequence <NUM>. Since roadmap deployment phase <NUM> may be a medical intervention, it may also be referred to as an online phase, while roadmap generation phase <NUM> may also be referred to as an offline phase.

Roadmap deployment phase <NUM> performs real-time fluoroscopy <NUM> in order to obtain a real-time fluoroscopy image <NUM> and corresponding real-time first fluoroscopy information <NUM>, real-time second fluoroscopy information <NUM> and imaging parameters <NUM>.

Real-time fluoroscopy image <NUM> may be an image obtained using real-time fluoroscopy <NUM>. During real-time fluoroscopy <NUM>, no contrast medium needs to be injected into vessel or a vessel tree given that vessel images are provided by vessel roadmap library <NUM>. Accordingly, since fluoroscopy is typically performed using X-ray, the only radio-opaque structure visible in the real-time fluoroscopy image <NUM>, apart from e.g. bones of the patient, is a fluoroscopy object. Like vessel image <NUM>, real-time fluoroscopy image <NUM> may be stored as a DICOM image.

First fluoroscopy information <NUM> may be any kind of fluoroscopy physiological information of a patient on whom real-time fluoroscopy <NUM> is performed and which may be used to overlay and align vessel roadmaps <NUM> with fluoroscopy image <NUM>. In some embodiments, first fluoroscopy information <NUM> may include an ECG recorded while real-time fluoroscopy <NUM> is performed. In such embodiments, first fluoroscopy information <NUM> may be processed by cardiac cycle detection <NUM> to identify one or more cardiac cycles based on the ECG. The identified one or more cardiac cycles may then also be included in the real-time fluoroscopy information <NUM>. Cardiac cycle detection <NUM> may detect one or more cardiac cycles in the fluoroscopy ECG in the same manner as cardiac cycle detection <NUM> may detect one or more cardiac cycles in the ECG recorded while vessel image sequence <NUM> is obtained.

The one or more cardiac cycles identified by cardiac cycle detection <NUM> may be processed by EDR detection <NUM>. EDR detection <NUM> may derive an EDR signal based on the identified one or more cardiac cycles in the same manner as describe with respect to EDR detection <NUM>. The derived EDR signal may then be included in second real-time fluoroscopy information <NUM>.

As discussed above, in some embodiments EDR detection <NUM> may be omitted. In such embodiments, EDR detection <NUM> may additionally derive an EDR signal based on the one or more cardiac cycles included in first alignment data <NUM> of a vessel roadmap <NUM> selected by roadmap selection <NUM>, which will be discussed in more detail below. It should be noted that EDR detection <NUM> may also be omitted if EDR detection <NUM> is omitted. In such embodiments, vessel roadmap selection <NUM> and vessel roadmap alignment <NUM> may operate without any EDR signals.

In addition to the EDR signal, second real-time fluoroscopy information <NUM> may include fluoroscopy object segmentation data identifying a position of a fluoroscopy object in fluoroscopy image <NUM>. The fluoroscopy object may be a fluoroscopy catheter, which may e.g., be used during a medical intervention, such as PCI. As opposed to the contrast application object, which may also be a catheter, the fluoroscopy catheter may typically not be used to inject contrast medium into a vessel, though it may still be configured for that purpose. The fluoroscopy segmentation data may be generated by fluoroscopy segmentation <NUM> based on a variety of image segmentation approaches as discussed above with regard to vessel segmentation <NUM> and contrast application object <NUM>, such as based on CNNs or MRFs.

Imaging parameters <NUM> may indicate one or more parameters associated with an imaging method used to obtain real-time fluoroscopy image <NUM>. Accordingly, imaging parameters <NUM> may correspond to imaging parameters <NUM> and may thus include at least one of a fluoroscopy angle and a contrast medium dosage.

The data recorded by real-time fluoroscopy <NUM> and processed by processing entities <NUM>, <NUM> and <NUM> is illustrated in <FIG>.

As shown in <FIG>, real-time fluoroscopy image <NUM> includes the fluoroscopy image recorded by real time fluoroscopy <NUM>. Typically, only the fluoroscopy object is visible in real-time fluoroscopy image <NUM>, unless other radio-opaque structures of the patient are visible.

First fluoroscopy information <NUM> includes fluoroscopy ECG 312a recorded during real-time fluoroscopy and one or more cardiac cycles 312b identified by cardiac cycle detection <NUM>. Analogously to first alignment data <NUM>, first fluoroscopy information <NUM> may include any cardiac information necessary to compensate vessel shifts when laying one of the vessel roadmap images <NUM> over real-time fluoroscopy image <NUM>. Accordingly, in some embodiments, first fluoroscopy information <NUM> may only include fluoroscopy ECG 312a or may include other or additional information to compensate vessel shifts when laying one of the vessel roadmap images <NUM> over real-time fluoroscopy image <NUM>. Further, while fluoroscopy ECG 312a is illustrated as the ECG curve recorded during real-time fluoroscopy <NUM>, in some embodiments only values corresponding to the point on the ECG curve may be included in first real-time fluoroscopy information <NUM>, e.g. a tuple or a triple, respectively, including the amplitude value of the complex lead of the fluoroscopy ECG 312a, the temporal position within the ECG and an indication of the identified cardiac cycle.

Second real-time fluoroscopy information <NUM> may include a fluoroscopy EDR signal 313a generated by EDR detection <NUM> and contrast application object segmentation data 313b generated by contrast application object segmentation <NUM>. EDR signal 313a is shown as a curve in <FIG> in order to illustrate fluoroscopy EDR signal 313a. However, similarly to the discussion of preceding discussions of curves, i.e. alignment ECG 212a, EDR signal 213a and fluoroscopy ECG 312a, fluoroscopy EDR signal 313a may in some embodiments be a data set indicating the value of fluoroscopy EDR signal 313a as well as the temporal position of the value within the fluoroscopy EDR signal curve. Further, as shown in <FIG>, fluoroscopy object segmentation data 313b may in some embodiments be an image indicating the vessel segmentation data. In some embodiments, fluoroscopy object segmentation data 313b may be data indicating the position of fluoroscopy object pixels.

The data obtained by real-time fluoroscopy <NUM> and vessel roadmap library <NUM> may be provided to a vessel roadmap selection <NUM>. Vessel roadmap selection <NUM> may select one of the vessel roadmaps <NUM> from roadmap library <NUM> based on comparing first real-time fluoroscopy information <NUM> with first alignment data <NUM> of each vessel roadmap <NUM> in roadmap library <NUM>.

More precisely, vessel roadmap selection <NUM> may compare alignment ECG 212a of each vessel roadmap <NUM> with fluoroscopy ECG 312a to determine a vessel roadmap <NUM> approximately corresponding, in terms of the respective alignment ECG 212a, to fluoroscopy ECG 312a. By selecting a vessel roadmap based on an ECG comparison, a vessel roadmap <NUM> can be chosen in which the position of the vessels is most similar, due to a similar shift caused by similar cardiac motion, to the position of the vessels in real time fluoroscopy image <NUM>.

In addition, in some embodiments vessel roadmap selection <NUM> may select a vessel roadmap <NUM> from roadmap library <NUM> based on comparing second real-time fluoroscopy information <NUM> with the second alignment data <NUM> of each vessel roadmap <NUM>. More precisely, vessel roadmap selection <NUM> may additionally compare EDR signal 213a of each vessel roadmap <NUM> with fluoroscopy EDR signal 313a to determine a vessel roadmap <NUM> approximately corresponding, in terms of the respective EDR signal 213a, to fluoroscopy EDR signal 313a. By selecting a vessel roadmap additionally based on an EDR signal comparison, a vessel roadmap <NUM> can be chosen in which the position of the vessels is most similar, due to a similar shift caused by similar breathing motion, to the position of the vessels in real time fluoroscopy image <NUM>.

In summary, vessel roadmap selection <NUM> may, based on an ECG comparison and in some embodiments based on an additional EDR comparison, select a vessel roadmap <NUM> from vessel roadmap library <NUM>. The vessels visible in the accordingly selected vessel roadmap <NUM> have experienced a similar shift due to cardiac motion as well as due to breathing motion. The position of the vessels visible in the accordingly selected vessel roadmap <NUM> may thus be approximately similar to the position of the vessels in real-time fluoroscopy image <NUM>.

Angle comparison <NUM> may compare the angiography angle included in the imaging parameters <NUM> of vessel roadmap <NUM> with the fluoroscopy angle included in imaging parameters <NUM> of real-time fluoroscopy <NUM>. The comparison performed by angle comparison <NUM> may ensure that the view provided by selected vessel roadmap image <NUM> and the view provided by real-time fluoroscopy image <NUM> are obtained at approximately the same C-arm position. If the angiography angle and the fluoroscopy angle are approximately the same, the views provided by selected vessel roadmap image <NUM> and real-time fluoroscopy image <NUM> are approximately similar. If the angiography angle and the fluoroscopy angle differ, e.g., differ by more than an angle difference threshold, the views provided by selected vessel roadmap image <NUM> and real-time fluoroscopy image <NUM> may be too different. Accordingly, in such cases angle comparison <NUM> may decide to refrain from overlaying vessel roadmap image <NUM> over real-time fluoroscopy image <NUM>. Due the deviation of the views in 3D corresponding to the difference of the angles, vessel roadmap overlay <NUM> and vessel roadmap alignment <NUM> may not be able properly overlay and align the views. The angle difference threshold may e.g., be <NUM>° or <NUM>°.

Angle comparison <NUM> may in some embodiments indicate, via a display, such as display <NUM>, to a medical specialist operating a medical imaging device, such as medical imaging device <NUM>, that the angiography angle and the fluoroscopy angle differ. Angle comparison <NUM> may further indicate how to correct the fluoroscopy angle by indicating the proper angular position to which C arm <NUM> of medical imaging device <NUM> should be moved. In some embodiments, angle comparison <NUM> may also be configured to control C arm <NUM> and may thus be configured to move C arm <NUM> to the proper angular position.

Vessel roadmap overlay <NUM> may lay the selected vessel roadmap image <NUM> of the selected vessel roadmap <NUM> over the real-time fluoroscopy image <NUM>. In some embodiments, vessel roadmap overlay <NUM> may perform the overlay by superimposing vessel image 211a over real-time fluoroscopy image <NUM>. Superimposing vessel image 211a may in some embodiments be achieved by transparent color blending, i.e. two pixel values from vessel image 211a, one corresponding to the pixel value as recorded originally in corresponding vessel image <NUM> and one corresponding to a color selected for vessel representation, can be simultaneously shown. In some embodiments, vessel roadmap image 211a may be overlaid with a variable level of opacity. In some embodiments, vessel segmentation data 211b may be integrated into real-time fluoroscopy image <NUM>, e.g. by changing the values of the pixels indicated as vessel pixels by vessel segmentation data 211b.

<FIG> provides an example of vessel roadmap overlay <NUM>. On the left, real-time fluoroscopy image <NUM> with a visible fluoroscopy object can be seen prior the overlay of a vessel roadmap image <NUM>. On the right, real-time fluoroscopy image <NUM> can be seen after the overlay of a vessel roadmap image <NUM>.

Roadmap image <NUM> laid over real-time fluoroscopy image <NUM> by vessel roadmap overlay <NUM> may be aligned by vessel roadmap alignment <NUM>. Vessel roadmap alignment <NUM> aligns overlaid vessel roadmap image <NUM> and real-time fluoroscopy image <NUM> based on second alignment data <NUM> and real time second fluoroscopy information <NUM>. In particular, vessel roadmap alignment <NUM> aligns the position of the contrast application object with the position of the fluoroscopy object based on contrast application object segmentation data 213b and fluoroscopy object segmentation data 313b. In other words, vessel roadmap alignment <NUM> aligns the positions of the contrast application object and the fluoroscopy object, which may both be catheters. For example, both object segmentation data 213b and fluoroscopy object segmentation data 313b may each indicate, in some embodiments, a centerline of the respective catheter. In such embodiments, vessel roadmap alignment <NUM> aligns vessel roadmap <NUM> with real-time fluoroscopy image <NUM> by minimizing the sum of squared distances between closest points on the centerlines of the catheters. It should be noted that aligning may include in-plane rotation, i.e. rotating the roadmap to achieve better alignment. By aligning the positions of the contrast application object with the position of the fluoroscopy object, any vessel shift caused by breathing motion may be further compensated in order to further improve the accuracy of overlaid vessel roadmap <NUM>.

<FIG> provides an example of an alignment by vessel roadmap alignment <NUM> based on catheter centerlines. As can be seen on the left side of <FIG>, prior to alignment by vessel roadmap alignment <NUM>, the centerlines are apart from one another. After alignment by vessel roadmap alignment <NUM>, the centerlines are approximately in the same position.

Finally, pathological vessel guidance <NUM> providing guidance for the fluoroscopy object to the pathological vessel detected by pathological vessel detection <NUM> based on the selected vessel roadmap <NUM> and second fluoroscopy information <NUM>. To this end, pathological vessel guidance <NUM> may determine a path from the fluoroscopy object to the pathological vessel based on the second real-time fluoroscopy information <NUM>, which includes the fluoroscopy segmentation data, and the vessel segmentation data 211b included in the selected vessel roadmap image <NUM>. In other words, the path may indicate the vessel segments the fluoroscopy object needs to pass through in order to reach the pathological vessel from the current position of the fluoroscopy object within the vessels of a patient. Accordingly, pathological vessel guidance <NUM> may provide guidance for the fluoroscopy object by indicating the vessel segments through which the fluoroscopy object needs to pass in order to reach the pathological vessel.

Pathological vessel guidance <NUM> may in some embodiments. provide the guidance to a medical specialist operating the fluoroscopy object e.g., by displaying the path determined by pathological vessel guidance <NUM> on a display, such as display <NUM>, to the medical specialist.

In summary, vessel roadmap selection <NUM>, vessel roadmap overlay <NUM> and vessel roadmap alignment <NUM> select, overlay and align one of the vessel roadmaps <NUM> with fluoroscopy image <NUM> in order to provide a vessel roadmap during roadmap deployment phase <NUM>. By taking into account first alignment data <NUM>, second alignment data <NUM>, the first fluoroscopy information <NUM> and the second fluoroscopy information <NUM>, a vessel roadmap can be selected, overlaid and aligned with fluoroscopy image <NUM>, which corresponds to the actual position of the vessels in fluoroscopy image <NUM> without having to inject contrast medium. Thus, vessel roadmap selection <NUM>, vessel roadmap overlay <NUM> and vessel roadmap alignment <NUM> compensate any motion of the vessels, such as cardiac motion or breathing motion, in order to correctly overlay one of the vessel roadmaps <NUM> over fluoroscopy image <NUM>. Further, the duration of the fluoroscopy may be reduced based on generated vessel roadmap library <NUM>, thereby reducing radiation exposure due to the guidance provided by pathological vessel guidance <NUM>, which provides direct guidance from the position of the fluoroscopy objection to the pathological vessel. This guidance reduces the durations of medical interventions, such as PCI, since guidance based on the properly selected, overlaid and aligned roadmap may enable fast and reliable navigation with the fluoroscopy object through the vessels.

<FIG> and <FIG> illustrate a pathological vessel guidance method <NUM> implemented by the pathological vessel guidance workflow <NUM> of <FIG>. Steps shown as boxes inside boxes of other steps indicate sub-steps. Steps illustrated as dashed boxes in <FIG> and <FIG> indicate optional steps of method <NUM>. To avoid repetitions, the discussion of <FIG> and <FIG> will only outline additional details if not already included in the discussion of the workflow <NUM> and will otherwise focus on identifying which processing entities of workflow <NUM> of <FIG> performs which steps of method <NUM>.

In step <NUM>, method <NUM> generates a vessel roadmap library. Step <NUM> is performed by processing entities <NUM> to <NUM> implementing roadmap generation phase <NUM> as discussed above.

In particular, step <NUM> may include a step <NUM>, in which method <NUM> may record, for each vessel roadmap image, one or more imaging parameters <NUM>.

Step <NUM> may further include a step <NUM>, in which method <NUM> may obtain a vessel image sequence <NUM> and associated imaging physiological information <NUM>.

Step <NUM> may further include a step <NUM>, in which method <NUM> detects, within vessel image sequence <NUM>, contrasted vessel images. Step <NUM> may be performed by contrast detection <NUM>.

Step <NUM> may further include a step <NUM>, in which method <NUM> may perform vessel segmentation on the contrasted vessel images to generate vessel segmentation data. Step <NUM> may be performed by vessel segmentation <NUM>.

Step <NUM> may further include a step <NUM>, in which method <NUM> may perform contrast application object segmentation on the contrasted vessel images to generate contrast application object segmentation data. Step <NUM> may be performed by contrast application object segmentation <NUM>.

Step <NUM> may include a step <NUM>, in which method <NUM> may generate a vessel roadmap <NUM> for each contrasted vessel image. More precisely, method <NUM> may in step <NUM> generate a vessel roadmap <NUM> by including the contrasted vessel image and the vessel segmentation data in vessel roadmap image <NUM>, the imaging physiological information in first alignment data <NUM> and the contrast application object segmentation data in second alignment data <NUM>.

Step <NUM> may finally include a step <NUM>, in which method <NUM> may identify one or more cardiac cycles within the vessel image sequence based on an ECG. Step <NUM> may be implemented by cardiac cycle detection <NUM>.

In step <NUM>, method <NUM> detects, within the vessel roadmap images of the vessel roadmap library, a pathological vessel. Step <NUM> is performed by pathological vessel detection <NUM>. Step <NUM> may include sub-steps <NUM>, <NUM> and <NUM>. In step <NUM>, method <NUM> may determine centerlines of the vessels included in each vessel roadmap image. In step <NUM>, method <NUM> may determine, based on the centerlines, lumina of the vessels included in each vessel roadmap image. In step <NUM> method <NUM> may detect the pathological vessel based on the lumina of the vessels included in each vessel roadmap image. Sub-steps <NUM> to <NUM> may likewise be performed by pathological vessel detection <NUM>.

In step <NUM>, method <NUM> obtains real-time fluoroscopy image <NUM> and corresponding real-time first fluoroscopy information <NUM> and real-time second fluoroscopy information <NUM>. Step <NUM> may be performed by roadmap deployment phase <NUM> using medical imaging device <NUM> of <FIG>. Step <NUM> may include sub-steps <NUM> to <NUM>. In step <NUM>, method <NUM> may obtain fluoroscopy physiological information associated with real-time fluoroscopy image <NUM>. The fluoroscopy physiological information may be included in first real-time fluoroscopy information <NUM>. Step <NUM> may, like step <NUM>, be performed by roadmap deployment phase <NUM> using medical imaging device <NUM> of <FIG>. In step <NUM>, method <NUM> may perform fluoroscopy object segmentation on the fluoroscopy image to generate fluoroscopy object segmentation data. Step <NUM> may be performed by fluoroscopy object segmentation <NUM>. In step <NUM>, method <NUM> may identify one or more cardiac cycles based on the ECG in embodiments, in which first fluoroscopy information <NUM> includes an ECG. Step <NUM> may be performed by cardiac cycle detection <NUM>.

In step <NUM>, method <NUM> selects a vessel roadmap from the roadmap library based on comparing first real-time fluoroscopy information <NUM> with first alignment data <NUM> of each vessel roadmap <NUM>. Step <NUM> is performed by vessel roadmap selection <NUM>.

In step <NUM>, method <NUM> may compare the angiography angle with the fluoroscopy angle and may refrain from overlaying real-time fluoroscopy image <NUM> with selected vessel roadmap image <NUM> if the angiography angle and the fluoroscopy angle differ by more than an angle difference threshold. Step <NUM> may be performed by angle comparison <NUM>.

In step <NUM>, method <NUM> may overlay real-time fluoroscopy image <NUM> with the selected vessel roadmap image <NUM>. Step <NUM> is performed by vessel roadmap overlay <NUM>.

In step <NUM>, method <NUM> aligns the selected vessel roadmap image <NUM> and real-time fluoroscopy image <NUM> based on second alignment data <NUM> and real time second fluoroscopy information <NUM>. Step <NUM> may include a sub-step <NUM>, in which method <NUM> may align the position of the contrast application object with the position of the fluoroscopy object. Both step <NUM> and <NUM> are performed by vessel roadmap alignment <NUM>.

In step <NUM>, method <NUM> provides guidance for the fluoroscopy object to the pathological vessel based on the selected vessel roadmap <NUM> and the second fluoroscopy information <NUM>. Step <NUM> may include a sub-step <NUM>, in which method <NUM> may determine a path from the fluoroscopy object to the pathological vessel based on the second real-time fluoroscopy information <NUM> and the vessel segmentation data. Steps <NUM> and <NUM> are performed by pathological vessel guidance <NUM>.

In step <NUM>, method <NUM> may display the path to the pathological vessel on display <NUM> of medical imaging system <NUM> to guide an operator or medical specialist operating the fluoroscopy object to the pathological vessel. Step <NUM> may be performed by pathological vessel guidance <NUM>.

As briefly discussed above, <FIG> show exemplary medical imaging system <NUM>. Medical imaging system <NUM> may be used for both angiography and fluoroscopy. However, angiography and fluoroscopy may also be performed on separate systems with largely identical elements. In <FIG>, medical imaging system <NUM> is in neutral position P<NUM>. In <FIG>, medical imaging system <NUM> is in a rotated position P<NUM>. As discussed above, the angle between the two positions is referred to as the angiography angle or the fluoroscopy angle, depending on the imaging process currently performed by medical imaging system <NUM>. Medical imaging system <NUM> includes C arm <NUM>, on which X-ray emission means <NUM> and X-ray detection means <NUM> may be mounted. C arm <NUM> and thereby X-ray emission means <NUM> and X-ray detection means <NUM> are positioned to center around patient surface <NUM>. X-ray emission means <NUM> may emit X-rays which may penetrate through a patient positioned on patient surface <NUM>. X-ray detection means <NUM> detects the X -rays emitted from X-ray emission means <NUM>. When a patient on patient surface <NUM> is injected with a radio-opaque contrast agent into the patient's vessels, some of the X-rays emitted by X-ray emission means <NUM> are absorbed by the radio-opaque contrast agent, leading X-ray detection means <NUM> to detect an image of the vessels filled with the radio-opaque contrast agent, i.e. an angiogram. X-ray emission means <NUM> and X-ray detection means <NUM> may also collectively be referred to as x-ray imaging means.

C arm <NUM> may be coupled to C arm rotation unit <NUM>. C arm rotation unit <NUM> may be any motorized means configured to rotate C arm <NUM> according to an angiography angel or a fluoroscopy angle as either instructed by the medical specialist or angle comparison <NUM>. C arm rotation unit <NUM> may be attached to and controlled by C arm control until <NUM>. C arm control unit <NUM> may be any kind of circuitry capable of controlling C arm <NUM>. For example, C arm control unit <NUM> may include computing device <NUM> of <FIG> or may be configured to interface with computing device <NUM>.

Medical imaging system <NUM> may further include a control panel <NUM> mounted onto a side surface of patient surface support <NUM>. Control panel <NUM> may be used to control C arm <NUM> in embodiments of the present invention in which method <NUM> displays real-time fluoroscopy image <NUM> with an overlaid vessel roadmap image <NUM> including the path to the one or more pathological vessels to the medical specialist in order to guide the medical specialist to the one or more pathological vessels. <FIG> does not show any connections between control panel <NUM> and C arm <NUM> to simplify the depiction of exemplary medical imaging system <NUM>. In some embodiments, the connection may be wireless. In some embodiments, the connection may be wired and may e.g., be integrated into the ceiling of the room where medical imaging system <NUM> is located.

Medical imaging system <NUM> may finally also include a display <NUM>. Display <NUM> may be used to display information to the medical specialist, such as real-time fluoroscopy image <NUM> with an overlaid vessel roadmap image <NUM> including the path to the one or more pathological vessels. Further, display <NUM> may be used to display the vessel segmentation data included in overlaid vessel roadmap image <NUM>, including labels for the various vessel segments visible on display <NUM>. In some embodiments, display <NUM> may be a touch screen, which may be used to toggle the display of the vessel segmentation data on and off. In some embodiments, display <NUM> may further display a confidence level indicating the confidence of roadmap selection <NUM>, vessel comparison <NUM> and vessel roadmap alignment <NUM> in the accuracy of the overly and the alignment. In some embodiments, display <NUM> may also display, to the medical specialist, the appropriate angular position of C arm <NUM> during fluoroscopy to enable proper overlay and alignment of vessel roadmap image <NUM> as determined by angle comparison <NUM>.

<FIG> shows a computing device <NUM> configured to perform method <NUM>. Computing device <NUM> may include a processor <NUM>, a graphics processing unit (GPU) <NUM>, a memory <NUM>, a bus <NUM>, a storage <NUM>, a removable storage <NUM>, an medical imaging system control interface <NUM> and a communications interface <NUM>.

Processor <NUM> may be any kind of single-core or multi-core processing unit employing a reduced instruction set (RISC) or a complex instruction set (CISC). Exemplary RISC processing units include ARM based cores or RISC V based cores. Exemplary CISC processing units include x86 based cores or x86-<NUM> based cores. Processor <NUM> may further be an application specific integrated circuit (ASIC) or a field-programmable gate-array specially tailored to or programmed, respectively, to perform workflow <NUM> and method <NUM>. Processor <NUM> may perform instructions causing computing device <NUM> to perform workflow <NUM> and method <NUM>. Processor <NUM> may be directly coupled to any of the components of computing device <NUM> or may be directly coupled to memory <NUM>, GPU <NUM> and bus <NUM>.

GPU <NUM> may be any kind of processing unit optimized for processing graphics related instructions or more generally for parallel processing of instructions. As such, GPU <NUM> may perform part or all of method <NUM> to enable fast parallel processing of instructions relating to method <NUM>. It should be noted that in some embodiments, processor <NUM> may determine that GPU <NUM> need not perform instructions relating to method <NUM>. GPU <NUM> may be directly coupled to any of the components of computing device <NUM> or may be directly coupled to processor <NUM> and memory <NUM>. GPU <NUM> may also be coupled to a display, such as display <NUM> of medical imaging system <NUM>, via connection 920C. In some embodiments, GPU <NUM> may also be coupled to bus <NUM>.

Memory <NUM> may be any kind of fast storage enabling processor <NUM> and GPU <NUM> to store instructions for fast retrieval during processing of the instructions well as to cache and buffer data. Memory <NUM> may be a unified memory coupled to both processor <NUM> and GPU <NUM> enabling allocation of memory <NUM> to processor <NUM> and GPU <NUM> as needed. Alternatively, processor <NUM> and GPU <NUM> may be coupled to separate processor memory 930a and GPU memory 930b.

Storage <NUM> may be a storage device enabling storage of program instructions and other data. For example, storage <NUM> may be a hard disk drive (HDD), a solid state disk (SSD) or some other type of non-volatile memory. Storage <NUM> may for example store the instructions of method <NUM> as well as the e.g. vessel image sequence <NUM> and vessel roadmap library <NUM>.

Removable storage <NUM> may be a storage device which can be removably coupled with computing device <NUM>. Examples include a digital versatile disc (DVD), a compact disc (CD), a Universal Serial Bus (USB) storage device, such as an external SSD, or a magnetic tape. Removable storage <NUM> may for example be used to provide the vessel image sequence <NUM> to computing device <NUM> and thereby to method <NUM> or to store the generated vessel roadmap library <NUM>. It should be noted that removable storage <NUM> may also store other data, such as instructions of method <NUM>, or may be omitted.

Storage <NUM> and removable storage <NUM> may be coupled to processor <NUM> via bus <NUM>. Bus <NUM> may be any kind of bus system enabling processor <NUM> and optionally GPU <NUM> to communicate with storage device <NUM> and removable storage <NUM>. Bus <NUM> may for example be a Peripheral Component Interconnect express (PCIe) bus or a Serial AT Attachment (SATA) bus.

Medical imaging system control interface <NUM> may enable computing device <NUM> to interface with medical imaging system <NUM> via connection 870C to control C arm <NUM> in accordance with method <NUM>. For example, medical imaging system control interface <NUM> may be dedicated logic circuitry configured to control rotation of C arm <NUM>. In some embodiments, medical imaging system control interface <NUM> may be C arm control unit <NUM>. In some embodiments, medical imaging system control interface <NUM> may also be omitted and computing device <NUM> interfaces with medical imaging system <NUM> solely via communications interface <NUM>. In such embodiments, processor <NUM> may control C arm directly via communications interface <NUM>.

Communications interface <NUM> may enable computing device <NUM> to interface with external devices, either directly or via network, via connection 980C. Communications interface <NUM> may for example enable computing device <NUM> to couple to a wired or wireless network, such as Ethernet, Wifi, a Controller Area Network (CAN) bus or any bus system appropriate in medical systems. For example, computing device <NUM> may be coupled with medical imaging system <NUM> via connection 980C in order to receive vessel image sequence <NUM> or real-time fluoroscopy image <NUM> or to provide, overlay and align a selected vessel roadmap image <NUM>. Communications interface may also be a USB port or a serial port to enable direct communication with an external device.

As stated above, computing device <NUM> may be integrated with medical imaging system <NUM>. For example, computing device <NUM> may be integrated with C arm control unit <NUM> or may be placed inside patient surface support <NUM>.

The disclosure may further be illustrated by the following examples.

In an example, a computer-implemented pathological vessel guidance method, comprises the steps of generating a vessel roadmap library, the vessel roadmap library including a plurality of vessel roadmaps, wherein each vessel roadmap comprises a vessel roadmap image and first and second alignment data, detecting, within the vessel roadmap images of the vessel roadmap library, a pathological vessel, obtaining a real-time fluoroscopy image and corresponding real-time first and second fluoroscopy information, selecting a vessel roadmap from the roadmap library based on comparing the first real-time fluoroscopy information with the first alignment data of each vessel roadmap of the roadmap library, overlaying the real-time fluoroscopy image with the selected vessel roadmap image of the vessel roadmap library, aligning the vessel roadmap image of the selected vessel roadmap and the real-time fluoroscopy image based on the second alignment data and the real time second fluoroscopy information; and providing guidance for a fluoroscopy object to the pathological vessel based on the selected vessel roadmap and the second fluoroscopy information.

In an example, detecting the pathological vessel may include determining centerlines of the vessels included in each vessel roadmap image of the vessel roadmap library, determining, based on the centerlines, lumina of the vessels included in each vessel roadmap image of the vessel roadmap library and detecting the pathological vessel based on the lumina of the vessels included in each vessel roadmap image of the vessel roadmap library.

In an example, generating a vessel roadmap library may include recording, for each vessel roadmap image of the vessel roadmap library, one or more imaging parameters, the imaging parameters indicating one or more parameters associated with an imaging method used to obtain the vessel roadmap image.

In an example, the one or more imaging parameters may include at least one of an angiography angle used to obtain each vessel roadmap image and a contrast medium dosage used to obtain each vessel roadmap image.

In an example, the one or more imaging parameters may include at least an angiography angle used to obtain each vessel roadmap image and the method may include comparing the angiography angle with a fluoroscopy angle, the fluoroscopy angle being used to obtain the real-time fluoroscopy image, and, if the angiography angle and the fluoroscopy angle differ by more than an angle difference threshold, the method refrains from overlaying the real-time fluoroscopy image with the selected vessel roadmap image.

In an example, providing guidance for the fluoroscopy object to the pathological vessel may include determining a path from the fluoroscopy object to the pathological vessel based on the second real-time fluoroscopy information and the vessel segmentation data.

In an example, the method may further comprise the step of displaying the path on a display of a medical imaging system to guide an operator operating the fluoroscopy object to the pathological vessel.

In an example, the second alignment data may be derived from the corresponding vessel roadmap image.

In an example, generating the vessel roadmap may further include obtaining a vessel image sequence using an imaging method based on an in-flow of a contrast medium into a vessel tree via a contrast application object and im-aging physiological information associated with the vessel image sequence, detecting, within the vessel image sequence, contrasted vessel images, performing vessel segmentation on the contrasted vessel images to generate vessel segmentation data, performing contrast application object segmentation on the contrasted vessel images to generate contrast application object segmentation data identifying a posi-tion of the contrast application object in the contrasted vessel images; and for each contrasted vessel image, generating a vessel roadmap, the generated vessel roadmap comprising the contrasted vessel image and the vessel segmentation data as the vessel roadmap image, the imaging physiological information in the first alignment data and the contrast application object segmentation data in the second alignment data.

In an example, the imaging physiological information may include an electrocardiogram (ECG) and generating the vessel roadmap may further include identifying one or more cardiac cycles within the vessel image se-quence based on the ECG, wherein the generated vessel roadmap further comprises the identified one or more cardiac cycles in the first alignment data.

In an example, obtaining a real-time fluoroscopy image and corresponding real-time first and second fluoroscopy information may include obtaining fluoroscopy physiological information associated with the fluoroscopy image and performing fluoroscopy object segmentation on the fluoroscopy image to generate fluoroscopy object segmentation data identifying a position of the fluoroscopy object in the fluoroscopy image, wherein the fluoroscopy physiological information may be included in the first real-time fluoroscopy information, and wherein the generated fluoroscopy object segmentation data may be included in the second real-time fluoroscopy information.

In an example, the fluoroscopy physiological information may include an electrocardiogram (ECG) and wherein obtaining the real-time fluoroscopy image and corresponding real-time first and second fluoroscopy information may further comprise identifying one or more cardiac cycles based on the ECG, wherein the identified one or more cardiac cycles are included in the first real-time fluoroscopy information.

In an example, aligning the vessel roadmap image and the real-time fluoroscopy image based on the second alignment data and the real time second fluoroscopy information may include aligning the position of the contrast application object with the position of the fluoroscopy object.

In an example, a computer-readable medium comprising instructions configured to be executed by a computer including at least one processor, the instructions causing the processor to perform the method according to any one of the preceding examples.

Claim 1:
Computer-implemented method for providing an operator with guidance to a pathological vessel (<NUM>) comprising the steps of:
generating (<NUM>) a vessel roadmap library (<NUM>), the vessel roadmap library (<NUM>) including a plurality of vessel roadmaps (<NUM>), wherein each vessel roadmap (<NUM>) comprises a vessel roadmap image (<NUM>) and first and second alignment data (<NUM>, <NUM>);
detecting (<NUM>), within the vessel roadmap images (<NUM>) of the vessel roadmap library (<NUM>), a pathological vessel; obtaining (<NUM>) a real-time fluoroscopy image (<NUM>) and corresponding real-time first fluoroscopy information (<NUM>) and real-time second fluoroscopy information (<NUM>);
selecting (<NUM>) a vessel roadmap (<NUM>) from the roadmap library (<NUM>) based on comparing the real-time first fluoroscopy information (<NUM>) with the first alignment data (<NUM>) of each vessel roadmap (<NUM>) of the roadmap library (<NUM>);
overlaying (<NUM>) the real-time fluoroscopy image (<NUM>) with the selected vessel roadmap image (<NUM>) of the vessel roadmap library (<NUM>);
aligning (<NUM>) the vessel roadmap image of the selected vessel roadmap and the real-time fluoroscopy image based on the second alignment data and the real-time second fluoroscopy information; and
providing an operator with guidance for a fluoroscopy object to the pathological vessel based on the selected vessel roadmap and the real-time second fluoroscopy information.