Patent ID: 12211208

DETAILED DESCRIPTION

Embodiments of the invention provide methods and systems for automated analysis of vessels from images of the vessels, or portions of the vessels, and display of the analysis results.

Analysis, according to embodiments of the invention, may include information about a property of the vessel, such as geometry related information of a vessel. The analysis may further include functional measurements which may possibly be calculated from one or more property of the vessel. Analysis may also include diagnostic information, such as presence of a pathology, identification of the pathology, location of the pathology, etc. The analysis results, which may include a functional measurement, vessel properties and/or a calculation, diagnosis or other information based on images of the vessel, may be displayed to a user.

A “vessel” may include a tube or canal in which body fluid is contained and conveyed or circulated. Thus, the term vessel may include blood veins or arteries, coronary blood vessels, lymphatics, portions of the gastrointestinal tract, etc.

An image of a vessel may be obtained using suitable imaging techniques, for example, X-ray imaging, ultrasound imaging, Magnetic Resonance imaging (MRI) and others suitable imaging techniques.

“Vessel properties” may include, for example, anatomical characteristics (e.g., shape and/or size of parts of the anatomy) of a vessel and/or of a pathology in the vessel. For example, pathologies may include a narrowing of the vessel (e.g., stenosis or stricture), lesions within the vessel, etc. Thus, vessel properties may include, for example, shape and/or size of vessels and/or parts of vessels, angles of bends in vessels, diameters of vessels (e.g., proximal and distal to a stenosis), minimal lumen diameter (e.g., at the location of a stenosis), lesion length, entrance angle of the stenosis, entrance length, exit angle of the stenosis, exit length, percentage of the diameter blocked by the stenosis, percentage of the area blocked by the stenosis, etc. A pathology or indication of a pathology and/or other diagnosis may be calculated based on these properties.

A “functional measurement” is a measurement of the effect of a pathology on flow through the vessel. Functional measurements may include measurements such as FFR, instant flow reserve (iFR), coronary flow reserve (CFR), quantitative flow ratio (QFR), resting full-cycle ratio (RFR), quantitative coronary analysis (QCA), and more.

In the following description, various aspects of the present invention will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well known features may be omitted or simplified in order not to obscure the present invention.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “using”, “analyzing”, “processing,” “computing,” “calculating,” “determining,” “detecting”, “identifying” or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulates and/or transforms data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices. Unless otherwise stated, these terms refer to automatic action of a processor, independent of and without any actions of a human operator.

In one embodiment, which is schematically illustrated inFIG.1, a system for analysis of a vessel includes a processor102in communication with a user interface device106. Processor102receives one or more images103of a vessel113, each of which may be capturing the vessel113from a different angle. Processor102then performs analysis on the received image(s) and communicates analysis results and/or instructions or other communications, based on the analysis results, to a user, via the user interface device106. In some embodiments, user input can be received at processor102, via user interface device106.

Vessels113may include one or more vessel or portion of a vessel, such as a vein or artery, a branching system of arteries (arterial trees) or other portions and configurations of vessels.

Processor102may include, for example, one or more processors and may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a microprocessor, a controller, a chip, a microchip, an integrated circuit (IC), or any other suitable multi-purpose or specific processor or controller. Processor102may be locally embedded or remote, e.g., on the cloud.

Processor102is typically in communication with a memory unit112. In one embodiment the memory unit112stores executable instructions that, when executed by the processor102, facilitate performance of operations of the processor102, as described below. Memory unit112may also store image data (which may include data such as pixel values that represent the intensity of reflected light as well partial or full images or videos) of at least part of the images103.

Memory unit112may include, for example, a random access memory (RAM), a dynamic RANI (DRAM), a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.

The user interface device106may include a display, such as a monitor or screen, for displaying images, instructions and/or notifications to a user (e.g., via graphics, images, text or other content displayed on the monitor). User interface device106may also be designed to receive input from a user. For example, user interface device106may include or may be in communication with a mechanism for inputting data, such as, a keyboard and/or mouse and/or touch screen, to enable a user to input data.

All or some of the components of the system may be in wired or wireless communication, and may include suitable ports such as USB connectors and/or network hubs.

In one embodiment, processor102can determine a property of a vessel from an image of the vessel, typically by applying computer vision techniques such as by applying shape and/or color detection algorithms, object detection algorithms and/or other suitable image analysis algorithms on at least part of one or more of images103. In some embodiments machine learning models can be used to detect portions of vessels and to determine properties of vessels from images103. In some embodiments, a pathology and/or a functional measurement for the vessel (e.g., at the location of a pathology) can be determined based on the determined vessel properties.

In some embodiments, a pathology and/or a functional measurement can be determined directly from one (or more) images of the vessel. For example, a pathology and/or a functional measurement can be determined based on a single 2D image of the vessel, without having to determine a property of the vessel.

In some embodiments, properties of vessels and/or functional measurements may be determined by using a combination of structural and temporal data obtained from images of the vessels, e.g., as described below, with reference toFIG.6.

Typically, each of images103captures vessel113from a specific angle or point of view.

In one embodiment, determining properties of a vessel may include receiving 2D images of the vessel and extracting 3D related features from the images, without constructing a 3D model of the vessel, for example, without using voxels and/or point clouds or other 3D representations.

The 3D related features are image features, which may be specific structures in the image such as points, edges or objects, or any other information in the image which can be used to determine a property of the vessel from the image. In some embodiments, 3D related features are extracted from images obtained from different views. Features from these images can be used to teach a machine learning model to detect properties of vessels from 2D images. For example, features extracted from 2D images obtained from different views can be combined using a neural network, e.g., a long short term (LS™) neural network, that can compute a feature for each imaged element, integrate the features, keep a representation of the features in memory (hidden state) and update its output as more images are input. Such a neural network may be used for learning properties of vessels and can then be used to determine a property of a vessel and/or a functional measurement from 2D images without having to reconstruct a full 3D representation or use a 3D model of the vessel or use voxels and/or point clouds or other 3D representations.

Extracting 3D related features from a 2D image and determining a vessel property and/or functional measurement from the 3D related features, without constructing a 3D representation and/or without using 3D representations of the vessel, provides a quick process, with minimal cost to accuracy.

In one embodiment, an indication of vessel properties and/or functional measurements that are determined from one or more images of the vessel (e.g., based on extracted 3D related features) can be displayed via user interface device106.

Medhub's AutoCathFFR™ is, for example, a system for automated calculation of FRR of vessels from images of the vessels. Medhub's AutoCathIFR™ is, for example, a system for automated calculation of an iFR procedure from images of a vessel.

In one embodiment, a system, as described above, includes a processor, e.g., processor102, that implements a method, which is schematically illustrated inFIG.2.

A sequence of images of a vessel, such as a video movie of angiogram images, is analyzed by the system. Processor102selects, using computer vision techniques, a first image from the sequence of images (step202) and detects a pathology, such as a stenosis or lesion, in the first image (step204). Detection of the pathology and/or the location of the pathology is done by using computer vision techniques, without requiring user input regarding a location of the pathology. The processor can automatically detect the pathology in a second image of the vessel (step206), the second image being captured at an angle different than the first image, and may then cause the first and/or second images of the vessel to be displayed on a user interface device, such as user interface device106, with an indication of the pathology (step208).

An indication of a pathology displayed on a user interface device may include, for example, graphics, such as, letters, numerals, symbols, different colors and shapes, etc., that can be superimposed on an image.

Once a pathology is detected in a first image, the pathology may be tracked throughout the images of the sequence (e.g., video), such that the same pathology can be detected in each of the images, even if it's shape or other visual characteristics change in between images.

One method of tracking a pathology in between images, is schematically illustrated inFIG.3. As described above, a first image from a sequence of images is selected (step302) and a pathology is detected in the first image (step304). A virtual mark is attached to the pathology by the processor (step306).

In some embodiments the virtual mark is location based, e.g., based on location of the pathology within portions of the vessel which are automatically detected by the processor102. In some embodiments, a virtual mark includes the location of the pathology relative to a structure of the vessel. A structure of a vessel can include any visible indication of anatomy of the vessel, such as junctions of vessels and/or specific vessels typically present in patients. Processor102may detect the vessel structure in the image by using computer vision techniques, and may then index a detected pathology based on its location relative to the detected vessel structures.

For example, a segmenting algorithm can be used to determine which pixels in the image are part of the pathology and the location of the pathology relative to structures of the vessel can be recorded, e.g., in a lookup table or other type of virtual index. For example, in a first image a stenosis is detected at a specific location (e.g., in the distal left anterior descending artery (LAD)). A stenosis located at the same specific location (distal LAD) in a second image, is determined to be the same stenosis that was detected in the first image. If, for example, more than one stenosis is detected within the distal LAD, each of the stenoses are marked with their relative location to additional structures of the vessel, such as, in relation to a junction of vessels, enabling to distinguish between the stenoses in a second image.

Thus, the processor102creates a virtual mark which is specific per pathology, and in a case of multiple pathologies in a single image, distinguishes the multiple pathologies from one another.

The pathology can then be detected in a second image of the vessel (step308), based on the virtual mark. The processor102may then cause display of the indication of the pathology (e.g., as described above) based on the virtual mark. In some embodiments the processor may assign a name to a pathology based on the location of the pathology within the vessel and the indication of pathology can include the name assigned to the pathology, as further demonstrated below.

In some cases, a vessel or group of vessels may include more than one stenosis or other pathology, making detecting a same pathology in different images, more difficult. In some embodiments, a processor detects a plurality of pathologies in the first image and creates a distinct virtual mark for each of the plurality of pathologies. The processor may then cause display of the indication of each of the pathologies based on the virtual mark. In some embodiments the indications are displayed together on a single display.

Thus, a processor according to embodiments of the invention, may determine a functional measurement (e.g., FFR value) of the pathology based on first and second images, e.g., based on the pathology detected in the first and second images, and may display an indication of the functional measurement, e.g., on a user interface device

In some embodiments, the processor can determine a level of accuracy of the functional measurement and can calculate a third image required to improve the level of accuracy. The processor can then cause an indication of the third image to be displayed on a user interface, to advise a user which image to add in order to improve accuracy of results.

The first, second and third images are typically each captured at different angles and the indication displayed on the user interface device includes the angle of the third image.

In one embodiment, an optimal frame chosen from a sequence of images of the vessel is used as the first image discussed above.

In an example, which is schematically illustrated inFIG.4, a video of angiogram images is received (step402) and an optimal image is detected (step404) from the video. A pathology is detected in the optimal image (step406). The pathology can then be tracked in the images of the sequence and can thus be detected in another frame (step408) enabling to display an indication of the pathology in all images (step410).

An optimal image is typically an image showing the most detail. In the case of angiogram images, which include contrast agent injected to a patient to make vessels (e.g., blood vessels) visible on an X-ray image, an optimal image may be an image of a blood vessel showing a large/maximum amount of contrast agent. Thus, an optimal image can be detected by applying image analysis algorithms on the images of the sequence.

In one embodiment an image captured at a time corresponding with maximum heart relaxation is an image showing a maximum amount of contrast agent. Thus, an optimal image may be detected based on capture time of the images compared with, for example, measurements of electrical activity of the heartbeat (e.g., ECG printout) of the patient.

In one embodiment the processor can calculate a value of a functional measurement, such as an FFR value, for each pathology and may cause the value(s) to be displayed.

In some embodiments, processor102calculates a level of accuracy of the functional measurement value (e.g., FFR value), based on an angle of capture of the first image, and can cause an indication of the level of accuracy to be displayed on the user interface device106.

In one embodiment, which is schematically illustrated inFIG.5, processor102receives an image (e.g., image103) of a vessel (step502) and provides an analysis (e.g., determines a property of the vessel and/or a functional measurement) from the image (step504). For example, processor102may apply computer vision algorithms (e.g., as described above) on the received image(s)103to determine one or more properties such as, shape and/or size of vessels and/or parts of vessels, angles of bends in vessels, diameters of vessels, minimal lumen diameter, lesion length, entrance angle of the stenosis, entrance length, exit angle of a stenosis, exit length, percentage of the diameter blocked by a stenosis, percentage of the area blocked by a stenosis, etc. Processor102may then determine a functional measurement based on the property of the vessel. In other embodiments, processor102determines a functional measurement directly from image103, e.g., by employing a machine learning model to run a regression algorithm to predict a value of a functional measurement (e.g., FFR) from an image of the vessel.

In some embodiments, processor102calculates a level of accuracy (also referred to in terms of “margin of error”) of the analysis, based on the image(s)103(step506) and may cause an indication of the level of accuracy to be displayed on the user interface device106(step508).

Calculating a level of accuracy can be done, for example, by obtaining a functional measurement for a vessel by using known methods (e.g., physical measurements) and comparing the obtained functional measurement to a functional measurement obtained from an image of the vessel according to embodiments of the invention. A deviation from the measurement obtained by known methods can be used to determine the level of accuracy of the determination based on embodiments of the invention. This can be done for images obtained at all possible angles, thereby creating a map or regression analysis connecting different angle images and/or combination of images to different accuracy levels. This analysis may be performed by carrying out empirical experiments or by using, for example, a predictive model to create a mapping function from an angle of an image to a level of accuracy.

Thus, a processor according to embodiments of the invention may receive an image of the vessel, the image capturing the vessel at an angle, and may calculate a level of accuracy of an analysis (e.g., determination of a property of the vessel and/or of a functional measurement), based on the angle.

Since, according to embodiments of the invention, any image, obtained at any possible angle, can be mapped to a level of accuracy, depending on the required level of accuracy, functional measurements and other analyses, can be obtained based on a single 2D image.

Because processor102can detect a specific pathology in different images of a vessel (e.g., images captured from different angles) and can determine a level of accuracy for each pathology based on the different images, processor102can calculate which (if any) additional images (captured at which angles) are necessary to adjust, e.g., improve, the accuracy of the analysis.

In one embodiment, the indication of the level of accuracy displayed on the user interface device in step508, includes instructions or notice for the user (e.g., health professional) regarding how many additional images to add, typically specifying the angle of each additional image, in order to improve the accuracy of the analysis results and lower the margin of error.

In one embodiment processor102can provide indication of a single angle of image, which when added to the images already supplied by the user, can provide the most improved accuracy level.

In one embodiment, which is schematically illustrated inFIG.6, a sequence of images, e.g., video603of angiogram images, is analyzed, e.g., for determining properties of an imaged vessel and/or for calculating a functional measurement for the vessel. A processor obtains structural data604of the vessel from at least one image from video603. The processor also obtains temporal data605of the vessel from images of video603. The structural data604and the temporal data605are combined and the combined information is analyzed e.g., by encoder610, to obtain relevant features from which to determine a property of a vessel and/or from which to calculate a functional measurement for the vessel.

In one embodiment, a processor determines a pathology from an image of a vessel and may cause an indication of the pathology to be displayed on the user interface device606.

In some embodiments, a functional measurement for the vessel can be calculated based on the property of the vessel or based on the relevant features obtained by encoder610. Indication of the functional measurement can then be displayed on a user interface device606.

The relevant features calculated by encoder610may also be used to determine a property of a vessel, such as a shape or size of a part of a vessel.

In all cases, an indication of the level of accuracy of the displayed analysis (pathology, functional measurement, properties of vessels, etc.) can be calculated and displayed on user interface device606.

As described above, angiogram images include contrast agent injected to a patient to make vessels (e.g., blood vessels) visible on an X-ray image. Thus, in one embodiment, an image chosen from the angiogram video from which to obtain structural data604, may be an optimal image, e.g., an image of a blood vessel showing a large/maximum amount of contrast agent.

Temporal data605may be obtained from a flow map estimating flow velocity of blood (visible as contrast agent) at points within the vessel. Computing the flow and producing a flow map may be done by applying on the video603motion detection algorithms and/or using a neural network trained to estimate motion and output an optical flow map.

Structural data604may be obtained by using computer vision techniques, such as by applying a segmenting algorithm on at least one image from the video (e.g., an image showing a maximum amount of contrast agent), to detect in the image a vessel and/or a pathology and/or geometry related or other information.

In some embodiments, portions of a vessel may be detected and a location of a pathology in the vessel can be determined based on the relevant features calculated by encoder610. The location of the pathology and/or other indications may then be displayed on user interface device606.

In one embodiment, an example of which is schematically illustrated inFIG.7, a processor, such as processor102, receives an image of a vessel (step702) and determines a pathology (e.g., lesion or stenosis) from the image of the vessel (step704). For example, the pathology may be determined from a property of a vessel and/or from relevant features extracted from images of the vessel, e.g., as described above. The processor may then calculate a level of significance of the pathology (step706) and may cause an indication of the pathology and/or a functional measurement of the pathology to be displayed based on the level of significance. For example, the level of significance of a pathology may be determined based on parameters of the pathology, such as, size and/or shape of the pathology and/or percentage of the diameter blocked by the pathology, percentage of the area blocked by the pathology, etc.

In one embodiment, if the level of significance is above a threshold, e.g., a predetermined threshold (step707) the pathology and/or functional measurement related to the pathology will be displayed to a user (708). However, if the level of significance is below the predetermined threshold (step707) then the pathology and/or functional measurement may not be displayed to the user (step710). In some embodiments, the pathologies can be rated based on their significance and can be displayed to the user together with their rate, e.g. each pathology can be displayed in a table listing its significance, as described below.

In another embodiment, a level of significance may be calculated by comparing a number of pathologies to each other and/or to a predetermined standard.

Thus, a system for analysis of a vessel includes a processor in communication with a user interface device. The processor determines a pathology of the vessel from an image of the vessel, and calculates a significance level of the pathology and controls a device based on the calculated significance level. For example, the processor can control a user interface device to control its display based on the calculated significance level.

In some embodiments a processor, such as processor102, can classify a pathology based on one or both of location of the pathology within the vessel and based on a functional measurement, e.g., FFR value. The processor may accept a user request for pathologies based on location within the vessel and/or based on FFR value and may display the pathology according to the classification.

Examples of user interfaces according to embodiments of the invention are schematically illustrated inFIGS.8A and8B.

In one embodiment, which is schematically illustrated inFIG.8A, an image capturing a coronary blood vessel e.g., LAD803at a specific angle, is displayed on monitor816.

In one embodiment, a functional measurement value, FFR801, is displayed on monitor816of a user interface device, together with an indication804of the number of images used to calculate the functional measurement FFR801. In one embodiment, a single image may be used from which a functional measurement is obtained. In some embodiments, even if several images were used to calculate a functional measurement, only one image of LAD803, which includes the best or most visible features, is displayed on monitor816.

An indication of one or more pathology, e.g., stenosis807, can be displayed as graphics superimposed on the displayed image. In some embodiments, the displayed image is a representation of the vessel. For example, the displayed representation may include a combination of images of the vessel, such as a combined image of several images (typically obtained at different angles) or an average image of a few images (obtained at different angles). The angle(s)805at which the displayed image(s) were obtained can be indicated on display816.

In some embodiments, as illustrated inFIG.8B, the image of the vessel which is displayed on monitor816is a graphical illustration813of the vessel, rather than an actual image.

Graphics, which may include, for example, letters, numerals, symbols, different colors and shapes, etc., can be displayed superimposed on the representation of the vessel. For example, references811can be made to different parts of a LAD and to one or more pathologies. The references811can be used to assist a user in locating pathologies, as shown in table812. For example, the first row of table812relates to vessel number7in the medial LAD, both of which are shown as references811on the graphical illustration813of the vessel.

The reference811can be assigned by a processor to different vessel portions which were detected by the processor based on computer vision techniques.

In some embodiments, monitor816includes a button808to enable a user to at least partially hide graphics superimposed on the representation (e.g., image or graphical illustration) of the vessel, so that the user can view the vessel unobstructed by the different graphics. For example, activating button808may cause all or specified graphics to fade or become transparent.

The level of accuracy, margin of error802, of the value of FFR801is also displayed on monitor816. As described above, the margin of error802may be known for each image obtained at a known angle. The accuracy level may similarly be known or calculated for combinations of images obtained at different angles. Thus, adding images obtained at different angles may change the level of accuracy of a currently displayed functional measurement. For example, a user (e.g., a health professional) may add images obtained at angles different than angles805, in order to change the margin of error802. In some embodiment, monitor816includes a window815for displaying to the user indication of the angles of additional images that should be input to the system in order to improve the level of accuracy or minimize the margin of error of the analysis results provided by the system.

In one embodiment, processor102can classify a pathology based on one or both of location of the pathology within the vessel and its FFR value. In one embodiment the processor may accept a user request for display of pathologies based on location within the vessel and/or based on FFR value.

Because embodiments of the invention enable automatic detection of pathologies in images of vessels and enable marking a location of a pathology in relation to an anatomical structure, pathologies may be identified and marked retroactively, even in images of vessels captured prior to identification of the pathology. Thus, a processor, according to embodiments of the invention, may detect in an image captured prior to detection of the pathology in a first image, the same pathology as in the first image, based on a virtual mark attached to the pathology in the first image. This enables a user to work off-line as well as at the capture time of the images. Working off-line may include retroactively marking pathologies in images as well as classifying images based on a desired parameter and displaying results based on the classification. Additionally, working off-line may include gathering analytics, as described below. For example, a user may request to see all stenoses detected in the medial LAD. The processor may then control a user interface device to display the stenoses according to the requested classification, e.g., to display only images or representations of medial LAD vessels and their respective information (e.g., first row in table812).

In another example, a user may request to see stenoses having an FFR value above a threshold, in which case processor102may cause only stenoses having relevant FFR values to be displayed or indicated on a user interface device.

Embodiments of the invention may be used with images obtained by any suitable imaging methods, e.g., images obtained using quantitative angiography methods, such as, quantitative superficial femoral angiography, ultrasound methods, such as intravascular ultrasound (IVUS), tomography, such as, optical coherence tomography (OCT), and more.

Embodiments of the invention provide systems and methods for obtaining functional measurements such as FFR, the accuracy of which can be improved in real-time, and can be tailored to specific user requirements.

In some embodiments, medical data (such as life expectancy and life longevity) and/or other data (such as age, gender, medical history, etc.) can be input to a system and can be used, together with images of vessels and pathologies in vessels, to create big data. For example, embodiments of the invention enable providing a user with analytics involving functional measurements, such as FFR, gathered from many subjects, e.g., angiograms of all patients examined at a specific facility or network of facilities. Such analytics may include, for example, FFR per gender and per age and FFR per anatomical area and/or per artery. A user interface device according to embodiments of the invention, may provide buttons for user requests for such analytics and/or windows displaying numerical and/or graphical representations of such analytics.

Big data may be used, for example, to predict the risk level of pathologies and possibly best treatment practices per pathology, in the long term. Medhub's AutoCathPred™ is, for example, a system for providing predictions of risky pathologies and best treatment for pathologies, based on big data analysis.