Patent ID: 12251261

DETAILED DESCRIPTION

The invention relates to a method1000of assisting in generating a sequence of ultrasound images. The invention is particularly advantageous for generating a sequence of ultrasound images of the heart. This is because the heart is in an organ subjected to a cycle.

The cardiac cycle consists of two periods: one during which the heart muscle relaxes and fills with blood, called diastole, followed by a period of vigorous contraction and pumping blood, called systole. After emptying, the heart immediately relaxes and expands to receive another inflow of blood returning from the lungs and other systems in the body, before contracting again to pump blood to the lungs and these systems. A normally functioning heart should be fully expanded before it can pump efficiently again.

In order to analyze possible conditions, it is important for the doctor to obtain a sequence of images (or video) of the entire cardiac cycle. The invention advantageously makes it possible to assist the practitioner in acquiring an image sequence.

However, the invention may also find advantages in the ultrasound acquisition of other organs or an image sequence is desirable, for example in the field of obstetric ultrasound.

The invention also relates to an associated device. An example of such device1is shown inFIG.2.

Device1may comprise a tablet, a smartphone, a computer, or any other device comprising at least one display and a processor associated with a memory. According to another aspect, the invention also relates to a system2comprising a device1according to the invention and an ultrasonic probe SECH connected to said device1.

In one embodiment, the device comprises software and hardware means for implementing the method according to the invention described below. Said device preferably comprises at least one receptor REC, a classifier, and a display.

Reception

The method comprises a step of receiving100ultrasound images11in real time. Said ultrasound images are received by a receptor REC of the device. In one embodiment, the receiver REC may comprise a buffer memory wherein images received11by the receiver REC are temporarily stored prior to transmission.

The received ultrasound images11are displayed in real time, preferably on a display AFF of the device. The received ultrasound images can therefore be transmitted to a display for display in real time.

The device includes a display AFF. The display AFF may comprise a screen such as a monitor, a touchscreen tablet or a smartphone.

The display AFF is connected to the receiver REC and/or the classifier CLASS and/or the processor. The display AFF is configured to display in real time the ultrasound images received by the receiver REC. The display AFF advantageously allows the operator to have real-time feedback on the ultrasound images that they are acquiring. The display AFF is configured to show additional indicators or data that will be described later in this description. The display AFF displays an image1comprising an ultrasound image received11by the receiver REC. Preferably, the ultrasound image displayed is the last ultrasound image received by the receiver REC. The display AFF is thus configured to display in real time the images captured by the ultrasonic probe SECH connected to the device according to the invention.

Classification

The method comprises a step of classifying200of the ultrasound images11received by the receptor REC in real time. Classification is performed by a classifier CLASS of the device.

The classification of an ultrasound image comprises associating said ultrasound image with a class14, preferably a class among a predefined class group.

In one embodiment, the classification further comprises generating300a quality indicator13according to said class associated with the ultrasound image. The quality indicator may be representative of a class or predefined set of classes.

In one embodiment, the step of classifying further comprises generating a confidence value. The confidence value may represent the confidence rate of the classification performed. The confidence value is generated by the classifier CLASS.

The classification step is preferably performed by a classifier CLASS of the device. Classifier means an algorithmic function executed by a processor or a calculator associated with a memory.

The classifier CLASS is configured to receive images received by the receiver REC.

The classifier is configured to classify in real time at least part of the images received by the receiver REC. The classifier is configured to associate a class with an ultrasound image. The classifier generates a quality indicator13according to the associated class. The quality indicator13is preferably associated with said classified ultrasound image.

Preferably, the image processing means device for processing the images received before providing them to the classifier CLASS. These processing means may comprise image filters or contrast functions.

Preferably, the classifier CLASS is configured to classify an ultrasound image of the heart among a plurality of classes comprising, on the one hand, classes each representing a particular view of the heart and on the other hand, a class representing a view of insufficient quality.

Thus, when an image is classified in a class representing a particular view of the heart, this image is considered to be of sufficient quality to advantageously allow the clinician to perform predefined measurements from these images. Indeed, cardiologists use particular well identified views for their diagnosis.

In one embodiment, each class representing a particular view of the heart represents one of:The view of a parasternal long axis sliceThe view of a parasternal short axis sliceThe view of an apical sliceThe view of a subcostal sliceThe view of a suprasternal sliceThe view of a straight paraphernal section

Each of these views is well known to cardiologists and is used to visualize different parts of the heart, calculate or measure specific data, and identify certain pathologies.

Each of these views can be characterized by the presence of one or more specific parts of the heart. For example, the view of a parasternal short axis slice is characterized by the presence of the section of the left and right ventricles in the image. From this view, the cardiologist can calculate the shortening fraction and calculate lung pressures.

In one embodiment, the class representing a view of insufficient quality represents ultrasound views of the heart that do not belong to one of the specific views of the other classes or the specific ultrasound views listed above, but whose image quality does not allow the necessary measurements related to that specific view.

In one embodiment, the classifier CLASS is implemented by means of a learning function trained from supervised and/or machine learning. The learning function preferably comprises a neural network. The learning function is preferably trained with a series of labelled ultrasound images of the heart.

In one example, the learning function has been configured from a training comprising submitting a plurality of ultrasound images each associated with a label to the classifier. In one embodiment, the classifier has been trained with ultrasound images of a specific view of the heart of sufficient quality, each image being associated with a label representing said specific view. The classifier has also been trained with ultrasound images of views of the heart other than those listed above or not having a sufficient quality to enable the cardiologist to perform measurements from those images, each of these images being associated with a quality label representing a view of insufficient quality. The labels may also include information characteristic of a shot of an image of an organ having a characteristic angle of view and/or size, an image quality, the presence of a specific portion of an image of the heart, the presence of a measurable anatomical shape contour. Preferably, the labels include the name of the particular view of the heart or a name associated with a poor quality view.

The classifier CLASS may be executed in a processor that is itself associated with a memory. The classifier CLASS may be recorded in a computer readable medium such as a memory associated with said processor.

In one embodiment, the step of classifying an ultrasound image comprises generating a match score for each class. The correspondence score may comprise a probability that the ultrasound image belongs to each class. The method then includes selecting the class with the highest score. Preferably, the confidence value is generated from said correspondence score of the class associated with said ultrasound image.

In an embodiment, the classifier CLASS classifies 100% of the images received in real time.

In an alternative embodiment, the classifier CLASS is configured to classify only one rate of received images. The classifier is then configured to classify a portion of the received ultrasound images in real time. This embodiment is particularly advantageous when the classification speed is less than the image acquisition speed (in images per unit of time).

In one embodiment, the classification of an ultrasound image comprises the classification of the last received ultrasound image. When said image has been classified, the method again takes the last received ultrasound image. As a result, some ultrasound images may not be classified. However, this execution mode advantageously makes it possible to classify ultrasound images in real time, regardless of the speed of the classifier or the frequency of reception of the ultrasound images.

Quality Indicator

The method according to the invention comprises generating and displaying in real time on the display AFF of the quality indicator13generated. Preferably, the quality indicator13displayed corresponds to the last quality indicator13generated.

In one embodiment, this quality indicator13is generated by the classifier CLASS. Such a classifier is configured to, upon receiving an ultrasound image of the heart, classify said image into one of the above-mentioned classes. The method then includes generating a quality indicator representative of the classification of that image. Said quality indicator may be associated with said classified ultrasound image.

Alternatively, the quality indicator13can be generated by a remote processor connected to the display AFF and receiving information from the classifier.

The display of the quality indicator13may comprise the display of a colored indicator the color of which depends on the class associated with the classified ultrasound image. The color of the indicator may be representative of a single class. Preferably, the color of the colored indicator may be of two distinct colors, a first color representative of a group of classes including classes representing a particular view of the heart such as those set out above, and a second color representative of a class representing a view of insufficient quality. Thus, the operator can advantageously visualize more quickly whether the ultrasound image he or she is taking is of sufficient quality or not.

In one embodiment, the classifier is also configured to, upon receiving an ultrasound image of the heart, generate a confidence value. The confidence value may be representative of a level of certainty that the image has been correctly classified. In one embodiment, the quality indicator may comprise said confidence value. Said quality indicator21is preferably displayed on the display AFF.

The quality indicator13associated with the classified image may include said confidence indicator. In another alternative embodiment, a confidence indicator representative of the confidence value is generated and displayed in real time.

In a first example illustrated inFIG.1, the displayed quality indicator13comprises a colored frame, which preferentially extends around the displayed ultrasound image. The operator can then advantageously visualize whether the image he or she is taking is the expected one without having to look away from the image. The confidence indicator can be displayed as a numerical value as illustrated inFIG.1.

In a second example illustrated inFIG.3, the quality indicator13comprises a colored indicator. The color of the colored indicator is representative of the classification of the last ultrasound image. At least one of the dimensions of said colored indicator is a function of the confidence value. In the example illustrated inFIG.3, the quality indicator13is a bar whose length varies according to the confidence value and whose color depends on the class associated with the last ultrasound image.

Such an indicator advantageously allows the operator to visualize whether the image he or she is taking is of sufficient quality and also allows him or her to visualize without looking away whether the indicator is degraded or improved as a function of the movement of the ultrasound probe SECH. The purpose of such an indicator is to allow the operator, when moving the probe slightly, to visualize whether this movement increases or decreases the confidence value, and therefore increases or decreases the likelihood of obtaining a validated sequence of images described below.

Sequence

The method comprises automatically recording400a sequence20of images in a memory MEM. This sequence20is automatically recorded when a sequence of a predefined number of received or classified images includes a rate of images associated with a same class14greater than a predefined rate. Preferably, said class is a class representing a particular view of the heart.

By “sequence of images” it is meant a series of ultrasound images which follow each other in the chronological order of their acquisition. The term “video sequence” will therefore also be used to refer to such a sequence of images.

Thus, when a sufficiently long video comprises, for example, a majority of ultrasound images classified in the same class representing a particular view of the heart, the video sequence20is automatically recorded in a memory. This automatic recording advantageously automatically generates videos of a particular view of the heart that can be analyzed by a cardiologist without validation by the operator.

The predefined number of received or classified images can be understood as a predefined minimum duration, as long as the image acquisition frequency of the ultrasound probe SECH is constant. The advantage of this threshold is to ensure that the video sequence20is long enough to be analyzed by a cardiologist. In another embodiment, the predefined number of images may be replaced by a number of received or classified images. The predefined number may be parameterized to correspond to a predefined number of cardiac cycles of the subject's heart.

The rate of images associated with a same class in said video sequence greater than a predefined rate advantageously allows automatic recording despite a negligible number of images not having sufficient quality in said sequence. Indeed, this negligible number of images in a video sequence can come from a classification error, or noise linked to a particular image. This tolerance advantageously consists of a good compromise between the ease of automatic generation of a sequence and the quality of said sequence.

In one embodiment, the predefined rate is at least greater than 50%. In another embodiment, the predefined rate can be set by the operator, for example by a user interface of the device.

In one embodiment, the automatic recording step comprises buffering the continuously received ultrasound images as well as the generated quality indexes associated with said images. In this way, when a video sequence meeting the above criteria is detected, the images belonging to this sequence can be transferred from the buffer memory to another memory, and/or grouped in a same file to generate a video sequence.

For the sake of clarity, the generation and recording of such a sequence will be referred to as the “validation” of such a sequence in the following.

Time Indicator

In one embodiment, the method comprises a step of generating and displaying in real time a time indicator12.

The displayed time indicator12is representative of the number of remaining images to be received or classified in order to reach a sequence of a predefined number of received images comprising a rate of images associated with a same class greater than a predefined rate.

This indicator advantageously allows the operator to visualize the time remaining to validate a video sequence of a particular view of the heart. Advantageously, this indicator allows the operator to visualize the remaining time during which he or she has to maintain an ultrasound image of sufficient quality, said quality being displayed to him/her by the quality indicator13and/or the confidence indicator.

The time indicator12may comprise a numerical value such as a countdown. The time indicator12may comprise a gauge that fills or empties as a function of the time remaining to validate a video sequence.

In one embodiment, the method comprises, when a first image is classified into a class representing a particular view of the heart, automatically generating and displaying the time indicator12.

With each new classified image, the time indicator12is preferentially updated in real time:if the number of classified images since the first image includes a rate of images associated with the same class that is greater than the predefined rate, then the time indicator12is updated indicating the remaining time or number of remaining images to be received to reach the predefined number of images and the recording of the video sequence20;if the number of classified images since the first image includes a rate of images associated with the same class that is less than the predefined rate, then the time indicator12is updated to indicate the failure of the validation, for example by resetting the counter or gauge to zero or by making the displaying of such a counter or gauge disappear.

The method comprises displaying in real time the received ultrasound images, the quality indicator13, and optionally the time indicator12. Such displaying is described below with reference toFIG.2. The image displayed by the display AFF comprises the ultrasound image11received in real time or taken in real time by the ultrasound probe SECH, the quality indicator13and the time indicator12.

Program

According to one embodiment, the method according to the invention comprises selecting a first class from among the classes representing a particular view of the heart such as one of the above-mentioned views.

The selection of the first class generates displaying a first pre-recorded ultrasound image15of the heart representing such a view on the display AFF. Preferably, the label17which may include the name of said particular view is also displayed on the display AFF. Selecting the first class also generates the display of a first setpoint image14. The first setpoint image14is preferably pre-recorded. The setpoint image shows a position and/or orientation setpoint of the ultrasound probe SECH on a patient to obtain said view corresponding to the class selected. The operator is thus advantageously guided and assisted in taking such a view. On the same display AFF, they can view an example of the view that they must take, the position and orientation of the probe to achieve this. Finally, the operator can view in real time whether the view acquired is of sufficiently good quality by the quality indicator13, and can advantageously view the influence of the movements they perform on the quality of the image acquired thanks to the confidence indicator.

Finally, once the operator has found an image of satisfactory quality, they are assisted in real time by the time indicator12representative of the time during which they must hold the probe to acquire images of sufficient quality. The operator is also assured, by visualizing the time indicator12, that a first sequence20is going to be recorded by maintaining its position.

Once the first sequence has been recorded or validated, the method may include automatically selecting a second class from among the classes representing a particular view of the heart such as one of the above-mentioned views. Again, the selection of said second class automatically generates displaying a second pre-recorded target image and displaying a second pre-recorded ultrasound image illustrating an ultrasound view.

According to one embodiment, the method further comprises generating and displaying a progress indicator18. The progress indicator may represent the number of sequences that have been recorded. For example, the progress indicator is representative of the number of classes representing a particular view of the heart for which a sequence20of images has been automatically generated and/or recorded according to the method of the invention.

According to one embodiment, the method comprises displaying the recorded sequence and further comprises a second manual validation by the operator after displaying said sequence.

Device

According to one aspect, the device according to the invention comprises software and hardware means for implementing the method such as described above.

An embodiment of the device according to the invention is now described in reference toFIG.2.

The device includes a receptor REC. The receiver REC is intended to be connected to an ultrasonic probe SECH in such a way as to receive continuously and in real time ultrasound images11acquired by said ultrasonic probe SECH.

The receiver REC can be connected to the ultrasonic probe SECH by a wired or wireless link, for example by a Bluetooth connection or a WI-FI connection or any other data exchange protocol known to those skilled in the art.

The receiver REC may comprise or be associated with one or more memories to temporarily store the images received. The receiver REC is directly or indirectly connected to the display AFF to transmit to the display AFF the ultrasound images acquired11.

The device further comprises means for implementing a classifier CLASS such as described above. The classifier is configured to receive ultrasound images11, associate a class14with ultrasound images in real time, and generate the quality indicator13in real time. The classifier is connected directly or indirectly to the display AFF to transmit the indicators generated to the display AFF.

According to an alternative, the classifier is implemented by remote electronic equipment, such as a remote server. In this case, the device comprises an interface to exchange data with the remote equipment in order to transmit data and retrieve the result of the processed, i.e., classified, data.

Finally, in the present invention, it is understood that when the classification is implemented wholly in part by remote equipment, the device of the invention can be interpreted as being the system comprising on the one hand the local device described in the present application and the remote means making it possible to implement the classification function.

The device further comprises at least one processor or calculator CALC associated with a memory for performing at least part of the steps of the method according to the invention. For example, said processor may be configured to perform the steps of classifying200, generating the display of the quality indicator300and the time indicator500and/or automatically recording300the sequence20of images. Said processor may be connected to the display AFF to transmit the time indicator12to the display AFF.

The device1may also comprise a plurality of processors, each associated with one or more memories, and configured to perform such steps together. In one embodiment, the processor(s) may be remote and connected to the display via a data network.

The device further comprises a memory for storing or recording sequences generated by the method according to the invention. In one embodiment, the device further comprises a transmitter EMM connected to said memory MEM to transmit said sequences recorded on said memory MEM to a data network.

The display AFF may comprise means to receive the different information11,13,21,12received by the different means REC, CLASS, PROC of the device to generate the final image to display.