Patent Description:
Many medical imaging modalities (e.g., ultrasound US, magnetic resonance imaging MRI, CT, positron emission tomography PET, etc.) provide temporal data describing functions of various body organs. One of the main application of the temporal functional imaging is the diagnosis and monitoring of heart disease. Because the heart is in constant periodic motion, the temporal imaging is extensively used to characterize cardiac function by analysis of the cardiac deformation.

Echocardiography (echo) as known in the art of the present disclosure is one of the most popular techniques used to capture the temporal data of a beating heart. Echo has several advantages over other imaging modalities that include low cost and portability. Echo real-time imaging and does not use any ionizing radiation.

There are two (<NUM>) different acquisition modes, the most widely utilized two-dimensional (2D) mode and a less popular three-dimensional mode (3D) mode.

For 2D echo, an ultrasound transducer is positioned close to the sternum and images in 2D planes intersecting the heart are acquired at <NUM> to <NUM> per second frame rate. These movies (temporal sequence of planar echocardiac images) are visualized live for the sonographer and can be saved and sent for later interpretation/diagnosis (e.g. PACS). 2D echo requires an acquisition of several different planes going through the heart to cover the entire volume of the myocardium.

For 3D echo, a more sophisticated transducer is used and temporal sequence of volumetric echocardiac images of beating heart are acquired.

An electrocardiogram (ECG) as known in the art of the present disclosure increases an ability to detect abnormal cardiovascular conditions (e.g., cardiomyopathies) that may lead to sudden cardiac arrest. The result of the ECG is a waveform that indicates the electrical activity of the heart during the heart cycle, and an ECG is simultaneously performed with an echo to enhance the cardiac diagnosis.

The main application of Echo relevant to the present disclosure is the detection and characterization of cardiovascular disease (CVD). The disease may be a result of occlusion in one or more coronary arteries which results in reduced contractility of one or more of the segments of the heart. In clinical applications of echo, the abnormalities in cardiac wall motion are detected based on temporal echo images and quantified. In the current practice, this quantification is done by subjective visual examination of the temporal images and detection of cardiac wall motion and thickening abnormalities per myocardial segment. The interpretation of the Echo may be done either during the examination as the images are visualized real-time or post examination at the reading console (e.g. PACS). There are many other types of cardiac diseases that come from abnormalities in cardiac function either electrical or mechanical in nature. The common dominator of those diseases if that they manifest in either the cardiac structure or/and in function (electrical/mechanical).

There is a substantial research effort done into modelling of cardiac deformation as evidenced by echo images. Majority of those efforts are based on image analysis. For example, detection of endocardial wall may be utilized and then quantified. Also, segmentation, speckle tracking, non-rigid registration approaches may be utilized to automatically track the cardiac motion and determine the motion abnormalities. A publication dated <NUM>-<NUM>-<NUM> with the title "<NPL>et al discloses a machine-learning algorithm, in particular an artificial neural network, for differentiating athlete's heart (ATH) from hypertrophic cardiomyopathy (HCM). However, all of those approaches suffer from a problem of severe noise in ultrasound images which prevents the robust implementation of these algorithms.

A different approach to this problem is to use a different data acquisition model involving a Doppler acquisition of ultrasound in which motion of tissues can be quantified. For this approach however, the motion can only be quantified in beam direction and results are dependent on the signal-to-noise ratio.

One of the major problems of the aforementioned Echo procedures is the diagnosis of CVD based on motion of the cardiac wall is done in a completely subjective manner. An echocardiographer eyeballs the temporal views and based on those views determines which segments exhibit motion abnormalities indicative of a reduced cardiac fiber contractility due to CVD.

The visual assessment that is used today is highly dependent on experience and training of an echocardiographer. It follows that inter-observer and intra-observer variability is significant. The other difficulty with interpretation of Echo is that it requires highly trained professionals needed for interpretation of echo images. If they are not promptly available or not available all the utility of Echo is substantially reduced for instant diagnosis.

Moreover, as previously stated, echo examinations are typically accompanied by the acquisition of ECG waveforms. However, the echo and the ECG are interpreted separately reducing the synergy of these tests.

To improve upon detection and characterization of cardiovascular disease (CVD) via acquisition of echo cardiac images, the inventions of the present disclosure provides systems, devices, controllers and methods for standardizing a classification/quantification of abnormal cardiac conditions (e.g., heart wall motion abnormalities) evidenced by echo cardiac images which may be combined with electrocardiograms to thereby standardize a diagnosis of CVD using echo.

Generally, the inventions of the present disclosure is premised on application of a deep convolutional neural network to an echocardiogram based on a modelling of temporal changes in the echocardiogram.

The inventions of the present disclosure is a convolutional neural cardiac diagnostic system comprising an ultrasound device for generating echocardiogram data and an echocardiogram controller for controlling a generation of an echocardiogram derived from the echocardiogram data. The echocardiogram includes a temporal sequence of echocardiac cycles.

The convolutional neural cardiac diagnostic system further includes a cardiac diagnostic controller for controlling a diagnosis of the echocardiogram. To this end, the cardiac diagnostic controller includes a periodic volume generator for generating an echocardiogram diagnostic volume including a periodic stacking of the temporal sequence of echocardiac cycles and further includes a diagnostic convolutional neural network for classifying(quantifying) the echocardiogram as one of a normal echocardiogram or an abnormal echocardiogram based on a convolutional neural analysis of the echocardiogram diagnostic volume.

Disclosed is that the convolutional neural diagnostic echo system further comprises a lead system for generating electrocardiogram data, and an electrocardiogram controller for controlling a generation of an electrocardiogram derived from the electrocardiogram data. The electrocardiogram includes a temporal sequence of electrocardiogram waves.

The periodic volume generator further generates an electrocardiogram diagnostic volume including a periodic stacking of the temporal sequence of electrocardiogram waves, and the diagnostic convolutional neural network classifies(quantifies) the echocardiogram as one of the normal echocardiogram or the abnormal echocardiogram based on a convolutional neural analysis of both the echocardiogram diagnostic volume and the electrocardiogram diagnostic volume.

The invention of the present disclosure is a convolutional neural cardiac diagnostic method comprising an ultrasound device generating echocardiogram data, and an echocardiogram controller controlling a generation of an echocardiogram derived from the echocardiogram data. The echocardiogram includes a temporal sequence of echocardiac cycles.

The convolutional neural cardiac diagnostic method further comprises a cardiac diagnostic controller controlling a diagnosis of the echocardiogram by generating an echocardiogram diagnostic volume including a periodic stacking of the temporal sequence of echocardiac cycles, and further by classifying(quantifying) the echocardiogram as one of a normal echocardiogram or an abnormal echocardiogram based on a convolutional neural analysis of the echocardiogram diagnostic volume.

Disclosed is that the convolutional neural diagnostic echo method comprises a lead system generating electrocardiogram data, and an electrocardiogram controller controlling a generation of an electrocardiogram derived from electrocardiogram data. The electrocardiogram includes a temporal sequence of electrocardiogram waves.

Disclosed is that the convolutional neural cardiac diagnostic method further comprises the cardiac diagnostic controller controlling the diagnosis of the echocardiogram by generating an electrocardiogram diagnostic volume including a periodic stacking of the temporal sequence of electrocardiac waves and by further classifying(quantifying) the echocardiogram as one of a normal echocardiogram or an abnormal echocardiogram based on a convolutional neural analysis of both the echocardiogram diagnostic volume and the electrocardiogram diagnostic volume.

For purposes of describing and claiming the inventions of the present disclosure,.

The foregoing embodiments and other embodiments of the inventions of the present disclosure as well as various features and advantages of the present disclosure will become further apparent from the following detailed description of various embodiments of the present disclosure read in conjunction with the accompanying drawings. The detailed description and drawings are merely illustrative of the present disclosure rather than limiting, the scope of the present disclosure being defined by the appended claims.

The inventive principles of the present disclosure are applicable to any type of cardiac diagnostic procedure including, but not limited to, echocardiography, cardiac CT, cardiac MRI, angiography, cardiac positron emission tomography (PET) and cardiac single photon computed emission tomography (SPECT). To facilitate an understanding of the inventive principles of the present disclosure, the inventions of the present disclosure will be described in the context of an echocardiography application. From this description, those having ordinary skill in the art will appreciate how to apply the general inventive principles of the present disclosure for any type of cardiac diagnostic procedure.

In particular to echocardiography, <FIG> illustrates six (<NUM>) standard echocardiogram views consisting of a four chamber echocardiogram view <NUM>, a two chamber echocardiogram view <NUM>, a long axis echocardiogram view <NUM>, a base echocardiogram view <NUM>, a mid-echocardiogram view <NUM> and an apex echocardiogram view <NUM>.

As shown in <FIG>, four chamber echocardiogram view <NUM> illustrates an apical cap segment <NUM>, an apical septum segment <NUM>, an apical lateral segment <NUM>, a mid inferoseptum segment <NUM>, a mid anterolateral segment <NUM>, a basal inferoseptum <NUM> and a basal anteroplateral segment <NUM>.

As shown in <FIG>, two chamber echocardiogram view <NUM> illustrates an apical cap segment <NUM>, an apical inferior segment <NUM>, an apical anterior segment <NUM>, a mid inferior segment <NUM>, a mid anterior segment <NUM>, a basal inferior segment <NUM> and a basal anterior segment <NUM>.

As shown in <FIG>, long axis echocardiogram view <NUM> illustrates an apical cap segment <NUM>, an apical lateral segment <NUM>, an apical septum segment <NUM>, a mid inferolateral segment <NUM>, a mid anteroseptum segment <NUM>, a basal inferolateral <NUM> and a basal anteroseptum segment <NUM>.

As shown in <FIG>, base echocardiogram view <NUM> illustrates an anterior segment <NUM>, an anterolateral segment <NUM>, an inferolateral segment <NUM>, an inferior segment <NUM>, an inferoseptum segment <NUM> and an anteroseptum segment <NUM>.

As shown in <FIG>, mid echocardiogram view <NUM> illustrates an anterior segment <NUM>, an anterolateral segment <NUM>, an inferolateral segment <NUM>, an inferior segment <NUM>, an inferoseptum segment <NUM> and an anteroseptum segment <NUM>.

As shown in <FIG>, apex echocardiogram view <NUM> illustrates an anterior segment <NUM>, a lateral segment <NUM>, an inferior segment <NUM> and a septal segment <NUM>.

The inventions of the present disclosure as applied to echocardiography provide for a detection and classification(quantification) of cardiovascular disease (CVD) involving an occlusion in one or more arteries, which results in contractility of one or more of the segments shown in <FIG>. More particularly, any abnormalities in cardiac wall motion are detected and classification(quantification) on a segment basis by a convolutional neural analysis of one or more of the echocardiogram views shown in <FIG>.

To facilitate an understanding of a convolutional neural cardiac training aspect of the inventions of the present disclosure, the following description of <FIG> teaches general inventive principles of an convolutional neural cardiac training of the present disclosure. From this description, those having ordinary skill in the art will appreciate how to apply the general inventive principles of the present disclosure for implementing numerous and various embodiments of convolutional neural cardiac training of the present disclosure.

Referring to <FIG>, a convolutional neural cardiac training controller <NUM> employs a training periodic volume generator <NUM> and a training convolutional neural network <NUM> for training a detection and classification(quantification) of CVD, particularly on a segment basis. For training purposes, convolutional neural cardiac training controller <NUM> may further employ a database manager <NUM> and a training database <NUM> as shown, or alternatively be in communication with database manager <NUM> for purposes of accessing training database <NUM>.

Training database <NUM> stores a set <NUM> of echocardiograms <NUM> demonstrating normal cardiac wall motion (and/or any other normal cardiac function) and a set <NUM> of echocardiograms <NUM> demonstrating abnormal cardiac wall motion (and/or any other abnormal cardiac function). Training database <NUM> may further store a set of electrocardiograms (not shown) corresponding to normal echocardiogram set <NUM>, and a set of electrocardiograms (not shown) corresponding to abnormal echocardiogram set <NUM>.

In practice, echocardiograms <NUM> and <NUM> may include a temporal sequence of 2D planar echo slices and/or a temporal sequence of 3D volume images.

As shown in <FIG>, echocardiograms as stored on training database <NUM> may range on an echo scale <NUM> extending between an ideal normal echocardiogram <NUM> and a fatal abnormal echocardiogram <NUM> with a midline echocardiogram <NUM> representative of at a transitional state between a normal echocardiogram and an abnormal echocardiogram.

In practice, each normal echocardiograms <NUM> is positioned on echo scale <NUM> between deal normal echocardiogram <NUM> and midline echocardiogram <NUM> with a degree of cardiac wall motion normality, and each abnormal echocardiogram <NUM> is positioned on echo scale <NUM> between midline echocardiogram <NUM> and fatal abnormal echocardiogram <NUM> with a degree of cardiac wall motion abnormality.

Also in practice, set <NUM> of normal echocardiograms <NUM> and set <NUM> of abnormal echocardiograms <NUM> may include a single segmental echocardiogram view (<FIG>) or alternatively include subsets for two or more segmental echocardiogram views (<FIG>).

Referring back to <FIG>, training periodic volume generator <NUM> is an application module structurally configured to generate one or more normal echocardiogram training volume(s) <NUM> and one or more abnormal echocardiogram training volume(s) <NUM> in accordance with the inventive principles of the present disclosure.

Specifically, in practice, each normal echocardiogram <NUM> and each abnormal echocardiogram <NUM> may include a temporal sequence of echocardiac cycles.

For example, <FIG> illustrates an echocardiac cycle <NUM>EC consisting of a temporal sequence of 2D planar echocardiac image slices over a single heartbeat extending between a first echocardiac slice <NUM>ESF and a last echocardiac slice <NUM>ESL. Each echocardiogram <NUM> and each abnormal echocardiogram <NUM> includes a temporal sequence of echocardiac cycles <NUM>EC. Training periodic volume generator <NUM> implement digital imaging processing technique(s) as known in the art of the present disclosure for a periodic stacking of the temporal sequence of echocardiac cycles <NUM>EC whereby a last echocardiac slice <NUM>ESL of any given echocardiac cycle <NUM>EC is a neighbor of a first echo echocardiac slice <NUM>ESF of any succeeding echocardiac cycle <NUM>EC.

For example, <FIG> illustrates a normal echocardiogram training volume 111a of the present disclosure derived from a periodic stacking of the temporal sequence of an X number of echocardiac cycle <NUM>EC, X ≥ <NUM>, of a normal echocardiogram <NUM> whereby a last echocardiac slice <NUM>ESL of any given echocardiac cycle <NUM>EC is a neighbor of a first echo echocardiac slice <NUM>ESF of any succeeding echocardiac cycle <NUM>EC.

In practice, training periodic volume generator <NUM> generates a normal echocardiogram training volume 111a for one or more of the echocardiogram segmental views of a normal echocardiogram <NUM> whereby the normal echocardiogram training volume 111a may consist of a single degree or a multiple-degree of normality of a cardiac wall motion per scale <NUM> (<FIG>). For example, <FIG> illustrates six (<NUM>) normal echocardiogram training volumes 111a corresponding to the six (<NUM>) echocardiogram segmental views of <FIG>.

Similarly, <FIG> illustrates an abnormal echocardiogram training volume 112a of the present disclosure derived from a periodic stacking of the temporal sequence of an X number of echocardiac cycle <NUM>EC, X ≥ <NUM>, of an abnormal echocardiogram <NUM> whereby a last echocardiac slice <NUM>ESL of any given echocardiac cycle <NUM>EC is a neighbor of a first echo echocardiac slice <NUM>ESF of any succeeding echocardiac cycle <NUM>EC.

In practice, training periodic volume generator <NUM> generates an abnormal echocardiogram training volume 112a for one or more of the echocardiogram segmental views of an abnormal echocardiogram <NUM> whereby the abnormal echocardiogram training volume 112a may consist of a single degree or a multiple-degree of normality of a cardiac wall motion per scale <NUM> (<FIG>). For example, <FIG> illustrates six (<NUM>) abnormal echocardiogram training volumes 112a corresponding to the six (<NUM>) echocardiogram segmental views of <FIG>.

Referring back to <FIG>, training periodic volume generator <NUM> may be further structurally configured to generate one or more electrocardiogram training volume(s) <NUM> in accordance with the inventive principles of the present disclosure.

Specifically, as previously described, training database <NUM> may store an electrocardiogram corresponding to each normal echocardiogram <NUM> and each abnormal echocardiogram <NUM> whereby each electrocardiogram includes a temporal sequence of ECG waves.

For example, <FIG> illustrates an ECG wave <NUM>CC over a single heartbeat. Training periodic volume generator <NUM> implement digital imaging processing technique(s) as known in the art of the present disclosure for a periodic stacking of the temporal sequence of ECG waves <NUM>. For example, <FIG> illustrates an electrocardiogram training volume 113a of the present disclosure derived from a periodic stacking of the temporal sequence of an X number of ECG waves <NUM>CC, X ≥ <NUM>.

Referring back to <FIG>, in practice, training periodic volume generator <NUM> may generate an electrocardiogram training volume 113a for each generated normal echocardiogram training volume <NUM> and each generated abnormal volume <NUM>. For example, <FIG> illustrates six (<NUM>) abnormal echocardiogram training volume 112a for the six (<NUM>) echocardiogram segmental views of <FIG> with each abnormal echocardiogram training volume 112a having a corresponding electrocardiogram training volume 113a.

Referring back to <FIG>, in practice, each normal echocardiogram <NUM> and each abnormal echocardiogram <NUM> as stored in training database <NUM> may alternatively include a temporal sequence of echocardiac cycles of three-dimensional (3D) volumetric echocardiac images.

For example, <FIG> illustrates an echocardiac cycle <NUM>EC consisting of a temporal sequence of 3D volumetric echocardiac images <NUM> over a single heartbeat extending between a first volumetric echocardiac image <NUM>VEF and a last volumetric echocardiac image <NUM>VEL. Each echocardiogram <NUM> and each abnormal echocardiogram <NUM> includes a temporal sequence of echocardiac cycles <NUM>EC. Training periodic volume generator <NUM> implement digital imaging processing technique(s) as known in the art of the present disclosure for a periodic stacking of the temporal sequence of echocardiac cycles <NUM>EC whereby a last volumetric echocardiac image <NUM>ESF of any given echocardiac cycle <NUM>EC is a neighbor of a first volumetric echocardiac image <NUM>ESF of any succeeding echocardiac cycle <NUM>EC.

For example, <FIG> illustrates an X number of echocardiac cycles <NUM>EC, X ≥ <NUM>, extending over one or more heartbeats. Training periodic volume generator <NUM> implement digital imaging processing technique(s) as known in the art of the present disclosure for a periodic stacking of the temporal sequence of echocardiac cycles <NUM>EC <NUM> to form a normal echocardiogram training volume 111b from a normal 3D echocardiogram <NUM> or an abnormal echocardiogram training volume 112b from an abnormal 3D echocardiogram <NUM>.

In practice, training periodic volume generator <NUM> generates a normal echocardiogram training volume 111b for one or more of the echocardiogram segmental views of a normal 3d echocardiogram <NUM> whereby the normal echocardiogram training volume 111b may consist of a single degree or a multiple-degree of normality of a cardiac wall motion per scale <NUM> (<FIG>). For example, <FIG> illustrates six (<NUM>) normal echocardiogram training volume 111b for the six (<NUM>) echocardiogram segmental views of <FIG>.

Similarly in practice, training periodic volume generator <NUM> generates an abnormal echocardiogram training volume 112b for one or more of the echocardiogram segmental views of an abnormal 3D echocardiogram <NUM> whereby the abnormal echocardiogram training volume 112b may consist of a single degree or a multiple-degree of abnormality of a cardiac wall motion per scale <NUM> (<FIG>). For example, <FIG> illustrates six (<NUM>) normal echocardiogram training volume 112a for the six (<NUM>) echocardiogram segmental views of <FIG>.

Also in practice, electrocardiogram training volumes 113a (<FIG>) may be co-generated with volumes 111b/112b as previously described in the present disclosure for volumes 111a/112b (<FIG>).

Referring back to <FIG>, training convolutional neural network (CNN) <NUM> is an application module structurally configured for processing a normal echocardiogram training volume <NUM> to generate a normal echocardiogram classifier <NUM> indicative of a normal cardiac wall motion, and for processing an abnormal echocardiogram training volume <NUM> to output an abnormal echocardiogram classifier <NUM> indicative of an abnormal cardiac wall motion. If corresponding electrocardiograms are utilized, training CNN <NUM> processes volumes <NUM> and <NUM> to output normal echocardiogram classification <NUM>, and training CNN <NUM> processes volumes <NUM> and <NUM> to output abnormal echocardiogram classification.

In practice, training CNN <NUM> may execute any type of CNN known in the art of the present disclosure for delineating a connectivity pattern between motion features of volumes <NUM>, <NUM> and <NUM> (if applicable) that facilitates a classification of motion within volumes <NUM>, <NUM> and <NUM> (if applicable).

In one embodiment, training CNN <NUM> executes a basic spatial-temporal CNN involving a connectivity between layers via local filters, and a parameter sharing via convolutions. In the training process, the CNN is learned to recognize patterns in the echo images (and ECG) which is indicative of the cardiac abnormalities. The type of abnormality that CNN is trained to recognize is defined during the training process by using training cases (images and ECG signals) with the abnormality present. The training can be commenced either with or without ECG signal depending on availability of ECG data.

For example, <FIG> illustrates a training CNN 170a for processing a segmental view of a normal echocardiogram training volume <NUM> or of an abnormal echocardiogram training volume <NUM> involving a 3D convolution and subsampling stage 171a of filters <NUM> and <NUM> for combining spatial and temporal information of volume <NUM> or volume <NUM> to establish a fully connected stage 173a of motion features 175a for classification. In practice, a particular setup of training CNN 170a in terms of the number and types of layers and kernels will be dependent upon (<NUM>) a size of volume <NUM> or of volume <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 170a and (<NUM>) a type of abnormality that CNN 170a is designed to classify/quantify.

By further example, <FIG> illustrates a training CNN 170b for processing an additional segmental view of a normal echocardiogram training volume <NUM> or of an abnormal echocardiogram training volume <NUM> involving a 3D convolution and subsampling stage 171b of filters <NUM> and <NUM> for combining spatial and temporal information of the additional volume <NUM> or volume <NUM> to establish a fully connected stage 173b of motion features 175a and motion features 175b to output motion features 175c for classification. In practice, a particular setup of training CNN 170b in terms of the number and types of layers and kernels will be dependent upon (<NUM>) a size of volumes <NUM> or of volumes <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 170b and (<NUM>) a type of abnormality that CNN 170b is designed to classify/quantify.

By further example, <FIG> illustrates a training CNN 170c for additionally processing an electrocardiogram training volume <NUM> involving a 3D convolution and subsampling stage 171c of filters <NUM> and <NUM> for combining spatial and temporal information of volume <NUM> to establish a fully connected stage 173c of motion features 175a and wave features 175d to output motion features 175e for classification. In practice, a particular setup of training CNN 170c in terms of the number and types of layers and kernels will be dependent upon (<NUM>) a size of volumes <NUM>/<NUM> or of volumes <NUM>/<NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 170c and (<NUM>) a type of abnormality that CNN 170c is designed to classify/quantify.

Referring back to <FIG>, in a second embodiment, training CNN <NUM> executes a multiple stream CNN involving an execution of a spatial-temporal CNN for each echocardiogram slice of a normal echocardiogram training volume <NUM> or an abnormal echocardiogram training volume <NUM> (i.e., a spatial stream CNN) and an execution of spatial-temporal CNN for a motion flow of normal echocardiogram training volume <NUM> or abnormal echocardiogram training volume <NUM> (i.e., a temporal stream CNN). The multiple (dual) streams are combined by a late fusion of scores (e.g., an averaging a linear SVM, another neural network). The information from the multiple (dual) streams can also be combines by using shared convolutional kernels between different streams.

For example, <FIG> illustrates a training CNN 180a for executing a spatial stream CNN 182a for each eco echocardiogram slice 181a of a segmental view of a normal echocardiogram training volume <NUM> or of an abnormal echocardiogram training volume <NUM>, and for executing a temporal stream CNN 184a for a motion flow 183a of volume <NUM> or of volume <NUM>. The multiple streams 182a and 184a are combined by a late score fusion <NUM>. In practice, a particular setup of training CNN 180a in terms a complexity of spatial stream CNN 182a and temporal stream CNN 184a will be dependent upon (<NUM>) a size of volume <NUM> or of volume <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 180a and (<NUM>) a type of abnormality that CNN 180b is designed to classify/quantify.

By further example, <FIG> illustrates a training CNN 180b for executing a spatial stream CNN 182b for each eco echocardiogram slice 181b of an additional segmental view of a normal echocardiogram training volume <NUM> or of an abnormal echocardiogram training volume <NUM>, and for executing a temporal stream CNN 184b for a motion flow 183b of the additional volume <NUM> or of the additional volume <NUM>. The multiple streams 182a and 184a and the multiple streams 182b and 184b are combined by a late score fusion <NUM>. In practice, a particular setup of training CNN 180b in terms a complexity of spatial stream CNNs 182a and 182b and of temporal stream CNN 184a and 184b will be dependent upon (<NUM>) a size of volumes <NUM> or of volumes <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 180b and (<NUM>) a type of abnormality that CNN 180b is designed to classify/quantify.

By further example, <FIG> illustrates a training CNN 180c for executing a spatial stream CNN 182c for each electrocardiac wave <NUM> of an electrocardiogram training volume <NUM>, and for executing a temporal stream CNN 184b for a wave flow <NUM> of volume <NUM>. The multiple streams 182a and 184a and the multiple streams 182c and 184c are combined by a late score fusion <NUM>. In practice, a particular setup of training CNN 180b in terms of a complexity of spatial stream CNNs 182a and 182c and of temporal stream CNN 184a and 184c will be dependent upon (<NUM>) a size of volumes <NUM>/<NUM> or of volumes <NUM>/<NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 170c and (<NUM>) a type of abnormality that CNN 170c is designed to classify/quantify.

Referring back to <FIG>, in a third embodiment, training CNN <NUM> executes a memory recurrent CNN involving an execution of a spatial-temporal CNN for each echocardiogram slice or a sliced 3D volume of a normal echocardiogram training volume <NUM> or an abnormal echocardiogram training volume <NUM>, a mean polling of the outputs of the spatial temporal CNNs, and an execution of a recurrent neural network (RNN) of the mean polling to obtain a scoring output.

For example, <FIG> illustrates a memory recurrent CNN 190a involving an execution of a mean polling 192a of a spatial-temporal CNN 191a for each echocardiogram slice of a segmental view of a normal echocardiogram training volume <NUM> or an abnormal echocardiogram training volume <NUM>, followed by an execution of a Long Short Term Memory (LSTM) RNN 193a and LSTM RNN 194a to obtain a scoring output 195a. In practice, a particular setup of training CNN 190a in terms of a complexity of spatial-temporal CNN 191a, LSTM RNN 193a and LSTM RNN 194a will be dependent upon (<NUM>) a size of volume or of volume <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 190a and (<NUM>) a type of abnormality that CNN 190a is designed to classify/quantify.

By further example, <FIG> illustrates a memory recurrent CNN 190b involving an execution of a mean polling 192b of a spatial-temporal CNN 191b for each echocardiogram slice of an additional segmental view of a normal echocardiogram training volume <NUM> or an abnormal echocardiogram training volume <NUM>, followed by an execution of a Long Short Term Memory (LSTM) RNN 193b and LSTM RNN 194b to obtain a scoring output 195b. In practice, a particular setup of training CNN 190b in terms of a complexity of spatial-temporal CNNs 191a and 191b, and LSTM RNNs 193a, 193b, 194a and 194b will be dependent upon (<NUM>) a size of volumes <NUM> or of volumes <NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 190b and (<NUM>) a type of abnormality that CNN 190b is designed to classify/quantify.

By further example, <FIG> illustrates a memory recurrent CNN 190c involving an execution of a mean polling <NUM> of a spatial-temporal CNN <NUM> for each electrocardiac wave of an electrocardiogram training volume <NUM>, followed by an execution of a Long Short Term Memory (LSTM) RNN <NUM> and LSTM RNN <NUM> to obtain a scoring output 195c. In practice, a particular setup of training CNN 190c in terms of a complexity of spatial-temporal CNNs 191a and <NUM>, and LSTM RNNs 193a, 193b, <NUM> and <NUM> will be dependent (<NUM>) a size of volumes <NUM>/<NUM> or of volumes <NUM>/<NUM> (whichever is being processed), (<NUM>) a desired detection accuracy of CNN 190c and (<NUM>) a type of abnormality that CNN 190c is designed to classify/quantify.

Referring back to <FIG>, normal echocardiogram classifier(s) <NUM> and abnormal echocardiogram classifier(s) <NUM> as generated by training convolution neural network <NUM> are utilized by a diagnostic convolution neural network for a real-time detection and characterization of any abnormality of a cardiac wall motion as will be further described in the present disclosure.

In practice, controller <NUM> may be installed in a workstation, accessible over a network by a workstation or distributed across a network.

For example, <FIG> illustrates a workstation <NUM> employing a monitor <NUM>, an input device <NUM> and a computer <NUM> having controller <NUM> installed therein.

By further example, <FIG> illustrates a workstation <NUM> employing a monitor <NUM>, an input device <NUM> and a computer <NUM> having a convolutional neural cardiac training device <NUM> installed therein. Device <NUM> employs training periodic volume generator <NUM> (<FIG>) and training CNN <NUM> (<FIG>) whereby database <NUM> (<FIG>) as managed by database manager <NUM> is accessible by periodic volume generator <NUM> via a network <NUM> of any type known in the art of the present disclosure.

Also in practice, controller <NUM> and device <NUM> may include a processor, a memory, a user interface, a network interface, and a storage interconnected via one or more system buses.

The processor may be any hardware device, as known in the art of the present disclosure or hereinafter conceived, capable of executing instructions stored in memory or storage or otherwise processing data. In a non-limiting example, the processor may include a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other similar devices.

The memory may include various memories, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, L1, L2, or L3 cache or system memory. In a non-limiting example, the memory may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

The user interface may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with a user such as an administrator. In a non-limiting example, the user interface may include a command line interface or graphical user interface that may be presented to a remote terminal via the network interface.

The network interface may include one or more devices, as known in the art of the present disclosure or hereinafter conceived, for enabling communication with other hardware devices. In an non-limiting example, the network interface may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, the network interface may implement a TCP/IP stack for communication according to the TCP/IP protocols. Various alternative or additional hardware or configurations for the network interface will be apparent\.

The storage may include one or more machine-readable storage media, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media. In various non-limiting embodiments, the storage may store instructions for execution by the processor or data upon with the processor may operate. For example, the storage may store a base operating system for controlling various basic operations of the hardware. The storage may further store one or more application modules in the form of executable software/firmware. Particularly, the storage stores executable software/firmware for training periodic volume generator <NUM> and training CNN <NUM>.

To facilitate an understanding of a convolutional neural cardiac diagnostic aspect of the inventions of the present disclosure, the following description of <FIG> teaches general inventive principles of an convolutional neural cardiac diagnostic aspect of the present disclosure. From this description, those having ordinary skill in the art will appreciate how to apply the general inventive principles of the present disclosure for implementing numerous and various embodiments of convolutional neural cardiac diagnostics of the present disclosure.

Referring to <FIG>, a convolutional neural cardiac diagnostic system <NUM> of the present disclosure employs an echocardiogram controller <NUM>, a ECG wave controller <NUM>, an echocardiogram diagnostic controller <NUM> and one or more output devices <NUM> (e.g., a display, a printer, a speaker and/or LED indicator(s)). In practice, controllers <NUM>, <NUM> and <NUM> may be fully or partially integrated, or segregated as shown.

Echocardiogram controller <NUM> is linked to and/or incorporates any necessary hardware/software interface to an ultrasound transducer 350a or an ultrasound probe 350b positioned relative to a heart <NUM> of a patient <NUM> for receiving echocardiogram data to thereby generate an echocardiogram as known in the art of the present disclosure. The echocardiogram includes a temporal sequence of echocardiac cycles <NUM> with echocardiac cycle <NUM> includes a temporal sequence of 2D echo slices as shown or a 3D echo image. Echocardiogram controller <NUM> sequentially communicates a temporal sequence of echocardiac cycles <NUM> of echocardiogram <NUM> via wired and/or wireless channel(s) to echocardiogram diagnostic controller <NUM> as shown and to output device(s) <NUM> for display.

ECG controller <NUM> is linked to and/or incorporates any necessary hardware/software interface to a cable connector <NUM> for receiving electrode signals from a lead system connected to patient <NUM> (e.g., a standard <NUM>-lead system, Mason-Likar lead system as shown or a reduced lead system like the EASI lead system) to thereby generate an electrocardiogram waveform <NUM> as known in the art of the present disclosure. Electrocardiogram waveform <NUM> includes a temporal sequence of ECG waves <NUM> as shown. Echocardiogram controller <NUM> sequentially communicates each ECG wave <NUM> of ECG waveform <NUM> via wired and/or wireless channel(s) to echocardiogram diagnostic controller <NUM> as shown and to output device(s) <NUM> for display.

Echocardiogram diagnostic controller <NUM> implement inventive principles of the present disclosure for the detection and classification(quantification) of any abnormality of cardiac wall motion of heart <NUM> and for generating an echocardiogram classification report <NUM> indicating a normal or an abnormal cardiac wall motion of heart <NUM>. In practice report <NUM>, may be displayed or printed with textual and/or graphical information by output device(s) <NUM>.

As shown in <FIG>, an echocardiogram diagnostic controller <NUM> employs a diagnostic periodic volume generator 331a and a diagnostic convolutional neural network (CNN) 333a.

Periodic volume generator 331a is an application module structurally configured for processing echo cardio cycles <NUM> to generate an echocardiogram training volume <NUM> in accordance with the inventive principles of the present disclosure previously described for training periodic volume generator <NUM> (<FIG>). In practice, echocardiogram training volume <NUM> consists of an X number of echo cardiac cycles <NUM>, whereby X may be unlimited or have a maximum limitation of echo cardiac cycles <NUM> involving a first in, first out implementation of echo cardiac cycles <NUM>.

The normality or the abnormality of echocardiogram training volume <NUM> is unknown.

Diagnostic CNN 333a therefore is an application module structurally configured for processing echocardiogram training volume <NUM> to generate an echocardiogram classification report 336a informative/illustrative of a normality or an abnormality of the cardiac wall motion of heart <NUM>. More particularly, diagnostic CNN 333a executes a CNN whereby an output of the CNN is compared to a training normal echocardiogram classifier 334a and an abnormal training echocardiogram classifier 335a to detect and classify(quantify) a normality or an abnormality of the cardiac wall motion of heart <NUM>.

In practice, diagnostic CNN 333a may execute any type of CNN known in the art of the present disclosure for delineating a connectivity pattern between motion features of echocardiogram training volume <NUM> that facilitates a classification of motion echocardiogram training volume <NUM>. For example, diagnostic CNN 333a may execute a spatial-temporal CNN, a multiple stream CNN and/or a memory recurrent CNN as previously described in the present disclosure for training CNN <NUM> (<FIG>).

Also in practice, diagnostic CNN 333a may implement any technique as known in the art for use the CNN outputs to train diagnostic models based on a normal echocardiogram 334a and an abnormal echocardiogram 335a. For example, diagnostic CNN 333a may employ a neural network, SVM networks developed/trained from outputs of CNN for normal echocardiogram classifier 334a and an abnormal training echocardiogram classifier 335a
As shown in <FIG>, an echocardiogram diagnostic controller <NUM> employs a diagnostic periodic volume generator 331b and a diagnostic convolutional neural network (CNN) 333b.

Periodic volume generator 331b is an application module structurally configured for additionally processing ECG waves <NUM> to generate an electrocardiogram training volume <NUM> in accordance with the inventive principles of the present disclosure previously described for training periodic volume generator <NUM> (<FIG>). In practice, electrocardiogram training volume <NUM> consists of an X number of ECG waves <NUM>, whereby X may be unlimited or have a maximum limitation of ECG waves <NUM> involving a first in, first out implementation of ECG waves <NUM>.

Diagnostic CNN 333b therefore is an application module structurally configured for processing both echocardiogram training volume <NUM> and electrocardiogram training volume <NUM> to generate an echocardiogram classification report 336b informative/illustrative of a normality or an abnormality of the cardiac wall motion of heart <NUM>. More particularly, diagnostic CNN 333b executes a CNN whereby an output of the CNN is compared to a training normal echocardiogram classifier 334b and an abnormal training echocardiogram classifier 335b to detect and classify(quantify) a normality or an abnormality of the cardiac wall motion of heart <NUM>.

In practice, diagnostic CNN 333a may execute any type of CNN known in the art of the present disclosure for delineating a connectivity pattern between motion features of echocardiogram training volume <NUM> and wave features of electrocardiogram training volume <NUM> that facilitates a classification of motion echocardiogram training volume <NUM>. For example, diagnostic CNN 333b may execute a spatial-temporal CNN, a multiple stream CNN and/or a memory recurrent CNN as previously described in the present disclosure for training CNN <NUM> (<FIG>).

Also in practice, diagnostic CNN 333a may implement any technique as known in the art for use the CNN outputs to train diagnostic models based on a normal echocardiogram 334a and an abnormal echocardiogram 335a. For example, diagnostic CNN 333a may employ a neural network, SVM networks developed/trained from outputs of CNN for normal echocardiogram classifier 334a and an abnormal training echocardiogram classifier 335a.

Referring back to <FIG>, in practice, echocardiogram controller <NUM>, ECG controller <NUM> and echocardiogram diagnostic controller <NUM> may be installed in a workstation, accessible over a network by a workstation or distributed across a network.

For example, <FIG> illustrates a workstation <NUM> employing a monitor <NUM>, an input device <NUM> and a computer <NUM> having controller suite <NUM> installed therein. Controller suite <NUM> includes controllers <NUM>, <NUM> and <NUM>.

By further example, <FIG> illustrates a workstation <NUM> employing a monitor <NUM>, an input device <NUM> and a computer <NUM> having echocardiogram controller <NUM> and <NUM> installed therein, and further illustrates a workstation <NUM> employing a monitor <NUM>, an input device <NUM> and a computer <NUM> having echocardiogram diagnostic controller <NUM> installed therein. Controllers <NUM>, <NUM> and <NUM> communicate over a network <NUM> of any type as known in the art of the present disclosure.

Also in practice, controllers <NUM>, <NUM> and <NUM> may include a processor, a memory, a user interface, a network interface, and a storage interconnected via one or more system buses.

The storage may include one or more machine-readable storage media, as known in the art of the present disclosure or hereinafter conceived, including, but not limited to, read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media. In various non-limiting embodiments, the storage may store instructions for execution by the processor or data upon with the processor may operate. For example, the storage may store a base operating system for controlling various basic operations of the hardware. The storage may further store one or more application modules in the form of executable software/firmware. Particularly, for echocardiogram diagnostic controller <NUM>, the storage stores executable software/firmware for training periodic volume generator <NUM> and training CNN <NUM>.

As previously described in the present disclosure, the inventive principles of the present disclosure are applicable to any type of cardiac diagnostic procedure including, but not limited to, echocardiography, CT heart scans and cardiac MRI echocardiography, cardiac CT, cardiac MRI, angiography, cardiac positron emission tomography (PET) and cardiac single photon computed emission tomography (SPECT). Thus, while the inventions of the present disclosure were described in the context of an echocardiography application, <FIG> illustrates a medical imaging modality <NUM> representative of an application of any type of cardiac diagnostic procedure for detecting and classifying(quantifying) a normality or an abnormality of a cardiogram applicable to the particular cardiac diagnostic procedure.

Specifically, examples of medical imaging modality <NUM> includes, but are not limited to, an ultrasound imaging modality, a X-ray computed tomography imaging modality, a magnetic resonance imaging modality, a fluoroscopic imaging modality, a position emission tomography imaging modality and a single-photo emission computed tomography imaging modality. Any embodiment of medical imaging modality <NUM> employs applicable imaging device(s) <NUM> and controller(s) <NUM> for generating cardiograms as known in the art of the present disclosure. Thus, the training and diagnostic aspects of the present disclosure are based on the particular type of cardiac imaging. In practice, the particular type of cardiac imaging may generate 2D planar and 3D volume images as exemplary shown herein and/or generate high dimensional imaging as known in the art of the present disclosure.

Referring to <FIG>, those having ordinary skill in the art will appreciate numerous benefits of the present disclosure including, but not limited to, (<NUM>) a reduction of intra-observer and inter-observer variability in interpreting echo image, (<NUM>) an allowance for a robot real-time diagnosis of a cardiovascular disease, (<NUM>) an improvement in reader confidence and a reduction in reading time of echo image and (<NUM>) an improvement in an accuracy of cardiovascular disease diagnosis by combining information contained in echo image with electrocardiogram waves.

Furthermore, as one having ordinary skill in the art will appreciate in view of the teachings provided herein, features, elements, components, etc. described in the present disclosure/specification and/or depicted in the drawings of the present disclosure may be implemented in various combinations of electronic components/circuitry, hardware, executable software and executable firmware, particularly as application modules of a controller as described in the present disclosure, and provide functions which may be combined in a single element or multiple elements. For example, the functions of the various features, elements, components, etc. shown/illustrated/depicted in the drawings of the present disclosure can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared and/or multiplexed. Moreover, explicit use of the term "processor" should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor ("DSP") hardware, memory (e.g., read only memory ("ROM") for storing software, random access memory ("RAM"), non-volatile storage, etc.) and virtually any means and/or machine (including hardware, software, firmware, circuitry, combinations thereof, etc.) which is capable of (and/or configurable) to perform and/or control a process.

It will be appreciated by one having ordinary skill in the art in view of the teachings provided herein that any block diagrams presented herein can represent conceptual views of illustrative system components and/or circuitry embodying the principles of the invention. Similarly, one having ordinary skill in the art should appreciate in view of the teachings provided herein that any flow charts, flow diagrams and the like can represent various processes which can be substantially represented in computer readable storage media and so executed by a computer, processor or other device with processing capabilities, whether or not such computer or processor is explicitly shown.

Furthermore, exemplary embodiments of the present disclosure can take the form of a computer program product or application module accessible from a computer-usable and/or computer-readable storage medium providing program code and/or instructions for use by or in connection with, e.g., a computer or any instruction execution system. In accordance with the present disclosure, a computer-usable or computer readable storage medium can be any apparatus that can, e.g., include, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. Such exemplary medium can be, e.g., an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include, e.g., a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), flash (drive), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk - read only memory (CD-ROM), compact disk - read/write (CD-R/W) and DVD. Further, it should be understood that any new computer-readable medium which may hereafter be developed should also be considered as computer-readable medium as may be used or referred to in accordance with exemplary embodiments of the present disclosure and disclosure.

Claim 1:
A convolutional neural cardiac diagnostic system, comprising:
a cardiogram controller structurally configured to control a generation of a cardiogram derived from a generation of cardiac imaging data by a medical imaging modality device, the cardiogram including a temporal sequence of cardiac images over a heartbeat; and
a cardiogram diagnostic controller (<NUM>) structurally configured to control a diagnosis of the cardiogram, wherein the cardiogram diagnostic controller (<NUM>) includes:
a diagnostic periodic volume generator (<NUM>) structurally configured to generate a cardiogram diagnostic volume derived from the generation of the cardiogram by the cardiogram controller, the cardiogram diagnostic volume including a periodic stacking of the temporal sequence of cardiac images over a heartbeat; and
a diagnostic convolutional neural network (<NUM>) structurally configured to classify the cardiogram as one of a normal cardiogram or an abnormal cardiogram based on a convolutional neural analysis of the cardiogram diagnostic volume as generated by the diagnostic periodic volume generator (<NUM>),
wherein the medical imaging modality device is an ultrasound imaging device (<NUM>), wherein the cardiogram is an echocardiogram, wherein the cardiogram controller is an echocardiogram controller (<NUM>), wherein the cardiogram diagnostic controller is an echocardiogram diagnostic controller, wherein the cardiogram data is echocardiogram data, wherein the cardiac images are echocardiac images, wherein the cardiogram diagnostic volume is an echocardiogram diagnostic volume.