Patent Description:
Image segmentation is used in digital image processing for partitioning a digital image or volume into multiple segments (e.g. groups of pixels/voxels), each covering a region of the image or volume. The different segments are typically identified and distinguished based on similar shared graphical properties of the pixels/voxels contained in that segmented region, e.g. color, intensity, or texture.

In the field of medical imaging, segmentation is valuable for identifying various anatomical structures or features such as organs, air and fluid passageways (e.g. blood vessels or digestive passages), valves, or chambers. The resulting contours of the segments enable more rapid assessment of medical images or volumes by clinicians (enabling quicker diagnosis or treatment decisions). The segmentation can also be useful for feeding in to subsequent processing techniques.

One important field of medical imaging is ultrasound imaging. Ultrasound imaging plays an important role in assessment and diagnosis for a wide range of areas of the body. Diagnosis, treatment and planning are often based on utilizing a clear delineation of relevant anatomical sites or features within ultrasound images and segmentation may be used to automate this delineation process.

Automatic segmentation and quantification of organs from 3D ultrasound volumes is an active field of research, but has not yet reached clinical routine. Some methods have been proposed, such as classification techniques or use of landmark-based approaches. However these methods often are not precise and typically contain segmentation leaks.

Automatic segmentation of 3D organs from 3D ultrasound arrays suffers from a main problem related to image acquisition. The acquired volumes are stored in rectangular volumes but contain information only within so-called cone shapes. This results in an inefficient representation of the target anatomies.

More precisely, acquisition methods of ultrasound data have the following disadvantages, which have consequences on the training of automated segmentation algorithms, especially when they involve deep learning:.

Therefore, a method for overcoming these acquisition disadvantages, with the final objective to train automatic segmentation algorithms in ultrasound imaging, is needed.

<CIT> discloses a segmentation method for an inner membrane in a blood vessel of an intravascular ultrasound image.

<NPL>), discloses a method for segmenting a vessel wall from intravascular ultrasound images.

<NPL>), discloses a method for detecting and localizing fetal anatomical regions in 2D ultrasound images.

According to examples in accordance with an aspect of the invention, there is provided a method for segmenting a target anatomy in 3D ultrasound volumes, the method comprising:.

Ultrasound volumes typically contain "empty" data which has no meaningful information. Ultrasound images used to generate ultrasound volumes are typically represented as cone shapes when obtained with phased arrays or curved arrays, however they are displayed on rectangular (or square) screens. Thus, a large proportion of the pixels (or voxels for the 3D volumes) are empty and contain no meaningful information.

The de-scanned space represents the data size needed to store the scan-converted ultrasound volume in the initial coordinate system. For example, in Cartesian coordinates, the data for an image may be stored in a matrix, wherein each matrix element represents the color of a pixel in RGB. However, for scan-converted ultrasound images, the meaningful data is in a conical shape due to the acquisition of the ultrasound volume, thus a proportion of the data stored in the matrix (for a scan-converted ultrasound image) is empty data.

The ultrasound volume is obtained in the scan-converted space (e.g. a cube including the cone of ultrasound volume) in the Cartesian coordinate space with empty data. It is obtained in the Cartesian coordinate system as it represents how the image would be seen in real life, and is thus more intuitive to a clinician.

However, due to the empty data, a segmentation algorithm may give more weight to the shape of the cone and areas with empty data instead of segmenting the target anatomy during training. Thus, the scan-converted ultrasound volume is "de-scanned", which relates to performing an inverse scan conversion on the scan-converted ultrasound volume. This is done by transforming the scan-converted ultrasound volume to the Toroidal coordinate system (which may thus be considered to be the de-scanned coordinate system) which represents how the ultrasound volume was acquired. By transforming the scan-converted ultrasound volume to a coordinate system which represents how it was captured, it may significantly reduce the proportion of empty data.

A segmentation algorithm can then be trained and applied to the de-scanned ultrasound volume in the de-scanned space with more accuracy and precision than in the acquisition space.

The segmentation method is for example based on a convolutional neural network. The de-scan conversion method improves the convolutional neural network based segmentation method through efficient use of memory.

The method may further comprise transforming the de-scanned ultrasound volume in the de-scanned space back to the scan-converted space after the segmentation has been performed.

Once the de-scanned ultrasound volume is segmented for the target anatomy (e.g. kidney, fetus etc.), the segmentation data (in the de-scanned space) can be transformed back to the Cartesian coordinate system such that it can be overlaid on the initial image. The transformation of the segmentation thus provides a visualization of the segmented parts with the correct Cartesian geometry. This way, the clinician can analyze the ultrasound volume with no warping, with the added benefit that the data has now been segmented (e.g. to calculate the area/volume of the segmented target anatomy).

The method may further comprise displaying one or more of:.

The method may further comprise determining the volume of the target anatomy based on the segmentation of the target anatomy.

The de-scanned coordinate system is the Toroidal coordinate system.

The use of the Toroidal coordinate system is used because it best represents the acquisition of the ultrasound volume. For example, on phased arrays and curved arrays, the ultrasound probe is rotated around the center of a rotor (i.e. the motor that makes the ultrasound probe rotate) when 3D volumes are captured. Thus, the Toroidal coordinate system provides a lower proportion of empty data.

The method may further comprise estimating the acquisition geometry of the ultrasound volume thereby to derive the transformation for transforming the scan-converted ultrasound volume to a de-scanned ultrasound volume, and wherein estimating the acquisition geometry comprises one or more of:.

In general imaging acquisition, it is likely that the imaging lines of the scan-converted ultrasound volume are not saved with the scan-converted ultrasound volume, thus performing the de-scanning may not be directly possible due to the lack of acquisition geometry needed to perform the transformation. In this case, the acquisition geometry may need to be estimated from the scan-converted ultrasound volume before transforming the scan-converted ultrasound volume to the de-scanned space.

If there are separate optical and mechanical centers, the toroidal transformation results.

The estimations may be obtained by image analysis of the scan-converted ultrasound volume in the Cartesian coordinate system.

The invention also provides a computer program product comprising computer program code means which, when executed on a computing device having a processing system, cause the processing system to perform all of the steps of the method mentioned above.

The invention also provides a system for segmenting a target anatomy in de-scanned ultrasound volumes, the system comprising:
a processor configured to:.

The processor may be further configured to transform the segmentation data (in the de-scanned space) back to the scan-converted space after the segmentation has been performed.

The system may further comprise a display for displaying one or more of:.

The processor may be further configured to determine the volume of the target anatomy based on the segmentation of the target anatomy.

The processor may be configured to transform the ultrasound volume to a de-scanned space within the Toroidal coordinate system by estimating the acquisition geometry of the ultrasound volume thereby to derive the transformation for transforming the scan-converted ultrasound volume to a de-scanned ultrasound volume, and wherein the processor is configured to estimate the acquisition geometry based on one or more of:.

The invention provides a method for segmenting a target anatomy in ultrasound data. Scan-converted ultrasound data is obtained within a scan-converted space in the Cartesian coordinate system. The scan-converted ultrasound data is transformed to de-scanned ultrasound data within a de-scanned space in the Toroidal coordinate system. The de-scanned ultrasound data is an estimate of the ultrasound data as obtained by an original acquisition procedure. A segmentation of a target anatomy can thus be performed on the ultrasound data in the de-scanned space. The resulting segmentation data can then be re-scanned back to the Cartesian coordinate system for display with the ultrasound data.

<FIG> shows scan-converted ultrasound data <NUM> in a rectangular scan-converted space. Scan lines are typically used to create viewable ultrasound data <NUM> (e.g. images or volumes) from raw data based on how the ultrasound data <NUM> was acquired (based on the acquisition procedure) which allows the data to be visualized in a true Cartesian domain where distances and angles are measurable. Thus, the raw data captured by an ultrasound probe is scan-converted with help of the scan lines to Cartesian coordinates to provide a "real life" view of the ultrasound data. Additionally, the scan-converted ultrasound data <NUM> is typically stored in the scan-converted space (in Cartesian coordinates) and any analysis or data processing performed post-acquisition must typically be done in the scan-converted space.

Scan-converted ultrasound data <NUM> in Cartesian coordinates typically contains meaningful data <NUM> in a cone-like shape surrounded by "empty" (or meaningless) data <NUM>. Even though a significant proportion of the scan-converted ultrasound data <NUM> does not contain meaningful data <NUM>, all of the scan-converted space needs to be stored due to the nature of image data storage (typically a bitmap of image) in the Cartesian coordinates.

3D scan-converted ultrasound volumes typically contain significant proportions of empty data <NUM> (black voxels) which are not required for clinical information but are required for the scan-converted ultrasound volume to be stored in Cartesian coordinates.

<FIG> shows a histogram of the percentage of meaningful data <NUM> in a large database of volumetric kidney acquisitions. The large database of volumetric kidney acquisitions contained <NUM> volumes. The ratio of the volume within the imaging cone (meaningful data <NUM>) over the whole Cartesian volume was calculated for each volumetric kidney acquisition (in percentage %).

The x-axis shows the ratio (in percentage % and in steps of <NUM>%) of the meaningful data <NUM> to the total data stored in the scan-converted space. The y-axis shows the number of cases for each ratio range.

As can be seen, the range of ratios is from <NUM>% to <NUM>% with the average ratio of meaningful data <NUM> being around <NUM>%. Thus, the proportion of empty data <NUM> in the volumetric kidney acquisitions (which is only required for storing the data in Cartesian coordinates) is around <NUM>%.

This means that an average of <NUM>% of all of the data stored (for volumetric kidney acquisitions) is, in essence, wasted storage space. Data compression may be used to make the data storage more efficient, but for use with a deep learning algorithm the voxels will require memory storage. Additionally, any machine learning algorithm (e.g. segmentation algorithm) which is trained with this data will place an unjustified high importance on the empty data <NUM> due to the high proportion of empty data <NUM> in the an converted ultrasound data <NUM>.

<FIG> shows a flow diagram of the method for performing a segmentation on 3D ultrasound volumes. Once a 3D ultrasound volume is obtained, step <NUM>, an acquisition geometry can be estimated, step <NUM>, if it is not known. The acquisition geometry may include estimating the scan lines of the ultrasound data and finding the optical center and mechanical center of the acquisition process.

Estimating the acquisition geometry, step <NUM>, is based on estimating how the ultrasound data was acquired. For example, if an ultrasound volume was obtained with a phased array ultrasound probe rotated around the skin of a subject, the ultrasound volume will have a minimum depth and a maximum depth (relative to the ultrasound transducer array), an optical center (where the ultrasound transducers reside), a mechanical center (where the rotor of the ultrasound probe resides), an angle of view (based on the angle of an ultrasound image obtained by the ultrasound probe) and an angle of sweep (based on the rotation of the ultrasound probe around the skin by the rotor of the ultrasound probe). These quantities of acquisition geometry can be estimated from the ultrasound volume. With the acquisition geometry, the volume can be "de-scanned", step <NUM>.

De-scanning is the process of "un-doing" the scan conversion process, in order to get rid of as much empty data as possible. This can be done by choosing a coordinate system which simulates the acquisition process. The Toroidal coordinate system is a good approximation, as it requires two centers (equating to the optical center and mechanical center of the acquisition in ultrasound imaging).

Once the volume is de-scanned, a segmentation algorithm can be applied, step <NUM>, to the volume in order to segment a target anatomy. Working in the Toroidal coordinate system for the segmentation process is more powerful than other coordinate systems and provides better performance of the automatic segmentation. The target anatomy may include one or more of organs, blood vessels, heart chambers, bones, muscles etc..

The segmentation algorithm would have also been trained with de-scanned volumes. The segmentation data can then be returned to the Cartesian coordinate system (re-scanned), step <NUM>, such that viewing the volume (e.g. on a display) is more intuitive for a clinician. The original volume is for example shown with the segmentation data having been re-scanned. Optionally, the image data of the original volume could be re-scanned together with the segmentation data, but it is preferred to use the original image data (without de-scanning and re-scanning).

<FIG> shows an example of how a scan-converted ultrasound image 102a may be de-scanned for the purposes of illustrating the general steps. 2D Ultrasound images are used in this example instead of ultrasound volumes in order to simplify the example, whilst showing the de-scanning steps.

<FIG> shows the scan-converted ultrasound image 102a in Cartesian coordinates. The empty data <NUM> is needed in order to store the meaningful image 104a in a rectangular format, such as in a bitmap. A target anatomy <NUM> is shown to aid in explaining how the de-scan process works.

<FIG> shows the scan-converted ultrasound image 102a interpolated with imaging lines <NUM>. It may be that the imaging lines <NUM> are stored with the scan-converted ultrasound image 102a. However, in most cases, stored scan-converted ultrasound data <NUM> does not contain data on the imaging lines <NUM>. In this case, the imaging lines <NUM> can be estimated from the scan-converted ultrasound images 102a. Two imaging lines <NUM> can be estimated from the side edges from the scan-converted ultrasound image 102a. From these imaging lines <NUM>, an optical center <NUM> can be estimated and further imaging lines <NUM> may be constructed. An angle of view can also be estimated from the imaging lines <NUM> as the angle between the two furthest imaging lines <NUM> determined at the edges.

<FIG> shows a de-scanned ultrasound image <NUM> in polar coordinates (again because a simplified 2D image is used as an example). Thus, the the radial component is based on the distance of each data point from the optical center <NUM> found in <FIG>. The angular component of each data point can also be found by measuring the angle between an imaging line containing the data point and an edge imaging line for the scan-converted ultrasound image 102a in the Cartesian coordinates. As can be seen, there is no empty data <NUM> in the de-scanned ultrasound image <NUM>, whilst it can still be stored in rectangular form (e.g. RGB matrix bitmap). Additionally, the target anatomy <NUM> represents a higher proportion of the overall data (compared to the scan-converted ultrasound image 102a).

The de-scanned coordinate system used is instead the Toroidal coordinate system. <FIG> shows part of a toroid <NUM> in the Cartesian and Toroidal coordinate systems. The transform is inspired from the acquisition process as it performs a kind of de-scan conversion, which may be seen as a dedicated pre-processing step for a segmentation algorithm. Using the acquisition geometry parameters, a Cartesian to Toroidal transformation can be applied to the scan-converted ultrasound data <NUM> in order to mimic ultrasound lines that could have generated a similar volume. The Toroidal coordinate system can be used to represent meaningful 3D ultrasound data 104b.

3D ultrasound volumes are typically determined from many 2D ultrasound images. An optical center <NUM> can be obtained for the 2D ultrasound images (at least two). The optical centers will create a section of circle, and the center of the circle is the mechanical center <NUM>.

Once the mechanical center <NUM> and optical center <NUM> are obtained, the distance between them can be calculated - R<NUM>. The Cartesian coordinates (x, y, z) can thus be converted to the Toroidal coordinates (r, ϕ, Θ): <MAT> <MAT>.

Equations (<NUM>) and (<NUM>) can be used to transform a volume from Cartesian coordinates into Toroidal coordinates. In the acquisition volume, a point is named (x, y, z). In order to find its corresponding intensity at point (r, ϕ, θ) in the transformed domain, the equations calculate (r, ϕ, θ) as a function of (x, y, z) where (r, ϕ, θ) ∈ [R, Θ, Φ ].

[R, Θ, Φ ] are the ranges of values of the parameters selected for de-scan. This defines the size of the de-scanned volume that is computed as input to the segmentation training and testing algorithm.

The true values of these parameters are usually not known because there is no storage of the pre-scan conversion data. Thus, there is freedom to set these values. This is an advantage in a learning approach. For example, for a rough and fast algorithm, small values can be selected such as (<NUM>,<NUM>,<NUM>). In practice, larger ranges of values are preferably chosen that can still enable the deep learning algorithm to be trained in memory. It is also possible to define unequal values for these parameters such as (<NUM>,<NUM>,<NUM>). This can reflect the proportional distances in the original acquired volumes.

The de-scan can thus be applied with multiple choices of these parameters. This means that the method can be scaled to the available memory on computers when learning the network for a given organ. At the end of this step, a 3D dataset of dimension (R, Θ, Φ) is obtained, where each voxel carries significant information.

The minimum and maximum value of r can be calculated from the minimum depth and maximum depth of acquisition respectively, the value of θ can be calculated from the angle of view and the value of ϕ can be calculated from the angle of sweep.

Once the de-scanned ultrasound data <NUM> has been segmented in the de-scanned space, it can be returned to the scan-converted space in Cartesian coordinates using equations (<NUM>) and (<NUM>). <MAT> <MAT>.

<FIG> shows a scan-converted ultrasound volume 102b in a scan-converted space. The white cone represents meaningful data 104b, while the dotted space represents zero values which do not code any useful information <NUM>. The larger dotted circle outside the scan-converted space represents the mechanical center <NUM> and the smaller black circle represents the optical center <NUM> in an A-plane.

Using the meaningful data 104b, the parameters of the acquisition geometry are estimated. They represent: the center of the A-plane imaging beams (optical center <NUM>), the minimum and maximum depth of imaging, the angle of view in A-plane, the center of the sweep for imaging other planes (mechanical center <NUM>), and the corresponding sweep angle.

<FIG> shows an ultrasound volume 104b in spherical coordinates. The spherical coordinate system is an extreme case of the Toroidal coordinate system for which the optical center <NUM> and the mechanical center <NUM> are both at the same place (or can be approximated as being in the same place). The ultrasound volume 104b can in such cases be fully defined in the spherical coordinate system without the need of empty data. However, the use of the spherical coordinate system will not be as accurate as the use of the Toroidal coordinate system due to the (sometimes unrealistic) assumption that the optical center <NUM> and the mechanical center <NUM> are in the same place. Thus, the use of the Toroidal coordinate system allows for scanned data with different optical and mechanical centers and is thus able to provide effective operation for a greater variety of initial scan conditions.

<FIG> shows organ segmentations <NUM> in the de-scanned coordinate system and in the acquisition coordinate system. The left hand side images <NUM> show ultrasound images from an ultrasound volume with kidney segmentations in Cartesian coordinates. The right hand side images <NUM> show de-scanned ultrasound images from an ultrasound volume after the ultrasound volume has been converted into the Toroidal coordinate system with the corresponding de-scanned segmentations. On the right hand side images <NUM>, the horizontal axis represents the radius r and the vertical axis represents the poloidal angle θ of the Toroidal coordinate system. The de-scanned segmentation is generated for ground-truth segmentations and may be used for network training.

Large quantities of data are needed to train the segmentation algorithm. Data augmentation is a method used to create large quantities of data from an initial small size of data. For example, geometric transformations (e.g. rotations and translations) and color transformations may be used (e.g. grayscale transformation) can be used on a single ultrasound image/volume to create multiple training images.

The de-scanned ultrasound data in the Toroidal coordinate system allows for more intuitive and less error prone geometric transformations to create large databases for training.

Additionally, when training the segmentation algorithm, data biases must be avoided if possible (e.g. shape of data). In scan-converted ultrasound data, the cone shape containing the meaningful data is a source of data bias for the segmentation algorithm. The segmentation algorithm is also trained with the empty (meaningless) data which may cause the segmentation algorithm to ignore certain sections of the data in future segmentations or to look for certain patterns of empty data.

De-scanned ultrasound volumes (e.g. in the Toroidal coordinate system) can be fully described without the need of empty data for storage, thus the data bias due to the empty data is removed.

The algorithm may be any type of standard deep-learning approach such as U-Net segmentation. At the output of segmentation, the segmented ultrasound data may be converted back to the Cartesian coordinate system. The acquisition geometry parameters are re-used in the inverse transformation, in order to convert the segmentation mask from, for example, the Toroidal coordinate system back to the image Cartesian coordinate system.

<FIG> shows the results of organ segmentations <NUM> in the Toroidal coordinate system <NUM> and in the Cartesian coordinate system <NUM>. The same cases are shown, segmented without the coordinate transformation on the left hand side images <NUM>, and segmented with the coordinate transformation to the de-scanned domain on the right hand side images <NUM>. Using the de-scanned ultrasound images <NUM> provides more accurate results. The approximate ground truth <NUM> is shown in black circles and the segmentation algorithm result <NUM> are shown by white perimeters. The stripped arrows <NUM> show segmentation leaks in the results without the de-scan transformation in the scan-converted ultrasound images <NUM>. The corresponding arrows <NUM> are shown in the images with the de-scan transformation, showing that segmentation leaks are significantly reduced in the de-scanned ultrasound images <NUM>.

The method has also been applied in the domains of fetal abdomen and adult kidney segmentations with success. The results obtained with this method significantly out-perform approaches that do not used the transformation into the toroidal domain. The table below shows the results, showing a significant improvement over conventional segmentation (not using de-scan).

As discussed above, the system makes use of processor to perform the data processing. The processor can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

Claim 1:
A method for segmenting a target anatomy in a 3D ultrasound volume, the method comprising:
(<NUM>) obtaining a scan-converted ultrasound volume within a scan-converted space in a Cartesian coordinate system;
(<NUM>) transforming the scan-converted ultrasound volume to a de-scanned ultrasound volume within a de-scanned space in a Toroidal coordinate system, wherein the two centers of the Toroidal coordinate system equate to the optical center and the mechanical center of the ultrasound volume as obtained by an original acquisition procedure, respectively, thereby significantly reducing the proportion of empty data; and
(<NUM>) performing a segmentation of a target anatomy on the ultrasound volume in the de-scanned space to generate segmentation data.