Patent Description:
Embodiments of the disclosure relate to planar medical images and, more particularly, to using a system and method for identifying, delineating, and segmentation of structures in planar medical images.

In the diagnosis and treatment of certain human diseases by medical imaging, clinicians seek to make certain measurements and delineate the boundaries of certain structures (for example, cancer lesions, tumors, etc.) and normal organs in the body. The measurements are subject to the expert interpretation of a trained radiologist, but are suggested by patterns of contrast contained in planar medical images. Oncologists seek more quantitative measurements such as faster review times, better capturing of volumes, masses, etc. from the radiologists. Lesion delineation can be a source of uncertainly, since typically, the lesion delineation process involves an experienced physician, interpreting, and manually contouring computed tomography (CT) alone or combined with position emission tomography (PET) imaging, on a slice-by-slice basis. As a result, advanced quantitative metrics and automation are needed for the trained professionals reviewing the medical images. Current tools are too slow for the radiologists to provide these metrics for the oncologists on all patients. <CIT> discloses how spatial segmentation of lymph nodes in a <NUM>-D medical image may be automatically determined, based on a set of inputs provided by a user which define a low number of initial conditions for segmentation. Patent publication <CIT> discloses a method and a system for image processing. The technique includes performing a post-processing of a result of a segmentation and managing storage of data. <NPL> discloses the segmentation of brain tumors. <CIT> discloses a method for lesion segmentation in 3D digital images, includes selecting a 2D region of interest from a 3D image.

The disclosure provided herein allows a clinician to interactively define and visualize certain artifacts such as segmentation, the long axis, and the short axis. The segmentation of planar medical images is important in medical diagnosis because segmentation provides information associated to anatomical structures as well as potential abnormal tissues necessary to treatment planning and patient follow-up.

One embodiment provides a method as defined in claim <NUM>.

Another embodiment provides a system as defined in claim <NUM>.

Yet another embodiment provides a non-transitory machine-readable storage medium as defined in claim <NUM>.

For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present disclosure.

The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the accompanying drawings.

Radiological scans generate a stack of image slices considered to form an image volume. A medical image dataset could be a single volume, such as a Computed Tomography (CT) scan. Or it could be a set of several volumes, such as a Magnetic Resonance (MR) scan. Multi-spectral MR scans the patient several times using different scanning protocols which highlight various type of tissue types. For example, edema can be distinguished by its darker appearance on a T1-weigthed image and brighter appearance on a T2-weighted image. A medical image dataset could also be a fusion of multiple imaging modalities, such as a CT scan with Positron Emission Tomography (PET). By aligning multi-modal image volumes, the functional information conveyed by the PET can be understood in the context of the anatomic information conveyed by the CT. Digital Imaging and Communications in Medicine (DICOM) is a standard for handling, storing, printing, and transmitting information in medical imaging. DICOM includes a file format definition and a network communications protocol.

<FIG> is an example planar medical image <NUM> of a lesion <NUM> in a lung <NUM>. The lesion <NUM> includes a long axis <NUM>, a short axis <NUM>, and a two dimensional (2D) contour <NUM>. The 2D contour <NUM> delineates the boundary of the lesion <NUM>.

Radiologists traditionally read medical image datasets on a slice-by-slice basis. Not long ago, radiologists viewed images on films on a light box. Image assessments were qualitative and any quantitative measurements of structures had to be simple such as length from placing a ruler on the image. Modern technology has digitized the process and now, doctors can read images displayed on computers, also known as a Picture Archiving and Communication Systems (PACS). The PACS systems have the capability of showing the images in one of three planes, axial, sagittal, or coronal. Multi Plane Reformat (MPR) can present simultaneous image views of all three planes. However, the workflow still follows the traditional process of reading it slice-by-slice instead of treating it as a 3D volume. Therefore any quantitative measurements, even when using a PACS workstation, are still confined to single slices.

Standard measurements of structures in planar medical images include the long axis and the short axis. The long axis is the longest one dimensional line segment of a structure in one planar medical image in which the structure is the longest. The long axis does not refer to the true longest 3D segment, but rather the longest axis identified within the plane of a planar medical image. The short axis is defined as the diameter that is perpendicular to the long axis on the same slice as the long axis. The short axis is the longest one dimensional line segment in same plane of the planar medical image containing the long axis. The short axis can have a length less than or equal to the long axis depending upon the circularity of the structure.

The long and short axis measurements serve as the building blocks for computing standard metrics for tracking the progress of a disease. For example, one commonly used standard is Response Evaluation Criteria in Solid Tumors (RECIST) that is used to monitor the response to therapy. Another example is Lung CT Screening Reporting and Data (Lung-RADS), which is a quantitative score that is used to measure the extent and progression of lung lesions. Both these measurements rely upon the long axis and short axis calculations.

An emerging field of interest to the medical imaging community is Radiomics. Radiomics is the discovery of correlation between lesion phenotypes and quantifiable image metrics. On a given lesion, hundreds of quantitative measurements are possible such as intensity, shape, and texture. These measurements can be combined to yield a Radiomics "signature," i.e., an imaging biomarker.

The fundamental requirement for extracting the image metrics is the ability to perform a volumetric segmentation of the image set. One method of calculating the volume is for the radiologist to manually delineate the boundaries of the lesion on each slice where it appears. This is a time-consuming task that no radiologist can fit into their fast paced workflow. What is needed is a method for computing volumetric segmentation that is as fast and easy for a radiologist as drawing a long axis and the short axis. A study of semi-automated segmentation using state-of-the-art research software (described by <NPL>, show that it still takes approximately <NUM> minutes to produce the volumetric segmentation of a lung lesion.

In some embodiments, systems and methods described herein provide a semi-automated segmentation process where the user's interaction closely mimics the process of drawing the long axis. Dragging the end point of an axis directly influences the 3D volume segmentation. The work flow provided herein is not only fast, but also very familiar to the radiologists as they are very accustomed to measuring diameters. The process ensures flexibility. The system and methods described herein produce volumetric segmentation of a lung lesion in approximately one minute or less.

<FIG> is a diagram an example system <NUM> for determining volumetric segmentation of structures in planar medical images. The system <NUM> may combine hardware, software, and firmware, to implement methods described herein. In the embodiment illustrated in <FIG>, the system <NUM> includes an electronic processor <NUM>, a memory <NUM>, a data storage <NUM>, a display <NUM>, a user interface <NUM>, a communication interface <NUM>, and a bus <NUM>. In some embodiments, the system <NUM> includes fewer or additional components in configurations different from the one illustrated in <FIG>. For example, in some embodiments, the system <NUM> includes multiple electronic processors, displays, or combinations thereof.

The electronic processor <NUM> may include at least one processor or microprocessor that interprets and executes a set of instructions stored, for example, in the memory <NUM>.

The memory <NUM> may include volatile memory elements (for example, random access memory (RAM)), nonvolatile memory elements (for example, ROM), and combinations thereof. The memory <NUM> may have a distributed architecture, where various components are situated remotely from one another, but may be accessed by the electronic processor <NUM>. The memory <NUM> may include an operating system <NUM> and software programs <NUM>. The software programs <NUM> may be configured to implement the methods described herein. The memory <NUM> may also store temporary variables or other intermediate information used during the execution of instructions by the electronic processor <NUM>.

The data storage <NUM> may include a non-transitory, tangible, machine-readable storage medium that stores machine-readable code or instructions. In one example, the data storage <NUM> stores a set of instructions detailing the methods described herein that when executed by the electronic processor <NUM> cause the electronic processor <NUM> to perform the methods. The data storage <NUM> may also include a database or a database interface for storing an application module. In some embodiments, the data storage <NUM> is located external to the system <NUM>.

The display <NUM> is a suitable display, for example, a liquid crystal display (LCD) touch screen, or an organic light-emitting diode (OLED) touch screen. In some embodiments, the system <NUM> implements a graphical user interface (GUI) (for example, generated by the electronic processor <NUM>, using the operating system <NUM> stored in the memory <NUM>, and presented on the display <NUM>), that enables a user to interact with the system <NUM>.

The user interface <NUM> can include any combination of digital and analog input devices required to achieve a desired level of control for the system <NUM>. In some embodiments, the user interface <NUM> includes one or more electronic devices for receiving user input such as a keyboard, a mouse, a trackpad, and the like. Alternatively or in addition, the user interface <NUM> includes a touch sensitive interface. For example, in some embodiments, the display <NUM> is a touch-screen display that receives user input using detected physical contact (for example, detected capacitance or resistance). Based on user input, the display <NUM> outputs signals to the electronic processor <NUM> which indicate positions on the display <NUM> currently being selected by physical contact. In some embodiments, the user interface <NUM> is located external to the system <NUM>.

The communication interface <NUM> provides the system <NUM> a communication gateway with an external network (for example, a wireless network, the internet, etc.). The communication interface <NUM> may include, for example, an Ethernet card or adapter or a wireless local area network (WLAN) card or adapter (for example, Institute of Electrical and Electronic Engineers standard <NUM> a/b/g/n). The communication interface <NUM> may include address, control, and/or data connections to enable appropriate communications on the external network.

The bus <NUM>, or other component interconnection, may permit communication among the components of the system <NUM>. The bus <NUM> may be, for example, one or more buses or other wired or wireless connections, as is known in the art. The bus <NUM> may have additional elements, which are omitted for simplicity, such as controllers, buffers (for example, caches), drivers, repeaters and receivers, or other similar components, to enable communications. The bus <NUM> may also include address, control, data connections, or a combination of the foregoing to enable appropriate communications among the aforementioned components.

<FIG> illustrates an example planar medical image <NUM> showing a structure <NUM>. <FIG> includes an inclusion region <NUM> (shown in <FIG> at as an ellipse). The inclusion region <NUM> is positioned completely within the structure <NUM> such that the inclusion region <NUM> includes at least a portion of the structure <NUM>. <FIG> also includes a containment region <NUM> (shown in <FIG> as a circle). The containment region <NUM> is positioned such that it includes the entirety of the structure <NUM>. <FIG> also includes a background region <NUM> (shown in <FIG> as a circle). The background region <NUM> is positioned such that it does not include any portion of the structure <NUM>.

As will be described in more detail below, the system <NUM> initially determines the inclusion region <NUM>, the containment region <NUM>, and the background region <NUM>. In some embodiments, the sizes and positions of the inclusion region <NUM>, the containment region <NUM>, and the background region <NUM> can later be manipulated by the user. As will be described in more detail below, the system <NUM> uses the inclusion region <NUM>, the containment region <NUM>, and the background region <NUM> perform a volumetric segmentation of the structure <NUM>. In other words, the system <NUM> uses the inclusion region <NUM>, the containment region <NUM>, and the background region <NUM> to determine a three dimensional (3D) contour <NUM> of the structure <NUM>. The system <NUM> also determines a long axis <NUM> of the structure <NUM> and a short axis <NUM> of the structure <NUM> using the inclusion region <NUM>, the containment region <NUM>, the background region <NUM>, the 3D contour <NUM>, or a combination thereof.

<FIG> illustrates an example method <NUM> for performing volumetric segmentation of a structure in a plurality of planar medical images. The method <NUM> is described with respect to <FIG> and <FIG>. The method <NUM> is described as being performed by the system <NUM> and, in particular, the electronic processor <NUM>. However, it should be understood that in some embodiments, portions of the method <NUM> may be performed by other devices included in the system <NUM>.

At block <NUM>, the electronic processor <NUM> receives a plurality a planar medical images. The plurality of planar medical images forms a 3D volume that includes the structure <NUM>. The plurality of planar medical image includes, for example, one or more computed tomography (CT) images, positron emission tomography (PET) images, magnetic resonance imaging (MRI) images, X-ray images, or a combination thereof. In some embodiments, the system <NUM> imports the plurality of planar medical images from a computer network (for example, a server) or a file system. In some embodiments, the imported medical images includes one or more of a set of CT, PET, multi-spectral MRI images all of which define the Image Pixel and Image Plane module of the DICOM PS <NUM> specification and are assembled to create a 3D rectilinear image volume.

At block <NUM>, the electronic processor <NUM> displays one of the plurality of planar medical images (for example, a first planar medical image) on the display <NUM>.

At block <NUM>, the electronic processor <NUM> receives user input from the user interface <NUM> indicating a line segment in the first planar medical image. In some embodiments, the line segment is a straight line. In alternate embodiments, the line segment is a non-straight line. The line segment is an approximation of the long axis by the user that is drawn through a stroke gesture across the first planar medical image with a pointing device (for example, a cursor). In some embodiments, the user interface <NUM> detects a user selection at one location in the first planar medical image (for example, a first location), and subsequently detects a user deselection at a different location in the first planar medical image (for example, a second location). The electronic processor <NUM> then determines the line segment as a line between the first location and the second location.

In the some embodiments, a user selection includes the user interface <NUM> receiving a selection signal generated by a user input device. For example, the user interface <NUM> receives a selection signal generated by a mouse responsive to a user clicking a button on the mouse or generated by a keyboard responsive to a user clicking a key on the keyboard. In such embodiments, the electronic processor <NUM> may store the location of a cursor on the display <NUM> as the first location when the selection signal is received. Alternatively or in addition, the user selection includes physical contact with the display <NUM>. For example, the user selection includes a user touching the display <NUM> with their finger. In such embodiments, the electronic processor <NUM> may store the location of the initial physical contact on the display <NUM> as the first location.

In the some embodiments, the user deselection includes a termination of the selection signal generated by a user input device. For example, the user deselection may include the user releasing a button on a mouse or releasing a key on a keyboard. In such embodiments, the electronic processor <NUM> may store the location of the cursor on the display <NUM> as the second location when the termination of the selection signal is detected. Alternatively or in addition, the user deselection includes the termination of physical contact with the display <NUM>. For example, the user deselection includes the user removing their finger from the display <NUM>. In such embodiments, the electronic processor <NUM> may store the location of the latest physical contact on the display <NUM> as the second location when the user removes their finger.

Returning to <FIG>, at block <NUM>, the electronic processor <NUM> determines an inclusion region <NUM> of the 3D volume using the line segment. The inclusion region <NUM> includes only the structure <NUM> and, in the most usual cases, only a portion of the structure <NUM>. The line segment describes the size of the structure <NUM> along one dimension and the initial layout of the inclusion region <NUM> may be an oblong region oriented along the line segment. In some embodiments, the electronic processor <NUM> determines the size of the structure <NUM> along other dimensions by analyzing orthogonal scout planes given statistical sampling along the line segment. For example, the electronic processor <NUM> may use probability distributions that are modeled parametrically (for example, as Gaussian Mixture Models (GMMs)) based on the line segment while determining the inclusion region <NUM>.

At block <NUM>, the electronic processor <NUM> determines a containment region <NUM> of the 3D volume using the line segment. The containment region <NUM> includes all of the structure <NUM>. The initial layout of the containment region <NUM> may be more spherical than the inclusion region <NUM>. The containment region <NUM> may be automatically constrained to circumscribe the inclusion region <NUM>. In some embodiments, the electronic processor <NUM> determines three 2D contours of the structure using the line segment. Each of the three 2D contours is in a different plane of the 3D volume. For example, the electronic processor <NUM> determines a first 2D contour for an axial plane of the 3D volume, a second 2D contour for a sagittal plane of the 3D volume, and a third 2D contour for a coronal plane of the 3D volume. Next, the electronic processor <NUM> determines the containment region <NUM> as an ellipsoid in the 3D volume that completely encompasses the three 2D contours. The inclusion region <NUM> may be reshaped such that each of its three intersections with the three planes is inscribed in the 2D contour associated with that plane. In some embodiments, the electronic processor <NUM> determines the 2D contours using probability distributions (for example, Gaussian) for image brightness derived from statistical sampling along the line segment. Alternatively or in addition, the electronic processor <NUM> determines the 2D contours using samples of boundary profiles that may be determined based on the end points of the line segment.

At block <NUM>, the electronic processor <NUM> determines a background region <NUM> of the 3D volume using the line segment. The background region <NUM> excludes the structure. The initial layout of the background region <NUM> may be to a location outside the containment region <NUM> that has low probability of belonging to the structure <NUM>. In some embodiments, the electronic processor <NUM> determines the background region <NUM> by searching the vicinity outside the containment region <NUM>, and within the body outline, while maximizing the Mahalanobis distance from the inclusion region <NUM>. Intentionally seeking statistical separation between the background region <NUM> and the inclusion region <NUM> is more effective than placing the background region <NUM> based on spatial criteria alone.

In some embodiments, the electronic processor <NUM> determines more than one background region of the 3D volume. Multiple background regions allow sampling of disparate objects. For example, the electronic processor <NUM> may determine initial layouts of two background regions by selecting locations with intensities both above and below those of the inclusion region <NUM>.

At block <NUM>, the electronic processor <NUM> determines a 3D contour <NUM> of the structure <NUM> using the inclusion region <NUM>, the containment region <NUM>, and the background region <NUM>. The 3D volume includes a plurality of voxels. Each voxel is a unit of graphic information that defines a point in the 3D volume. As will be described in more detail below, the electronic processor <NUM> classifies voxels in the plurality of voxels as belonging to either a foreground class or a background class.

The voxels in the 3D volume that make up the structure <NUM> belong to the foreground class. The other voxels in the 3D volume belong to the background class. The electronic processor <NUM> classifies voxels (for example, a first set of voxels) located within the inclusion region <NUM> as belonging to the foreground class. The electronic processor <NUM> classifies voxels (for example, a second set of voxels) located within the background region <NUM> as belonging to the background class.

To classify voxels (for example, a third set of voxels) located within the containment region <NUM> and outside the inclusion region <NUM>, the electronic processor <NUM> statistically samples all (or a portion) of the plurality of voxels. The voxels located within the inclusion region <NUM> (i.e., the first set of voxels) statistically typify the foreground class. The voxels located within the background region <NUM> (i.e., the second set of voxels) statistically typify the background class. In some embodiments, the electronic processor <NUM> statistically samples the first set of voxels and the second set of voxels to classify the third set of voxels. Alternatively or in addition, the electronic processor <NUM> statistically samples the voxels located within the containment region <NUM> and the voxels located within the background region <NUM> to classify the third set of voxels.

In some embodiments, the electronic processor <NUM> classifies the third set of voxels using Bayesian classification wherein prior probabilities are spatially varying and derived from region boundaries and may be a function of the distance from the inclusion region <NUM> and the containment region <NUM>. In some embodiments, the electronic processor <NUM> uses the voxels located within the inclusion region <NUM> and the voxels located within the background region <NUM> to perform Parzen windowing to estimate the likelihoods for Bayesian classification. Conditional densities (for example, likelihoods) may be derived from a function of the histogram of the voxels in each region. Noise and artifacts in the planar medical images vary greatly by dose level and choice of reconstruction kernel. Thus, in some embodiments, the electronic processor <NUM> augments Bayesian classification with Markov Random Fields and with at least three iterations of a mean-field approximation. In order to achieve a smoothly varying structure, the electronic processor <NUM> may perform regularization. Examples of regularization processing include connected component analysis (for example, remove islands and fill holes), morphological operations (for example, dilation, erosion, opening, and closing), active contours (for example, snakes and level sets), and fitting a 3D mesh to the voxel classification by adapting vertices connected by virtual springs to their neighbors to provide a regularizing force that smooths the surface. Super-sampling the image voxel data is another way to produce smoother results, especially since medical image voxels tend to have anisotropic shape.

The voxels assigned to the foreground class define the boundary of the structure <NUM>. After classifying each of the plurality of voxels as belonging to the foreground class or the background class, the electronic processor <NUM> determines a border in the 3D volume between the voxels belonging to the foreground class and the voxels belonging to the background class. The electronic processor <NUM> may determine a 3D contour <NUM> of the structure <NUM> based on this border. For example, the electronic processor <NUM> may determine the 3D contour <NUM> of the structure <NUM> to be the border between the voxels belonging to the foreground class and the voxels belonging to the background class.

As described herein, the structure <NUM> is wholly contained with the containment region <NUM>. As such, some (or all) of the voxels located outside the containment region <NUM> may not be relevant to defining the 3D contour <NUM> of the structure <NUM>. Thus, in some embodiments, the electronic processor <NUM> does not classify every voxel of the plurality of voxels as belonging to the foreground class or the background class when determining the 3D contour <NUM> of the structure <NUM>. For example, the electronic processor <NUM> may only classify voxels located within the inclusion region <NUM> (i.e., the first set of voxels), voxels located within the background region <NUM> (i.e., the second set of voxels), voxels located within the containment region <NUM> and outside the inclusion region <NUM> (i.e., the third set of voxels), or a combination thereof. In general, the 3D volume can include a large quantity of voxels (for example, about <NUM> million voxels). Classifying every single voxel in the 3D volume requires a lot of processor power and processor time to complete. By not classifying every voxel in the 3D volume, the electronic processor <NUM> is able to determine the 3D contour <NUM> of the structure <NUM> much faster.

As described herein, the containment region <NUM> is used to narrow the region of interest in the 3D volume. However, the containment region <NUM> is not the same as a bounding box. In general, bounding boxes are manually drawn by users. As such, bounding boxes are much larger than necessary and have no orientation. On the other hand, as described herein, the containment region <NUM> is determined based on 2D contours of the structure <NUM>. As such, the containment region <NUM> is shaped to more accurately represent the region of interest than a bounding box.

Returning to <FIG>, at block <NUM>, the electronic processor <NUM> determines a long axis <NUM> of the structure <NUM> using the 3D contour <NUM> of the structure <NUM>. In some embodiments, the electronic processor <NUM> determines the long axis <NUM> of the structure <NUM> for the plurality of planar medical images. For example, the electronic processor <NUM> determines the long axis <NUM> of the structure <NUM> as the longest one dimensional line segment that is present in all of the plurality of planar medical images. In other words, the electronic processor <NUM> identifies the planar medical image with the longest one dimensional line segment and sets the long axis <NUM> as this line segment. Alternatively or in addition, the electronic processor <NUM> determines the long axis <NUM> of the structure <NUM> for a specific planar medical image. For example, the electronic processor <NUM> determines the long axis <NUM> of the structure <NUM> as the longest one dimensional line segment that is present in a single, specific planar medical image.

At block <NUM>, the electronic processor <NUM> outputs a dimension of the long axis <NUM> of the structure <NUM>. A dimension (for example, a first dimension) of the long axis <NUM> of the structure <NUM> may include a measurement of the length of the long axis <NUM>, a position of the long axis <NUM>, a specific planar medical image that includes the long axis <NUM>, or a combination (or derivative) thereof. In some embodiments, the electronic processor <NUM> outputs the dimension of the long axis <NUM> by transmitting data indicating the dimension via the communication interface <NUM>. For example, the electronic processor <NUM> may transmit data via the communication interface <NUM> indicating the length of the long axis <NUM>. Alternatively or in addition, the electronic processor <NUM> outputs the dimension of the long axis <NUM> by storing data indicating the dimension. For example, the electronic processor <NUM> may store data indicating the length of the long axis <NUM> in the memory <NUM>, the data storage <NUM>, or both. Alternatively or in addition, the electronic processor <NUM> outputs the dimension of the long axis <NUM> by displaying the dimension on the display <NUM>. For example, the electronic processor <NUM> may cause the display <NUM> to display a visual indicator of the long axis <NUM> on a planar medical image (as illustrated in <FIG>). As a further example, the electronic processor <NUM> may cause the display <NUM> to display the length of the long axis <NUM>.

In some embodiments, the electronic processor <NUM> determines a short axis <NUM> of the structure <NUM> using the 3D contour <NUM> of the structure <NUM>. In some embodiments, the electronic processor <NUM> determines the short axis <NUM> of the structure <NUM> as the longest one dimensional line segment perpendicular to the long axis <NUM> in the same planar medical image as the long axis <NUM>. In some embodiments, the electronic processor <NUM> outputs a dimension (for example, a second dimension) of the short axis <NUM> of the structure <NUM>. For example, the electronic processor <NUM> may cause the display <NUM> to display a visual indicator of the short axis <NUM> on a planar medical image (as illustrated in <FIG>).

In some embodiments, the electronic processor <NUM> causes the display <NUM> to display a visual indication of the 3D contour <NUM>. For example, display <NUM> may display a boundary delineating the 3D contour <NUM> of the structure <NUM> on a planer medical image (as illustrated in <FIG>).

In some situations, a structure of interest (for example, a tumor or lesion) may be located near a different structure. For example, <FIG> is planar medical image <NUM> of a blood vessel <NUM> positioned near a nodule <NUM>. The blood vessel <NUM> is positioned close enough to the nodule <NUM> such that the electronic processor <NUM> may inadvertently determine the blood vessel <NUM> to be part of the nodule <NUM> when performing volumetric segmentation of the nodule <NUM>. Thus, in some embodiments, the electronic processor <NUM> determines an avoidance region <NUM> to separate nearby structures that are not part of the structure of interest. <FIG> includes an example avoidance region <NUM> around the blood vessel <NUM>. The voxels (for example, a fourth set of voxels) located within the avoidance region <NUM> are not part of the structure of interest. Thus, the electronic processor <NUM> classifies the voxels located within the avoidance region <NUM> as belonging to the background class. The values (for example, the intensity values) of the voxels located within the avoidance region <NUM> may be similar to values of the voxels of the structure of interest. This similarity can cause the voxels located within the avoidance region <NUM> to negatively impact overall statistical sampling. Thus, in some embodiments, the electronic processor <NUM> classifies the voxels located within the containment region and outside the inclusion region as belonging to either the foreground class or the background class without statistically sampling the voxels located within the avoidance region <NUM>.

As described herein, in some embodiments, the line segment received via the user interface <NUM> can be generated by a user dragging a cursor across a structure. <FIG> are an example series of screen shots of the display <NUM> illustrating a user inputting a line segment <NUM> across a structure <NUM> in a planar medical image. The descriptions of <FIG> included herein are described as being performed by a user with a mouse. These descriptions are not limiting and merely provide one example implementation. First, the user positions a cursor <NUM> at a first location <NUM> in the planar medical image on the border of the structure <NUM>, as illustrated in <FIG>. Next, the user presses and holds a button of the mouse while dragging the cursor <NUM> across the structure <NUM>, as illustrated in <FIG>. As the cursor <NUM> is being dragged, the display <NUM> displays the line segment <NUM> as a line between the first location <NUM> and the current location of the cursor <NUM>. The user moves the cursor <NUM> to a second location <NUM> in the planar medical image on the opposite border of the structure <NUM>, as illustrated in <FIG>. The user releases the button on the mouse to complete the line segment <NUM>, as illustrated in <FIG>.

As described herein, the electronic processor <NUM> determines 2D contours of the structure using a line segment received via user input received by the user interface <NUM>. Further, the electronic processor <NUM> determines the inclusion region, the containment region, and the background region based in part on the 2D contours of the structure. As such, the accuracy of the volumetric segmentation techniques described herein improve when user input inputs a line segment that closely represents the long axis. In order to improve the quality line segments input by the user, in some embodiments, the system <NUM> determines and displays the 2D contours in real-time as the user is drawing the line segment.

<FIG> are an example series of screen shots of the display <NUM> illustrating real-time determination and display of 2D contours while a user inputs a line segment <NUM> across a structure <NUM> in a planar medical image. The descriptions of <FIG> included herein are described as being performed by a user with a mouse. These descriptions are not limiting and merely provide one example implementation. First, the user positions a cursor <NUM> at a first location <NUM> in the planar medical image on the border of the structure <NUM>, as illustrated in <FIG>. Next, the user presses and holds a button of the mouse while dragging the cursor <NUM> across the structure <NUM>, as illustrated in <FIG>. As the cursor <NUM> is dragged across the structure <NUM>, the electronic processor <NUM> determines a 2D contour <NUM> of the structure <NUM> based on the line segment <NUM> between the first location <NUM> and the current location of the cursor <NUM> (for example, a third location). The display <NUM> displays visual indicators of the line segment <NUM> and the 2D contour <NUM> on the planar medical image, as illustrated in <FIG>. When the cursor <NUM> is positioned at a location <NUM> on the opposite border of the structure <NUM>, the 2D contour <NUM> substantially encompasses the entire structure <NUM>, as illustrated in <FIG>. By determining and displaying 2D contours in real-time as the line segment is being drawn, as illustrated in <FIG>, a user is able to provide a line segment that more closely represents the long axis.

In some situations, the 2D contours determined by the electronic processor <NUM> using the line segment may not contain the entire structure. The line segment (for example, a first line segment) may not be a good approximation of the long axis, and thus the 2D contours may not include a portion of the structure. In some embodiments, the electronic processor <NUM> adjusts the 2D contours based on additional user input. For example, in some embodiments, the electronic processor <NUM> receives additional user input (for example, a second user input) from the user interface <NUM> indicating a second line segment in a planar medical image. The first line segment and the second line segment may be from the same planar medical image. The second line segment includes, for example, the portion of the structure that is not included in the 2D contour or a second approximation of the long axis. The electronic processor <NUM> adjusts the 2D contour using the first line segment and the second line segment. In some such embodiments, the electronic processor <NUM> may determine the inclusion region, the containment region, and the background region using the first line segment, the second line segment, the adjusted 2D contour, or a combination thereof. For example, the inclusion region may embody the adjusted 2D contour. For example, the containment region may be shaped such that its intersection with the planar medical image conforms to the adjusted 2D contour. For example, the background region may be placed by performing statistical clustering of the voxels in the vicinity of the adjusted 2D contour in order to find the clusters that best bracket the probability distribution of the inclusion region. In some embodiments, the electronic processor <NUM> adds an avoidance region around the adjusted 2D contour. In general, the containment region and the avoidance region remove voxels from consideration during classification that could be distracting because of these voxel's similarity and brightness to the voxels of the target structure. Since it may be difficult to discern the voxels located outside the containment region and the voxels located within the avoidance region by brightness alone, these voxels can be described by spatial position.

In some embodiments, the electronic processor <NUM> may receive additional user input (for example, a second user input) from the user interface <NUM> indicating an edit to the 2D contour. Edits to the 2D contour may include, for example, the user dragging a portion of the 2D contour with a cursor or the user drawing a corrected 2D contour with a cursor. In some such embodiments, the electronic processor <NUM> may determine the inclusion region, the containment region, and the background region using the first line segment, the edit to the 2D contour, or a combination thereof.

As described herein, in some embodiments, the electronic processor <NUM> determines the inclusion region, the containment region, and the background region using a single line segment in a single planar medical image. In alternate embodiments, the electronic processor <NUM> determines the inclusion region, the containment region, and the background region using multiple line segments in different planar medical images. For example, the electronic processor <NUM> may display a first planar medical image and a second planar medical image on the display <NUM>. The second planar medical image is an image slice along a different plane of the 3D volume than the first planar medical image. For example, the first planar medical image may be an image slice along an axial plane of the 3D volume, and second planar medical image may be an image slice along a sagittal plane of the 3D volume. The electronic processor <NUM> receives a first user input from the user interface <NUM> indicating a first line segment in the first planar medical image. The electronic processor <NUM> also receives a second user input from the user interface <NUM> indicating a second line segment in the second planar medical image. The electronic processor <NUM> determines the inclusion region, the containment region, and the background region using the first line segment and the second line segment.

In some embodiments, after the 3D contour (for example, a first 3D contour) is determined, the electronic processor <NUM> may receive additional user input (for example, a second user input) from the user interface <NUM> indicating an edit to the 3D contour. Edits to the 3D contour may include, for example, the user dragging a portion of the 3D contour with a cursor, the user drawing a corrected 3D contour with a cursor, or alterations to the containment region, the inclusion region, the background region, the long axis, and the short axis. The user may alter one of the containment, inclusion, and background regions, for example, by dragging a portion of a region with a cursor, or by drawing a corrected portion of a region with a cursor. The user may alter the long axis or the short axis, for example, by dragging an endpoint of an axis with a cursor. The electronic processor <NUM> determines a new inclusion region (for example, a second inclusion region) and a new containment region (for example, a second containment region) using the line segment and the edit to the 3D contour. Using the second inclusion region, the second containment region, and the background region, the electronic processor <NUM> classify the voxels located within the second containment region as belonging to either the foreground class or the background class. Next, the electronic processor <NUM> determines a new 3D contour of the structure (for example, a second 3D contour of the structure) based on a border in the 3D volume between the voxels belonging to the foreground class and the voxels belonging to the background class.

<FIG> illustrates an example method <NUM> for segmentation of a foreground structure in a plurality of planar medical images. The method <NUM> is described as being performed by the system <NUM> and, in particular, the electronic processor <NUM>. However, it should be understood that in some embodiments, portions of the method <NUM> may be performed by other devices included in the system <NUM>.

At block <NUM>, the electronic processor <NUM> receives a plurality of planar medical images. The plurality of planar medical images forms a 3D volume that includes a structure. The plurality of planar medical image includes, for example, one or more computed tomography (CT) images, positron emission tomography (PET) images, magnetic resonance imaging (MRI) images, X-ray images, or a combination thereof. In some embodiments, the system <NUM> imports the plurality of planar medical images from a computer network (for example, a server) or a file system. In some embodiments, the imported medical images includes one or more of a set of CT, PET, multi-spectral MRI images all of which define the Image Pixel and Image Plane module of the DICOM PS <NUM> specification and are assembled to create a 3D rectilinear image volume.

At block <NUM>, segmentation is initialized by a stroke in any one plane, providing an approximate diameter of the region of interest. In some embodiments, a multi-planar reformatting is performed that allows presentation of three image slice planes along the axial, sagittal, and coronal planes. The user interface <NUM> lets the user scroll to use any of the planar medical images on any of these three planes. The segmentation process can be initiated by either one stroke on any of these planes or by two different strokes, each on a different plane. These strokes provide image pixel brightness and spatial seed points for the segmentation process.

At block <NUM>, the electronic processor <NUM> estimates the extent of the foreground structure along two other dimensions. In some embodiments, the electronic processor <NUM> performs a quick segmentation based on statistics derived from the strokes provided at block <NUM>. This can constitute the approximation of the extent along all dimensions.

At block <NUM>, the electronic processor <NUM> determines an inclusion region, a containment region, and one or more background regions. The inclusion region is smaller than the approximate segmentation and is contained wholly inside it. The containment region is larger than the approximate segmentation region and wholly contains it. The background regions are positioned outside the containment region. The electronic processor <NUM> searches the plurality of planar medical images prior to the determination of the background regions. In some embodiments, the electronic processor <NUM> determines a first background region in tissue that is darker, and a second background region in tissue that is brighter. Alternatively, the electronic processor <NUM> determines background regions in statistically distinct places. In some embodiments, the inclusion region and the containment region are ellipsoidal shapes and the background regions are spherical shapes.

At block <NUM>, the electronic processor <NUM> computes volumetric segmentation. In some embodiments, the segmentation process performs Bayesian classification wherein prior probabilities are spatially varying and derived from region boundaries and may be a function of the distance from the inclusion region and the containment region. The conditional densities (for example, the likelihoods) may be derived from sampling the voxels within the background and inclusion regions in order to perform Parzen window density estimation. In some embodiments, the electronic processor <NUM> analyzes multi-spectral images using multi-dimensional Bayesian classification such that there are two categories (i.e., background and foreground), and there are as many dimensions of feature space as there are spectrums. Each plurality of planar medical images can be sampled inside the inclusion region and the background region. One classification, given all series as multi-variate input, can be performed inside the containment region. In some embodiments, the electronic processor <NUM> computes 2D contours in real time and computes 3D contours in the background. In such embodiments, the 3D contours are displayed on the display <NUM> only after the electronic processor <NUM> finishes calculating the 3D contours. <FIG> is an example screen shot of the display <NUM> illustrating an example multi-planar view of three image slice planes along the axial, sagittal, and coronal planes with the 2D contours <NUM> displayed. <FIG> is an example screen shot of the display <NUM> illustrating an example multi-planar view of three image slice planes with the calculated 3D contours <NUM> displayed.

At block <NUM>, the electronic processor <NUM> determines the long axis and the short axis. In some embodiments, the electronic processor <NUM> identifies the planar medical image that contains the long axis and displays that planar medical image to the user on the display <NUM>. Additionally, as the user scrolls through the planar medical images, the electronic processor <NUM> updates and displays the long axis and the short axis on any of these planar medical images.

At block <NUM>, the volumetric segmentation is edited by dragging an endpoint of the long axis or the short axis. For example, the user can select the end point of the long axis or the short axis on the display <NUM> and drag the end point to edit the volumetric segmentation. The act of dragging the end points creates a deformation of the containment region and the inclusion region, and thus, alters the volumetric segmentation. The volumetric segmentation can also by edited by the user dragging a side of a bounding box that, in some embodiments, is displayed on the display <NUM> in a planar medical image. <FIG> is an example screen shot of the display <NUM> illustrating an example multi-planar view of three image slice planes with a 3D contour <NUM>, a long axis <NUM>, a short axis <NUM>, and a bounding box <NUM> displayed. The bounding box contains the extent of the segmentation of the foreground structure. Altering the size of the bounding box contributes to the deformation of the containment region, thus affecting the segmentation. In some embodiments, movement of the bounding box is constrained to not impinge on the inclusion region.

At block <NUM>, the volumetric segmentation is confirmed, for example, by the user. The user evaluates the 3D contours of the segmentation. If the user agrees, the user is prompted to confirm it and the system <NUM> stores the new segmentation, for example, in the data storage <NUM>. One reason to store the volumetric segmentation is to compute various quantitative metrics to be used for Radiomics. If the user disagrees, the system <NUM> stores the long axis measurement and/or the short axis measurement, for example, in the data storage <NUM>.

In some embodiments, normal organs can be segmented reliably and automatically without any intervention. They can be pre-computed before the user starts interacting with the system <NUM> to segment the lesions. The segmentation methods described herein can be organ specific and organ-aware. In some embodiments, the electronic processor <NUM> detects organs in planar medical images. In other words, the electronic processor <NUM> can determine the organ in which a lesion lies. In some embodiments, the electronic processor <NUM> accounts for known lesions during segmentation. For example, lymph nodes are known to be spherical in shape and the segmentation can be constrained to be rounded. RECIST guidelines treat the short axis differently for lymph nodes, so automatic organ identification saves time and manual data entry for the user. In some embodiments, the electronic processor <NUM> excludes the lung vessels from lung lesions. For example, vessels can be identified by their brightness, connectedness, and tubular shape, as measured by Eigen-analysis. In some embodiments, the lung vessel segmentation will also be included in normal organ segmentation. Inflammation and bullae are additional examples of lung-specific structures than can be considered for removal from the structure. Inflammation, often precipitated by the lesion, may be excluded from the lesion volume. Inflammation may be identified by brightness, shape, and position as emanations relative to solid mass. Lung nodules may contain air bubbles, involve bronchioles, or grow adjacent to bullae. Radiologists may decide whether to include each air pocket on a case-by-case basis. For example, bubbles whose outlines correspond significantly with the outline of the lesion could be included, whereas bubbles whose outlines show little overlap could be excluded.

In some embodiments, the methods provided herein are performed by a software executed by a server, and a user may access and interact with the software application using a computing device. Also, in some embodiments, functionality provided by the software application may be distributed between a software application executed by a local device and a software application executed by another electronic process or device (for example, a server) external to the local device.

However, one of ordinary skill in the art appreciates that various modifications and changes may be made without departing from the scope of the disclosure as set forth in the claims below.

Moreover in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by "comprises. a," "includes. a," or "contains. a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms "a" and "an" are defined as one or more unless explicitly stated otherwise herein. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

Claim 1:
A method for volumetric segmentation of an anatomical structure (<NUM>) in a plurality of planar medical images (<NUM>), the method comprising:
receiving, at an electronic processor (<NUM>), the plurality of planar medical images (<NUM>), wherein the plurality of planar medical images (<NUM>) form a three dimensional, 3D, volume including the structure (<NUM>);
displaying, on a display (<NUM>), a first planar medical image (<NUM>) from the plurality of planar medical images (<NUM>);
receiving, with a user interface (<NUM>), a user input indicating a line segment in the first planar medical image (<NUM>), wherein the line segment provides an approximate diameter of a region of interest;
determining, with the electronic processor (<NUM>), an inclusion region (<NUM>) of the 3D volume using the line segment, wherein the inclusion region (<NUM>) consists of a portion of the structure (<NUM>);
determining, with the electronic processor (<NUM>), a containment region (<NUM>) of the 3D volume using the line segment, wherein the containment region (<NUM>) includes the structure (<NUM>);
determining, with the electronic processor (<NUM>), a background region (<NUM>) of the 3D volume using the line segment, wherein the background region (<NUM>) excludes the structure (<NUM>);
determining, with the electronic processor (<NUM>), a 3D contour (<NUM>) of the structure (<NUM>) using
conditional densities from a function of the histogram of a first set of voxels located within the inclusion region (<NUM>),
the containment region (<NUM>), and
conditional densities from a function of the histogram of a second set of voxels located within the background region (<NUM>), and
prior probabilities of classifying a voxel as being a member of a foreground class or of a background class, the prior probabilities varying as a function of a distance from the inclusion region (<NUM>) and the containment region (<NUM>);
determining, with the electronic processor (<NUM>), a long axis (<NUM>) of the structure (<NUM>) using the 3D contour (<NUM>) of the structure (<NUM>); and
outputting, with the electronic processor (<NUM>), a dimension of the long axis (<NUM>) of the structure (<NUM>).