Patent Description:
The invention is a method, apparatus, computer program, and computer-readable medium as defined in the appended claims.

<CIT> discloses a method and system for regression-based 4D mitral valve segmentation from 2D+T magnetic resonance imaging slices.

<CIT> discloses a method of segmenting an entity in a digital image, more specifically an anatomic entity in a digital medical image.

<CIT> relates to anatomy detection and to a system and method for transformation invariant landmark detection for anatomical primitives.

In an effort to satisfy this need in the art, the inventors disclose an exemplary apparatus for automatically delineating a structure of interest within image data, the image data comprising a subject image of a region of interest, the region of interest including the structure of interest, the image comprising a plurality of data points, the data points comprising a plurality of intensity values, the apparatus comprising a processor configured to (<NUM>) compute a plurality of features for a plurality of the data points, the features being indicative of intensity variations over a plurality of windows of the data points, (<NUM>) detect a plurality of locations for a plurality of landmarks within the image based on an application the computed features to a trained landmark detector, (<NUM>) generate a shape estimate for the structure of interest based on the detected landmark locations, and (<NUM>) refine the shape estimate according to a shape refinement tool to thereby compute a refined shape estimate for the structure of interest. With such an apparatus in an exemplary embodiment, the landmark detection can provide a useful initial rough approximation of structure shape based on a wide area of the image while the shape refinement tool can be used to refine an initial shape approximation using a narrower area of the image. Thus, in an exemplary embodiment, both local and global aspects of the image can be used to refine the shape for the structure of interest.

In accordance with another aspect, the inventors disclose an exemplary method for automatically delineating a structure of interest within image data, the image data comprising a subject image of a region of interest, the region of interest including the structure of interest, the image comprising a plurality of data points, the data points comprising a plurality of intensity values, the method comprising: (<NUM>) computing a plurality of features for a plurality of the data points, the features being indicative of intensity variations over a plurality of windows of the data points, (<NUM>) detecting a plurality of locations for a plurality of landmarks within the image based on an application the computed features to a trained landmark detector, (<NUM>) generating a shape estimate for the structure of interest based on the detected landmark locations, and (<NUM>) refining the shape estimate according to a shape refinement tool to thereby compute a refined shape estimate for the structure of interest, and wherein the method steps are performed by a processor.

Further still, the inventors disclose an exemplary computer program product for automatically delineating a structure of interest within image data, the image data comprising a subject image of a region of interest, the region of interest including the structure of interest, the image comprising a plurality of data points, the data points comprising a plurality of intensity values, the computer program product comprising a plurality of instructions that are resident on a non-transitory computer-readable storage medium and executable by a processor to (<NUM>) compute a plurality of features for a plurality of the data points, the features being indicative of intensity variations over a plurality of windows of the data points, (<NUM>) detect a plurality of locations for a plurality of landmarks within the image based on an application the computed features to a trained landmark detector, (<NUM>) generate a shape estimate for the structure of interest based on the detected landmark locations, and (<NUM>) refine the shape estimate according to a shape refinement tool to thereby compute a refined shape estimate for the structure of interest.

In accordance with an aspect described herein, the inventors disclose an apparatus for training a landmark detector using a plurality of atlas images, the atlas images including location information for a landmark with respect to a structure of interest, the apparatus comprising a processor configured to (<NUM>) collect a plurality of positive samples and a plurality of negative samples from the atlas images, (<NUM>) compute a plurality of Haar-like features for the collected positive and negative samples, and (<NUM>) apply the computed Haar-like features and location data associated with the computed Haar-like features to a machine-learning algorithm to train a landmark detector to detect the landmark. A corresponding method and computer program product are also disclosed.

Still further, the inventors disclose an apparatus comprising a processor configured to (<NUM>) resolve a plurality of candidate locations for a landmark with respect to a structure in an image to a single landmark location based on a probability map, the probability map being defined according to a Gaussian distribution model for the landmark, (<NUM>) repeat the resolving operation for a plurality of different landmarks, (<NUM>) initialize a shape estimate for the structure based on the single landmark locations, and (<NUM>) iteratively refine the shape estimate. A corresponding method and computer program product are also disclosed.

These and other features and advantages of the present invention will be apparent to those having ordinary skill in the art upon review of the teachings in the following description and drawings.

Various embodiments will now be described that relate to both training an automated landmark detector using a machine learning algorithm and performing automated contouring of a structure of interest within image data using trained landmark detectors and a shape refinement tool.

It should be understood that the images processed using the techniques described herein can be take any of a number of forms. In various exemplary embodiments, the images can be medical images such as CT images. However, it should be understood that images of different types can be employed. For example, image types such as magnetic resonance (MR) images and ultrasound images could also be processed using the techniques described herein. The images can comprise a plurality of image data points, whose locations can be expressed through a coordinate system.

<FIG> depicts an exemplary embodiment for automatically delineating a structure shape within image data. As shown in <FIG>, a processor <NUM> can be configured to implement processing logic <NUM> whereby a new subject image <NUM> is processed with the aid one or more trained landmark detectors <NUM> and a shape refinement tool <NUM> to generate one or more refined contours <NUM> for the structure of interest. The refined contour data <NUM> can take any of a number of forms. For example, the contour data may comprise a plurality of image data points that fall lie on the boundary for the structure of interest (e.g., <NUM> data points evenly distributed (on the contour) where each point is represented by its coordinates within the image. Also, it should be understood that the subject image <NUM> can be either a two-dimensional (2D) image or three-dimensional (3D) image.

The processor <NUM> can be any processor with sufficient computational capabilities to implement the automated delineation features described herein. It should be understood that processor <NUM> may comprise multiple processors, optionally distributed via a network. The programming instructions for implementing the processing logic <NUM> can be resident on a non-transitory computer-readable storage medium (e.g., memory <NUM>) for access and execution by the processor <NUM>. It should be understood that the memory <NUM> may comprise multiple memory devices, optionally multiple distributed memory devices and/or memory devices of different types.

The trained landmark detectors <NUM> are configured to process data points of the image <NUM> to automatically detect the presence and location of certain landmarks within the image <NUM>. In an exemplary embodiment, each trained landmark detector <NUM> is configured to detect a different landmark, although this need not be the case. The landmark detectors <NUM> can take any of a number of forms, such as a set of machine-executable rules. Furthermore, if desired by a practitioner, the trained landmark detectors <NUM> can be configured to process multiple points of an image in parallel, although this need not be the case. As discussed below, the landmarks can be detected via analysis of various attributes of the image data points. It is expected that different landmarks will be used for different structures and other factors. The detected landmarks can then be used to generate an initial shape estimate for the structure as discussed below.

The processor <NUM> can leverage the shape refinement tool <NUM> and the output of the trained landmark detectors <NUM> to automatically estimate the boundary for the structure of interest within the image <NUM>. An example of a shape refinement tool <NUM> that can be employed is a boundary detector, which can take any of a number of forms, including a trained boundary detector that comprises a set of machine-executable rules. Another example of a shape refinement tool that can be employed is a shape deformation algorithm. Furthermore, if desired by a practitioner, the shape refinement tool <NUM> can be configured to process multiple points of an image in parallel, although this need not be the case. The shape refinement tool <NUM> can operate iteratively to estimate and adjust the estimated boundary for the structure of interest to generate the refined contour data <NUM>.

<FIG> depicts an exemplary process flow that expands on how subject images <NUM> can be processed to generate refined contours <NUM>. The left half of <FIG> (relative to the vertical dashed line) illustrates a process flow generally corresponding to the operations performed by processing logic <NUM>. The right half of <FIG> (relative to the vertical dashed line) illustrates a process flow that is performed to train the landmark detectors <NUM> and the boundary detector <NUM> using machine-learning algorithms. In an exemplary embodiment, the right half of <FIG> is performed offline, while the left half of <FIG> is performed online. That is to say, the tasks of training the various detectors can be performed prior to the new subject image <NUM> being generated or processed. Thus, the trained landmark detectors <NUM> and a trained boundary detector can already be ready to process image data by the time there is a need to process a new subject image <NUM>. Accordingly, the processing logic <NUM> will be able to generate refined contour estimates in an efficient manner, which can be particularly useful in instances where there is a desire to use the refined contour data quickly after the new subject image is taken. However, it should be understood that, if desired, a practitioner could nevertheless choose to implement the tasks of training the detectors in-line with the processing of the new subject image.

For the offline operations, training data such as atlas images <NUM> can be processed using machine-learning algorithms to create the trained landmark and boundary detectors. The atlas images <NUM> preferably include annotations that serve as reference data for information of interest to the training process. For example, the atlases <NUM> used to train the landmark detectors can include an identification of where the landmarks are located in those atlas images. Similarly, the atlases <NUM> used to train the boundary detector can include an identification of where the boundary for the structure of interest is located. In some instances, the atlas images may include both the landmark information and the boundary information, in which case such atlas images can be used to train both the landmark detectors and the boundary detector, but this need not be the case. The annotations included in the atlases can be provided by trained expert personnel through manual techniques or provided through automated techniques, preferably after confirmation of the automated results for accuracy. As such, in an exemplary embodiment, the corpus of training data can serve as reliable identifications of where landmarks and boundaries are present in prior images. In exemplary embodiments, the atlas images <NUM> can be images of people other than the person who is the subject of the new image <NUM>, but this need not be the case. In some instances, the atlases <NUM> may be prior images of the subject himself/herself.

At step <NUM>, the atlas images are aligned so as to create a common frame of reference for assessing landmarks and boundaries. Any of a number of techniques can be used for the image alignment/registration operation. For example, a rigid transformation technique can be used at step <NUM>. Such rigid transformation can be performed pair-wise on slices of atlas image volume data. An example of a suitable rigid transformation technique is similarity transformation, although other techniques such as mutual information-based registration, affine transformation, etc. can also be employed, as described by <NPL>), the There are <NUM> degrees of freedom: scale, rotation, x-directional translation, and y-directional translation.

At step <NUM>, the aligned atlas images are processed to train the landmark detectors <NUM> using a machine-learning algorithm. As shown in the example of <FIG>, a different training operation can be performed for each landmark of interest. <FIG> and <FIG> elaborate on how this training can be performed. <FIG> depicts an exemplary image <NUM> of a prostate region showing the locations of different landmarks of interest <NUM> thereon. A different detector <NUM> can be trained to find each landmark <NUM>.

In this example, the different landmarks <NUM> are five anatomical points on the prostate region. These landmark points can be selected so as to closely approximate the expected prostate anatomical structure where there is a good contrast relative to the neighboring non-prostate region. In an exemplary embodiment, for each atlas image slice, there will be an annotated ground-truth contour of the prostate. Such a ground-truth contour can be represented by <NUM> evenly distributed points (on the contour), where each point is represented by its (x, y) coordinates within the atlas image slice.

The top central landmark <NUM><NUM> can be manually selected from one of the contour points by an expert or other appropriately skilled person. In an exemplary embodiment, the top central landmark <NUM><NUM> is located on the ground-truth contour and in the middle region of the image at around the same distance to the left and right of the pubic bones. That is, a trained person can select the location along the ground-truth contour that is approximately equidistant between the leftmost portion of the right pubic bone and the rightmost portion of the left pubic bone (with reference to the image perspective shown by <FIG>) to serve as the top central landmark <NUM><NUM>. The remaining <NUM> landmark points can be selected automatically according to a selection criteria roughly corresponding to distance. For example, the remaining <NUM> landmark points can be selected from the contour points so as to create gaps between landmark points of, moving clockwise from the top central landmark point <NUM><NUM>, <NUM> contour points (for landmark point <NUM><NUM>), <NUM> contour points (for landmark point <NUM><NUM>), <NUM> contour points (for landmark point <NUM><NUM>), and <NUM> contour points (for landmark point <NUM><NUM>), thus leaving <NUM> contour points between landmark points <NUM><NUM> and <NUM><NUM>. It should be understood that different spacing criteria between landmark points could be employed if desired by a practitioner.

<FIG> illustrates a processor <NUM> and memory <NUM> that are configured to cooperate with each other to execute processing logic <NUM>, where the processing logic implements steps <NUM> and <NUM> of <FIG>. The processor <NUM> and memory <NUM> can optionally be the same as processor <NUM> and memory <NUM>, although this need not be the case. The processing logic <NUM> can be resident on a non-transitory computer-readable storage medium (e.g., memory <NUM>) for access and execution by the processor <NUM>. <FIG> depicts an exemplary process flow for processing logic <NUM> to implement steps <NUM> and <NUM>.

In the example of <FIG>, the different atlases <NUM> (e.g., <NUM><NUM>, <NUM><NUM>,. ) comprise 3D image volumes. At step <NUM>, the processor extracts 2D slices from the 3D volume data. At step <NUM>, the 2D slices from a common atlas are aligned, preferably in a pair-wise manner. As noted, this alignment process can use a rigid transformation or other suitable image registration techniques. The aligned slices will depict a region of interest that includes the structure of interest. For example, in an embodiment where the structure of interest is the prostate, it can be expected that the image slices will encompass the full prostate area, all or part of the symphysis pubis, and all or part of the rectum.

At step <NUM>, the processor collects a plurality of training samples from the aligned 2D slices. Both positive training samples and negative training samples can be collected. For example, the landmark point itself and a plurality of points that are deemed to be spatially near the landmark point can be collected from a 2D slice to define the set of positive samples for that 2D slice. As an example, a <NUM> distance from the landmark point can be used to govern the region from which positive samples are selected. The positive samples can be selected randomly within this region. For negative samples, the processor can select a plurality of points from the 2D slices that are deemed to not be near the landmark of interest. A larger distance threshold relative to the landmark point can be used to define this negative region. The processor can also be configured to randomly select the negative samples from this negative region. Moreover, the processor can make these selections to achieve a ratio between positive and negative samples of around <NUM>:<NUM>.

At step <NUM>, the processor computes features for the collected training samples that are indicative of intensity variation over windows of image data points. For example, the features can be Haar-like features. The Haar-like features can be used to represent rectangular regions at each sample location. An example is shown by <FIG> where a rectangular region <NUM> (or window) of image <NUM> is selected. The Haar-like feature can be computed as the difference between the sum of pixel intensities inside the darkened "+" region <NUM> and the sum of pixel intensity values inside the white "-" region <NUM>. Any of a number of patterns for the window <NUM> with corresponding "+" and "-" regions can be employed, as indicated by <FIG>. In an exemplary embodiment, the window pattern shown in the bottom left of <FIG> can be employed. With such an embodiment specifically, step <NUM> can consider <NUM> adjacent rectangular regions centered at the subject sample point in a detection window. The pixel intensities within each rectangular region are summed, and then the differences between these sums are calculated. Each difference is then recorded as a value in a feature vector for that sample. The size of the extracted feature vector will vary with the size of the detection window. In an exemplary embodiment, the window size can be <NUM> by <NUM>, although it should be understood that different window sizes could be employed. A description of how Haar-like features can be computed is found at <NPL>), the.

At step <NUM>, the processor applies the collected samples and their computed attributes to a machine-learning algorithm to train a landmark detector to distinguish between points that qualify and do not qualify as the landmark of interest. Because a Haar-like feature is a weak learner or classifier, a relatively large number of Haar-like features are preferably used to describe an object with accuracy. Therefore, the Haar-like features can be organized and learned by a cascade of simple classifiers to form a strong learner. In an exemplary embodiment, the machine-learning algorithm used for this purpose can be the LogitBoost machine-learning algorithm.

The LogitBoost algorithm can be used to learn from the weak classifiers. Specifically, the weak classifiers are trained sequentially. The weight distribution of the training set can be updated between iterations according to the accuracy of classification of the previous classifiers. The weight of misclassified samples is increased for the next iteration, whereas the weight of the correctly classified samples is decreased. The next classifier is then trained with such as re-weighted distribution. The amount of change on the weight of each classifier is proportional to the classification error of the classifier. With the LogitBoost algorithm, adaptive Newton steps can be used to fit an adaptive symmetric logistic model. This approach provides an advantage in that it places less emphasis on samples that are poorly classified, since those samples are most likely to be outliers that should be excluded from the calculation. Step <NUM> thus operates to produce a trained landmark detector <NUM>i for the landmark i of interest that is flexible and robust. The trained landmark detector <NUM>i for the landmark i can be a group of trained simple classifiers, where each classifier has a trained threshold and a trained weight value that defines its importance. As discussed below, these parameters can be leveraged during the detection phase.

A description of the LogitBoost machine-learning algorithm can be found at <NPL>), However, it should be understood that other machine-learning algorithms could be employed to train the landmark detectors <NUM>, such as the AdaBoost, FloatBoost, Any Boost, and MarginBoost machine-learning algorithms.

Once again, it should be understood that the process flow of <FIG> can be performed separately for each landmark of interest. Optionally, different processors and memories can be configured to train the different landmark detectors <NUM>.

Returning to <FIG>, at step <NUM>, the aligned atlas images are processed to train a boundary detector using a machine-learning algorithm. This operation can be performed by a processor and memory in a manner similar to that described by <FIG>. In an exemplary embodiment, the machine-learning algorithm for step <NUM> can be the random forests (RF) machine learning algorithm (see <NPL>. <FIG> depict examples of atlas images <NUM> that show boundary information for a structure of interest (the boundary being defined by the transition between the white regions and black regions within each image <NUM>.

The RF algorithm in this instance operates to produce a trained boundary detector that is a collection of decision trees. Each decision tree is a set of decision rules organized in a tree-like structure. Each node of the tree applies a decision rule, which is often called a test function or a split function. Each test function takes an attribute or feature value as input and produces a binary (yes/no) output. Based on the yes/no answer, the input data is sent to either the left or the right child-node of the current node. The child node will run another test based on a new feature value. This is repeated until the so-called "leaf-node" is reached. A "leaf-node" is a tree node without "children" nodes. Each leaf-node has a classification label associated with it but sometimes it can also be a probability value indicating the likelihood of belonging to a particular classification (e.g., boundary status). The rules can be expressed as test functions with binary outputs, e.g.: <MAT> where vi denotes the i-th feature value, and ai,bi are two thresholds. Thus, with a boundary detector that was trained using the RF algorithm, the detector can take the form of an ensemble of decision trees, where each tree is a set of decision rules organized in a tree or flowchart-like structure, where each internal (non-leaf) node denotes a test on an attribute (i.e., a decision rule), each branch represents an outcome of the test, and each leaf (or terminal) node holds a classification label (e.g., boundary status).

During step <NUM>, the trees can be built in parallel if desired since each tree is trained independent of the others. The training samples are used by the RF algorithm to "learn" the tree, i.e., to decide which decision rule to use at each internal tree node. For the RF algorithm, each tree is built using a random subset of the training data such that different trees are highly uncorrelated. Once training samples are collected and their attributes are computed for a tree, the tree is built recursively by adding one node at a time. At each node, the RF algorithm aims to find the best decision rule that most efficiently splits the training data arriving at the current node. In the case of binary classification, "best splitting" means that each branch of the node should contain as many samples from the same class as possible. Thus, the training or learning process with the RF algorithm aims to determine which feature to use at the current node and what threshold values to apply to best split the training data. With the RF algorithm, only a small, random subset of all features are considered at each node, and the "best" feature is selected among this random subset instead of using all features. This randomization again aims to make the trees as independent as possible. Each newly added node splits the incoming (training) data into two branches (two subsets), and each subset will be tested again at the subsequent child node. Thus, each (non-leaf) node can have two children node. The tree continues growing until the training data arriving at each child node all belong to the same class. The child node then becomes a leaf node of the tree, and the class label of the training data arrived at the child node becomes the label of that leaf node.

There are a number of RF algorithm implementations that are publicly-available, for example the Weka machine-learning software package is available online, and it includes an RF algorithm software package. These software packages also include known interfaces through which training samples can be applied to the machine learning algorithm. Moreover, the trained boundary detector produced by such software packages can take a form such as a text file that expresses the decision tree as machine-executable rules.

The RF algorithm can thus naturally handle multiple classes if desired, i.e., one detector to classify several structures (plus the background). The output of a RF-based detector can be a probability estimation of which class the input data belongs to, which is also preferred over a hard decision as some other learning algorithms would produce. In addition, the RF algorithm is fast in both detector training and detector application, and it can deal with very large dimensions of input data.

However, it should be understood that other machine-learning algorithms could also be employed at step <NUM> if desired by a practitioner. Examples of other machine-learning algorithms that can be employed at step <NUM> include the LogitBoost algorithm as well as those described in <NPL>), such as the Support Vector Machine (SVM) or AdaBoost machine learning algorithms.

Once the landmark detectors <NUM> and the boundary detector have been trained, the system is ready to process new subject images <NUM>. Turning to the online portion of <FIG>, at step <NUM>, the new subject image is generated. Any suitable image generation technique and equipment can be used at step <NUM>. Furthermore, it should be understood that the generated image <NUM> can be a 2D or 3D image. If a 3D image volume is generated, step <NUM> can also include generating 2D slices of the 3D image volume as well as performing alignment of the 2D slices (using an image registration technique as previously discussed).

At step <NUM>, the processor processes the new subject image data using the trained landmark detectors <NUM> to detect the locations of landmarks of interest within the image data. In doing so, step <NUM> can compute features that are indicative of intensity variation of windows of the subject image data points, and then apply these computed features to the trained landmark detectors. <FIG> and <FIG> depict process flows for step <NUM>. It should be understood that the process flow of <FIG> can be repeated for each landmark detector <NUM> employed by the system. Thus, if <NUM> landmark detectors are used to detect <NUM> different landmarks, then the process flow of <FIG> (or at least steps <NUM>-<NUM> of <FIG>) can be repeated for each landmark of interest. Similarly, the process flow of <FIG> can be repeated for each landmark of interest.

At step <NUM>, the processor selects a 2D slice of the new subject image <NUM> to process. At step <NUM>, the processor applies points of the selected image slice to the trained landmark detector <NUM> for the landmark of interest. This step can operate on all of the points of the selected image slice or on a targeted subset of the image points. For example, if it is known that the landmark of interest will reside in a definable portion of the image slice (e.g., the left half of the image), then step <NUM> can be configured to select only points within the desired image portion for processing. With step <NUM>, the processed image points are scored with regard to their suitability as candidates for the landmark of interest. To score the points, the features described above for training the landmark detector <NUM> are computed for the image slice points (e.g., Haar-like features). The image slice points and their computed features are then applied to the landmark detector <NUM> for the landmark of interest to compute a score for each point that is indicative of its suitability as a candidate for the landmark of interest.

At step <NUM>, the computed point scores are compared with a defined threshold. This threshold can be configurable based upon the desires of a practitioner. Those points having scores above the threshold can be classified as landmark points (step <NUM>).

More specifically, for steps <NUM>-<NUM>, a window of a defined target size is moved over the image slice, and for each subsection of the image, the Haar-like feature is calculated. <FIG> illustrates this sliding window concept. A sliding window corresponding to the size of the Haar-like feature window slides over all desired locations of the image <NUM>. <FIG> shows examples of the sliding window in three different positions corresponding to <NUM>, <NUM>, and <NUM>. The Haar-like features are computed in the same fashion as they were during the training phase. For each computed Haar-like feature from the image slice, each classifier of the trained landmark detector 108i (for the subject landmark i) makes a binary decision according to its trained threshold value. Thus, if the trained landmark detector <NUM> includes a set of <NUM> classifiers, each classifier will reach a binary decision based on its associated threshold value. The final decision for the landmark detector with respect to the Haar-like feature for a given sliding window position is a weighted sum of the decisions from the individual classifiers (using each classifier's associated weight). If this weighted sum passes a defined configurable threshold, the center of that sliding window position will be marked as a detected landmark point.

In some instances, it may be the case that the computed Haar-like features for multiple positions of the sliding window cause multiple points to be detected as landmark points for a given landmark. In other instances, the detected landmark point may fall out of a normal/expected range (an outlier point). To help resolve the landmark location in such scenarios, each detected landmark point for a subject landmark votes for the candidate locations of all other landmarks based on a distribution model for the landmarks.

This distribution model is a Gaussian model of how landmark points are distributed according to their relative distances between each other amongst the slices of the training data. With such a model, instead of a landmark point voting for a single candidate location each of the other landmark points, it will vote for a probability map which follows the built Gaussian distribution, where the voted values on the probability map sum to <NUM>.

Then, after the voting stage (and after the process flow of <FIG> has been repeated to detect landmark points for all of the subject landmarks with respect to an image slice), the processor can make a final determination of landmark locations according to the process flow of <FIG>. At step <NUM>, the processor selects a slice of the new subject image. As noted, at this stage, the process flow of <FIG> will have been repeated to detect landmark points and collect votes for candidate landmark locations with respect to all of the landmarks for the selected slice. At step <NUM>, the processor selects the detected landmark point(s) for the subject landmark with respect to the selected slice (see step <NUM>). At step <NUM>, the processor reviews the votes for candidate locations of the subject landmark that were cast by the detected landmark points of the other landmarks (see step <NUM>). As part of this, the probability maps from the automated voting process for the subject landmark with respect to the slice can be merged such that the merged probability map sums to <NUM>. The processor will select candidate locations from the merged probability map that are above a defined and configurable threshold. Next, at step <NUM>, the processor determines the location for the subject landmark by averaging the location(s) of the detected landmark point(s) from step <NUM> and the location(s) of the candidate(s) that passed the threshold set by step <NUM> (if any). This approach can effectively group multiple landmark detections together while greatly reducing the impact of outlier detections. As another example of combining the detected landmark point with the voted landmark points, the merged probability map can also include the detected landmark point from the step <NUM>. A relative weighting between the detected landmark point and the voted landmark locations can be assigned, preferably with a stronger weight being given to a detected landmark point than to an individual voted landmark location (e.g., a weight of " <NUM>" for the detected landmark point(s) and a weighting for the voted landmark locations such that their aggregated weights sum to "<NUM>"). Once again, the highest scoring location can be selected as the landmark location or an average location of all locations scoring above a certain threshold can be selected as the landmark location.

The process flow of <FIG> and <FIG> can then return to step <NUM> to process the next image slice if necessary. Upon processing all of the image slices for the new subject image <NUM>, each slice can have determined locations for the landmarks of interest. <FIG> depicts an example of results from applying <NUM> trained landmark detectors <NUM> to different image slices <NUM>, where each image slice <NUM> shows the detected locations <NUM> for the landmarks of interest.

The detected landmark locations from step <NUM> can be used as a shape initialization input into a shape modeling algorithm. Step <NUM> then operates to initialize a shape estimate from the detected landmark points and then iteratively refine the shape estimate. <FIG> depicts an exemplary process flow for step <NUM>.

The shape initialization and refinement can employ, for example, a shape modeling algorithm such as that described by <NPL>, At <NUM>, the shape estimate is initialized using the detected landmark points and a shape dictionary <NUM>. The initial shape can be inferred by S in Equation (<NUM>) below.

Thus, a sparse shape model can be employed as the shape prior method to infer this shape. It selects a sparse set of shapes in the shape dictionary <NUM> and composes them together to infer/refine an input shape. This model leverages two sparsity observations of the input shape instance: (<NUM>) the input shape can be approximately represented by a sparse linear combination of shapes in the shape dictionary; (<NUM>) parts of the input shape may contain gross errors but such errors are sparse. It is able to alleviate three problems of shape prior modeling, i.e., modeling complex shape variations, handling non-Gaussian errors and preserve local detail information of the input shape.

Then, at step <NUM>, for each refinement iteration, the algorithm minimizes the following optimization function: <MAT>.

Where vs is a subset of points on the input shape, D is the shape dictionary <NUM> that represents all training shapes, T(vs,β) is a global transformation operator with parameter β, which aligns the input shape to the same space of D. x denotes the weight coefficient of the linear combination, and e is a vector that modes the large residual errors. S is a binary diagonal matrix which indicates if the a certain point is in the subset vs. When S becomes very sparse and only includes a few points, the equation becomes the formula of landmark-based shape initialization, which is the first step <NUM> of the refinement process. Later on, once the edge points get refined by the trained boundary detector <NUM>, there will be more points available to put into the equation, in which case S becomes more dense, but the optimization process is essentially the same. The solved shape is then sent back to the boundary detector <NUM> for another round of edge refinement. Each iteration, the trained boundary detector <NUM> can be used to process a plurality of points near a shape point (e.g., <NUM> points along the normal direction of the point). For each point, the trained boundary detector <NUM> can generate a probability as to whether that point is "on the boundary" between the structure and non-structure. The processor can then select the point with the highest probability to be an updated edge point. <FIG> illustrates an example of a refined shape <NUM> relative to a ground-truth shape <NUM> for a slice. The iterative process stops once (<NUM>) it reaches a certain number of iterations (e.g., <NUM> iterations), or (<NUM>) it reaches a certain minimal residual error.

Upon the conclusion of the iterative refinement, step <NUM> produces the refined contour estimate <NUM>. Step <NUM> can also operate on a slice-by-slice basis to generate a refined contour <NUM> for each slice of the new subject image <NUM>. A 3D volume of the structure of interest can then be generated from the refined 2D contours <NUM> for delivery to treatment planning software to calculate/update a treatment plan. <FIG> depicts an example of such a 3D volume for an embodiment where the structure of interest is the prostate. However, it should be understood that the refined contour data need not be limited to treatment planning uses, and may also be used to interventionally guide treatment delivery as well.

If desired by a practitioner, dictionary learning techniques can be employs to train a compact shape dictionary <NUM> instead of using all training shapes and thus improve computational efficiency. One assumption of the sparse linear combination strategy is that the input shape lies within the convex hull defined by training shapes, i.e., an assumption that the training shapes should be sufficiently representative. However, this assumption may not hold in this all segmentation problems, such as the prostate segmentation problem. It is desirable to adaptively increase the representation capacity of the sparse shape model, so it is able to handle new types of shapes. A solution is to include newly segmented shapes in the shape dictionary <NUM> or re-train the dictionary <NUM> with all available shapes. However, this approach will significantly reduce the computational efficiency when the data scales up. As another solution, shown by <FIG>, an online learning method is applied to adaptively and efficiently incorporate new shapes into the shape dictionary <NUM>. When new training shapes come, instead of re-constructing the dictionary from scratch, the existing dictionary <NUM> is updated using a block-coordinates descent approach. Using the dynamically updated dictionary, the sparse shape model can be gracefully scaled up to model shape priors from a large number of training shapes without sacrificing run-time efficiency. See <NPL>), This method starts from constructing an initial shape dictionary using the K-SVD algorithm. When new shape comes, it iteratively employs two stages until converge, sparse coding <NUM> and dictionary update <NUM>. Sparse coding <NUM> aims to find the sparse coefficient for each signal, and the dictionary update stage <NUM> aims to update the dictionary based all discovered coefficients.

Based on stochastic approximation, the dictionary <NUM> is updated efficiently using block-coordinates descent. It is a parameter-free method and does not require any learning rate tuning. It is important to note that in the dictionary update step <NUM>, instead of requiring all training shapes, only exploits a small batch of newly coming data. The dictionary update thereby becomes much faster. In this way, the shape dictionary can be efficiently updated online by using new data as selected. Using this online updated dictionary, the run-time efficiency of shape composition is not sacrificed with much more training shapes. In addition, it can be gracefully scaled-up to contain shape priors from, theoretically, an infinite number of training shapes.

Claim 1:
A computer implemented method comprising:
training a landmark detector using a plurality of aligned atlas images, the atlas images including location information for a landmark with respect to a structure of interest, wherein the training comprises:
<NUM>) Collecting a plurality of positive training samples and a plurality of negative training samples from the atlas images,
<NUM>) Computing a plurality of Haar-like features for the collected positive and negative samples, and
<NUM>) Applying the computed Haar-like features and location data associated with the computed Haar-like features to a machine-learning algorithm to train a landmark detector to detect the landmark,
wherein the method further comprises:
for each landmark detected by the trained landmark detector:
determining votes for a candidate location of the landmark cast by all other landmarks based on a distribution model for the landmarks wherein the distribution model is a Gaussian model of how landmark points are distributed according to their relative distances between each other amongst the slices of training data wherein a landmark votes for a probability map and wherein the values on the probability map sum to <NUM>,
and wherein the method further comprises, for a selected detected landmark point of a selected slice:
merging with respect to the slice the probability maps from the voting process cast by all other landmarks,
selecting candidate locations from the merged probability map that are above a set threshold,
determining the location of the selected landmark by averaging the location of the selected landmark and the locations of the candidates that passed the threshold set step, initializing a shape estimate for the structure based on the landmark locations determined for the plurality of landmarks, and
iteratively refining the shape estimate wherein the refinement is performed by estimating boundaries for the shape estimate using a trained boundary detector.