accurate and automatic segmentation of medical images is an essential component of a computer - aided diagnosis ( cadx ) system . however , medical image segmentation is typically a difficult task due to noise resulting from the image acquisition process , irregular shape and variable size of anatomical objects , as well as the characteristics of the object 's neighborhood . for example , a lung nodule or a colon polyp is usually embedded in a complex surrounding region . in ct imaging , the intensities of such lesions ( e.g. , juxtavascular nodules , juxtapleural nodules , or colonic polyps ) are usually very similar to their adjacent tissues . in this case , traditional intensity - based or morphological methods [ 15 ] may fail to accurately segment the lesion . in particular , graph cut methods [ 612 ] , which construct an image - based graph and achieve a global minimum of energy functions are often found in image segmentation . two key issues in the graph cut technique are as follows : ( 1 ) how to define an energy function so its minimization leads to the desired result ; ( 2 ) how to represent the energy function in terms of the graph construction .   in most graph cut methods , the graph vertices are constructed at the image pixels , and the segmentation energy is composed of intensity terms only . proposed a framework to simultaneously segment and register the lungs and nodules in ct data . for segmentation , a 2d pixel - based graph cut algorithm was applied to 3d lung and nodule datasets . it is noted that , when a graph vertex is placed at each image pixel , the number of nodes in the graph increases exponentially with the image size , dramatically increasing the computation time . to improve the efficiency , li et al . it is known that pixel intensity can be locally erroneous due to noise and other image acquisition issues , such as partial volume effect ( pve ) . xu et al .   presented a graph cuts - based active contour approach to object segmentation , which slabaugh and unal   extended by incorporating a shape prior . these methods show promising results , but require manual initialization and quantitative evaluation on different types of lesions which are not available . liu et al .   applied ordering constraints into an energy smoothness term based on an initial labeling . zheng et al .   constructed a graph laplacian matrix and solved linear equations for the estimation of ground glass opacity ( ggo ) nodules in ct , with assistance from some manually drawn scribbles for which the opacity values are easy to determine manually . more general - purpose graph cut methods show much utility in image segmentation , however , are ideally suited to nonmedical images . other medical imaging approaches apply to nodule segmentation but are evaluated only on very small ct datasets . further , most available methods require user interaction to identify initial foreground and background seeds . the goal of this paper is to develop an automatic and robust superpixel - based graph cut method for accurate segmentation of different types of lesions in ct imaging including solid and ggo nodules , as well as colonic polyps . one of the original contributions of this paper is that our graph is built on mean - shift superpixels .   in this paper , we obtain superpixels by using five - dimensional mean shift clustering that incorporates joint spatial , intensity , and shape ( jsis ) features . the superpixels express the local structure of the data in the five - dimensional feature space . mean shift clustering reduces the data variation , thereby helping to produce accurate segmentations . by connecting superpixels instead of pixels , a second original contribution is our energy function , which incorporates the image intensity and the shape feature into a markov random field ( mrf ) minimized with graph cuts . in particular , the intensity and shape information appear in both our unary and pairwise terms . compared to our previous work in , we have improved our unary term in the energy function by considering both image intensity and shape features and investigated the method on a relatively large dataset that contains different types of nodules , including juxtavascular , juxtapleural , as well as ggo . moreover , the experimental results , evaluated both visually and quantitatively , demonstrate high performance for ct lung nodule segmentation . initial results on colon polyps demonstrate the generalization of the method to other anatomies . the paper is organized as follows : section 2 describes the proposed algorithm , which covers five - dimensional mean shift clustering based on jsis features ( section 2.1 ) and automatic graph cut segmentation of the mean shift jsis superpixels ( section 2.2 ) . section 3 presents our experimental results , followed by a discussion and future work in section 4 and conclusion in section 5 . our approach is a combination of five - dimensional mean shift clustering followed by energy minimization based on graph cuts . the method is also guided by prior knowledge about the lesion ( e.g. , nodule / polyp ) . the flow chart of our method is illustrated in figure 1 . in the following sections , the method first computes the jsis features , which are then clustered in a five - dimensional space using mean shift . in this section , we review the volumetric shape index feature and our five - dimensional mean shift approach . the volumetric shape index ( si ) at voxel p can be defined as follows [ 14 , 15 ] : 
 ( 1)si(p)=121arctank1(p)+k2(p)k1(p)k2(p ) , 
									 where k1(p ) and k2(p ) are the principal curvatures , defined as 
 ( 2)k1(p)=h(p)+h2(p)k(p),k2(p)=h(p)h2(p)k(p ) , 
									 where k(p ) and h(p ) are the gaussian and mean curvatures , respectively . the calculation of the gaussian and mean curvatures is based on the first and second fundamental forms of differential geometry . a practical approach for computing these forms is to use the smoothed first and second partial derivatives of the image as suggested in . in this paper , prior to shape index calculation , a single - scale 3d gaussian filter , with standard deviation of 1.5 , is applied to obtain a smoothed image . every distinct shape , except for the plane , corresponds to a unique shape index . for example , a shape index value of 1.0 indicates a sphere - like shape ( e.g. , lung nodule or colonic polyp ) , while 0.75 indicates a cylinder - like shape ( e.g. , vessel or colonic fold ) . based on the definition , the volumetric shape index directly characterizes the topological shape of an isosurface in the vicinity of each voxel without explicitly calculating the isosurface . this feature provides rich information that complements the image intensity and is useful for automated segmentation .   in particular , lesions in a ct image may appear within an area of complicated anatomy ( such as a lung nodule neighboring a blood vessel or colonic polyp attached to the colon wall ) where adjacent structures have similar image intensities but different shapes . note that we use the term spherical concentration throughout the paper to describe a spatially connected set of voxels that have a high shape index value .      for each voxel , the 3d spatial location , intensity , and volumetric shape index features are concatenated in the joint spatial - intensity - shape domain of dimension d = 5 . given n data points pi ,   i = 1,  ,n in a five - dimensional space , the multivariate kernel is defined as the product of three radially symmetric kernels 
 ( 3)khshrhsi(p)=ck,5k(||pshs||2)k(||prhr||2)k(||psihsi||2 ) , 
									 where ck,5 is a normalization constant which assures that k(p ) integrates to 1 . p , p , and p are the spatial location , intensity , and shape index features , associated with each voxel p ; hs , hr , and hsi are the kernel bandwidths for spatial , intensity , and shape index kernel functions . the normal kernel is used in this paper , where k(p ) = ( 2)exp  ( 1/2||p|| ) .   in the mean shift framework , the shape index feature can be combined with the intensity and spatial features for clustering . the mean shift vector with three kernel bandwidths ( spatial , intensity , and shape index ) is defined as 
 ( 4)mhshrhsi(p )  = i=1npig(||pis / hs||2 )  g(||pir / hr||2)g(||pisi / hsi||2)i=1ng(||pis / hs||2 )  g(||pir / hr||2)g(||pisi / hsi||2)p , 
									 where g(p ) = k(p ) . the mean shift vector always points in the direction of the maximum increase in the density function . it is noted that the mean shift algorithm estimates the modes ( the densest regions ) of the multivariate distribution underlying the feature space . the set of points that converge to the same mode is defined as the attraction basin . mean shift maps all the data samples to the local maxima of their corresponding attraction basin based on five - dimensional features . each superpixel has a constant shape index and intensity , represented with mode maps , namely , an intensity mode map ( mi ) and a shape index mode map ( msi ) .   additionally , there is a spatial mode map ( ms ) that is not incorporated directly in our energy function ; figure 2 shows a nodule attached to a vessel and its corresponding intensity and shape index mode maps . it can be noted that the mode maps ( ( c ) and ( d ) ) from jsis mean shift clustering can be seen as  filtered  images and are less contaminated by outliers . in the following section , graph cut - based segmentation is applied to these superpixels . the mode map obtained by the above jsis mean shift technique expresses the local structure of the data in a given region of the feature space . the number of resulting superpixels depends on the kernel bandwidth and the data itself ; a key advantage of the mean shift clustering is its ability to locally group regions of similar intensity and shape , reducing the data variance in superpixel . the aim of this section is to group superpixels into two classes : foreground ( lesion ) and background ( nonlesion ) . it is known that the image labeling problem can be formulated using an energy function in a bayesian framework in the context of maximization a posteriori ( map ) and mrf theory and can be solved by energy minimization . the energy function includes both unary and pairwise terms , the latter providing smoothing by modeling the interaction between neighboring superpixels . in this section , a novel energy formulation that incorporates shape and intensity in both unary and pairwise terms is defined on superpixels and minimized with graph cuts using the maxflow / mincut method . two key issues are addressed in the following subsections : ( 1 ) how to initialize the segmentation and ( 3 ) how to define the energy function for minimization . in this paper , we assume that a nodule ( or polyp ) is generally either spherical or has a local spherical concentration , while a blood vessel ( or colonic fold ) is usually oblong . the initial seeds for the graph cut are computed automatically based on a spherical concentration .   a spherical concentration s is defined , using hysteresis thresholding , as a 3d connected region that satisfies the following criteria . all voxels in the region have a shape index greater than or equal to a low threshold tl . at least one voxel in the region has a shape index greater than or equal to a high threshold th . typical thresholds are in the range  tl  [ 0.8,0.9 ) and  th  [ 0.9,1 ] . the background seeds are determined using adaptive enlargement of the spherical concentration .   a distance transform   of s is computed and then adaptively enlarged ( e.g. , factorfdistancemax , where factor is the enlargement factor and the enlargement of the foreground region is designed so that the background region does not cover the foreground object . figure 3 shows an example of the initialization for a juxtavascular nodule using the above method , where tl and th were set to 0.82 and 0.92 , respectively . figures 3(b ) and 3(c ) are the initial foreground and background regions , respectively , where factor is set to be 10 . the thresholds of tl and th are fixed throughout the paper and are based on the assumption that a nodule which may not be entirely spherical has at least spherical elements concentrated within the nodule . in fact , using shape index to extract the spherical concentration is the first step in our automatic lung nodule detection algorithm . very high detection sensitivity can be achieved at this stage .   in most cases ( e.g. , high contrast solid nodule ) , the sphericity concentration covers the core part of the nodule . however , in the case of some part - solid or ggo nodules , the initial foreground region might spread in the nodule object . figures 4 and 5 show examples of two different segmented nodules using automatic calculation of foreground and background regions , where , in figure 4(b ) , the initial foreground contains the core part of the solid nodule object , while in figure 5(b ) , the foreground region scatters in the part - solid nodule ( it is noted , the foreground region is a 3d connected region but it may appear as several small regions in each 2d slice as shown in figure 5(b ) ) . figures 4(d ) and 5(d ) show the segmentation results by using the proposed graph cut - based method based on the initial foreground ( b ) and background seeds ( c ) . however , occasionally the above initialization may fail to give good foreground and background seeds , as the spherical concentration may contain part of the nodule as well as lung tissue . more discussion for the initial foreground seeds will be given in section 4 . in this section , we employ a graph g = ( v , e ) , defined with vertices v  v representing superpixels determined from five - dimensional mean shift clustering , and edges   e connecting adjacent superpixels . a key difference from the usual graph construction is that we connect superpixels instead of the original pixels . as a result , the number of vertices in g is greatly reduced compared to the original number of pixels in the image . the lesion segmentation problem is formulated as a binary labeling problem , so the goal is to assign a unique label li  { 0,1 } ( where 0 is background and 1 represents foreground ( lesion ) ) to each superpixel i by minimizing the following energy function e(l ) : 
 ( 5)e(l)=ive1(li)+(i , j)e2(li , lj ) , 
									 where e1(li ) is the unary data term representing the cost to assign the label li to the superpixel i , e2(li , lj ) is the pairwise smoothness term representing the cost of assigning the labels li and lj to adjacent superpixels i and j , and  is a weighting factor . the details of energy minimization via the graph cut algorithm for binary labeling can be found in . below we unary data termwe are given the initial foreground { mm } and background regions{mt } , automatically calculated from the previous section , where m and t are the superpixel indices for initial foreground and background , respectively . figure 6 shows the schematic diagram for different types of superpixels , for example , foreground , background , and uncertain superpixels . for each superpixel i , the unary term is defined as 
 ( 6)e1(li=1)=cifcif+cib ,  e1(li=0)=cibcif+cib , 
										 where ci and ciare the costs of a superpixel i to the foreground superpixels and background superpixels , determined from the initialization . the foreground cost ( ci ) is calculated as 
 ( 7)cif = min  m{1exp  ( (si(i)mmf)2i2 ) }  (1exp  ( (ssi(i)1.0)2si2 ) ) , 
										 where si(i ) and ssi(i ) are the intensity value and shape index value at superpixel i , determined from mean shift clustering . it is noted that the intensity and shape features are normalized to the same scale . also , in this paper , the parameters i and si are determined experimentally ( e.g. , it can be seen that ci encourages a superpixel to have the same label as the initial foreground superpixels if the superpixel has similar intensity to the foreground superpixels and also its shape feature is closer to one . similarly , the background cost ci is defined as 
 ( 8)cib = min  t{1exp  ( (si(i)mtb)2i2 ) }  exp  ( (ssi(i)1.0)2si2 ) . we are given the initial foreground { mm } and background regions{mt } , automatically calculated from the previous section , where m and t are the superpixel indices for initial foreground and background , respectively . figure 6 shows the schematic diagram for different types of superpixels , for example , foreground , background , and uncertain superpixels . for each superpixel i , the unary term is defined as 
 ( 6)e1(li=1)=cifcif+cib ,  e1(li=0)=cibcif+cib , 
										 where ci and ciare the costs of a superpixel i to the foreground superpixels and background superpixels , determined from the initialization . the foreground cost ( ci ) is calculated as 
 ( 7)cif = min  m{1exp  ( (si(i)mmf)2i2 ) }  (1exp  ( (ssi(i)1.0)2si2 ) ) , 
										 where si(i ) and ssi(i ) are the intensity value and shape index value at superpixel i , determined from mean shift clustering . it is noted that the intensity and shape features are normalized to the same scale . also , in this paper , the parameters i and si are determined experimentally ( e.g. , it can be seen that ci encourages a superpixel to have the same label as the initial foreground superpixels if the superpixel has similar intensity to the foreground superpixels and also its shape feature is closer to one . similarly , the background cost ci is defined as 
 ( 8)cib = min  t{1exp  ( (si(i)mtb)2i2 ) }  exp  ( (ssi(i)1.0)2si2 ) . pairwise smoothing term     the second term e2(li , lj ) in ( 5 ) is defined as 
 ( 9)e2(li , lj)=|lilj|(wsi(ei+esi)+(1wsi)eiesi ) , 
										 where ei is an intensity energy term denoting the intensity difference between two adjacent superpixels i and j , defined as ei(li , lj ) = 1/(||si(i )  si(j)|| + 1 ) . this means that superpixels with similar intensities have a larger ei , which leads to assigning them the same labels . e 
 si is the shape energy term denoting the shape difference between two adjacent superpixels , defined as esi(li , lj ) = 1/(||ssi(i )  ssi(j)|| + 1 ) . similar to the intensity term , the shape energy term captures the shape features for the two adjacent superpixels . if adjacent superpixels have similar shape , esi is larger , with high probability , both superpixels have the same label.as can be seen from ( 9 ) , the intensity term and shape term are combined through a weighting function wsi which is defined as  
											 ( 10)wsi(ssi(i),ssi(j))=exp  ( (ssi(i)ssi(j)si)2).it is noted that when two adjacent superpixels have the same shape , namely , wsi = 1 , the energy depends on the first term of ( 9 ) . however , when two adjacent superpixels have very different shapes , wsi is small , so the ( 9 ) depends on the second term . figure 7 shows a juxtavascular nodule segmentation that uses different pairwise smoothing terms . due to pve in ct imaging , part of the nodule 's pixels ( e.g. , figure 7(a3 ) ) has relatively low intensities , compared to those on other slices . with just the first term of ( 9 ) , the pixels with low intensity ( but similar shape ) can still be correctly identified as being part of the nodule object , as seen in figure 7(b3 ) . however , some small amounts of vessels ( which has similar intensity but a different shape feature value ) are also included in the segmentation , as seen in figures 7(b1b3 ) . this is because the first term equally considers the similarity for both of the intensity and shape features . figures 7(c1c3 ) are the results using the second term only in ( 9 ) . it can be seen that the shape feature is only used as a weighting to the intensity feature . superpixels with different shapes result in a lower weighting ; this is why part of the nodule can be properly separately from the adjoining vessel despite the similar intensity , as shown in figures 7(c1c2 ) . however , the superpixels with lower intensity due to pve are wrongly identified as background due to the different intensities , compared to that on the other slices , as shown in figure 7 ( c3 ) . figures 7(d1d3 ) are the results by combining both of terms as in ( 9 ) , in which the nodule boundary can be properly delineated despite the pve and the presence of vessels with similar intensities . figure 8 shows a step - by - step example of using different cost functions for the energy minimization . figures 8(a1 ) and 8(b1 ) are two contiguous slices for one 3d vascular nodule . figures 8(a2 ) and 8(b2 ) are the unary cost   ( 7 ) for each slice , respectively . the dark area in the image indicates lower cost for being foreground , while the bright area means high cost . figures 8(a3 ) and 8(b3 ) are the pairwise smoothing term ( 9 ) for each slice , respectively . it can be seen that there are low costs for the boundary , while high costs are usually within the regions ; figures 8(a4 ) and 8(b4 ) are the minimization results in which the energy function uses both of unary term ( 6 ) and pairwise smoothing term ( 9 ) . the second term e2(li , lj ) in ( 5 ) is defined as 
 ( 9)e2(li , lj)=|lilj|(wsi(ei+esi)+(1wsi)eiesi ) , 
										 where ei is an intensity energy term denoting the intensity difference between two adjacent superpixels i and j , defined as ei(li , lj ) = 1/(||si(i )  si(j)|| + 1 ) . this means that superpixels with similar intensities have a larger ei , which leads to assigning them the same labels . e 
 si is the shape energy term denoting the shape difference between two adjacent superpixels , defined as esi(li , lj ) = 1/(||ssi(i )  ssi(j)|| + 1 ) . similar to the intensity term , the shape energy term captures the shape features for the two adjacent superpixels . if adjacent superpixels have similar shape , esi is larger , with high probability , both superpixels have the same label . as can be seen from ( 9 ) , the intensity term and shape term are combined through a weighting function wsi which is defined as  
											 ( 10)wsi(ssi(i),ssi(j))=exp  ( (ssi(i)ssi(j)si)2 ) . it is noted that when two adjacent superpixels have the same shape , namely , wsi = 1 , the energy depends on the first term of ( 9 ) . however , when two adjacent superpixels have very different shapes , wsi is small , so the ( 9 ) depends on the second term . figure 7 shows a juxtavascular nodule segmentation that uses different pairwise smoothing terms . due to pve in ct imaging , part of the nodule 's pixels ( e.g. , figure 7(a3 ) ) has relatively low intensities , compared to those on other slices . with just the first term of ( 9 ) , the pixels with low intensity ( but similar shape ) can still be correctly identified as being part of the nodule object , as seen in figure 7(b3 ) . however , some small amounts of vessels ( which has similar intensity but a different shape feature value ) are also included in the segmentation , as seen in figures 7(b1b3 ) . this is because the first term equally considers the similarity for both of the intensity and shape features . figures 7(c1c3 ) are the results using the second term only in ( 9 ) . it can be seen that the shape feature is only used as a weighting to the intensity feature . superpixels with different shapes result in a lower weighting ; this is why part of the nodule can be properly separately from the adjoining vessel despite the similar intensity , as shown in figures 7(c1c2 ) . however , the superpixels with lower intensity due to pve are wrongly identified as background due to the different intensities , compared to that on the other slices , as shown in figure 7 ( c3 ) . figures 7(d1d3 ) are the results by combining both of terms as in ( 9 ) , in which the nodule boundary can be properly delineated despite the pve and the presence of vessels with similar intensities . figure 8 shows a step - by - step example of using different cost functions for the energy minimization . figures 8(a1 ) and 8(b1 ) are two contiguous slices for one 3d vascular nodule . figures 8(a2 ) and 8(b2 ) are the unary cost   ( 7 ) for each slice , respectively . the dark area in the image indicates lower cost for being foreground , while the bright area means high cost . figures 8(a3 ) and 8(b3 ) are the pairwise smoothing term ( 9 ) for each slice , respectively . it can be seen that there are low costs for the boundary , while high costs are usually within the regions ; figures 8(a4 ) and 8(b4 ) are the minimization results in which the energy function uses both of unary term ( 6 ) and pairwise smoothing term ( 9 ) . in this section , we present results demonstrating the effectiveness of the proposed algorithm applied to clinical ct scans . the first , visual analysis is performed on several different types of ct pulmonary nodules and colonic polyps to provide insight into the method 's performance for any visual gross missegmentation , such as a failure in separating nodules from vasculature . the second , quantitative experiments evaluate the segmentation results on large nodule datasets , containing 101 solid nodules and 80 ggo nodules . it is noted that , all the nodules in our database are confirmed by three experienced thoracic radiologists . however , each nodule boundary was manually delineated by one qualified radiologist .      in all experiments , the three kernel bandwidths ( spatial hs , intensity hr , and shape index hsi in ( 4 ) ) in the five - dimensional jsis mean shift clustering were set to 3.0 , 6.5 , and 3.0 , respectively . the proposed graph cut algorithm was applied to the mean - shift superpixels using the energy formulation based on ( 5)(10 ) , where the weighting factor  was set to be 1 . the mrf parameters were selected manually based to maximize the overlap ratios with the ground truth data . 
 attached nodules 
 figure 9 demonstrates a detailed example of the segmentation of a solid nodule attached to a small vessel . the figure shows ( a ) the original ct lung nodule on five contiguous slices , ( b ) , and ( c ) corresponding intensity mode and shape index mode values on two of the slices of the superpixels resulting from the five - dimensional jsis mean shift clustering .   since each superpixel is an attraction basin in the 5d feature space , each superpixel has a different pair of intensity and shape index mode values . in figure 9(b ) , after five - dimensional jsis mean shift clustering , most of voxels within the nodule on the same slice have the same mode ( such as the value of in intensity mode map of the first slice and  5  of the third slice ) . it can be seen that , although the intensity modes within nodule are very different , their shape modes are quite similar . the nodule superpixels can be merged using the proposed graph cut method . figure 9(d ) shows the segmented nodule resulting from a graph cut without the shape feature , for which the intensity energy term ei was only used in the pairwise term ( 9 ) . the algorithm correctly segments most of the nodule ( specifically , the part of nodule in the middle three slices ) . however , the other parts of the nodule with lower intensity ( e.g. , pixels on the first and the last slice ) are wrongly identified as background due to the different intensities . it can be seen that , by combining both the intensity energy ei and shape energy esi into the pairwise energy term ( 9 ) , the parts of the nodule with low intensity can be successfully segmented due to the similar shapes on the first and last slices . the intensity of the nodule is very similar to that of the adjoining vessel and heart wall . figure 10(a ) shows the nodule on three continuous ct slices ; figures 10(b ) , and 10(c ) are two intensity mode maps corresponding to the same nodule slice produced by four - dimensional mean shift clustering ( without the shape index feature ) and five - dimensional mean shift clustering ( with the shape index feature ) , respectively . it can be seen that , without considering the shape feature , a large amount of nonnodule voxels    ( e.g. , heart ) have a same superpixel as that of nodule voxels ( e.g. , however , using the proposed five - dimensional jsis feature , most of the nonnodule object is separated from the nodule . this is due to the different local shapes of the nodule and the surrounding anatomy ( such as the adjoining vessel and heart wall ) . the proposed five - dimensional mean shift clustering not only considers spatial and intensity information , but also shape features . figure 10(d ) shows the final segmented nodule using graph cut on the above four - dimensional mean shift superpixels , for which only the intensity term is used in the energy function . it is noted that the superpixel with intensity mode value of  24  in figure 10(b ) produced from four - dimensional mean shift without taking   into account the shape feature contains part of the nodule but also nonnodule objects ( such as heart ) . by representing each superpixel as a node in the graph construction and minimizing the energy function defined in ( 5 ) , this superpixel is identified as nonnodule object . that is why most of the nodule voxels within this super pixel were wrongly identified as background ( nonnodule object ) . for comparison , it is noted that , by using the proposed method , the misclassified nodule part in figure 10(c ) can be successfully identified as being part of nodule object . figure 9 demonstrates a detailed example of the segmentation of a solid nodule attached to a small vessel . the figure shows ( a ) the original ct lung nodule on five contiguous slices , ( b ) , and ( c ) corresponding intensity mode and shape index mode values on two of the slices of the superpixels resulting from the five - dimensional jsis mean shift clustering .   since each superpixel is an attraction basin in the 5d feature space , each superpixel has a different pair of intensity and shape index mode values . in figure 9(b ) , after five - dimensional jsis mean shift clustering , most of voxels within the nodule on the same slice have the same mode ( such as the value of  570  in intensity mode map of the first slice and  5  of the third slice ) . it can be seen that , although the intensity modes within nodule are very different , their shape modes are quite similar .   the nodule superpixels can be merged using the proposed graph cut method . figure 9(d ) shows the segmented nodule resulting from a graph cut without the shape feature , for which the intensity energy term ei was only used in the pairwise term ( 9 ) . the algorithm correctly segments most of the nodule ( specifically , the part of nodule in the middle three slices ) . however , the other parts of the nodule with lower intensity ( e.g. , pixels on the first and the last slice ) are wrongly identified as background due to the different intensities . it can be seen that , by combining both the intensity energy ei and shape energy esi into the pairwise energy term ( 9 ) , the parts of the nodule with low intensity can be successfully segmented due to the similar shapes on the first and last slices . the intensity of the nodule is very similar to that of the adjoining vessel and heart wall . figure 10(a ) shows the nodule on three continuous ct slices ; figures 10(b ) , and 10(c ) are two intensity mode maps corresponding to the same nodule slice produced by four - dimensional mean shift clustering ( without the shape index feature ) and five - dimensional mean shift clustering ( with the shape index feature ) , respectively . it can be seen that , without considering the shape feature , a large amount of nonnodule voxels    ( e.g. , heart ) have a same superpixel as that of nodule voxels ( e.g. , 24  in figure 10(b ) ) .   however , using the proposed five - dimensional jsis feature , most of the nonnodule object is separated from the nodule . this is due to the different local shapes of the nodule and the surrounding anatomy ( such as the adjoining vessel and heart wall ) . the proposed five - dimensional mean shift clustering not only considers spatial and intensity information , but also shape features . figure 10(d ) shows the final segmented nodule using graph cut on the above four - dimensional mean shift superpixels , for which only the intensity term is used in the energy function . it is noted that the superpixel with intensity mode value of  24  in figure 10(b ) produced from four - dimensional mean shift without taking   into account the shape feature contains part of the nodule but also nonnodule objects ( such as heart ) . by representing each superpixel as a node in the graph construction and minimizing the energy function defined in ( 5 ) , this superpixel is identified as nonnodule object . that is why most of the nodule voxels within this super pixel were wrongly identified as background ( nonnodule object ) . for comparison , it is noted that , by using the proposed method , the misclassified nodule part in figure 10(c ) can be successfully identified as being part of nodule object . pleural nodulea pleural nodule is defined as a nodule attached to the pleural wall . similar to the above two examples , figures 11(b ) and 11(c ) compare results based on the 4d approach ( no shape feature ) to the 5d approach ( using the shape feature ) . it can be seen that , without considering the shape feature , the algorithm is unable to segment the nodule on the last slice , while this missing part can be identified as part of the nodule by using our method . figure 11(a ) is a 3d nodule in four contiguous ct slices . similar to the above two examples , figures 11(b ) and 11(c ) compare results based on the 4d approach ( no shape feature ) to the 5d approach ( using the shape feature ) . it can be seen that , without considering the shape feature , the algorithm is unable to segment the nodule on the last slice , while this missing part can be identified as part of the nodule by using our method . it is challenging to properly segment ggo boundaries due to the faint contrast , irregular shape , and fuzzy margins of ggo nodules . figure 12 shows an example of the five - dimensional method on a large ggo nodule ( containing four slices as shown in figure 12(a ) ) , attached to small vessels with similar intensity . the segmentation results shown in figure 12(c ) demonstrate good performance of the method . for comparison , it can be seen that , without considering the shape feature , a large amount of small adjoining vessels are wrongly identified as part of the nodule object . however , using the shape index feature in both the mean shift clustering and the graph cut energy function , the nodule boundary can be properly delineated from the background despite the low intensity of the nodule and the presence of other nontarget structures with similar intensity.quantitative evaluation of ggo nodule segmentation and further discussion will be given in next subsection . it is challenging to properly segment ggo boundaries due to the faint contrast , irregular shape , and fuzzy margins of ggo nodules . figure 12 shows an example of the five - dimensional method on a large ggo nodule ( containing four slices as shown in figure 12(a ) ) , attached to small vessels with similar intensity . the segmentation results shown in figure 12(c ) demonstrate good performance of the method . for comparison , it can be seen that , without considering the shape feature , a large amount of small adjoining vessels are wrongly identified as part of the nodule object . however , using the shape index feature in both the mean shift clustering and the graph cut energy function , the nodule boundary can be properly delineated from the background despite the low intensity of the nodule and the presence of other nontarget structures with similar intensity . quantitative evaluation of ggo nodule segmentation and further discussion will be given in next subsection . the segmentation of colonic polyps in ct images is a complex task due to the irregularity of polyp morphology and the complexity of surrounding regions . results showing the proposed method applied to colonic polyp segmentation appear in figures 13 and 14 . similar to the nodule segmentation mentioned above , for comparison , figures 13(b ) and 14(b ) show results without the shape feature , while figures 13(c ) and 14(c ) are the results based on the proposed method . it can be seen that , by considering the shape index feature in both the mean shift clustering and the graph cut energy , both polyp boundaries can be properly delineated and separated from nonpolyp tissues . the performance of our superpixel - based graph cut algorithm was also compared with that of a standard pixel - based method , where the graph was constructed on each pixel and only the intensity was considered in the energy function . figure 15 shows segmentation results based on the two methods on three different nodules : ( a ) one large vascular ggo nodule , ( b ) one part - solid nodule , and ( c ) one solid nodule . testing was performed on a system with a 2.39ghz cpu and 2 gb memory . table 1 shows the total number of vertices used in the graph cut algorithm and the computation time for both of pixel - based and superpixel - based methods . it is noted that the computation time in table 1 includes construction of the graph and energy minimization . the majority of the computation time is in the graph construction , which includes the calculation of the both energy terms for each vertex . for the superpixel - based method , , it can be seen that , on the superpixel - based graph , the fewer vertices result in a much faster run - time . furthermore , from figure 15 , it is noted that on the pixel - based graph , only the core part of the nodule object is segmented while the outside part is wrongly labeled as a nonnodule object as shown in the middle row for each nodule figures 15(a)15(c ) . however , the proposed method shown in the third row for each nodule can correctly identify the nodules , separating nodules from the adjoining vessels . since the superpixels from the five - dimensional jsis mean shift algorithm express the local structure of the data , it produces better results and improves the speed significantly . in this section , the proposed algorithm has been evaluated on a large set of 130 thoracic ct scans with a slice thickness ranging from 0.5  mm to 2.0  mm . each scan was read individually by three experienced thoracic radiologists to produce a gold standard of 181 nodules ( 101 solid nodules and 80 ggo nodules ) . among those solid nodules , 28 nodules are isolated nodule ( in this paper , isolated nodule is a nodule not located near distracting structures of similar intensity , like large blood vessels , the pleural wall , etc . ) , 53 are vascular nodules , and another 20 are pleural nodules . the size of the nodules ranged between 5  mm to 20  mm in diameter . to produce the ground truth , dice 's coefficient ( r ) between each segmented nodule and the ground truth nodule is calculated as follows : 
 ( 11)r=2||vsvg||||vs||+||vg|| , 
							 where vs and vg are the segmented nodule object and the ground truth object . the notation |||| gives the total number of voxels in the region , and the operator  is an intersection . figure 17 shows the overall distribution of   the coefficients ( r ) calculated based on ( 11 ) for all solid nodules ( 101 nodules ) using the proposed method and the method without the shape feature . it is noted that , without the shape feature , the mean dice coefficient for the whole dataset is 0.71 with standard deviation (  ) of 0.1 . however , using the proposed method , the mean dice coefficient has been increased to 0.79 with the  decreasing to 0.06 .   for comparison , we split all the solid nodules into three different types ( isolated , vascular and pleural nodules ) and   evaluated the segmentation performance for each type separately . table 2 is the summary of dice coefficients for each of the types of nodules , where the mean and standard deviation are calculated for the two different methods .   for the isolated nodules , both methods give good results . for these nodules , the mean dice coefficient for the 4d method is 0.80 , and 0.81 for the 5d method . as can be seen in figure 16(a ) , in general , isolated nodules have high contrast and well - defined boundaries , so the image intensity is the predominant feature , hence both methods obtain similarly good results . however , the proposed 5d method provides better results in the presence of pve .   in such cases , by considering the shape feature , the parts of the nodule with low intensity but similar shape can be successfully segmented .    for both vascular nodules and pleural nodules , the proposed method gives a much better mean dice coefficient . especially for vascular nodules , the mean dice coefficient increased from 0.68 to 0.8 with  decreasing from 0.1 to 0.06 using the proposed method . in general , vascular nodule is embedded in a complex surrounding anatomy with similar intensities , and considering the shape feature provides rich information for the accurate segmentation . for pleural nodules , the mean dice coefficient is 0.73 for the proposed method and 0.67 without the shape feature . among these 20 pleural nodules , however , figure 18 shows an example of a small pleural nodule segmentation based on the proposed method , where the segmented nodule missed a small part of the nodule on the last slice , and figure 19 shows a case where the segmented nodule boundary includes a small amount of lung wall tissue . the proposed method has also been evaluated on 80 ggo nodules ( with pure and part opacified components ) . most ggo nodules exhibit low contrast , irregular shapes and are often attached to vessels with very similar intensities . figure 20 shows the overall distribution of the dice coefficients computed between each segmented ggo nodule and the ground truth , based on the proposed method . the proposed method provides a robust approach that combines both of local shape features and intensity . experimental results presented in the previous section demonstrate the promise and generalization of the proposed method to different types of lung nodules in ct .   most of nodules can be properly delineated from the lung parenchyma despite the presence of other nontarget structures such as vessels or the lung wall .    based on the quantitative evaluation shown in table 2 , the mean dice coefficient of the proposed method for both isolated nodules ( 0.81 ) and vascular nodules ( 0.8 ) is on average higher than that of pleural nodules . moreover , for the vascular and pleural nodules , the mean dice coefficient has been greatly improved when considering the shape feature , compared to without shape feature . figures 18 and 19 show two small pleural nodules for which the proposed method either missed a small amount of nodule or included a small amount of adjoining tissues . we further investigated these cases for probable causes .   in the mean shift clustering step , different fixed bandwidths are used in ( 3 ) for both the intensity and shape features , respectively . however , a fixed bandwidth might not be ideal , as a bandwidth that is too large might create superpixels that are too large , grouping small differences in intensity or shape that should otherwise be ungrouped . this can lead extraneous parts included in the final segmentation . to improve the performance , variable bandwidths for mean figures 12 and 15(a ) show our visual results on two ggo nodules , and figure 20 provides a quantitative measure on 80 ggo nodules . the mean dice coefficient of ggo nodules is , as expected , relatively lower compared to that of solid nodules . however , it is more difficult to segment ggo nodules due to the large variations in the intensities and vague boundaries .   as an initial study , both in terms of a visual analysis and a quantitative evaluation on a relatively large dataset in general , ggo nodule has shown certain opacity pattern as shown in figures 12 or   15(a ) ; texture features such as grey level cooccurrence ( glcm ) can be incorporated into mean shift clustering for the further improvement of the ggo segmentation .   also , in this paper , the initial foreground seeds for graph cut segmentation are automatically obtained based on spherical concentration . for our data , the initialization , described in section 3 , never failed completely , that is , the foreground initialization always returned some seeds within the nodule object for subsequent processing . in most cases , the initialization provided good foreground and background seeds for the graph construction and cut . the concentration of the high shape index values can either cover the core part of the object ( e.g. , solid nodules ) or scatter within the nodule ( e.g. , some part - solid or ggo nodules ) . however , in the case of some part - solid or ggo nodules , the spherical concentration might be on the nodule boundary , which contains part of the nodule as well as the lung tissue . an example can be seen in figure 21 , where the initial foreground is on the top of the nodule border that contains some nonnodule parts , while the automatically obtained initial background also contains part of the nodule object . that can lead to the final segmentation missing part of the lesion . to resolve this issue , we are investigating a robust method which not only considers the shape feature but also intensity for the automatic generation initial seeds . the polyp boundaries can be accurately delineated from the adjoining nonpolyp tissues with very similar intensity . however , further analysis on a large dataset with different types of colonic polyps is required to test the generalization of the proposed method on this anatomy . a five - dimensional jsis mean shift clustering is firstly used to produce superpixels comprised of intensity and shape index mode maps . a graph cut algorithm is then applied using a novel energy formulation that considers not only image intensity but also shape . the jsis features provide rich information for lesion segmentation . both by visual inspection on different types of lesions ( lung nodules and colonic polyps ) , as well as using a quantitative evaluation on 101 solid nodules and 80 ggo nodules , demonstrate the potential of the proposed method . the method can not only successfully segment lesions adjacent to structures of similar intensity but different shape , but also can correctly identify some part of lesions with different intensity ( due to pve in ct imaging ) but similar shape .
this paper presents a new , automatic method of accurately extracting lesions from ct data . 
 it first determines , at each voxel , a five - dimensional ( 5d ) feature vector that contains intensity , shape index , and 3d spatial location . 
 then , nonparametric mean shift clustering forms superpixels from these 5d features , resulting in an oversegmentation of the image . finally , a graph cut algorithm groups the superpixels using a novel energy formulation that incorporates shape , intensity , and spatial features . 
 the mean shift superpixels increase the robustness of the result while reducing the computation time . 
 we assume that the lesion is part spherical , resulting in high shape index values in a part of the lesion . from these spherical subregions , foreground and background seeds for the graph cut segmentation 
 can be automatically obtained . 
 the proposed method has been evaluated on a clinical ct dataset . 
 visual inspection on different types of lesions ( lung nodules and colonic polyps ) , as well as a quantitative evaluation on 101 solid and 80 ggo nodules , both demonstrate the potential of the proposed method . 
 the joint spatial - intensity - shape features provide a powerful cue for successful segmentation of lesions adjacent to structures of similar intensity but different shape , as well as lesions exhibiting partial volume effect .