Patent Publication Number: US-11657502-B2

Title: Systems and methods for segmentation of anatomical structures for image-guided surgery

Description:
RELATED APPLICATIONS 
     This patent application is a continuation of U.S. application Ser. No. 16/289,103, filed Feb. 28, 2019, which claims priority to and the benefit of the filing date of U.S. Provisional Patent Application No. 62/637,232, entitled “Segmentation of Renal Structures for Image-Guided Surgery,” filed Mar. 1, 2018 and of U.S. Provisional Patent Application 62/638,831, entitled “Segmentation of Anatomical Structures for Image-Guided Surgery,” filed Mar. 5, 2018, all of which are incorporated by reference herein in their entirety. 
    
    
     TECHNICAL FIELD 
     The present disclosure is directed to systems and methods for segmenting an anatomic image. 
     BACKGROUND 
     Minimally invasive surgical systems enable surgeons to perform minimally invasive surgeries more dexterously and precisely and provide a platform to enable image-guided surgery. For example, a minimally invasive surgical system may include an endoscope held by a teleoperational arm which provides a stable view of the patient anatomy and is tracked in real-time through the robotic kinematics. Images from the endoscope may be augmented with pre-operative or intra-operative images generated by other imaging modalities and displayed to an operator to integrate image information from multiple sources. 
     Anatomic models generated from pre-operative or intraoperative images, especially three-dimensional models, may illustrate the relationship of the different structures and help with planning the best strategy of the surgery. During the procedure, the model can provide some real-time guidance. For example, by identifying arteries and a tumor, it is possible to find out which major arteries feed the tumor to enable selective arterial clamping. In another example, a model enables the awareness of critical structures (major blood vessels and collecting systems) close to the tumor and helps reduce surgical complications. In another example, images of tumors not visible from outside of an organ are overlaid on the image of the organ to help the surgeons to quickly localize them. 
     Efficient and low-cost segmentation of anatomical structures is a prerequisite for many image guidance systems. Manual segmentation by using interactive tools can be very time consuming which prohibits the wide spread of such use. Techniques for efficiently segmenting anatomical image data are needed. 
     SUMMARY 
     Some embodiments of the invention are best summarized by the claims that follow the description. 
     Consistent with some embodiments, a method for image segmentation comprises receiving volumetric image data for an anatomical region and generating a first volumetric patch from the volumetric image data. The method also includes generating a second volumetric patch from the first volumetric patch by weighting a plurality of volumetric units in the first volumetric patch and receiving the second volumetric patch as an input to a convolutional neural network. The method also comprises conducting a down-sampling filter process and conducting an up-sampling filter process within the convolutional neural network. 
     Consistent with some embodiments, a non-transitory machine-readable medium comprises a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising receiving volumetric image data for an anatomical region and generating a first volumetric patch from the volumetric image data. The method also includes generating a second volumetric patch from the first volumetric patch by weighting a plurality of volumetric units in the first volumetric patch and receiving the second volumetric patch as an input to a convolutional neural network. Within the convolutional neural network, a down-sampling filter process and an up-sampling filter process are conducted. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description. 
    
    
     
       BRIEF DESCRIPTIONS OF THE DRAWINGS 
         FIG.  1    illustrates a method for image segmentation according to some embodiments. 
         FIG.  2 A  illustrates an anatomic image generated from anatomic image data according to some embodiments. 
         FIG.  2 B  illustrates the anatomic image of  FIG.  2 A  with segmented and localized anatomical features according to some embodiments. 
         FIG.  3 A  illustrates an anatomic image generated from anatomic image data according to some embodiments. 
         FIGS.  3 B and  3 C  illustrate the anatomy of  FIG.  2 A  with segmented and localized anatomical features according to some embodiments. 
         FIG.  4    illustrates a method for training an artificial neural network according to some embodiments. 
         FIG.  5    illustrates a multi-scale entropy map of anatomical features according to some embodiments. 
         FIG.  6    illustrates a background sampling approach according to some embodiments. 
         FIG.  7    illustrates a residual architecture for a directed acyclic graph according to some embodiments. 
         FIG.  8    illustrates a UNet architecture according to some embodiments. 
         FIG.  9    illustrates a residual UNet architecture according to some embodiments. 
         FIG.  10    illustrates a simplified diagram of a robotic or teleoperated medical system according to some embodiments. 
     
    
    
     Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same. 
     DETAILED DESCRIPTION 
     In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional. 
       FIG.  1    illustrates a method  100  for image segmentation according to some embodiments. The method  100  is illustrated as a set of operations or processes  102  through  108 . Not all of the illustrated processes  102  through  108  may be performed in all embodiments of method  100 . Additionally, one or more processes that are not expressly illustrated in  FIG.  1    may be included before, after, in between, or as part of the processes  102  through  108 . In some embodiments, one or more of the processes may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes. 
     At a process  102 , anatomic image data for an anatomical region is received by a processor. Various imaging techniques may be used to acquire anatomical image data of a patient anatomy for use in a variety of medical procedures including surgical, diagnostic, therapeutic procedures. For example, anatomical image data may be acquired using imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and/or the like. Anatomical image data may be acquired preoperatively or intraoperatively. In some embodiments, the anatomic image data may be volumetric image data representing a three-dimensional model of a patient anatomic region. With reference to  FIGS.  2 A and  3 A , anatomic images  120 ,  130  may be generated from anatomic image data such as CT image data. Although the illustrations of this disclosure may refer to anatomical structures of the kidney, the systems and methods described herein may be applied to other anatomical regions including the lung, the heart, the colon, the intestines, the brain, the circulatory system including vasculature, and/or the like. 
     At a process  104 , training patches are generated from the anatomic image data. The training patches are comprised of a plurality of graphical units (e.g. pixels or voxels). In some embodiments the training patches are three-dimensional volumetric patches comprised of a plurality of voxels. In one embodiment, for example, each training patch may have a 96×96×96 voxel dimension. 
     At a process  106 , all or some subset of the training patches may be used as inputs to train an artificial neural network. The artificial neural network may be, for example, a deep convolutional neural network, including multiple convolution layers with each layer including a convolution filter kernel. In these layers, the filter kernels are applied to the input patches to thereby perform a convolution operation and generate a feature map. The convolution operation may be followed by a pooling operation to reduce the dimensionality of the feature map. In some embodiments, the artificial neural network may be a U-shaped network that performs down-sampling filter operations to detect features within the anatomical image data and up-sampling filter operations (i.e., deconvolution) to localize the detected features by assigning classification information to the detected features. 
     At a process  108 , one or more segmentation maps are generated by the trained convolutional neural network. The output segmentation maps include segmented anatomical features. In some embodiments the output segmentation maps may displayed on a display system. In some embodiments, the output segmentation maps may be displayed with the anatomic image data. For example, in some embodiments, all or some of the segmented anatomical features may be superimposed on the anatomic image data or displayed side-by-side with the anatomic image data. With reference to  FIGS.  2 B,  3 A, and  3 B , segmentation maps  121 ,  131 ,  133  are generated from trained convolutional neural networks. The segmentation map  121  is generated from the anatomic image  120 . Segmented anatomical features include arteries  124 , veins  122 , and ureter tissue  126 . In this embodiment, the segmented anatomical features  124 ,  122 ,  126  are displayed with the anatomic image  120 . The segmentation maps  131 ,  133  are generated from the anatomic image data used to generate anatomic image  130 . The segmentation map  133  includes the segmented anatomical features of parenchyma tissue  132 , arteries  134 , veins  136 , masses  138  including abnormal tissue such as tumors, kidney stones, or cysts, and ureter tissue  140 . In some embodiments, the displayed segmentation map may omit certain ones of the segmented features. For example, the segmentation map  133  may omit the parenchyma tissue  132  to allow the viewer to better observe the ureter tissue structure  140 . 
       FIG.  4    illustrates, in greater detail, the process  106  for training an artificial neural network according to some embodiments. The process  106  is illustrated as a set of operations or sub-processes  152  through  160 . Not all of the illustrated sub-processes may be performed in all embodiments of process  106 . Additionally, one or more sub-processes that are not expressly illustrated in  FIG.  4    may be included before, after, in between, or as part of the sub-processes  152  through  160 . In some embodiments, one or more of the sub-processes may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the sub-processes. 
     In some embodiments, all or a randomly generated subset of the plurality of volumetric patches may be used to train the artificial neural network. The patches and/or the voxels within the patches may be uniformly weighted such that no particular consideration is given to the content of the patches and voxels. This type of uniform weighting in model training may yield unsatisfactory segmentation results. For example, it will tend to label voxels into some classifications at percentages overwhelmingly greater than other classifications. Besides this inter-class unbalance, it may also create problems for intra-class voxels. For example, it will undesirably make small anatomical structures (e.g., thin vessels) less important than larger anatomical structures (e.g., thick vessels) simply because the smaller anatomical structures are comprised of fewer voxels. As described below, various techniques may be used to assign weights to voxels, groups of voxels, or entire patches to ensure the artificial neural network is trained to balance foreground and background classifications, emphasize structure edges, and emphasize complicated or detailed anatomical structures. Foreground classifications may be associated with anatomical structures of greatest interest. In the kidney for example, foreground classifications may be assigned to anatomical structures such as parenchyma tissue, ureter tissue, arteries, veins, and masses including abnormal tissue such as tumors, kidney stones, or cysts. 
     At a sub-process  152 , the content of the plurality of volumetric patches may be evaluated to determine weights for the voxels in each patch. In one embodiment and with reference to  FIG.  5   , a multi-scale categorical entropy map  170  is generated. The voxels associated with high entropy properties may be evaluated to have a greater value and therefore, be assigned a higher weighting. The voxel-wise weight map  170  may emphasize any of a variety of properties. For example, in the map  170  edges  172  may be important as they define a boundary. Complicated structures  174  (e.g., branching thin vessels), though possibly small, may be important as they indicate the source of blood flow to tissues. Additionally, the farther away a point is from the foreground, the less important it may be. The entropy map generally provides an indication of structural detail between scale levels. For example, if the difference in detail between scale levels is great (e.g., great detail in foreground structures between zoom resolution levels), entropy is high. If the difference in detail between scale levels is small (e.g., all background between zoom resolution levels) entropy is low. 
     Given a multi-channel probabilistic map y and a series of increasing scale levels σ 0 , . . . , σ n , the multi-scale entropy s is defined as: 
                       y   δ     =       k   σ     *   y       ,       for   ⁢           ⁢   σ     ∈     {       σ   0     ,   …   ⁢           ,     σ   n       }       ,           (   1   )                   RelativeEntropy     σ   ❘     σ     i   +   1           ⁡     (   p   )       =       ∑   c     ⁢           y     σ   i       ⁡     (   p   )       ⁡     [   c   ]       ⁢           y     σ   i       ⁡     (   p   )       ⁡     [   c   ]               y     σ     i   +   1         ⁡     (   p   )       ⁡     [   c   ]       ′                   (   2   )                   s     σ   ❘     σ     i   +   1           =       k            σ     i   +   1       -     σ   i              *     RelativeEntropy     σ   ❘     σ     i   +   1               ,           (   3   )                 s   =       ∑   i     ⁢     s     σ   ❘     σ     i   +   1               ,           (   4   )               
where K σ  is a Gaussian kernel of width δ, and y σi  (p)[c] is c-th channel&#39;s probability at a point p in y σi . Equation (1) generates a series of smoothed probabilistic maps y δ s from the original map y. Equation (2) calculates the relative entropy between two smoothed probabilistic maps y σi  and y σi+1 . It captures how complex the label map is at a scale. Equation (3) smooths the relative entropy map to the resolution of σ 0i+1 . Equation (4) sums up relative entropy at different scales. σ 0 , . . . , σ n  usually increase exponentially. The entropy map  170  shows an entropy map derived at σ=1, 2, 4, 8, 16, 32, 64, 128, and 256, emphasizing edges  172 , complicated structures  174  and areas near the foreground  176 .
 
     In another embodiment, voxels associated with certain foreground classifications may be evaluated to have a greater value and therefore, be assigned a higher weighting. Both intra-foreground and background-foreground classes may be balanced within volumetric patches. In many embodiments, a majority of patches are free of foreground structures. Even when a foreground class exists, it occupies a small percentage. To hinder bias against the foreground classes, voxels may be weighted based on their classification and/or based on patch significance. A major challenge in medical image processing is detecting small structures. Vessels in direct contact with organs are tiny and their resolution is limited by acquisition. Tiny vessels are challenging to annotate and more valuable to acquire. Thus, patches containing smaller foreground structures such as vessels are more significant and may be assigned a higher weight. Patch weight may be inversely proportional to the foreground structure volume (e.g., vessel volume) inside the patch. Foreground classes may also be assigned higher weights to hinder the network background bias. During the evaluation sub-process  152  the average foreground class volumes may be dynamically measured per patch. Then, higher weights may be assigned to classes with smaller average volumes and vice versa. 
     In one embodiment, a policy that a vessel volume×weight must equal 1/n is imposed where n is the number of classes including the background class. Thus, all classes contribute equally from a network perspective. To account for data augmentation, vessel volumes are measured during training. Enforcing equal contribution (volume×weight) from all classes may be an objective. To do so, a moving average procedure may be used: 
     
       
         
           
               
             
               
                   
               
               
                 Algorithm 1 
               
               
                 Our proposed moving average procedure assigns voxels  
               
               
                 dynamic weights (VW c ) based on their classes and patch weight. 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 Require: α : Convergence Rate  
               
               
                   
                 Require: P : Current 3D patch with axis x, y, z  
               
               
                   
                 Require: n : Number of classes (background included)  
               
               
                   
                 Require: V c  : Class (c) moving average volume  
               
               
                   
                 Require: PW : Current patch weight  
               
               
                   
                 Require: CW c  : Class (c) weight  
               
               
                   
                  for all c in classes do  
               
               
                   
                   // Measure class (c) volume in patch P  
               
               
                   
               
               
                   
                   
           V   c     ⁡     (   P   )       =       (         ∑   x     ⁢       ∑   y     ⁢       ∑   z     ⁢     P   ⁡     (     x   ,   y   ,   z     )             ==   c     )     /     size   ⁡     (   P   )             
 
               
               
                   
               
               
                   
                   // Update class (c) moving average volume  
               
               
                   
                   V c  = V c  × (1 − α) + V c (P) × α 
               
               
                   
                   // Update class weight based on its moving average volume 
               
               
                   
                   CW c  = 1/(n × V c ) 
               
               
                   
                  end for  
               
               
                   
                  // Set patch weight based on foreground volume  
               
               
                   
                  if P contains background only then  
               
               
                   
                   PW = 1  
               
               
                   
                  else  
               
               
                   
               
               
                   
                   
                       //     ⁢           ⁢   Foreground   ⁢           ⁢   volume   ⁢           ⁢       ∑     c   =   1       n   -   1       ⁢       V   c     ⁡     (   P   )           &lt;   1               PW   =     1   -     log   (       ∑     c   =   1       n   -   1       ⁢       V   c     ⁡     (   P   )         )                   
 
               
               
                   
               
               
                   
                  end if  
               
               
                   
                  VW c  = PW * CW c  (Voxel weight is function of PW and CW c ) 
               
               
                   
               
               
                 Patch Weight (PW) is inversely proportional to its foreground volume. 
               
               
                                     Class   ⁢           ⁢   weight   ⁢           ⁢     (   CW   )     ⁢           ⁢   is   ⁢           ⁢   inversely   ⁢           ⁢   proportional   ⁢           ⁢   to   ⁢           ⁢   its   ⁢           ⁢   average   ⁢           ⁢   volume   ⁢           ⁢   per               patch   ;       initially   ⁢           ⁢     V   c       =       1   n     ⁢           ⁢   for   ⁢           ⁢   every   ⁢           ⁢   class   ⁢           ⁢     c   .                     
Our settings α = 0.001, n = 4. 
               
            
           
         
       
     
     Because a large volume of voxels have background classifications, those voxels may be assigned a small class weight. Consequently, many of false classifications may be generated because it is relatively cheap to mis-classify a background voxel. To overcome this undesired phenomena, a random background sampling technique may be used to weight background voxels. With this technique, background voxels are sampled at random and assigned high weights. This technique may be effective in limiting false positives because a high loss is incurred for these mis-classified voxels. With reference to the sampling schema  190  of  FIG.  6   , a band  192  and a band  194  are constructed around a foreground anatomical structure  196  (e.g., a vessel) using morphological kernel, a binary circular radial distance of two and four voxels. Misclassifications within the band  192  (e.g., &lt;2 voxels away from the vessel) are considered marginal errors. In a given patch, sampled background voxels  198  are equivalent to the foreground vessel volume, where 20% and 80% come from the band  194  and the volume beyond this band respectively. If a patch is free foreground voxels, 1% of the background voxels are randomly sampled. 
     With reference again to the process  106  of  FIG.  4   , at a sub-process  154 , any of the weighting schema described above may be applied to the voxels in each patch to be used for training the artificial neural network. The weighted volumetric patch generated by weighting the first volumetric patch may be considered a second volumetric patch. 
     At a sub-process  156  the weighted volumetric patches may be input to the convolutional filters of the artificial neural network. In various embodiments, the artificial neural network may be a convolutional neural network such as a residual UNet. 
     At a process  158 , the residual UNet performs down-sampling processes as described below. At a process  160 , the residual UNet performs an up-sampling process and described below. 
     In greater detail with regards to the residual UNet architecture, a residual architecture for directed acyclic graphs is fully compatible with a sequential residual network. Within the residual architecture, let G={V,{right arrow over (E)}} be a directed acyclic graph where V is its vertices and {right arrow over (E)} is its directed edges. Given a vertex α∈V, let ancestors (α), excluding a itself, denote the vertices in V which can reach α via directed paths in {right arrow over (E)}. The residual graph derived from G is composed by functions on vertices. The function F α  on a vertex α takes the summation of ancestors (α) as its input, that is:
 
input of  f   α =Σ b∈ancestors(a)   f   b  
 
 FIG.  7    illustrates a residual graph  200  where the input of f 4  is the summation of f 1  and f 2 ; the input of f 5  is the summation of f 2  and f 3 ; and the input of f 6  is the summation of f 1 , f 2 , . . . , f 5 .
 
     In various embodiments, UNet used for image segmentation is a deep neural network using multi-scale analysis.  FIG.  8    illustrates the UNet architecture  210  according to some embodiments. In this embodiment, an image is processed at different scales. At one scale k, there are two filters f pre  and f post . The filter f pre  takes its input from a lower scale k−1, and then relays its output to a higher scale k+1. (Shifting from low scales to high scales is implemented by down-sampling or strided convolution.) In this way, features are summarized from low scales to high scales. The filter f post  takes its input from both f pre  and a higher scale k+1 and relays its output to a lower scale k−1. (Shifting from high scales to low scales is implemented by up-sampling or deconvolution convolution.) In this way, local information is combined with high scale context information. 
       FIG.  9    illustrates a ResUNet architecture  220  which includes aspects of the residual graph  200  with the UNet architecture  210 . Data flows from low scales to high scales through two channels: channel  222  which is scale-specific features and channel  224  which is the accumulation of supervision features. The channels  222 ,  224 , concatenated together, are projected to a high-dimensional feature space through f preproj  and then compressed by f prepsup . The output of f prepsup  is added to the channel  224  to keep the accumulation updated. The channels  222 ,  224 , after f preproj  and f prepsup , are forwarded to a higher scale k+1, respectively through strided convolution and down-sampling. Data flows from high scales to low scales through  222  and channel  226 , respectively through de-convolution and up-sampling. The channel  226  is the accumulation of features from the last scale to the current scale k. The channel  226  is added to the channel  224  to keep the accumulation updated. The accumulated features, concatenated with the channels  222 , are projected to a high-dimensional feature space through f postproj , and then compressed by f postsup . The outputs of f presup  and f postsup  added to the channel  226  to keep it as the accumulation from the current scale to the last scale. When the channel  226  travels to the first scale, it is the accumulation of all scales. 
     The ResUNet architecture  220  consists of two threads. The first thread is scale-specific features: the channels  222 . It follows a similar architecture to the UNet architecture  210 . It branches at f preproj  and then merges at f postproj . The second thread is the residual architecture, including the channels  224 ,  226 ,  228 . The pair of f preproj  and f prepsup  and the pair of f presup  and f postsup  are considered as processing units in the residual architecture. f presup  and f postsup  produces the output of a processing unit. f preproj  or f postproj  only produces intermediate features. The input to a processing unit always includes summation of f presup  or f postsup  from its ancestor processing units. 
     In some embodiments, image segmentation may be used in an image-guided medical procedure performed with a teleoperated medical system as described in further detail below. While some embodiments are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. In some embodiments, the segmented tree structures may be used for non-teleoperational procedures involving guidance of traditional manually operated medical instruments. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems and general robotic, general teleoperational, or robotic medical systems. As shown in  FIG.  7   , medical system  500  generally includes a manipulator assembly  502  for operating a medical instrument  504  in performing various procedures on a patient P positioned on a table T. The manipulator assembly  502  may be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with select degrees of freedom of motion that may be motorized and/or teleoperated and select degrees of freedom of motion that may be non-motorized and/or non-teleoperated. Master assembly  506  generally includes one or more control devices for controlling manipulator assembly  502 . Manipulator assembly  502  supports medical instrument  504  and may optionally include a plurality of actuators or motors that drive inputs on medical instrument  504  in response to commands from a control system  512 . The actuators may optionally include drive systems that when coupled to medical instrument  504  may advance medical instrument  504  into a naturally or surgically created anatomic orifice. Other drive systems may move the distal end of medical instrument  504  in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the actuators can be used to actuate an articulable end effector of medical instrument  504  for grasping tissue in the jaws of a biopsy device and/or the like. 
     Teleoperated medical system  500  also includes a display system  510  for displaying an image or representation of the surgical site and medical instrument  504  generated by sub-systems of sensor system  508 . Display system  510  and master assembly  506  may be oriented so operator O can control medical instrument  504  and master assembly  506  with the perception of telepresence. 
     In some embodiments, medical instrument  504  may include components of an imaging system, which may include an imaging scope assembly or imaging instrument that records a concurrent or real-time image of a surgical site and provides the image to the operator or operator O through one or more displays of medical system  500 , such as one or more displays of display system  510 . The concurrent image may be, for example, a two or three-dimensional image captured by an imaging instrument positioned within the surgical site. In some embodiments, the imaging system includes endoscopic imaging instrument components that may be integrally or removably coupled to medical instrument  504 . However, in some embodiments, a separate endoscope, attached to a separate manipulator assembly may be used with medical instrument  504  to image the surgical site. The imaging system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, which may include the processors of the control system  512 . 
     Teleoperated medical system  500  may also include control system  512 . The artificial neural network may operate within the control system  112 . Control system  512  includes at least one memory and at least one computer processor (not shown) for effecting control between medical instrument  504 , master assembly  506 , sensor system  508 , and display system  510 . Control system  512  also includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein, including instructions for providing information to display system  510 . 
     Control system  512  may optionally further include a virtual visualization system to provide navigation assistance to operator O when controlling medical instrument  504  during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based upon reference to an acquired preoperative or intraoperative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and/or the like. 
     One or more elements in embodiments of this disclosure may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code segments may be downloaded via computer networks such as the Internet, Intranet, etc. Any of a wide variety of centralized or distributed data processing architectures may be employed. Programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. In one embodiment, the control system supports wireless communication protocols such as Bluetooth. IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry. 
     Note that the processes and displays presented may not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein. In some instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. 
     While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.