Patent Publication Number: US-11648066-B2

Title: Method and system of determining one or more points on operation pathway

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Application No. 62/514,951, filed Jun. 4, 2017, U.S. Provisional Application No. 62/612,728, filed Jan. 2, 2018, and U.S. Provisional Application No. 62/618,053, filed Jan. 16, 2018, which are incorporated by reference in their entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     Field of the Invention 
     Embodiments of the present invention relate generally to methods and systems of determining one or more points on an operation pathway. 
     Description of the Related Art 
     Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section. 
     In an operation, a plan of an operation pathway is critical. The operation pathway may include multiple points, such as a safety point and a preoperative point away from the patient, an entry point on patient&#39;s tissues, and a target point at the target of the operation. 
     Robotic operation may offer a precise control of the operation pathway. Before the operation, patient is subjected to a medical scan (e.g., CT or MRI). The operation pathway to the desired anatomical region is planned. Artificial intelligence may be employed to suggest the surgeon with best routes that incur the least amount of damages. To perform the operation, the position of the patient may be matched to the perspective of the medical scan to accurate perform the operation along the planned operation pathway. Conventional approaches have relied on glued on or screwed in fiducial marks, which has adoption issues. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is an example figure showing the spatial relationships among several points that may be encountered during an operation; 
         FIG.  2    is a flow diagram illustrating an example process to drive robotic arm to one or more points on an operation pathway; 
         FIG.  3    illustrates an example of the operation pathway calculation; 
         FIGS.  4 A and  4 B  illustrate an image processing to process information obtained by a medical image scan; 
         FIGS.  5 A and  5 B  illustrate facial recognition method of selecting key features to identify landmark points; 
         FIG.  6    illustrates coordinate transformation from the constructed three-dimensional model coordinate system to the three-dimensional camera coordinate system; 
         FIG.  7    is a flow diagram illustrating an example process to transform coordinates; 
         FIG.  8    is a flow diagram illustrating an example process to register an optical apparatus in the robotic arm coordinate system; 
         FIG.  9    is a flow diagram illustrating an example process to move robotic arm from an initial point to a safety point on an operation pathway; 
         FIG.  10    is an example figure showing a system to register a surgical tool attached to a robotic arm flange in a robotic arm coordinate system; and 
         FIGS.  11  and  12    illustrate example images captured by a first camera and a second camera of the system to register a surgical tool attached to a robotic arm flange in a robotic arm coordinate system, all arranged in accordance with some embodiments of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein. 
       FIG.  1    is an example figure showing the spatial relationships among several points that may be encountered during an operation, arranged in accordance with some embodiments of the present disclosure. In  FIG.  1   , an operation pathway  110  may include safety point  120 , preoperative point  130 , entry point  140 , and target point  150 . 
       FIG.  2    is a flow diagram illustrating an example process  200  to drive robotic arm to one or more points on an operation pathway, arranged in accordance with some embodiments of the present disclosure. Process  200  may include one or more operations, functions, or actions as illustrated by blocks  210 ,  220 ,  230 ,  240 ,  250 ,  260 ,  270 , and/or  280 , which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. 
     Process  200  may begin at block  210 , “construct three-dimensional model based on medical image scan.” Before an operation is performed, some medical imaging techniques may be used to capture a snapshot of a patient&#39;s conditions, so that an operation plan may be formulated. The operation plan may include a planned operation pathway as set forth above. For example, the surgeon may order a medical image scan (e.g., CT or MRI) of the operation target. Such a medical image scan may be performed a few days (e.g., 3 to 5 days) prior to the operation. A three-dimensional model may be constructed based on the medical image scan data using some known approaches. Accordingly, points on the planned operation pathway may be identified in the three-dimensional model. 
     Block  210  may be followed by block  220  “perform intelligent operation pathway planning.” In some embodiments, an artificial intelligence engine may be employed to suggest the surgeon with one or more routes with minimized physical damages to the patient. Based on the patient&#39;s CT or MRI scan, the artificial intelligence engine may suggest one or more optimal operation pathway.  FIG.  3    illustrates an example of a calculation of operation pathway  310  to reach target point  320 , arranged in accordance with some embodiments of the present disclosure. The calculation may include transforming the standard brain-atlas data, and registering it to the patient&#39;s medical scan images to identify the brain regions. Some example brain regions include motor association area  331 , expressive speech area  332 , higher mental functions area  333 , motor area  334 , sensory area  335 , somatosensory association area  336 , global language area  337 , vision area  338 , receptive speech area  338 , receptive speech area  339 , association area  341 , and cerebellum area  342 . Moreover, common target tissues, such as sub-thalamic nucleus, may be automatically identified. In addition, each brain region set forth above may be assigned with a cost function for the artificial intelligence engine to suggest one or more routes to the target tissues. The blood vessels may be identified from the TOF (time-of-flight MRI) data. The points on the outer brain boundary are candidate for entry point. 
       FIGS.  4 A and  4 B  illustrate an image processing to process information obtained by the medical image scan, arranged in accordance with some embodiments of the present disclosure. In conjunction with  FIGS.  4 A and  4 B , block  220  may be followed by block  230 , “process information obtained by medical image scan,” in accordance with some embodiments of the present disclosure. The points obtained from the images captured by a three-dimensional optical apparatus are only associated with the patient&#39;s appearance information but are not associated with the patient&#39;s under-skin information. However, images obtained by medical image scan usually are associated with both the appearance and the under-skin information. Image processing is performed to remove the under-skin information from the medical image scan. Therefore, in block  260 , the appearance points obtained by the three-dimensional optical apparatus and the appearance points obtained from the medical image scan are used for the three-dimensional matching. 
     In  FIG.  4 A , in some embodiments, assuming the operation target is inside the patient&#39;s head, binary image  410  may be derived from an original MRI image along an axial direction from head to toes of the patient. Region  411  is the skull of the patient and is usually represented in white in the original MRI image. The outer periphery  412  of region  411  may refer to the patient&#39;s skin, which is associated with the patient&#39;s appearance. With thresholding approach, image  410  including region  413  outside of the skull (all black) and region  415  inside the skull may be created. Image  410  may be further processed to form image  420 . In image  420 , region  413  is assigned a gray scale to be differentiated from black and white to form region  423 . 
     In  FIG.  4 B , image  420  may be further processed to form image  430 . In image  430 , regions other than the gray scale of region  413  are assigned with black to form region  431 . Region  423  in image  430  may then be assigned with white to form image  440 . In some embodiments, points along periphery  441  of image  440  may be selected for matching points from images taken by the three-dimensional optical apparatus in block  240 . 
     Block  230  may be followed by block  240  “retrieve image of patient taken by three-dimensional optical apparatus.” The “three-dimensional optical apparatus” may generally refer to a camera or a scanner with depth sensing technologies. In some embodiments, the three-dimensional optical apparatus may include, but not limited to, at least two two-dimensional cameras. In some other embodiments, the three-dimensional optical apparatus may include, but not limited to, a three-dimensional depth sensing module and a two-dimensional camera module. In some embodiments, the three-dimensional scanner is configured to create a point cloud of geometric samples on a surface of an object to be scanned. These points may be used to extrapolate the shape of the object. In some embodiments, the three-dimensional scanner may be an intraoral scanner. 
     In some embodiments, in addition to information obtained with traditional two-dimensional optical apparatus, the three-dimensional optical apparatus may capture the depth information of the patient. In some embodiments, the three-dimensional optical apparatus may be configured to take images of the patient immediately prior to a surgical incision is made on the patient in the operating room. The three-dimensional optical apparatus may be fixed at a stationary device (e.g., a robotic arm, an operating trolley) in the operating room. 
     Block  240  may be followed by block  250  “select landmark points.” In some embodiments, one or more landmark points may be selected from the three-dimensional model constructed in block  210  and the image retrieved in block  240 . In some embodiments, 3 or more landmark points with strong features are selected. Examples of landmark points may include middle of eyes, nose tip, or mouth. In some embodiments, 3 landmark points may identify an initial three-dimensional coordinate that allows subsequent matching using iterative closest point (ICP) algorithm. In some embodiments, the landmark points may be in the same order in both the three-dimensional optical apparatus scan and the CT/MRI data (for example, 1. nose tip, 2. left eye, 3. right eye). 
     For clarity, the following discussions mainly use one non-limiting example of the three-dimensional optical apparatus, e.g., a three-dimensional camera, and its coordinate system, e.g., three-dimensional camera coordinate system, to explain various embodiments of the present disclosure. 
     In some embodiments,  FIGS.  5 A and  5 B  illustrate facial recognition methods of selecting key features to identify landmark points, arranged in accordance with some embodiments of the disclosure. Because a patient typically closes his or her eyes during surgical procedures, traditional facial recognition algorithms that rely on identifying his or her eyes with round irises cannot be used. Instead, the patient&#39;s nose, the most anterior structure, may be selected first. This is possible because the patient generally lies in the same position. From the nose tip with pre-determined z-offset, a region of interest (e.g., the rectangle illustrated in  FIG.  5 A ) is defined for selecting other landmark points (e.g., eyes). In some embodiments, the region of interest may include a region defined by the nose tip, the eyes, the forehead, and a region between the eyebrows. Accordingly, in some embodiments, from transverse (axial) cross sections, 3 or more landmark points may be identified (e.g., nose as the taller center peak, and the eye-balls as the smaller 2 peaks). 
     Block  250  may be followed by block  260 , “perform image matching.” In some embodiments, the image matching includes matching the landmark points selected in block  250 . In some embodiments, the landmark points selected from the images taken by the three-dimensional optical apparatus and the landmark points selected from the constructed three-dimensional model are matched, sometimes iteratively to minimize the differences between the two sets of landmark points. In some embodiments, the landmark points used for matching may encompass all the available surface data. In other embodiments, the landmark points used for matching may be a chosen subset of the surface data. For example, the landmark points for matching may be repeatedly selected from the region of interest (e.g., the rectangle illustrated in  FIG.  5 A ) of the images taken by the three-dimensional optical apparatus and the region of interest of the constructed three-dimensional model, instead of repeatedly selecting from all the available surface data. 
     Block  260  may be followed by block  270 , “transform coordinates.” In block  270 , selected points from the constructed three-dimensional model (e.g., P1, P2 and P3) are transformed from their original coordinate system (i.e., three-dimensional model coordinate system) to the coordinates of the images taken by the three-dimensional optical apparatus (i.e., three-dimensional camera coordinate system). The transformation may be based on some image comparison approaches, such as iterative closest point (ICP). Block  270  may further include additional coordinate transformations in which all points on the three-dimensional camera coordinate system are transformed to the coordinates of the robotic arm (i.e., robotic arm coordinate system). The details of transforming coordinates will be further described below. 
     Block  270  may be followed by block  280 , “drive robotic arm to points on operation pathway.” In block  280 , the coordinates of the planned operation pathway in three-dimensional model coordinate system may be transformed to the robotic arm coordinate system. Therefore, the robotic arm may move to the safety point, the preoperative point, the entry point, and/or the target point on the planned operation pathway. 
     In sum, a three-dimensional camera or scanner may be used to obtain a patient&#39;s facial features. The facial features may then be compared with processed image data associated with a medical image scan. To compare, landmark points are selected in the same sequence in both the images obtained by the three-dimensional camera and the images associated with the medical image scan. In some embodiments, example process  200  may be applied to various types of operations, such as brain operations, nervous system operations, endocrine operations, eye operations, ears operations, respiratory operations, circulatory system operations, lymphatic operations, gastrointestinal operations, mouth and dental operations, urinary operations, reproductive operations, bone, cartilage and joint operations, muscle/soft tissue operations, breast operations, skin operations, and etc. 
       FIG.  6    illustrates an example coordinate transformation from the constructed three-dimensional model coordinate system to the three-dimensional camera coordinate system, in accordance with some embodiments of the present disclosure. This figure will be further discussed below in conjunction with  FIG.  7   . 
       FIG.  7    is a flow diagram illustrating an example process  700  to transform coordinates, in accordance with some embodiments of the present disclosure. Process  700  may include one or more operations, functions, or actions as illustrated by blocks  710 ,  720 ,  730 , and/or  740 , which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. 
     In conjunction with  FIG.  6   , in block  710 , initial matrices are obtained. In some embodiments, a first initial matrix T MRI  and a second initial matrix T camera  are obtained. In some embodiments, 
                   TMRI   =     [           Vector     X   x             Vector     Y   x             Vector     Z   x             P     1   ⁢           ⁢   x                 Vector     X   y             Vector     Y   y             Vector     Z   y             P     1   ⁢           ⁢   y                 Vector     X   z             Vector     Y   z             Vector     Z   z             P     1   ⁢           ⁢   z               0       0       0       1         ]             (   1   )               
in which
 
 P   1   P′   2     norm     ×P   1   P′   3     norm   =Vector Y  
 
 P   1   P′   2     norm   ×Vector Y =Vector Z  
 
 P   1   P′   2     norm   =Vector X  
 
Vector Xx  is the x component of Vector X , Vector Xy  is the y component of Vector X , and Vector Xz  is the z component of Vector X . Similarly, Vector yx  is the x component of Vector y , Vector yy  is the y component of Vector y , and Vector yz  is the z component of Vector y . Vector Zx  is the x component of Vector Z , Vector Zy  is the y component of Vector Z , and Vector Zz  is the z component of Vector Z . P1 X  is the x coordinate of P1, P1 y  is the y coordinate of P1, and P1 z  is the z coordinate of P1.
 
     In some other embodiments, 
                   Tcamera   =     [           Vector     X   x   ′             Vector     Y   x   ′             Vector     Z   x   ′             P       1   ′     ⁢   x                 Vector     X   y   ′             Vector     Y   y   ′             Vector     Z   y   ′             P       1   ′     ⁢   y                 Vector     X   z   ′             Vector     Y   z   ′             Vector     Z   z   ′             P       1   ′     ⁢   z               0       0       0       1         ]             (   2   )               
in which
 
                       P   1     ⁢     P   2       →     norm     ×           P   1     ⁢     P   3       →     norm       =     Vector   Y                           P   1     ⁢     P   2       →     norm     ×     Vector   Y       =     Vector   Z                         P   1     ⁢     P   2       →     norm     =     Vector   X           
Vector X′x  is the x component of Vector X′ , Vector X′y  is the y component of Vector X′ , and Vector X′z  is the z component of Vector X′ . Similarly, Vector y′x  is the x component of Vector y′ , Vector y′y  is the y component of Vector y′ , and Vector y′z  is the z component of Vector y′ . Vector Z′x  is the x component of Vector Z′ , Vector Z′y  is the y component of Vector Z′ , and Vector Z′z  is the z component of Vector Z′ . P1′ X  is the x coordinate of P1′, P1′ y  is they coordinate of P1′, and P1′ z  is the z coordinate of P1′.
 
     Block  710  may be followed by block  720 , “obtain conversion matrix.” In some embodiments, the conversion matrix may be T camera  T MRI   −1  and P1, P2, and P3 are transformed to the three-dimensional camera coordinate system according to T camera  T MRI   −1 . Assuming P1, P2, and P3 are transformed to P1 transformed , P2 transformed , and P3 transformed , respectively, a distance metric associated with differences between P1 transformed  and P1′, P2 transformed  and P2′, and P3 transformed  and P3′ is calculated based on some feasible ICP approaches. 
     Block  720  may be followed by block  730 . In block  730 , whether the change of the distance metric reaches a threshold is determined. If the threshold is not reached, block  730  may go back to block  720  in which P1 transformed , P2 transformed , and P3 transformed  are selected to update T camera  and eventually obtain new conversion matrix T camera  T MRI   −1  If the threshold is reached, block  730  may be followed by block  740 . 
     In block  740 , a transform matrix is obtained to transform points from the three-dimensional camera coordinate system to the robotic arm coordinate system. In some embodiments, the transform matrix 
                     T   robot     =     [                                               P     C   x                           R                     P     C   y                                                     P     C   z               0       0       0       1         ]             (   5   )               in   ⁢           ⁢   which                             R   =     I   +       (     sin   ⁢           ⁢   θ     )     ⁢   K     +       (     1   -     cos   ⁢           ⁢   θ       )     ⁢     K   2           ⁢     
     ⁢     K   =     [         0           -     k   z              k   →                    k   y            k   →                        k   z            k   →                0           -     k   x              k   →                        -     k   y              k   →                    k   x            k   →                0         ]       ⁢     
     ⁢     θ   =          k   →            ⁢     
     ⁢     I   =     [         1       0       0           0       1       0           0       0       1         ]               (   4   )               
{right arrow over (k)} is a rotation vector associated with a camera center (e.g., origin of the camera coordinate system) in the robotic arm coordinate system;
 
kx is the x component of {right arrow over (k)}, ky is the y component of {right arrow over (k)}, and kz is the z component of {right arrow over (k)}; and Pcx is the x coordinate of the camera center, Pcy is the y coordinate of the camera center, and Pcz is the z coordinate of the camera center in the robotic arm coordinate system. In some embodiments, example approaches to register the camera center in the robotic arm coordinate system will be further described below in conjunction with  FIG.  8   .
 
     According to the transform matrix, points on the operation pathway in the three-dimensional model coordinate system may be transformed to the robotic arm coordinate system. Therefore, the robotic arm may move to one or more points on the operation pathway. 
     In some embodiments,  FIG.  8    is a flow diagram illustrating an example process  800  to register an optical apparatus (e.g., camera) in the robotic arm coordinate system. In some embodiments, the optical apparatus may be mounted at a flange of the robotic arm. To describe the optical apparatus in the robotic arm coordinate system with kx, ky, kz, Pcx, Pcy, and Pcz as set forth above, a point associated with the optical apparatus (e.g., origin of the camera coordinate system) may be registered in the robotic arm coordinate system first according to process  800 . Process  800  may include one or more operations, functions, or actions as illustrated by blocks  810 ,  820 ,  830  and/or  840 , which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. 
     Process  800  may begin with block  810 . In block  810 , the robotic arm is configured to move to a start position. In some embodiments, the start position is adjacent to and facing a base of the robotic arm. In some embodiments, at the start position, the optical apparatus is configured to capture one or more images of the base of the robotic arm. The captured images are associated with spatial relationships between a point of the optical apparatus and the base of the robotic arm. 
     Block  810  may be followed by block  820 . In block  820 , a mesh of the base of the robotic arm is obtained based on the captured images. 
     Block  820  may be followed by block  830 . In block  830 , a three-dimensional model of the base of the robotic arm is constructed based on certain physical information of the robotic arm. In some embodiments, the physical information may include the dimension, orientation and/or geometric features of the elements of the robotic arm. 
     Block  830  may be followed by block  840 . In block  840 , the obtained mesh and the constructed three-dimensional model are matched. Some technical feasible approaches may be used for the matching, for example, iterative closest points approach may be used to match points of the obtained mesh and points of the constructed three-dimensional model to satisfy a given convergence precision. In response to the given convergence precision is satisfied, the spatial relationships between the point of the optical apparatus and the base of the robotic arm can be calculated. Based on the calculation, the point of the camera may be registered in the robotic arm coordinate system. 
       FIG.  9    is a flow diagram illustrating an example process  900  to move robotic arm from an initial point to a safety point on an operation pathway, in accordance with some embodiments of the present disclosure. In some embodiments, the initial point is the point that an optical apparatus is configured to take images of the patient. In some embodiments, the initial point and the safety point are in the robotic arm coordinate system. Process  900  may include one or more operations, functions, or actions as illustrated by blocks  910 ,  920 , and/or  930 , which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. 
     In block  910 , a rotation matrix for rotating the robotic arm from the initial point to the safety point is obtained. In some embodiments, the rotation matrix R s =R X (β)R z (α)R 1 , in which 
     
       
         
           
             
               
                 R 
                 z 
               
               ⁡ 
               
                 ( 
                 α 
                 ) 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       cos 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                   
                   
                     
                       
                         - 
                         sin 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                   
                   
                     0 
                   
                 
                 
                   
                     
                       sin 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                   
                   
                     
                       cos 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                   
                   
                     0 
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
               ] 
             
           
         
       
       
         
           
             
               
                 R 
                 χ 
               
               ⁡ 
               
                 ( 
                 β 
                 ) 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       cos 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       β 
                     
                   
                   
                     0 
                   
                   
                     
                       sin 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       β 
                     
                   
                 
                 
                   
                     0 
                   
                   
                     1 
                   
                   
                     0 
                   
                 
                 
                   
                     
                       
                         - 
                         sin 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       β 
                     
                   
                   
                     0 
                   
                   
                     
                       cos 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       α 
                     
                   
                 
               
               ] 
             
           
         
       
         
         R 1  is the rotation matrix at the initial point; 
         α is the degrees of rotation along Z-axis at the initial point; and 
         β is the degrees of rotation along Y-axis at the initial point. 
       
    
     Block  910  may be followed by block  920 . In block  920 , coordinates of the safety point is obtained. In some embodiments, coordinates (X s ,Y s ,Z s ) of the safety point may be: 
     
       
         
           
             
               [ 
               
                 
                   
                     
                         
                     
                   
                   
                     
                       X 
                       1 
                     
                   
                 
                 
                   
                     
                       
                         R 
                         1 
                       
                       ⁡ 
                       
                         ( 
                         
                           3 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           X 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           3 
                         
                         ) 
                       
                     
                   
                   
                     
                       Y 
                       1 
                     
                   
                 
                 
                   
                     
                         
                     
                   
                   
                     
                       Z 
                       1 
                     
                   
                 
                 
                   
                     0 
                   
                   
                     1 
                   
                 
               
               ] 
             
             ⁡ 
             
               [ 
               
                 
                   
                     
                       X 
                       dis 
                     
                   
                 
                 
                   
                     
                       Y 
                       dis 
                     
                   
                 
                 
                   
                     
                       Z 
                       dis 
                     
                   
                 
               
               ] 
             
           
         
       
         
         R 1  is the rotation matrix at the initial point; 
         (X 1 ,Y 1 ,Z 1 ) are coordinates of the initial point; 
         X dis  is a moving distance on X-axis from the initial point; 
         Y dis  is a moving distance on Y-axis from the initial point; and 
         Z dis  is a moving distance on Z-axis from the initial point. 
       
    
     In some embodiments, X dis , Y dis , and Z dis  are associated with a length of the surgical instrument attached on the robotic arm. In some embodiments, the sum of the length of the surgical instrument and any of X dis , Y dis , and Z dis  is smaller than the distance from the robotic arm to the patient on any X-axis, Y-axis, and Z-axis, respectively. 
     Block  920  may be followed by block  930 . In block  930 , robotic arm is configured to rotate and move according to the rotation matrix obtained in block  910  and the coordinates obtained in block  920  from the initial point to the safety point. 
     In some embodiments, computing system  1060  of  FIG.  10    may be configured to perform the aforementioned process  900 . 
       FIG.  10    is an example figure showing a system  1000  to register a surgical tool attached to a robotic arm flange in the robotic arm coordinate system, in accordance with some embodiments of the present disclosure. In some embodiments, the robotic arm is at an initial point. In some embodiments, system  1000  may include flange  1010  of a robotic arm having edge  1011 , surgical instrument  1020  attached to flange  1010  having tip  1021 , first light source  1032 , first camera  1042 , second light source  1034 , second camera  1044 , and computing system  1060  coupled to the robotic arm either via a wired connection or a wireless connection. In some embodiments, light sources  1032  and  1034  may be back light plates perpendicularly disposed on plate  1050 . In some embodiments, plate  1050  is defined by the robotic arm coordinate system through a technical feasible calibration approach using a standard probe attached on flange  1010 . After calibration, the standard probe may be detached from flange  1010 , and surgical instrument  1020  is then attached to flange  1010 . Because plate  1050  is defined by the robotic arm coordinate system through the calibration, first back light plate  1032  may be defined by the x and z coordinates of the robotic arm coordinate system, and second back light plate  1034  may be defined by the y and z coordinates of the robotic arm coordinate system as shown in  FIG.  10   . 
     In some embodiments, light sources  1032  and  1034  are disposed perpendicularly with respect to each other. In some other embodiments, first camera  1042  is disposed on plate  1050  and is perpendicular to second light source  1034 , and second camera  1044  is disposed on plate  1050  and is perpendicular to first light source  1032 . Surgical instrument  1020  may be disposed among first light source  1032 , second light source  1034 , first camera  1042  and second camera  1044 . 
     In some embodiments, first light source  1032  is configured to generate back light. The back light is configured to pass the body of surgical instrument  1020 . First camera  1042  is configured to capture one or more images associated with surgical instrument  1020  (e.g., projections of surgical instrument on x-z plane of the robotic arm coordinate system) caused by the back light generated by first light source  1032 . 
     Similarly, second light source  1034  is also configured to generate back light. The back light is configured to pass the body of surgical instrument  1020 . Second camera  1044  is configured to capture one or more images associated with surgical instrument  1020  (e.g., projection of surgical instrument on y-z plane of the robotic arm coordinate system) caused by the back light generated by second light source  1034 . 
     In conjunction with  FIG.  10   ,  FIG.  11    illustrates example images  1101 ,  1102 , and  1103  captured by first camera  1042 , in accordance with some embodiments of the present disclosure. As set forth above, in some embodiments, first camera  1042  is configured to capture one or more projections of surgical instrument  1020  on x-z plane of the robotic arm coordinate system. To register surgical instrument  1020  in the robotic arm coordinate system, in response to a first x-z plane projection of surgical instrument  1020  at a first state, having a first x-z plane projection of tip  1021  away from center of image  1101 , computing system  1060  is configured to generate commands to move the robotic arm and flange  1010  and thus surgical instrument  1020  is moved to a second state. At the second state, a second x-z plane projection of surgical instrument  1020  is shown in image  1102  in which a second x-z plane projection of tip  1021  matches the center of image  1102 . However, at the second state, the second x-z plane projection of surgical instrument  1020  is not aligned with z axis of the robotic arm coordinate system. In response, computing system  1060  is configured to generate commands to rotate the robotic arm and flange  1010  and thus surgical instrument  1020  is rotated to a third state. At the third state, a third x-z plane projection of surgical instrument  1020  is shown in image  1103  in which a third x-z plane projection of tip  1021  matches the center of image  1103  and the third x-z plane projection of surgical instrument  1020  is aligned with z axis of the robotic arm coordinate system. 
     In addition to the steps described above, in conjunction with  FIG.  10    and  FIG.  11   ,  FIG.  12    illustrates example images  1201 ,  1202 , and  1203  captured by second camera  1044 , in accordance with some embodiments of the present disclosure. As set forth above, in some embodiments, second camera  1044  is configured to capture one or more projections of surgical instrument  1020  on y-z plane of the robotic arm coordinate system. To register surgical instrument  1020  in the robotic arm coordinate system, in response to a first y-z plane projection of surgical instrument  1020  at a fourth state, having a first y-z plane projection of tip  1021  away from center of image  1201 , computing system  1060  is configured to generate commands to move robotic arm and flange  1010  and thus surgical instrument  1020  is moved to a fifth state. At the fifth state, a second y-z plane projection of surgical instrument  1020  is shown in image  1202  in which a second y-z plane projection of tip  1021  matches the center of image  1202 . However, at fifth state, the second y-z plane projection of surgical instrument  1020  is not aligned with z axis of the robotic arm coordinate system. In response, computing system  1060  is configured to generate commands to rotate the robotic arm and flange  1010  and thus surgical instrument  1020  is rotated to a sixth state. At the sixth state, a third y-z plane projection of surgical instrument  1020  is shown in image  1203  in which a third y-z plane projection of tip  1021  matches the center of image  1203  and the third y-z plane projection of surgical instrument  1020  is aligned with z axis. 
     In some embodiments, at the sixth state, first camera  1042  is configured to capture one or more projections of surgical instrument  1020  on x-z plane of the robotic arm coordinate system. In response to the sixth state also shows a x-z plane projection substantially the same as image  1103 , computing system  1060  is configured to determine that a line corresponding to surgical instrument  1020 , having a start point at edge  1011  of flange  1010  and an end point at tip  1021 , is defined in the robotic arm coordinate system. Accordingly, coordinates of tip  1021  in the robotic arm coordinate system may be determined based on the coordinates of edge  1011  in the robotic arm coordinate system and geometry information (e.g., length) of surgical instrument  1021 . 
     On the contrary, in response to the sixth state shows a x-z plane projection not substantially the same as image  1103 , one or more steps described above for  FIG.  11    and  FIG.  12    are repeated until the sixth state shows a x-z plane projection substantially the same as image  1103 . 
     In some embodiments, after determining the coordinates of tip  1021  in the robotic arm coordinate system, the robotic arm may be configured to contact tip  1021  to a reference point having known coordinates in the robotic arm coordinate system by moving and/or rotating the flange  1010  and surgical instrument  1020 . In response to tip  1021  substantially touches the reference point, then computing system  1060  is configured to determine that coordinates of tip  1021  are verified. Otherwise, the steps set forth above may be repeated. 
     The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In some embodiments, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signal processors (DSPs), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of signal bearing medium used to actually carry out the distribution. Examples of a signal bearing medium include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a Compact Disc (CD), a Digital Versatile Disk (DVD), a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.). 
     From the foregoing, it will be appreciated that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various embodiments disclosed herein are not intended to be limiting.