Patent Description:
Embodiments of the present invention relate generally to methods and systems of tracking a position or an orientation of a patient in an operation.

To perform an operation, a plan of an operation pathway is critical. Robotic operation may offer a precise control of the operation pathway. Before the operation, patient is subjected to a medical scan (e.g., CT or MRI). The operation pathway to the desired anatomical region is planned based on the medical scan. Artificial intelligence may be employed to suggest the surgeon with optimal routes that incur the least amount of damages. To perform the operation, the position of the patient may be matched to the perspective of the medical scan to accurate perform the operation along the planned operation pathway.

However, in the operation, the position of the patient may be moved. Accordingly, the planned operation pathway may not be applicable in view of the moved position of the patient and may be updated in response to the moved position of the patient.

<CIT> discloses a recalibration device used during the acquisition of images of an anatomical area of a patient during robotassisted surgery, including a three dimensional body made of radiolucent material, said body having an upper surface and an opposite surface forming a bearing surface to be manually placed on a surface of said anatomical area of the patient, said body comprises fiducial markers made of radiopaque material, wherein said fiducial markers are arranged between the upper surface and the bearing surface in at least one specific geometrical pattern permitting a certain detection of the position and the orientation of the recalibration device in a three-dimensional digital model built from the images resulting from the acquisition of the anatomical area.

<CIT> discloses a method for defining a boundary surface for a robotic surgery system, comprising: identifying at least a portion of a skin surface of a patent in an image dataset of the patient' s anatomy; generating a boundary surface based on the identified skin surface; registering the image dataset and the boundary surface within a patient coordinate system; and controlling a robotic arm to prevent at least a portion of the robotic arm from crossing the boundary surface.

<CIT> discloses a surgical positioning device that comprises a positioning ruler, a host computer and a series manipulator arm with at least six degrees of freedom. The host computer is connected with the series manipulator arm, and the positioning ruler includes two opposite surfaces that are transparent to X-rays. The two opposite surfaces are fixedly connected by an X-ray-transparent connecting surface, and a group of marks are provided on the two opposite surfaces, and each group of marks includes at least four mark points that are not on a straight line, and the mark points are opaque to X-rays. Any opposite surface or connecting surface is fixedly connected to a ruler handle, and the ruler handle is connected with the end of a serial manipulator through an interface. The computer adjusts the position of the positioning ruler by controlling the movement of the serial manipulator for transmission. The angle is transformed, and the spatial positioning calculation is performed according to the marked points in the collected images to obtain the planned path.

<CIT> discloses a system for surgical navigation, comprising: a first array including tracking markers, the first array releasably secured with a first anatomical feature; and at least one camera configured to track the array and transmit the images of the array to a computer system including a processor, wherein the computer system is configured to display a simulation of the anatomical feature on a display screen.

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

<FIG> is an example figure showing operation system <NUM> configured to perform an operation on patient <NUM>, arranged in accordance with some embodiments of the present disclosure. In some embodiments, operation system <NUM> includes frame <NUM>, operating table <NUM> and robotic arm assembly <NUM>.

In some embodiments, frame <NUM> is configured to support and attach to an operation target (e.g., patient's head <NUM>) of patient <NUM>. In addition, frame <NUM> is also configured to attach to operating table <NUM>. In some embodiments, frame <NUM> further includes tag <NUM>. Tag <NUM> is fixed or integrally formed with frame <NUM>. Therefore, patient's head <NUM>, operating table <NUM>, frame <NUM> and tag <NUM> together may form a predetermined spatial relationship. In other words, in response to a first movement of patient's head <NUM>, tag <NUM> also has a corresponding second movement based on the predetermined spatial relationship. Accordingly, the movement of tag <NUM> may also be used to determine whether the patient's head has moved.

In some embodiments, frame <NUM> may include more than one tags. Each tag may include a unique identification pattern from each other. Accordingly, operating system <NUM> may process different requirements based on these tags.

In some embodiments, robotic arm assembly <NUM> may include surgical instrument <NUM>, optical apparatus <NUM>, first arm <NUM>, second arm <NUM> and third arm <NUM>. In some embodiments, optical apparatus <NUM> is an IR camera capable of capturing images of tag <NUM> at different times. These captured images are processed to identify whether a movement of tag <NUM> exceeds a threshold. In response to determining the movement of tag <NUM> exceeding the threshold, a processor (not shown) associated with robotic arm assembly <NUM> may determine that patient <NUM> has moved based on the predetermined spatial relationship set forth above and issue a command to robotic arm assembly <NUM> to stop the operation and move to a safety point which will not cause harms to patient <NUM>. In response to the determination that patient <NUM> has moved, the processor may calculate a new operation pathway and issue a command to robotic arm assembly <NUM> to move from the safety point along the new operation pathway.

<FIG> is an example figure showing tag <NUM>, arranged in accordance with some embodiments of the present disclosure. Tag <NUM> may correspond to tag <NUM> of <FIG>. In some embodiments, tag <NUM> includes point <NUM>, point <NUM>, point <NUM>, point <NUM>, point <NUM> and identification pattern <NUM>. Points <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and identification pattern <NUM> may emit lights having specific wavelengths which can be captured by an optical apparatus (e.g., optical apparatus <NUM>). On the contrary, other parts of tag <NUM> emit very few such lights and therefore may be barely captured by the optical apparatus. In some other embodiments, points <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and identification pattern <NUM> may reflect lights generated by a light source having specific wavelengths which can be captured by an optical apparatus (e.g., optical apparatus <NUM>). Other parts of tag <NUM> may absorb such lights and therefore may be barely captured by the optical apparatus.

<FIG> is an example image <NUM> of a tag captured by an optical apparatus, arranged in accordance with some embodiments of the present disclosure. Image <NUM> may include point images <NUM>, <NUM>, <NUM>, <NUM> and <NUM>, bar image <NUM> between points images <NUM> and <NUM> and identification image <NUM>. It has been a challenge to associate point images <NUM>, <NUM>, <NUM>, <NUM> and <NUM> to points <NUM>, <NUM>, <NUM>, <NUM> and <NUM> efficiently and correctly.

In some embodiments, point image <NUM> may correspond to an image of point <NUM>. Similarly, point image <NUM> may correspond to an image of point <NUM>, point image <NUM> may correspond to an image of point <NUM>, point image <NUM> may correspond to an image of point <NUM>, and point image <NUM> may correspond to an image of point <NUM>. Bar image <NUM> may correspond to parts of tag <NUM> between points <NUM> and <NUM>. Identification image <NUM> may correspond to identification pattern <NUM>.

<FIG> is an example processed image <NUM>' of a tag captured by an optical apparatus and <FIG> is a flow diagram illustrating an example process <NUM> to process a two-dimensional image associated with a tag, all arranged in accordance with some embodiments of the present disclosure. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM> and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. In conjunction with <FIG>, <FIG> will be further described in details below.

Process <NUM> may begin at block <NUM>, "assign coordinates. " Referring to <FIG>, in some embodiments, image <NUM> is a two-dimensional image associated with tag <NUM>. In <FIG>, image <NUM>' may be a processed image <NUM>. In some embodiments, a two-dimensional coordinate system is assigned to image <NUM>'. In some embodiments, the upper left corner <NUM> is assigned as the origin point (<NUM>, <NUM>) of the two-dimensional coordinate system. The two-dimensional coordinate system may be based on pixels of image <NUM>. Therefore, point images <NUM>, <NUM>, <NUM>, <NUM> and <NUM> on image <NUM> may be described by the two-dimensional coordinate system.

Block <NUM> may be followed by block <NUM> "identify coordinates associated with point images. " In some embodiments, to efficiently identify point images <NUM>, <NUM>, <NUM>, <NUM> and <NUM> from image <NUM>', thresholding techniques may be applied. For example, image <NUM>' is first processed with blocks including many pixels. In some embodiments, blocks not including any pixel having an intensity greater than a threshold (e.g., region <NUM> or bar image <NUM>) are discarded from further processing to save the computation resources. Blocks including pixels having an intensity greater than the threshold (e.g., region <NUM>) are then further processed on a pixel basis.

In some embodiments, region <NUM> may be enlarged with an enlargement factor. Therefore, point images <NUM>, <NUM>, <NUM>, <NUM> and <NUM> may be also enlarged with the enlargement factor. In some embodiments, additional thresholding techniques may be applied to pixels of the enlarged point images to identify one or more pixels that have a greater intensity. Image associated with the identified pixels may be shrunk back to the two-dimensional system based on a shrinkage factor (e.g., reciprocal of the enlargement factor). Therefore, new point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' with higher intensity may be identified on image <NUM> and coordinates of the two-dimensional coordinate system may be assigned to point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>'. In some embodiments, point image <NUM>' is assigned coordinates (X<NUM>', Y<NUM>') on the two-dimensional coordinate system. Similarly, point image <NUM>' is assigned coordinates (X<NUM>', Y<NUM>'), point image <NUM>' is assigned coordinates (X<NUM>', Y<NUM>'), point image <NUM>' is assigned coordinates (X<NUM>', Y<NUM>'), and point image <NUM>' is assigned coordinates (X<NUM>', Y<NUM>'), respectively.

Block <NUM> may be followed by block <NUM> "associate point images with tag points. " In some embodiments, point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' are associated with points <NUM>, <NUM>, <NUM>, <NUM> and <NUM>. In some embodiments, based on perspective-n-point approaches, coordinates of point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' on the two-dimensional coordinate system and coordinates of points <NUM>, <NUM>, <NUM>, <NUM> and <NUM> of tag <NUM> on a three-dimensional coordinate system which describes the spatial relationship among optical apparatus <NUM>, robotic arm assembly <NUM> and surgical instrument <NUM>, a conversion matrix describing a relationship between a point image in image <NUM>' and its associated point at tag <NUM> may be established.

<FIG> is a flow diagram illustrating an example process <NUM> to associate point images with points of a tag, arranged in accordance with some embodiments of the present disclosure. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

Process <NUM> may begin at block <NUM>, "obtain geometric center of point images. " In some embodiments, as set forth above, in the two-dimensional coordinate system, point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' have coordinates (X<NUM>', Y<NUM>'), (X<NUM>', Y<NUM>'), (X<NUM>', Y<NUM>'), (X<NUM>', Y<NUM>') and (X<NUM>', Y<NUM>'), respectively. A geometric center of point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' may have coordinates ((X<NUM>'+ X<NUM>'+ X<NUM>'+ X<NUM>'+ X<NUM>')/<NUM>, (Y<NUM>'+ Y<NUM>'+ Y<NUM>'+ Y<NUM>'+ Y<NUM>')/<NUM>).

Block <NUM> may be followed by block <NUM>, "obtain distance of point image from geometric center. " In some embodiments, a first distance between point image <NUM>' and the geometric center is obtained as <MAT> Similarly, a second distance between point image <NUM>' and the geometric center, a third distance between point image <NUM>' and the geometric center, a fourth distance between point image <NUM>' and the geometric center and a fifth distance between point image <NUM>' and the geometric center may be obtained.

Block <NUM> may be followed by block <NUM>, "obtain vector from geometric center to point image. " In some embodiments, a first vector <MAT> from the geometric center to point image <NUM>' is obtained as (X<NUM>' - (X<NUM>'+ X<NUM>'+ X<NUM>'+ X<NUM>'+ X<NUM>')/<NUM>, Y<NUM>'-(Y<NUM>'+ Y<NUM>'+ Y<NUM>'+ Y<NUM>'+ Y<NUM>')/<NUM>). Similarly, a second vector <MAT> from the geometric center to point image <NUM>', a third vector <MAT> from the geometric center to point image <NUM>', a fourth vector <MAT> from the geometric center to point image <NUM>' and a fifth vector <MAT> from the geometric center to point image <NUM>' may be also obtained.

Block <NUM> may be followed by block <NUM>, "select base point image. " In some embodiments, the first distance, the second distance, the third distance, the fourth distance and the fifth distance obtained in block <NUM> are compared to select which point image is the base point image. In some embodiments, the point image having the farthest distance from the geometric center in the two-dimensional coordinate system may be selected as the base point. Alternatively, the point image having the closest distance from the geometric center in the two-dimensional coordinate system may be selected as the base point. In some other embodiments, the base point may be selected based on a relationship between identification image <NUM> and point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' in the two-dimensional coordinate system. For illustration only, in some embodiments, point image <NUM>' is selected as the base point image.

Block <NUM> may be followed by block <NUM>, "index point images based on vectors. " In some embodiments, a cross product of the first vector (e.g., <MAT>) and the second vector (e.g., <MAT>) may be used to determine whether point image <NUM>' is along the clockwise direction from point image <NUM>'. In some embodiments, in response to a cross product of the first vector and the second vector being a vector pointing to the positive z direction with respect to the two-dimensional coordinate system, point image <NUM>' is determined along the clockwise direction from point image <NUM>'. In some other embodiments, a dot product of the first vector (e.g., <MAT>) and the second vector (e.g., <MAT>) may be used to determine an angle between the first vector and the second vector. Therefore, based on the cross product and the dot product, point image <NUM>' may be determined at a first angle along the clockwise direction from point image <NUM>'. Similarly, point images <NUM>', <NUM>' and <NUM>' may be determined at a second angle, a third angle and a fourth angle along the clockwise direction from point image <NUM>', respectively. In some embodiments, based on a comparison between the first angle, the second angle, the third angle and the fourth angle, a sequence of point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' along the clockwise direction may be determined. For example, in response to the first angle is smaller than the second angle, point image <NUM>' is determined to be closer to point image <NUM>' along the clockwise direction than point <NUM>'. After the sequence is determined, point images <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>' may be indexed according to the sequence.

Block <NUM> may be followed by block <NUM>, "obtain conversion matrix of point image and corresponding tag point. " In some embodiments, based on the index obtained in block <NUM> and perspective-n-point approaches, a conversion matrix which describes a mathematical relationship between coordinates of points of tag <NUM> (e.g., <NUM>, <NUM>, <NUM>, <NUM> and <NUM>) in a three-dimensional coordinate system and their corresponding point image (e.g., <NUM>', <NUM>', <NUM>', <NUM>' and <NUM>') may be established. Referring back to <FIG>, in some embodiments, the three-dimensional coordinate system may describe the spatial relationship among optical apparatus <NUM>, robotic arm assembly <NUM> and surgical instrument <NUM>.

<FIG> is a flow diagram illustrating an example process <NUM> to update an operation pathway in response to a movement of a patient based on a plurality of conversion matrices, arranged in accordance with some embodiments of the present disclosure. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM>, <NUM>, <NUM> and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

Process <NUM> may begin at block <NUM>, "obtain projection point based on conversion matrix. " In some embodiments, after obtaining the conversion matrix, a first projection point of point <NUM> according to the conversion matrix may be obtained. In some embodiments, the projection point has coordinates (X<NUM>", Y<NUM>") in the two-dimensional coordinate system. Similarly, a second projection point of point <NUM>, a third projection point of point <NUM>, a fourth projection point of point <NUM> and a fifth projection point of point <NUM> may also be obtained and has coordinates (X<NUM>", Y<NUM>"), (X<NUM>", Y<NUM>"), (X<NUM>", Y<NUM>") and (X<NUM>", Y<NUM>") in the two-dimensional coordinate system, respectively.

Block <NUM> may be followed by block <NUM>, "obtain distance between projection point and corresponding point image. " In some embodiments, a first distance between the first projection point having coordinates (X<NUM>", Y<NUM>") and point image <NUM>' having coordinates (X<NUM>', Y<NUM>') is calculated. Similarly, a second distance between the second projection point having coordinates (X<NUM>", Y<NUM>") and point image <NUM>' having coordinates (X<NUM>', Y<NUM>'), a third distance between the second projection point having coordinates (X<NUM>", Y<NUM>") and point image <NUM>' having coordinates (X<NUM>', Y<NUM>'), a fourth distance between the fourth projection point having coordinates (X<NUM>", Y<NUM>") and point image <NUM>' having coordinates (X<NUM>', Y<NUM>') and a fifth distance between the fifth projection point having coordinates (X<NUM>", Y<NUM>") and point image <NUM>' having coordinates (X<NUM>', Y<NUM>') may be also obtained, respectively.

Block <NUM> may be followed by block <NUM>, "distance greater than threshold?" In some embodiments, in response to a sum of the first distance, the second distance, the third distance, the fourth distance and the fifth distance is greater than a threshold, block <NUM> may be followed by block <NUM>, "determine patient is moved.

In some other embodiments, in response to a sum of the first distance, the second distance, the third distance, the fourth distance and the fifth distance not greater than a threshold, block <NUM> may be followed by block <NUM>, "obtain projection point based on conversion matrix. " After obtaining another conversion matrix associated with image <NUM>/<NUM>' captured at a second time, another set of projection points having coordinates in the two-dimensional coordinate system of points <NUM>, <NUM>, <NUM>, <NUM> and <NUM> may be obtained according to another conversion matrix.

Block <NUM> may be followed by block <NUM>, "update operation pathway. " In some embodiments, referring back to <FIG>, robotic arm assembly <NUM> may move to a safety point which will not cause harms to patient <NUM>. A new operation pathway may be updated in response to the movement of patient <NUM> and robotic arm assembly <NUM> is then configured to move from the safety point along the updated operation pathway.

In some other embodiments, the updated operation pathway may be verified so that the updated operation pathway will not cause a collision between surgical instrument <NUM> and robotic arm assembly <NUM>. Alternatively, referring back to <FIG>, the updated operation pathway may be verified so that the updated operation pathway will not cause a collision among first arm <NUM>, second arm <NUM> and third arm <NUM> of robotic arm assembly <NUM>.

In response to the updated operation pathway is verified to cause a collision of robotic arm assembly <NUM>, the updated operation pathway will be abandoned and a new updated operation pathway will be calculated.

Claim 1:
A method to update an operation pathway for a robotic arm assembly (<NUM>) to perform an operation on a patient (<NUM>) having a predetermined spatial relationship with a frame (<NUM>) including a tag (<NUM>, <NUM>) in response to a movement of the patient (<NUM>), comprising:
processing a two-dimensional (<NUM>) image associated with the tag (<NUM>, <NUM>) having the spatial relationship with the patient (<NUM>), wherein a corresponding movement of the tag (<NUM>, <NUM>) in response to the movement of the patient is determined based on the spatial relationship, and wherein the tag (<NUM>, <NUM>) includes a first point (<NUM>) and a second point (<NUM>), and the two-dimensional image (<NUM>) includes a first point image (<NUM>) and a second point image (<NUM>);
associating the first point image (<NUM>) with the first point (<NUM>) and the second point image (<NUM>) with the second point (<NUM>); and
identifying the movement of the patient (<NUM>) and updating the operation pathway, characterized in that said method comprises using a conversion matrix which describes a mathematical relationship between coordinates of said first and second points (<NUM>, <NUM>) of tag (<NUM>, <NUM>) in a three-dimensional coordinate system and their corresponding first and second point images (<NUM>, <NUM>);
and in that the processing the two-dimensional image (<NUM>) comprises:
discarding blocks (<NUM>) not including any pixel having an intensity greater than a threshold from further processing, and
enlarging a region (<NUM>) including said first and second point images (<NUM>, <NUM>) with an enlargement factor, applying additional thresholding techniques to the enlarged first and second point images to identify one or more pixels that have a greater intensity, and shrinking the image with the identified pixels back based on a shrinkage factor, thereby providing new first and second point images (<NUM>', <NUM>') with higher intensity.