Patent Description:
Embodiments of the present invention relate generally to methods and systems of determining one or more points on an operation pathway.

In an operation, a plan of an operation pathway is critical. The operation pathway may include multiple points, such as a safety point and a preoperative point away from the patient, an entry point on patient's tissues, and a target point at the target of the operation.

Robotic operation may offer a precise control of the operation pathway. Before the operation, patient is subjected to a medical scan (e.g., CT, MRI, PET, ultrasound etc.). The operation pathway to the desired anatomical region is planned. Artificial intelligence may be employed to suggest optimal routes with minimal damages to the surgeon. To perform the operation, the position of the patient may be matched to the perspective of the medical scan to accurate perform the operation along the planned operation pathway. Conventional approaches have relied on glued on or screwed in fiducial marks, which have not been widely adopted.

<CIT> discloses a A method to drive a robotic arm to one or more points on an operation pathway, comprising: constructing a three-dimensional model based on a medical image scan of a patient; planning an operation pathway according to the three-dimensional model; retrieving image information of the patient captured by a three-dimensional optical apparatus; selecting a first set of landmark points in the three-dimensional model; selecting a second set of landmark points in the retrieved image information; matching the first set of landmark points with the second set of landmark points; transforming coordinates from a first coordinate system associated with the three-dimensional model to a second coordinate system associated with the three-dimensional optical apparatus based on a result of matching the first set of landmark points with the second set of landmark points; and driving the robotic arm to the one or more points on the operation pathway in a third coordinate system associated with the robotic arm.

<CIT> discloses a method to determine an actual surgical pathway of a patient based on a planned surgical pathway, comprising: retrieving a first two-dimensional image of the patient associated with information collected at a first point in time, wherein a spatial relationship between the patient and an apparatus in an operating room is described by a first coordinate system; modifying a three-dimensional model of the patient including the planned surgical pathway with a first set of modification parameter values, wherein the three-dimensional model is described by a second coordinate system and constructed by information collected at a second point in time earlier than the first point in time; retrieving a second two-dimensional image from the modified threedimensional model; computing a first correlation between the first two-dimensional image and the second two-dimensional image; and in response to the first correlation exceeding a threshold, transforming the planned surgical pathway in the second coordinate system to the first coordinate system based on the first set of modification parameter values to determine the actual surgical pathway.

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

<FIG> is an example figure showing the spatial relationships among several points that may be encountered during an operation, arranged in accordance with some embodiments of the present disclosure. In <FIG>, an operation pathway <NUM> may include safety point <NUM>, preoperative point <NUM>, entry point <NUM>, and target point <NUM>.

<FIG> is a flow diagram illustrating an example process <NUM> to determine an operation pathway for a patient, arranged in accordance with some embodiments of the present disclosure. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

Process <NUM> may begin at block <NUM>, "construct 3D model based on medical image scan. " Before an operation is performed, some medical imaging techniques may be used to capture a snapshot of a patient's conditions, so that an operation plan may be formulated. The operation plan may include a planned operation pathway as set forth above. For example, the surgeon may order a medical image scan (e.g., CT or MRI) of the operation target. Such a medical image scan may be performed a few days (e.g., <NUM> to <NUM> days) prior to the operation. A three-dimensional model may be constructed based on the medical image scan data using some known approaches. Accordingly, points on the planned operation pathway may be identified in the three-dimensional model.

In some embodiments, an artificial intelligence engine may be employed to suggest to the surgeon one or more planned operation pathways with minimal physical damages to the patient. Based on the patient's CT or MRI scan, the artificial intelligence engine may suggest one or more optimal planned operation pathways. <FIG> illustrates an example of calculating planned operation pathway <NUM> to reach target point <NUM>, arranged in accordance with some embodiments of the present disclosure. The calculation may include transforming the standard brain-atlas data, and registering it to the patient's medical scan images to identify the brain regions. Some example brain regions include motor association area <NUM>, expressive speech area <NUM>, higher mental functions area <NUM>, motor area <NUM>, sensory area <NUM>, somatosensory association area <NUM>, global language area <NUM>, vision area <NUM>, receptive speech area <NUM>, receptive speech area <NUM>, association area <NUM>, and cerebellum area <NUM>. Moreover, common target tissues, such as sub-thalamic nucleus, may be automatically identified. In addition, each brain region set forth above may be assigned with a cost function for the artificial intelligence engine to suggest one or more planned operation pathways to the target tissues. The blood vessels may be identified from the TOF (time-of-flight MRI) data. The points on the outer brain boundary are candidate for entry point.

Block <NUM> may be followed by block <NUM> "generate 2D snapshot. " In some embodiments, a two-dimensional snapshot is generated based on the three-dimensional model constructed in block <NUM>. In some embodiments, the two-dimensional snapshot is a front view of the three-dimensional model of the patient. The front view of the patient includes at least some facial features of the patient.

Block <NUM> may be followed by block <NUM> "drive robotic arm to obtain patient's 2D facial image. " In some embodiments, at least two two-dimensional optical devices (e.g., cameras and/or scanners) are fixed on the robotic arm. By driving the robotic arm to different positions, the two two-dimensional optical devices may capture different images associated with the patient. In some embodiments, each of the two two-dimensional optical devices is configured to collect two-dimensional images associated with the patient. In some other embodiments, the two two-dimensional optical devices, in combination, are configured to collect a depth information associated with the patient. Therefore, the at least two two-dimensional optical devices may collect either two-dimensional images associated with the patient or three-dimensional images associated with the patient.

In conjunction with <FIG>, image <NUM> is a two-dimensional image associated with the patient collected at a first time. Image <NUM> has image center <NUM> with a coordinate (X, Y). An artificial intelligence engine may be employed to identify the patient in image <NUM>. In some embodiments, the artificial intelligence engine may identify the patient in frame <NUM> in image <NUM>. In addition, the artificial intelligence engine may identify facial central point <NUM> of the patient in image <NUM>. Facial central point <NUM> may have a coordinate (x, y). In some embodiments, the robotic arm is driven at least based on a first offset of (X-x) and a second offset of (Y-y).

For example, at the first time, either the first offset or the second offset is greater than one or more predetermined thresholds, and the robotic arm is driven to another position to decrease the first offset and the second offset.

In some embodiments, in response to the robotic arm being driven to a first updated position at a second time, in conjunction with <FIG>, image <NUM> is a two-dimensional image associated with the patient collected by the two-dimensional optical devices on the robotic arm. Image <NUM> has an image center <NUM> with an updated coordinate (X, Y). An artificial intelligence engine may be employed to identify the patient in image <NUM>. In some embodiments, the artificial intelligence engine may identify the patient in frame <NUM> in image <NUM>. In addition, the artificial intelligence engine may identify a facial central point <NUM> of the patient in image <NUM>. Facial central point <NUM> may have an updated coordinate (x, y). In some embodiments, the first offset of (X-x) and the second offset (Y-y) are both less than the one or more predetermined thresholds.

In some embodiments, at the first updated position at a third time, an artificial intelligence engine may be employed to identify at least three feature points <NUM>, <NUM> and <NUM>. The two-dimensional optical devices are configured to collect the depth information associated with the patient. The depth information may be assigned to feature points <NUM>, <NUM> and <NUM>, which can define a first plane in a three-dimensional space. In some embodiments, the robotic arm is driven to rotate and move to a second updated position at a fourth time so that the two-dimensional optical devices on the robotic arm is on a second plane substantially in parallel to the first plane in the three-dimensional space. In response to an average depth of image <NUM> at the second updated position in a predetermined range associated with operational parameters of the two-dimensional optical devices, image <NUM> is taken as the patient's 2D facial image and method <NUM> goes to block <NUM>.

In block <NUM> "select 2D feature points in 2D snapshot and 2D facial image," an artificial intelligence engine may be employed to select a first set of two-dimensional feature points in the two-dimensional snapshot generated in block <NUM> and a second set of two-dimensional feature points in the patient's two-dimensional facial image (e.g., image <NUM>) taken in block <NUM>.

In conjunction with <FIG>, image <NUM> is a two-dimensional snapshot generated in block <NUM>. In some embodiments, an artificial intelligence engine may be employed to identify glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> as these points are easier to be identified in image <NUM>. However, glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> may not be on the same two-dimensional plane for various races of people. Given image <NUM> is a two-dimensional snapshot, glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> are not suitable to be two-dimensional feature points on image <NUM>. Nevertheless, based on anatomy, a small region <NUM> on faces of various races of people is statistically planar. Therefore, in some embodiments, the artificial intelligence engine is employed to generate three lines <NUM>, <NUM> and <NUM> passing through glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM>, respectively and intersecting in region <NUM>. In some embodiments, intersected point <NUM> is selected as a two-dimensional feature point. In addition, in some embodiments, point <NUM> in region <NUM> on line <NUM> and point <NUM> in region <NUM> on line <NUM> are also selected as two-dimensional feature points. Points <NUM>, <NUM> and <NUM> may be a first set of two-dimensional feature points. In some embodiments, additional 2D points in region <NUM> may be selected to increase the number of the first set of two-dimensional feature points.

In conjunction with <FIG>, image <NUM> is a two-dimensional facial image taken in block <NUM>. In some embodiments, an artificial intelligence engine may be employed to identify glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> as these points are easier to be identified in image <NUM>. However, glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> may not be on the same two-dimensional plane for various races of people. Given image <NUM> is a two-dimensional facial image, glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM> are not suitable to be two-dimensional feature points on image <NUM>. Similarly, based on anatomy, a small region <NUM> on faces of various races of people is statistically planar. Nevertheless, in some embodiments, the artificial intelligence engine is employed to generate three lines <NUM>, <NUM> and <NUM> passing through glabella <NUM>, right endocanthion <NUM> and left endocanthion <NUM>, respectively and intersecting in region <NUM>. In some embodiments, the intersected point <NUM> is selected as a two-dimensional feature point. In addition, in some embodiments, a point <NUM> in region <NUM> on line <NUM> and a point <NUM> in region <NUM> on line <NUM> are also selected as two-dimensional feature points. Points <NUM>, <NUM> and <NUM> may be a second set of two-dimensional feature points. In some embodiments, additional 2D points in region <NUM> may be selected to increase the number of the second set of two-dimensional feature points.

Block <NUM> may be followed by block <NUM> "transform first set of 2D feature points to first set of 3D feature points. " In some embodiments, in block <NUM>, the first set of two-dimensional feature points (e.g., points <NUM>, <NUM> and <NUM>) are transformed to a first set of three-dimensional feature points. As set forth above, the first set of two-dimensional feature points are selected in a two-dimensional snapshot generated in a constructed three-dimensional model. Based on the algorithm taken the two-dimensional snapshot in the constructed three-dimensional model, a reverse operation may be performed to transform the first set of two-dimensional feature points on the two-dimensional snapshot (e.g., snapshot generated in block <NUM>) to a first set of three-dimensional feature points in the constructed three-dimensional model (e.g., three-dimensional model generated in block <NUM>). In some embodiments, the first set of three-dimensional feature points may identify a first initial three-dimensional coordinate that allows subsequent matching using iterative closest point (ICP) algorithm.

Block <NUM> may be followed by block <NUM> "transform second set of 2D feature points to second set of 3D feature points. " In some embodiments, in block <NUM>, the second set of two-dimensional feature points (e.g., points <NUM>, <NUM> and <NUM>) are transformed to a second set of three-dimensional feature points. As set forth above, the depth information associated with the patient may be collected by the two-dimensional optical devices. In some embodiments, the depth information may be added to the second set of two-dimensional feature points to transform the second set of two-dimensional feature points to a second set of three-dimensional feature points. In some embodiments, the second set of three-dimensional feature points may identify a second initial three-dimensional coordinate that allows subsequent matching using iterative closest point (ICP) algorithm.

Block <NUM> may be followed by block <NUM> "perform image matching between first set of three-dimensional feature points and second set of three-dimensional feature points. " In some embodiments, the first set of three-dimensional feature points and the second set of three-dimensional feature points are matched to determine a relationship that aligns the first set of three-dimensional feature points and the second set of three-dimensional feature points, sometimes iteratively to minimize the differences between the two sets of three-dimensional feature points.

For clarity, the following discussions mainly use one non-limiting example of the two two-dimensional optical devices (e.g., two two-dimensional cameras) and a three-dimensional coordinate system associated with the two two-dimensional optical devices, e.g., three-dimensional camera coordinate system, to explain various embodiments of the present disclosure.

Block <NUM> may be followed by block <NUM>, "transform coordinates. " In block <NUM>, the first set of three-dimensional feature points in the constructed three-dimensional model are transformed from their original coordinate system (i.e., three-dimensional model coordinate system) to the coordinates of the images taken by the two two-dimensional optical devices (i.e., three-dimensional camera coordinate system). The transformation may be based on some image comparison approaches, such as iterative closest point (ICP). Block <NUM> may further include additional coordinate transformations in which all points on the three-dimensional camera coordinate system are transformed to the coordinates of the robotic arm (i.e., robotic arm coordinate system). The details of transforming coordinates will be further described below.

Block <NUM> may be followed by block <NUM>, "determine operation pathway. " In block <NUM>, the coordinates of the planned operation pathway in three-dimensional model coordinate system may be transformed to the robotic arm coordinate system. Therefore, the robotic arm may move to the safety point, the preoperative point, the entry point, and/or the target point on the planned operation pathway.

In some embodiments, example process <NUM> may be applied to various types of operations, such as, without limitation, brain operations, nervous system operations, endocrine operations, eye operations, ears operations, respiratory operations, circulatory system operations, lymphatic operations, gastrointestinal operations, mouth and dental operations, urinary operations, reproductive operations, bone, cartilage and joint operations, muscle/soft tissue operations, breast operations, skin operations, and others.

In sum, at least two two-dimensional cameras or scanners may be used to obtain a patient's facial features. The facial features may then be compared with a two-dimensional snapshot of a three-dimensional model associated with a medical image scan. A first set of two-dimensional feature points are selected in the two-dimensional snapshot and a second set of two-dimensional feature points are selected in a two-dimensional patent's facial image obtained by the two-dimensional cameras or scanners, respectively. To compare, the first set of two-dimensional feature points and the second set of two-dimensional feature points are transformed to a first set of three-dimensional feature points in the three-dimensional model and a second set of three-dimensional feature points, respectively. In some embodiments, example process <NUM> may be applied to various types of operations, such as brain operations, nervous system operations, endocrine operations, eye operations, ears operations, respiratory operations, circulatory system operations, lymphatic operations, gastrointestinal operations, mouth and dental operations, urinary operations, reproductive operations, bone, cartilage and joint operations, muscle/soft tissue operations, breast operations, skin operations, and etc..

<FIG> illustrates an example coordinate transformation from the constructed three-dimensional model coordinate system to the three-dimensional camera coordinate system, in accordance with some embodiments of the present disclosure. This figure will be further discussed below in conjunction with <FIG>.

<FIG> is a flow diagram illustrating an example process <NUM> to transform coordinates, in accordance with some embodiments of the present disclosure. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM>, <NUM>, and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

In conjunction with <FIG>, in block <NUM>, initial matrices are obtained. In some embodiments, a first initial matrix TMRI and a second initial matrix Tcamera are obtained. In some embodiments, <MAT> in which <MAT> <MAT> <MAT> VectorXx is the x component of VectorX, VectorXy is the y component of VectorX, and VectorXz is the z component of VectorX. Similarly, Vectoryx is the x component of Vectory, Vectoryy is the y component of Vectory, and Vectoryz is the z component of Vectory. VectorZx is the x component of VectorZ, VectorZy is the y component of VectorZ, and VectorZz is the z component of VectorZ. P1X is the x coordinate of P1, P1y is the y coordinate of P1, and P1z is the z coordinate of P1.

In some other embodiments, <MAT> in which <MAT> <MAT> <MAT> VectorX'x is the x component of VectorX', VectorX'y is the y component of VectorX', and VectorX'z is the z component of VectorX'. Similarly, Vectory'x is the x component of Vectory, Vectory'y is the y component of Vectory', and Vectory,z is the z component of Vectory'. VectorZ'x is the x component of VectorZ', VectorZ'y is the y component of VectorZ', and VectorZ'z is the z component of VectorZ'. P1'X is the x coordinate of P1', P1'y is the y coordinate of P1', and P1'z is the z coordinate of P1'.

Block <NUM> may be followed by block <NUM>, "obtain conversion matrix. " In some embodiments, the conversion matrix may be <MAT> and P1, P2, and P3 are transformed to the three-dimensional camera coordinate system according to Tcamera <MAT>. Assuming P1, P2, and P3 are transformed to P1transformed, P2transformed, and P3transformed, respectively, a distance metric associated with differences between P1transformed and P1', P2transformed and P2', and P3transformed and P3' is calculated based on some feasible ICP approaches.

Block <NUM> may be followed by block <NUM>. In block <NUM>, whether the change of the distance metric reaches a threshold is determined. If the threshold is not reached, block <NUM> may go back to block <NUM> in which P1transformed, P2transformed, and P3transformed are selected to update Tcamera and eventually obtain new conversion matrix Tcamera <MAT>. If the threshold is reached, block <NUM> may be followed by block <NUM>.

In block <NUM>, a transform matrix is obtained to transform points from the three-dimensional camera coordinate system to the robotic arm coordinate system. In some embodiments, the transform matrix <MAT> in which <MAT> <MAT> <MAT> <MAT>.

According to the transform matrix, points on the operation pathway in the three-dimensional model coordinate system may be transformed to the robotic arm coordinate system. Therefore, the robotic arm may move to one or more points on the operation pathway.

In some embodiments, <FIG> is a flow diagram illustrating an example process <NUM> to register an optical device (e.g., camera) in the robotic arm coordinate system. In some embodiments, the optical device may be mounted at a flange of the robotic arm. To describe the optical device in the robotic arm coordinate system with kx, ky, kz, Pcx, Pcy, and Pcz as set forth above, a point associated with the optical device (e.g., origin of the camera coordinate system) may be registered in the robotic arm coordinate system first according to process <NUM>. Process <NUM> may include one or more operations, functions, or actions as illustrated by blocks <NUM>, <NUM>, <NUM> and/or <NUM>, which may be performed by hardware, software and/or firmware. The various blocks are not intended to be limiting to the described embodiments. The outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein.

Process <NUM> may begin with block <NUM>. In block <NUM>, the robotic arm is configured to move to a start position. In some embodiments, the start position is adjacent to and facing a reference point (e.g., robotic arm base) of the robotic arm. In some embodiments, at the start position, the optical device is configured to capture one or more images of the reference point of the robotic arm. The captured images are associated with spatial relationships between a point of the optical device and the reference point of the robotic arm.

Block <NUM> may be followed by block <NUM>. In block <NUM>, a mesh of the reference point of the robotic arm is obtained based on the captured images.

Block <NUM> may be followed by block <NUM>. In block <NUM>, a three-dimensional model of the reference point of the robotic arm is constructed based on certain physical information of the robotic arm. In some embodiments, the physical information may include the dimension, orientation and/or geometric features of the elements of the robotic arm.

Block <NUM> may be followed by block <NUM>. In block <NUM>, the obtained mesh and the constructed three-dimensional model are matched. Some technical feasible approaches may be used for the matching, for example, iterative closest points approach may be used to match points of the obtained mesh and points of the constructed three-dimensional model to satisfy a given convergence precision. In response to the given convergence precision is satisfied, the spatial relationships between the point of the optical device and the reference point of the robotic arm can be calculated. Based on the calculation, the point of the camera may be registered in and transformed to the robotic arm coordinate system.

Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In some embodiments, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signal processors (DSPs), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of signal bearing medium used to actually carry out the distribution. Examples of a signal bearing medium include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a Compact Disc (CD), a Digital Versatile Disk (DVD), a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).

Claim 1:
A method to determine an operation pathway (<NUM>, <NUM>) for a patient, comprising:
constructing a three-dimensional model based on a medical image scan of the patient;
obtaining image information of the patient with a set of two-dimensional optical devices;
generating a two-dimensional snapshot of the three-dimensional model;
selecting a first set of two-dimensional feature points (<NUM>, <NUM>, <NUM>) associated with the three-dimensional model in the two-dimensional snapshot;
selecting a second set of two-dimensional feature points (<NUM>, <NUM>, <NUM>) associated with the image information of the patient;
transforming the first set of two-dimensional feature points (<NUM>, <NUM>, <NUM>) to a first set of three-dimensional feature points;
transforming the second set of two-dimensional feature points (<NUM>, <NUM>, <NUM>) to a second set of three-dimensional feature points;
matching between the first set of three-dimensional feature points and the second set of three-dimensional feature points to determine a relationship that aligns the first set of three-dimensional feature points and the second set of three-dimensional feature points;
transforming coordinates from a first coordinate system associated with the three-dimensional model to a second coordinate system associated with the set of two-dimensional optical devices based on the relationship; and
determining the operation pathway (<NUM>, <NUM>) in a third coordinate system associated with a robotic arm based on the transformed coordinates in the second coordinate system.