Patent Description:
During surgical procedures, endoscopes have been used to visualize a surgical site. Particularly, in minimally invasive surgery including robotic surgery, stereoscopic image sensors have been used to allow a surgeon to visualize a surgical site.

A surgeon operating a robotic surgical system engages in a relatively small portion of organs inside of a patient and has a limited scope of view through a display, so the surgeon may not see the placement of persons assisting with the surgery. When a person is in close proximity to the robot arms, it is possible for the person to be contacted by or collide with an arm of the robotic system. Such collision can cause safety concerns. Therefore, there is continuing interest in developing and improving safety of robotic surgical systems.

<CIT> discloses a robotic system for collision detection and prevention.

The techniques of this disclosure generally relate to mitigating or reducing potential collisions between a person and a robotic surgical system. As described in more detail herein, the type of a human-held tool and/or the pose/orientation of the human-held tool is the starting point for inferring the position of the person holding the human-held tool. Thus, the techniques disclosed herein for determining the position of a person holding a non-robotic tool is not based on an image of the person. The inferred position of the person is compared to the swept volume of the robotic surgical system to identify potential collisions.

The invention is defined in appended independent claims <NUM>,<NUM> and <NUM>, further embodiments are described in the dependent claims.

In accordance with aspects of the disclosure, a robotic surgical system includes a robotic linkage including a plurality of joints, an endoscope coupled to a distal portion of the robotic linkage and configured to capture stereoscopic images, and a controller in operable communication with the endoscope. The controller is configured to execute instructions to cause the controller to analyze the stereoscopic images from the endoscope to identify a human-held tool in the stereoscopic images and to estimate a type and/or pose of the human-held tool, infer a position of a person holding the human-held tool based on the type and/or pose of the human-held tool, determine a spatial relationship between the person and the robotic linkage based on the inferred position of the person, and generate a warning of a potential collision between the person and the robotic linkage based on the determined spatial relationship.

In various embodiments of the system, the robotic linkage includes sensors configured to provide measurements of an angle and a velocity of each joint of the plurality of joints.

In various embodiments of the system, the robotic surgical system includes a robotic tool coupled to a distal portion of the robotic linkage. The controller is configured to execute the instructions to cause the controller to analyze the stereoscopic images from the endoscope to identify the robotic tool in the stereoscopic images and to estimate a pose of the robotic tool, and determine a swept volume of the robotic linkage based on the estimated pose of the robotic tool and based on the angle and the velocity of each joint of the plurality of joints, where the swept volume is determined without using any real-time image of the robotic linkage.

In various embodiments of the system, in analyzing the stereoscopic images, the instructions implement an artificial-intelligence learning machine to determine the pose of the robotic tool.

In various embodiments of the system, the swept volume is a physical space that the robotic linkage could move through in a particular time period based on the velocity of each j oint of the plurality of joints.

In various embodiments of the system, in generating the warning of the potential collision, the instructions when executed cause the controller to determine the potential collision based on an overlap between the inferred position of the person and the swept volume of the robotic linkage.

In various embodiments of the system, in analyzing the stereoscopic images, the instructions implement an artificial-intelligence learning machine configured to determine the type and/or pose of the human-held tool.

In various embodiments of the system, in inferring the position of the person holding the human-held tool, the controller executes the instructions to cause the controller to access information relating to how the human-held tool is typically held during a particular phase of a surgical procedure, and infer the position of the person holding the human-held tool based further on the information relating to how the human-held tool is typically held during the particular phase of the surgical procedure.

In various embodiments of the system, in inferring the position of the person holding the human-held tool, the controller executes the instructions to cause the controller to access physical attribute information for the person holding the human-held tool, and infer the position of the person holding the human-held tool based further on the physical attribute information for the person holding the human-held tool, where the position of the person holding the human-held tool is inferred without using any real-time image of the person holding the human-held tool.

In accordance with aspects of the disclosure, a method is disclosed for warning of a potential collision between a person holding a human-held tool and a robotic surgical system having a robotic linkage including a plurality of joints. The method includes accessing stereoscopic images obtained by an endoscope coupled to a distal portion of the robotic linkage, analyzing the stereoscopic images from the endoscope to identify the human-held tool in the stereoscopic images and to estimate a type and/or pose of the human-held tool, inferring a position of the person holding the human-held tool based on the type and/or pose of the human-held tool, determining a spatial relationship between the person and the robotic linkage based on the inferred position of the person, and generating a warning of a potential collision between the person and the robotic linkage based on the determined spatial relationship.

In various embodiments of the method, the robotic linkage includes sensors configured to provide measurements of an angle and a velocity of each joint of the plurality of joints.

In various embodiments of the method, the robotic surgical system includes a robotic tool coupled to a distal portion of the robotic linkage, and the method includes analyzing the stereoscopic images from the endoscope to identify the robotic tool in the stereoscopic images and to estimate a pose of the robotic tool, and determining a swept volume of the robotic linkage based on the estimated pose of the robotic tool and based on the angle and the velocity of each joint of the plurality of joints, where the swept volume is determined without using any real-time image of the robotic linkage.

In various embodiments of the method, analyzing the stereoscopic images includes using an artificial-intelligence learning machine to determine the pose of the robotic tool.

In various embodiments of the method, the swept volume is a physical space that the robotic linkage could move through in a particular time period based on the velocity of each j oint of the plurality of joints.

In various embodiments of the method, generating the warning of the potential collision includes determining the potential collision based on an overlap between the inferred position of the person and the swept volume of the robotic linkage.

In various embodiments of the method, analyzing the stereoscopic images includes using an artificial-intelligence learning machine configured to determine the type and/or pose of the human-held tool.

In various embodiments of the method, inferring the position of the person holding the human-held tool includes accessing information relating to how the human-held tool is typically held during a particular phase of a surgical procedure, and inferring the position of the person holding the human-held tool based further on the information relating to how the human-held tool is typically held during the particular phase of the surgical procedure.

In various embodiments of the method, inferring the position of the person holding the human-held tool includes accessing physical attribute information for the person holding the human-held tool, and inferring the position of the person holding the human-held tool based further on the physical attribute information for the person holding the human-held tool, where the position of the person holding the human-held tool is inferred without using any real-time image of the person holding the human-held tool.

In various embodiments of the method, a non-transitory computer readable medium includes computer executable instructions which, when executed by a controller, cause the controller to perform a method for warning of a potential collision between a person holding a human-held tool and a robotic surgical system having a robotic linkage including a plurality of joints. The method includes accessing stereoscopic images obtained by an endoscope coupled to a distal portion of the robotic linkage, analyzing the stereoscopic images from the endoscope to identify the human-held tool in the stereoscopic images and to estimate a type and/or pose of the human-held tool, inferring a position of the person holding the human-held tool based on the type and/or pose of the human-held tool, determining a spatial relationship between the person and the robotic linkage based on the inferred position of the person, and generating a warning of a potential collision between the person and the robotic linkage based on the determined spatial relationship.

In various embodiments of the non-transitory computer readable medium, the computer executable instructions, when executed by the controller, cause the controller to further perform the method for warning of a potential collision, including analyzing the stereoscopic images from the endoscope to identify a robotic tool in the stereoscopic images and to estimate a pose of the robotic tool, where the robotic tool is coupled to a distal portion of the robotic linkage, and determining a swept volume of the robotic linkage based on the estimated pose of the robotic tool and based on an angle and a velocity of each joint of the plurality of joints, where generating the warning of the potential collision includes determining the potential collision based on an overlap between the inferred position of the person and the swept volume of the robotic linkage.

Surgeries with a robotic surgical system use a non-robotic/human-held tool under certain situations, giving rise to potential collisions between the robotic surgical system and a person holding the non-robotic tool. In accordance with aspects of the present disclosure, such potential collisions can be mitigated by determining a spatial relationship between the person and the robotic surgical system. The person's position may be inferred based on stereoscopic images from an endoscope, and a swept volume for the robotic surgical system may be determined based on joint angles and velocities of the robotic surgical system. Based on an overlap between the position of a person and the swept volume, potential collisions can be determined. Appropriate controls may be performed to reduce the potential collisions. As described in more detail below, the type of the human-held tool and/or the pose/orientation of the human-held tool is the starting point for determining the position of the person holding the human-held tool. Thus, the techniques disclosed herein for determining the position of a person holding a non-robotic tool is not based on an image of the person.

Referring to <FIG> and <FIG>, there is shown a diagram of an exemplary robotic surgical system <NUM> used in conjunction with a non-robotic/human-held tool <NUM>. The robotic surgical system includes a surgical robot <NUM>, a processor <NUM>, and a user console <NUM>. The robotic surgical system <NUM> may not be able to completely perform a surgery by itself and may be supplemented by use of non-robotic/human-held tools <NUM>. The surgical robot <NUM> generally includes one or more robotic linkages or arms <NUM> and robot bases <NUM> which support the corresponding robotic linkages <NUM>. Each robotic linkage <NUM> is moveable at or around joints. For example, a distal robotic linkage <NUM> has an end <NUM> that supports the end effector or tool <NUM>, which is configured to act on a target of interest. In addition, an imaging device <NUM> for imaging a surgical site "S" may be installed at the end <NUM> of the distal robotic linkage <NUM>.

The user console <NUM> is in communication with the robot bases <NUM> through the processor <NUM>. In addition, each robot base <NUM> may include a controller <NUM>, which is in communication with the processor <NUM>, and an arm motor <NUM>, as shown in <FIG>. The robotic linkage <NUM> may include one or more arms, and joints between two adjoining arms. The arm motor <NUM> may be configured to actuate each joint of the robotic linkage <NUM> to move the end effector <NUM> to intended positions according to the clinician's controls.

In accordance with aspects of the present disclosure, each joint of the robotic linkages <NUM> may have one or more sensors configured to sense an angle between two adjoining arms and to sense a velocity of each arm or angular velocity of each joint. Such sensed information may be transmitted to the processor <NUM>, which then performs calculations to identify a swept volume of each of the robotic linkages <NUM>. The swept volume may indicate a volume of space that each of the robotic linkages <NUM> may occupy within a period of time.

The non-robotic/human-held tool <NUM> may be held by a person who occupies a space in the surgery room next to the robotic surgical system <NUM>. Thus, there is a possibility that the person and the robotic surgical system <NUM> might collide or interfere with each other. Such collisions may lead to unexpected movements of the robotic and non-robotic tools, resulting in potential injury to the patient or the person holding the non-robotic tool. As described later herein, the processor <NUM> may determine a possibility of potential collision between the person and the robotic linkages <NUM>. The processor <NUM> may further display a popup window on a display device <NUM> of the user console <NUM> to provide a warning of the potential collision. The warning may include an audible sound or haptic vibrations to an input handle <NUM>.

Now referring to <FIG>, the processor <NUM> may be a stand-alone computing device similar to the computing device <NUM> of <FIG>, or integrated into one or more of the various components of the robotic surgical system <NUM> (e.g., in the robot bases <NUM> or the user console <NUM>). The processor <NUM> may also be distributed to multiple components of the robotic surgical system <NUM> (e.g., in multiple robot bases <NUM>). The processor <NUM> of the robotic surgical system <NUM> generally includes a processing unit <NUM>, a memory <NUM>, the robot base interface <NUM>, a console interface <NUM>, and an image device interface <NUM>. The robot base interface <NUM>, the console interface <NUM>, and the image device interface <NUM> communicate with the robot bases <NUM>, the user console <NUM>, the imaging devices <NUM> via either wireless configurations, e.g., Wi-Fi, Bluetooth, LTE, and/or wired configurations. Although depicted as a separate module, the console interface <NUM>, the robot base interface <NUM>, and the image device interface <NUM> may be a single component in other embodiments.

The user console <NUM> also includes input handles <NUM> which are supported on control arms <NUM> which allow a clinician to manipulate the surgical robot <NUM> (e.g., move the robotic linkages <NUM>, the ends <NUM> of the robotic linkages <NUM>, and/or the tools <NUM>). Each of the input handles <NUM> is in communication with the processor <NUM> to transmit control signals thereto and to receive feedback signals therefrom. Additionally or alternatively, each of the input handles <NUM> may include input devices (not explicitly shown) which allow the surgeon to manipulate (e.g., clamp, grasp, fire, open, close, rotate, thrust, slice, etc.) the tools <NUM> supported at the ends <NUM> of the robotic linkages <NUM>.

Each of the input handles <NUM> is moveable through a predefined workspace to move the ends <NUM> of the robotic linkages <NUM>, e.g., tools <NUM>, within the surgical site "S". The three-dimensional images on the display device <NUM> are orientated such that the movement of the input handles <NUM> moves the ends <NUM> of the robotic linkages <NUM> as viewed on the display device <NUM>. The three-dimensional images remain stationary while movement of the input handles <NUM> is scaled to movement of the ends <NUM> of the robotic linkages <NUM> within the three-dimensional images. To maintain an orientation of the three-dimensional images, kinematic mapping of the input handles <NUM> is based on a camera orientation relative to an orientation of the ends <NUM> of the robotic linkages <NUM>. The orientation of the three-dimensional images on the display device <NUM> may be mirrored or rotated relative to the view captured by the imaging devices <NUM>. In addition, the size of the three-dimensional images on the display device <NUM> may be scaled to be larger or smaller than the actual structures of the surgical site "S" permitting the clinician to have a better view of internal structures within the surgical site "S". As the input handles <NUM> are moved, the tools <NUM> are moved within the surgical site "S" as detailed below. Movement of the tools <NUM> may also include movement of the ends <NUM> of the robotic linkages <NUM> which support the tools <NUM>.

The user console <NUM> further includes a computer <NUM>, which includes a processing unit or processor and memory, which includes data, instructions and/or information related to the various components, algorithms, and/or operations of the robot bases <NUM>, similar in many respects to the computing device <NUM> of <FIG>. The user console <NUM> may operate using any suitable electronic service, database, platform, cloud, or the like. The user console <NUM> is in communication with the input handles <NUM> and a display device <NUM>. Each input handle <NUM> may, upon engagement by the clinician, provides input signals to the computer <NUM> corresponding to the movement of the input handles <NUM>. Based on the received input signals, the computer <NUM> may process and transmit the signals to the processor <NUM>, which in turn transmits control signals to the robot bases <NUM>, and the devices of the robot bases <NUM>, to effect motion based at least in part on the signals transmitted from the computer <NUM>. The input handles <NUM> may be handles, pedals, or computer accessories (e.g., a keyboard, joystick, mouse, button, touch screen, switch, trackball, and the like).

The user console <NUM> includes the display device <NUM> configured to display two-dimensional and/or three-dimensional images of the surgical site "S", which may include data captured by the imaging devices <NUM> positioned on the ends <NUM> of the robotic linkages <NUM>. The imaging devices <NUM> may capture visual images, infra-red images, ultrasound images, X-ray images, thermal images, and/or any other known real-time images of the surgical site "S". The imaging devices <NUM> transmit captured imaging data to the processor <NUM> which creates three-dimensional images of the surgical site "S" in real-time from the imaging data and transmits the three-dimensional images to the display device <NUM> for displaying such.

The display device <NUM> may be connected to an endoscope installed on the end <NUM> of the robotic linkages <NUM> so that live view images from the endoscope may be displayed on the display device <NUM>. Further, as described above, a potential warning may be displayed in an overlapping manner over the live view images. The endoscope may capture images of the non-robotic/human-held tool <NUM>. Such captured images of the non-robotic/human-held tool <NUM> are transmitted to and processed by the processor <NUM> so that a pose and/or a type of the non-robotic/human-held tool <NUM> may be determined. Such information may be used to determine a volume occupied by the person or a position of the person who holds the non-robotic/human-held tool <NUM>. The person's volume/position and the swept volume may be compared to determine a potential collision between the robotic surgical system <NUM> and the person.

Referring now to <FIG>, a block diagram of an exemplary computing device is shown and is designated generally as a computing device <NUM>. Though not explicitly shown in the corresponding figures of the present disclosure, the computing device <NUM>, or one or more components thereof, may represent one or more components (e.g., the processor <NUM> or the computer <NUM>) of the robotic surgical system <NUM>. The computing device <NUM> may include one or more processors <NUM>, one or more memories <NUM>, input interface <NUM>, output interface <NUM>, network interface <NUM>, or any desired subset of components thereof.

The memory <NUM> includes non-transitory computer-readable storage media for storing data and/or software which include instructions that may be executed by the one or more processors <NUM>. When executed, the instructions may cause the processor <NUM> to control operation of the computing device <NUM> such as, without limitation, reception, analysis, and transmission of sensor signals received in response to movement and/or actuation of the one or more input handles <NUM>. The memory <NUM> may include one or more solid-state storage devices such as flash memory chips. Additionally, or alternatively, the memory <NUM> may include one or more mass storage devices in communication with the processor <NUM> through a mass storage controller and a communications bus (not shown). Although the description of computer readable media described in this disclosure refers to a solid-state storage device, it will be appreciated by one of ordinary skill that computer-readable media may include any available media that can be accessed by the processor <NUM>. More particularly, the computer readable storage media may include, without limitation, non-transitory, volatile, non-volatile, removable, non-removable media, and the like, implemented in any method of technology for storage of information such as computer readable instructions, data structures, program modules, or other suitable data access and management systems. Examples of computer-readable storage media include RAM, ROM, EPROM, EEPROM, flash memory, or other known solid state memory technology, CD-ROM, DVD, Blu-Ray, or other such optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store information and which can be accessed by the computing device <NUM>.

In embodiments, the memory <NUM> stores data <NUM> and/or one or more applications <NUM>. The data <NUM> can include information about various robotic tools and various human-held tools, such as dimensions of such tools. Use of such data <NUM> will be described in more detail in connection with <FIG>. The applications <NUM> may include instructions which are executed on the one or more processors <NUM> of the computing device <NUM>. The applications <NUM> may include instructions which cause an input interface <NUM> and/or an output interface <NUM> to receive and transmit sensor signals, respectively, to and from the various components of the robotic surgical system <NUM>. Additionally or alternatively, the computing device <NUM> may transmit the signals for analysis and/or display via the output interface <NUM>. For example, the memory <NUM> may include instructions which, when executed, generate a depth map or point cloud of the objects within the surgical environment based on the real-time image data received from the image devices of the robotic surgical system <NUM>. The depth map or point cloud may be stored in the memory <NUM> across multiple iterations for a later cumulative analysis of the depth maps or point clouds. In various embodiments, the computing device <NUM> can include data <NUM> and applications <NUM> which implement a trained learning machine, such as a convolutional neural network. In various embodiments, the trained learning machine can process images of a surgical site to identify surgical tools and determine the pose/orientation of the tools, as discussed in more detail in connection with <FIG>.

Additionally, in accordance with aspects of the present disclosure, the memory <NUM> may include instructions that, when executed by the processor <NUM>, identify potential collisions between a person and the robotic surgical system <NUM>. Techniques for identifying a potential collision will be described later herein. The output interface <NUM> may transmit the sensor signals to a display device such as the display device <NUM> of the user console <NUM>, or a remote display located in the surgical environment and in communication with the computing device <NUM>, to display an indication that a collision may occur.

The output interface <NUM> may further transmit and/or receive data via a network interface <NUM> via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi®, Bluetooth® (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE® <NUM>. <NUM>-<NUM> standard for wireless personal area networks (WPANs)). Although depicted as a separate component, the network interface <NUM> may be integrated into the input interface <NUM> and/or the output interface <NUM>.

With additional reference to <FIG>, there is shown a diagram of an exemplary surgical procedure <NUM> using the robotic surgical system <NUM> of <FIG>, including an endoscope <NUM> which is inserted through a body cavity of the patient "P" and is configured to provide optical views or images of the surgical site "S" and to transmit the optical images to the display device <NUM>. The endoscope <NUM> includes a camera <NUM> to capture images of the surgical site "S" during a procedure as detailed below. The surgical procedure <NUM> also includes a robotic tool <NUM> of the robotic surgical system and a human-held non-robotic/human-held tool <NUM>.

In an aspect, the camera <NUM> may be a stereoscopic camera such that the captured images provide depth information that can be used to determine positions/orientations of organs, the robotic tool <NUM>, and/or the non-robotic/human-held tool <NUM>. Further, the stereoscopic images having depth information may be used to identify a pose and/or a type of the non-robotic/human-held tool <NUM> captured therein.

The endoscope <NUM> is inserted through an opening, either a natural opening or an incision, to position the camera <NUM> within the body cavity adjacent the surgical site "S" to allow the camera <NUM> to capture images of the surgical site "S". The camera <NUM> transmits the captured images to the processor <NUM>. The processor <NUM> receives the images or data of the surgical site "S" from the camera <NUM> and displays the received images on the display device <NUM> such that a clinician can visualize the surgical site "S". In various embodiments, the endoscope <NUM> and/or camera <NUM> includes a sensor <NUM> that indicates the pose of the camera <NUM> as the images of the surgical site "S" are captured. The sensor <NUM> is in communication with the processor <NUM> such that the processor <NUM> receives the pose of the camera <NUM> from the sensor <NUM> and associates the pose of the camera <NUM> with the images captured by the camera <NUM>. In various embodiments, the sensor <NUM> may sense six degrees of freedom, including X, Y, and Z axes, as well as pitch, roll, and yaw.

The surgical diagram of <FIG> is exemplary, and in various embodiments, more than one endoscope <NUM> or more than one camera <NUM> can be used. For convenience, a procedure using a single endoscope and a single camera <NUM> will be described below. However, it will be understood that the aspects and embodiments described herein apply to procedures involving multiple endoscopes or cameras, as well.

In accordance with aspects of the present disclosure, the camera <NUM> may capture images of the surgical site "S" and can capture images of the non-robotic/human-held tool <NUM> and the robotic tool <NUM>. The images from the camera <NUM> may be used to identify a position, a pose, and/or a type of the non-robotic/human-held tool <NUM>. As mentioned above, the camera <NUM> can be a stereoscopic camera that provides depth information, which can be used to determine the pose/orientation of the non-robotic/human-held tool <NUM> in the surgical site S. In various embodiments, and as mentioned above, a computing device (e.g., <NUM>, <FIG>) can implement a trained learning machine that can process the images from the camera <NUM> to determine the pose and/or the type of the non-robotic/human-held tool <NUM>. In various embodiments, learning machines such as convolutional neural networks can be trained to recognize and identify objects, and such neural networks can be applied to identify the non-robotic/human-held tool <NUM>. Additionally, learning machines can also be trained to identify other aspects of the tool, such as pose/orientation. In various embodiments, depth information in the stereoscopic images captured by the camera <NUM> can be processed by a trained learning machine to identify a pose or orientation of the non-robotic/human-held tool <NUM>. In various embodiments, pose information from the sensor <NUM> of the endoscope <NUM> can also be input to and processed by the trained learning machine to determine the pose/orientation of the non-robotic/human-held tool <NUM>. The learning machine described above is exemplary, and other types or implementations of machine learning are contemplated to be within the scope of the present disclosure.

In accordance with aspects of the present disclosure, the position of the person holding the non-robotic/human-held tool <NUM> can be inferred based on the type and/or the pose of the non-robotic/human-held tool <NUM>. Thus, the type of the non-robotic/human-held tool <NUM> and/or the pose/orientation of the non-robotic/human-held tool <NUM> are the starting point for inferring the position of the person holding the non-robotic/human-held tool <NUM>. Accordingly, the disclosed techniques for determining the position of the person holding the non-robotic/human-held tool <NUM> are not based on an image of the person.

In various embodiments, identifying the type of the non-robotic/human-held tool <NUM> may be sufficient to infer the position of the person holding the tool. For example, if there is one particular way to orient and hold the identified non-robotic/human-held tool <NUM>, then identifying the particular non-robotic/human-held tool <NUM> would provide an inference with regard to the position of the person holding the non-robotic/human-held tool <NUM>.

In various embodiments, when the non-robotic/human-held tool <NUM> can have various orientations, the orientation of the non-robotic/human-held tool <NUM> can be determined from the stereoscopic images of the camera <NUM> using a trained learning machine, as described above. Different orientations of the non-robotic/human-held tool <NUM> can correspond to different ways of holding the non-robotic/human-held tool <NUM> and correspond to different positions of the person holding the non-robotic/human-held tool <NUM>. In various embodiments, such correspondence can be stored in a database (e.g., <NUM>, <FIG>), for each of the possible orientations of the non-robotic/human-held tool <NUM>. In various embodiments, a trained learning machine can predict a position of the person holding the non-robotic/human-held tool <NUM> based on the pose/orientation of the non-robotic/human-held tool <NUM>. Such a learning machine can be trained using training data relating to the pose/orientation of human-held tools and which are tagged with information relating to the position of the person holding such tools.

In an aspect, the position of the person holding the hand-held tool can be inferred based on personal dimensions of the person holding the tool, such as height dimensions and/or arm dimensions, among other dimensions. Such personal dimension information can be stored in a database of such information (e.g., <NUM>, <FIG>). The database can include such dimensional information for each person who is assigned to the surgical procedure and who could be tasked with holding the non-robotic/human-held tool <NUM>. The processor <NUM> may access the database and infer the position of the person holding the non-robotic/human-held tool <NUM> based on the personal information.

The position of a person holding a non-robotic tool can be represented in various ways. In various embodiments, the position can be represented as a cylindrical volume or as another three-dimensional volume. In various embodiments, the position can be represented by a more detailed model, such as a volume including protrusions corresponding to appendages or other body positions. Such possible ways of representing a person's position are exemplary, and other variations are contemplated to be within the scope of the disclosure.

In accordance with aspects of the disclosure, and with continuing reference to <FIG>, the techniques described above for identifying a type and/or pose of a human-held tool is also applied to identifying a type and/or pose of a robotic tool <NUM>. The physical position of the robotic linkage coupled to the robotic tool <NUM> (e.g., <NUM>, <FIG>) can be determined based on the type and/or the pose of the robotic tool <NUM>. Thus, the type of the robotic tool <NUM> and/or the pose/orientation of the robotic tool <NUM> are the starting point for determining the position of the robotic linkage coupled to the robotic tool <NUM>. Accordingly, the disclosed techniques for determining the position of the robotic linkage coupled to the robotic tool <NUM> are not based on an image of the robotic system (<NUM>, <FIG>) nor based on any information about the position of the robotic system in physical space. Rather, as explained below, the disclosed techniques are based on an image of the robotic tool <NUM> at the surgical site and are based on information of the robotic system <NUM> on the angle and velocity of each joint of the robotic linkages.

The images from the camera <NUM> can capture images of the robotic tool <NUM> at the surgical site. In various embodiments, a computing device (e.g., <NUM>, <FIG>) can implement a trained learning machine that can process the images from the camera <NUM> to determine the pose and/or the type of the robotic tool <NUM>. In various embodiments, learning machines such as convolutional neural networks can be trained to recognize and identify objects, and such neural networks can be applied to identify the robotic tool <NUM>. Additionally, learning machines can also be trained to identify other aspects of the tool, such as pose/orientation. In various embodiments, depth information in the stereoscopic images captured by the camera <NUM> can be processed by a trained learning machine to identify a pose or orientation of the robotic tool <NUM>. In various embodiments, pose information from the sensor <NUM> of the endoscope <NUM> can also be input to and processed by the trained learning machine to determine the pose/orientation of the robotic tool <NUM>. The learning machine described above is exemplary, and other types or implementations of machine learning are contemplated to be within the scope of the present disclosure.

With reference also to <FIG>, the processor <NUM> accesses information on angles between each joint of the linkages or robotic arms <NUM> and angular velocities of each joint. Based on the received information from the robotic linkages <NUM>, the processor <NUM> is to able apply such information to the type and/or pose of the robotic tool <NUM> to determine the position of each portion of the robotic linkages <NUM> and to determine a swept volume of the robotic surgical system <NUM>, which is a volume of space that the robotic surgical system <NUM> can occupy during a time period. In various embodiments, dimensions of the robotic tool <NUM> can be stored in a database (e.g., <NUM>, <FIG>), and dimension of the robotic linkages <NUM> can also be stored in the database. After identifying the type and/or pose of the robotic tool <NUM>, the dimensions of the robotic tool <NUM> can be retrieved from the database and the entire position of the robotic tool <NUM> can be determined. Using the joint angles and arm dimensions, the position of each linkage of the robotic arm coupled to the robotic tool <NUM> can be determined. Then, the velocity of each joint can be used to project the motion of the robotic linkages <NUM> forward in time, to determine the swept volume of the robotic linkages <NUM> and the robotic system <NUM>.

In various embodiments, when the distance between the swept volume of the robotic surgical system <NUM>/robotic linkage <NUM> and the inferred position of the person is less than or equal to a threshold distance, the processor <NUM> determines that a potential collision can occur. The threshold distance can be, for example, <NUM> inches or <NUM> inches, or another distance. The distance can be computed using, for example, the closest portions of the swept volume and the inferred position of the person. In various embodiments, when the swept volume of the robotic surgical system <NUM>/robotic linkage <NUM> overlaps with the inferred position of the person holding the non-robotic/human-held tool <NUM>, the processor <NUM> determines that a potential collision can occur. In case of a potential collision, the processor <NUM> may overlay a warning window over the display device <NUM>. In an aspect, the warning may be a haptic vibration feedback on the input handle <NUM>, an audio warning, red-flashes on the display device <NUM>, or any combination thereof.

When potential collisions are to be differentiated from an imminent collision, there are two thresholds - one for potential collisions and the other one for the imminent collision. For example, an imminent collision can be based on the swept volume and the inferred position of the person overlapping with each other, whereas a potential collision can be based on the swept volume and the inferred position of the person not overlapping but being within a threshold distance of each other. In this case of an imminent collision, the processor <NUM> may immediately stop movement of the robotic surgical system <NUM> to reduce the chance of such an imminent collision. In various embodiments, the processor <NUM> may decrease a maximum torque of the motor <NUM> as the robotic arm <NUM> is close to the imminent collision, or decrease speed limits, acceleration limits, or maximum external torques that can be produced, where the external torque is equal to subtract an actuator torque and a gravitational torque from a dynamic torque of the robotic arm <NUM>.

In an aspect, the processor <NUM> may change a damping gain in controlling the position or the angular velocity of the robotic arm <NUM> such that the controller <NUM> becomes smoother in the direction that the imminent/potential collision may occur. Also, the processor <NUM> may increase a scaling factor so as to decrease the angular velocity and to provide more reaction time as the haptic/visual/audio feedback is provided.

<FIG> is a flowchart illustrating a method <NUM> for mitigating a potential collision during a robotic surgery in accordance with embodiments of the disclosure. When an end portion of a robotic linkage is inserted through an incision or orifice of a patient, an endoscope disposed at the end portion of the robotic linkage may capture images of a surgical site. The method <NUM> starts by accessing images from the endoscope within the surgical site in step <NUM>. The endoscope may include a stereoscopic imaging device, which captures stereoscopic images.

The robotic linkage includes a plurality of arms, between which a joint connects. The link may include one or more sensors, which sense an angle between the connected arms and a velocity thereof. In step <NUM>, joint angles and velocities of a robotic arm are received. Such information may include an angle between two connected arms of a joint of the robotic linkage and a velocity of each joint. In an aspect, the velocity may be an angular velocity of each joint.

In step <NUM>, it is determined whether or not a non-robotic tool <NUM> is captured in the images. When it is determined that the non-robotic tool <NUM> is not captured in the images, the method <NUM> goes back to step <NUM>.

When it is determined that the non-robotic tool <NUM> is captured in the images in step <NUM>, a type and/or pose of the non-robotic tool <NUM> may be determined in step <NUM>. The stereoscopic images include depth information, which may be processed to identify the pose of the non-robotic tool <NUM> positioned within the surgical site. Based on the type and/or pose of the non-robotic tool <NUM>, a position of a person holding the non-robotic tool <NUM> may be inferred in step <NUM>. In a case when the position of the person cannot be determine based on the stereoscopic images, a predetermined or default position of the person holding the non-robotic tool <NUM> can be used as the inferred position of the person. This may occur, for example, when the type or pose of the non-robotic tool <NUM> is not identifiable in the stereoscopic images.

In step <NUM>, the swept volume of the robotic linkage <NUM> may be determined. The type and/or pose of a robotic tool <NUM> at the surgical site can be determined based on an image of the robotic tool <NUM> at the surgical site. The angle of each joint and dimension of each arm segment (or robotic linkage <NUM>) can be used to determine the positions of the robotic linkage <NUM> by starting the determination from the type and/or pose of the robotic tool <NUM>. The velocity of each joint can be used to determine the swept volume of the robotic linkage <NUM>. The swept volume is a volume of space that can be occupied by the robotic linkage <NUM> during a time frame. The time frame can vary and can be, for example, <NUM> second or <NUM> seconds or another time period. The angular velocity of each joint includes a direction and an angular speed. Thus, the angular travel distance of each joint and each arm or linkage can be calculated in the direction and the corresponding swept volume can be then calculated. Based on the configuration (e.g., connection structure of each joint) of the robotic linkage <NUM>, the swept volume of the robotic linkage <NUM> may be determined by combining the swept volume of each j oint. Thus, the swept volume of the robotic linkage <NUM> is determined from calculating the swept volume of each arm or linkage.

The distance or overlap between the swept volume and the inferred position of the person may be the shortest distance between the position of the person and the swept volume. At step <NUM>, the distance between the volume of the person and the swept volume is compared with the first threshold. When it is determined that the distance is less than or equal to the first threshold, a warning of potential collision is generated in step <NUM>. The warning may be haptic vibrations on an input handle of the robotic linkage, a visual warning overlaid on a display of the robotic surgical system, or an audible warning. The listing of warnings is not meant to be limiting but may include any other suitable means as readily appreciated by a person having ordinary skill in the art.

At step <NUM>, it is further determined whether or not the distance is less than or equal to a second threshold. In a case when the distance is determined to be less than or equal to the second threshold, the movement of the robotic linkage <NUM> is stopped in step <NUM>. The second threshold is a value indicating an imminent collision, meaning that the robotic linkage is about to collide with the person or another robotic linkage <NUM>. Thus, in this situation, the robotic linkage <NUM> stops movement to reduce the chance of the imminent collision, in step <NUM>. The method <NUM> then ends. In an aspect, after stopping movements of the robotic linkage <NUM>, the clinician may adjust the settings of the robotic linkage <NUM> or position of the person and re-initiate the method <NUM> again.

When it is determined that the distance is greater than the second threshold in step <NUM>, it is further determined whether or not the surgery is complete in step <NUM>. When the surgery is not completed, steps <NUM>-<NUM> are repeated until the surgery is completed.

When it is determined that the surgery is completed in step <NUM>, the method <NUM> ends. In this way, the surgery with the robotic surgical system and the non-robotic tool <NUM> can be performed while mitigating potential collisions between the robotic system <NUM> and a person holding a non-robotic tool <NUM>.

Claim 1:
A robotic surgical system comprising:
a robotic linkage including a plurality of joints;
an endoscope coupled to a distal portion of the robotic linkage and configured to capture stereoscopic images; and
a controller in operable communication with the endoscope, characterized in that the controller is configured to execute instructions to cause the controller to:
analyze the stereoscopic images from the endoscope to identify a human-held tool in the stereoscopic images and to estimate at least one of a type or pose of the human-held tool,
infer a position of a person holding the human-held tool based on the at least one of the type or pose of the human-held tool,
determine a spatial relationship between the person and the robotic linkage based on the inferred position of the person, and
generate a warning of a potential collision between the person and the robotic linkage based on the determined spatial relationship.