Patent Description:
Minimally-invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during a surgical procedure. For example, laparoscopic procedures typically involve creating a number of small incisions in the patient (e.g., in the abdomen), and introducing one or more tools and at least one endoscopic camera through the incisions into the patient. The surgical procedures are then performed by using the introduced tools, with the visualization aid provided by the camera. Generally, MIS provides multiple benefits, such as reduced patient scarring, less patient pain, shorter patient recovery periods, and lower medical treatment costs associated with patient recovery. In some embodiments, MIS may be performed with robotic systems that include one or more robotic arms for manipulating surgical instruments based on commands from an operator.

In MIS procedures, access is provided to the body cavity of a patient through a trocar. Once a distal end of a cannula of the trocar is properly positioned and inserted through tissue and into an interior region of the patient, for example, through the abdominal wall of the patient, a surgical robotic arm having a trocar docking interface at its distal end is manually maneuvered by a user until the interface is adjacent to and aligned with an attachment portion (e.g., a mating interface) on the proximal end of the trocar (outside the patient. ) The user then manually latches the arm and trocar docking interfaces to each other, thereby rigidly attaching the arm to the trocar. A surgical tool having an end effector at its distal end (e.g., scissors, grasping jaws, or camera) is then inserted into an outside opening of the cannula, and a transmission housing of the tool is then attached to the arm.

<CIT> and <CIT> are directed to systems and methods for navigating a medical instrument into an entry point of a patient, and more particularly to systems and methods for localizing the entry point.

Applicant has discovered a need for improved systems and methods for docking a surgical robotic arm to a trocar that has been inserted into a patient. Such techniques should obviate the challenges that are presented by some modalities of trocar docking. For example, some trocar docking procedures employ optical tracking through the use of visual imaging sensors that guide the surgical robotic arm to the trocar. However, visual sensors can be blocked by the sterile barriers or drapes that cover the surgical robotic arm and its surrounding environment. Additional examples of trocar docking techniques, for example, ultrasonic triangulation, inertial sensing, and the detection of generated electromagnetic fields, involve the use of electrically powered components on the trocar that generate signals that can be used to guide the robotic arm. However, such electrically powered equipment can reduce the lifespan of a trocar, as these components can degrade, for example, due to repeated use and/or through sterilization procedures.

The use of magnets, for example, non-electrically powered magnets such as permanent magnets, in the trocar can provide magnetic fields for detection by a sensor system in a surgical robotic arm, such that the robotic arm can be controlled to automatically align with a pose of the trocar where it can then be mechanically coupled to the trocar. The use of such magnetic sensing does not require a line-of-sight between the robotic arm and the trocar so that, for example, sterile barriers can be used to cover portions of the robotic arm without interfering with trocar docking procedures. In addition, the use of magnets in the trocar to generate the signals based on which the robotic arm is guided does not require electrically powered components and as such the trocar is more robust having increased lifespan and versatility.

In one embodiment, an arm-to-trocar docking capability of a surgical robotic system senses position, orientation or both (pose) of the trocar. The surgical robotic system includes a surgical robotic arm, magnetic field sensors on the arm, and a digital processor that implements a machine learning model (e.g., an artificial neural network "neural network"). The machine learning model is coupled to receive output data of the magnetic field sensors. The machine learning model is trainable to output a three-dimensional sensed position, a three-dimensional sensed orientation, or both (sensed pose, in six degrees of freedom), of a trocar that is producing a magnetic field. The sensing of three-dimensional position or orientation of the trocar is thus based on output data from the trocar-mounted magnetic field sensors that propagates through the machine learning model. In one version, the surgical robotic system performs a digital algorithm that automatically drives the motorized joints of the surgical robotic arm to guide the docking interface on the arm to dock with the trocar, based on its sensed position or orientation (e.g., both, as pose) of the trocar.

The above summary does not include an exhaustive list of all aspects of the present disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the Claims section. Such combinations may have particular advantages not specifically recited in the above summary.

Several aspects of the disclosure here are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to "an" or "one" aspect in this disclosure are not necessarily to the same aspect, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one aspect of the disclosure, and not all elements in the figure may be required for a given aspect.

Several aspects of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other aspects of the parts described are not explicitly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some aspects of the disclosure may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description.

Referring to <FIG>, this is a pictorial view of an example surgical robotic system <NUM> in an operating arena. The robotic system <NUM> includes a user console <NUM>, a control tower <NUM>, and one or more surgical robotic arms <NUM> that may be mounted to a surgical robotic platform <NUM>, e.g., a table, a bed, etc. The system <NUM> can incorporate any number of devices, tools, or accessories used to perform surgery on a patient <NUM>. For example, the system <NUM> may include one or more surgical tools <NUM> used to perform surgery. A surgical tool <NUM> may be an end effector that is attached to a distal end of a surgical arm <NUM>, for executing a surgical procedure.

Each surgical tool <NUM> may be manipulated manually, robotically, or both, during the surgery. For example, the surgical tool <NUM> may be a tool used to enter, view, or manipulate an internal anatomy of the patient <NUM>. In an embodiment, the surgical tool <NUM> is a grasper that can grasp tissue of the patient. The surgical tool <NUM> may be controlled manually, by a bedside operator <NUM>; or it may be controlled robotically, via actuated movement of the surgical robotic arm <NUM> to which it is attached. The robotic arms <NUM> are shown as a table-mounted system, but in other configurations the arms <NUM> may be mounted in a cart, ceiling or sidewall, or in another suitable structural support.

Generally, a remote operator <NUM>, such as a surgeon or other operator, may use the user console <NUM> to remotely manipulate the arms <NUM> and/or the attached surgical tools <NUM>, e.g., teleoperation. The user console <NUM> may be located in the same operating room as the rest of the system <NUM>, as shown in <FIG>. In other environments however, the user console <NUM> may be located in an adjacent or nearby room, or it may be at a remote location, e.g., in a different building, city, or country. The user console <NUM> may comprise a seat <NUM>, foot-operated controls <NUM>, one or more handheld user input devices, UID <NUM>, and at least one user display <NUM> that is configured to display, for example, a view of the surgical site inside the patient <NUM>. In the example user console <NUM>, the remote operator <NUM> is sitting in the seat <NUM> and viewing the user display <NUM> while manipulating a foot-operated control <NUM> and a handheld UID <NUM> in order to remotely control the arms <NUM> and the surgical tools <NUM> (that are mounted on the distal ends of the arms <NUM>.

In some variations, the bedside operator <NUM> may also operate the system <NUM> in an "over the bed" mode, in which the beside operator <NUM> (user) is now at a side of the patient <NUM> and is simultaneously manipulating a robotically-driven tool (end effector as attached to the arm <NUM>), e.g., with a handheld UID <NUM> held in one hand, and a manual laparoscopic tool. For example, the bedside operator's left hand may be manipulating the handheld UID to control a robotic component, while the bedside operator's right hand may be manipulating a manual laparoscopic tool. Thus, in these variations, the bedside operator <NUM> may perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient <NUM>.

During an example procedure (surgery), the patient <NUM> is prepped and draped in a sterile fashion to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of the robotic system <NUM> are in a stowed configuration or withdrawn configuration (to facilitate access to the surgical site. ) Once access is completed, initial positioning or preparation of the robotic system <NUM> including its arms <NUM> may be performed. Next, the surgery proceeds with the remote operator <NUM> at the user console <NUM> utilizing the foot-operated controls <NUM> and the UIDs <NUM> to manipulate the various end effectors and perhaps an imaging system, to perform the surgery. Manual assistance may also be provided at the procedure bed or table, by sterile-gowned bedside personnel, e.g., the bedside operator <NUM> who may perform tasks such as retracting tissues, performing manual repositioning, and tool exchange upon one or more of the robotic arms <NUM>. Non-sterile personnel may also be present to assist the remote operator <NUM> at the user console <NUM>. When the procedure or surgery is completed, the system <NUM> and the user console <NUM> may be configured or set in a state to facilitate post-operative procedures such as cleaning or sterilization and healthcare record entry or printout via the user console <NUM>.

In one embodiment, the remote operator <NUM> holds and moves the UID <NUM> to provide an input command to move a robot arm actuator <NUM> in the robotic system <NUM>. The UID <NUM> may be communicatively coupled to the rest of the robotic system <NUM>, e.g., via a console computer system <NUM>. The UID <NUM> can generate spatial state signals corresponding to movement of the UID <NUM>, e.g. position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals to control a motion of the robot arm actuator <NUM>. The robotic system <NUM> may use control signals derived from the spatial state signals, to control proportional motion of the actuator <NUM>. In one embodiment, a console processor of the console computer system <NUM> receives the spatial state signals and generates the corresponding control signals. Based on these control signals, which control how the actuator <NUM> is energized to move a segment or link of the arm <NUM>, the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the UID <NUM>. Similarly, interaction between the remote operator <NUM> and the UID <NUM> can generate for example a grip control signal that causes a jaw of a grasper of the surgical tool <NUM> to close and grip the tissue of patient <NUM>.

The surgical robotic system <NUM> may include several UIDs <NUM>, where respective control signals are generated for each UID that control the actuators and the surgical tool (end effector) of a respective arm <NUM>. For example, the remote operator <NUM> may move a first UID <NUM> to control the motion of an actuator <NUM> that is in a left robotic arm, where the actuator responds by moving linkages, gears, etc., in that arm <NUM>. Similarly, movement of a second UID <NUM> by the remote operator <NUM> controls the motion of another actuator <NUM>, which in turn moves other linkages, gears, etc., of the robotic system <NUM>. The robotic system <NUM> may include a right arm <NUM> that is secured to the bed or table to the right side of the patient, and a left arm <NUM> that is at the left side of the patient. An actuator <NUM> may include one or more motors that are controlled so that they drive the rotation of a joint of the arm <NUM>, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical tool <NUM> that is attached to that arm. Motion of several actuators <NUM> in the same arm <NUM> can be controlled by the spatial state signals generated from a particular UID <NUM>. The UIDs <NUM> can also control motion of respective surgical tool graspers. For example, each UID <NUM> can generate a respective grip signal to control motion of an actuator, e.g., a linear actuator, which opens or closes jaws of the grasper at a distal end of surgical tool <NUM> to grip tissue within patient <NUM>.

In some aspects, the communication between the platform <NUM> and the user console <NUM> may be through a control tower <NUM>, which may translate user commands that are received from the user console <NUM> (and more particularly from the console computer system <NUM>) into robotic control commands that transmitted to the arms <NUM> on the robotic platform <NUM>. The control tower <NUM> may also transmit status and feedback from the platform <NUM> back to the user console <NUM>. The communication connections between the robotic platform <NUM>, the user console <NUM>, and the control tower <NUM> may be via wired and/or wireless links, using any suitable ones of a variety of data communication protocols. Any wired connections may be optionally built into the floor and/or walls or ceiling of the operating room. The robotic system <NUM> may provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system.

To create a port for enabling introduction of a surgical instrument into the patient <NUM>, a trocar assembly may be at least partially inserted into the patient through an incision or entry point in the patient (e.g., in the abdominal wall). The trocar assembly may include a cannula or trocar <NUM> (<FIG>), an obturator, and/or a seal. In some variations, the trocar assembly can include an obturator such as a needle with a sharpened tip for penetrating through a patient's skin. The obturator may be disposed within the lumen of the trocar <NUM> when being inserted into the patient <NUM>, and then removed from the trocar <NUM> such that a surgical instrument may be inserted through the lumen of the trocar <NUM>. Once positioned within the body of the patient <NUM>, the trocar <NUM> may provide a channel for accessing a body cavity or other site within the patient <NUM>, for example, such that one or more surgical instruments or tools can be inserted into a body cavity of the patient <NUM>, as described further herein.

Turning to <FIG>, a portion of a robotic arm <NUM> is illustrated according to an exemplary embodiment of the disclosure. The robotic arm <NUM> and associated components described herein can form a surgical robotic assembly <NUM> according to an exemplary embodiment of the disclosure. The surgical robotic assembly <NUM> can be incorporated into the surgical robotic system <NUM> described above, or can form a portion of a different system. The robotic arm <NUM> can include a plurality of links and a plurality of actuated joint modules that enable relative movement between adjacent links. While a single robotic arm <NUM> has been illustrated, it will be understood that the robotic arm <NUM> can include additional arm portions or can be a component of a multi-arm apparatus without departing from the disclosure.

The robotic arm <NUM> can include a plurality of links (e.g., links 20A - 20E) and a plurality of joint modules (e.g., joints 21A-21E) for actuating the plurality of links relative to one another. The joint modules can include various joint types, such as a pitch joint or a roll joint, any of which can be actuated manually or by the robotic arm actuators <NUM>, and any of which may substantially constrain the movement of the adjacent links around certain axes relative to others. As also shown, a tool drive <NUM> is attached to the distal end of the robotic arm <NUM>. As described herein, the tool drive <NUM> can be configured with a docking interface <NUM> to receive and physically latch or lock with an attachment portion (e.g., a mating interface) of a trocar <NUM> such that one or more surgical instruments (e.g., endoscopes, staplers, etc.) can be guided through a lumen of the cannula of the trocar <NUM>. The plurality of the joint modules 21A-21E of the robotic arm <NUM> can be actuated to position and orient the tool drive <NUM> for robotic surgeries.

<FIG> is a schematic diagram illustrating an exemplary tool drive <NUM> without a loaded tool, in accordance with aspects of the subject technology. In one variation, the tool drive <NUM> may include an elongated base (or "stage") <NUM> having longitudinal tracks <NUM> and a tool carriage <NUM>, which is slidingly engaged with the longitudinal tracks <NUM>. The base <NUM> may be configured to couple to the distal end of a robotic arm <NUM> such that articulation of the robotic arm <NUM> positions and/or orients the tool drive <NUM> in space. The tool carriage <NUM> may be configured to receive a base of a tool whose elongated portion has been inserted into and extend through the trocar <NUM>. The tool carriage <NUM> may actuate a set of articulated movements of the tool through a cable system or wires manipulated and controlled by actuated drives (the terms "cable" and "wire" are used interchangeably throughout this application). The tool carriage <NUM> may include different configurations of actuated drives, such as a mechanical transmission.

The tool drive <NUM> is configured to receive different surgical tools (e.g., surgical tool <NUM>, as well as other detachable surgical tools - not shown) that can be selectively attached, either one at a time or in combination. Such surgical tools can be, for example, jaws, cutting tools, an endoscope, spreader, implant tool, energy emitter, etc. In this regard, the tool drive <NUM> can include one or more drive disks and/or other adapters that interface with and engage portions of the surgical tools that are attached thereto. The drive disks are actuatable, for example, through a mechanical transmission in the tool drive <NUM>, to transfer force or torque to the drive disks <NUM> to effect operation of the attached surgical tool.

Referring now to <FIG>, an example of a docking interface <NUM> that is coupled to the base <NUM> of a tool drive <NUM> is shown. The trocar <NUM> can be coupled to the tool drive <NUM> or to another location on the arm <NUM> at the docking interface <NUM>. The docking interface <NUM> in this example is located at a distal block of the elongated base <NUM> - see also <FIG>. The docking interface <NUM> is configured to receive a portion of the trocar <NUM> such that the docking interface <NUM> is configured as a trocar docking interface, a trocar attachment device, or a trocar mounting device. The docking interface <NUM> can provide a reliable and quick way to removably attach the trocar <NUM> to the arm <NUM>.

The docking interface <NUM> can define a chamber <NUM> that is accessible through a mouth or frontal opening <NUM> of the docking interface <NUM> and which can include first and second clamp components <NUM>, <NUM> (e.g., arms, plates, levers, members) arranged about a receiver <NUM> that defines a receiving space <NUM> for receiving a portion of the trocar <NUM> (e.g., a mating interface formed in an attachment portion of a cannula located in a proximal portion of the cannula). At least one of the clamp components <NUM>, <NUM> may be pivotable between an open position and a closed position such that an attachment portion <NUM> of the trocar <NUM> can be inserted into the receiving space <NUM> between the clamp components <NUM>, <NUM> so that a portion of the trocar <NUM> is held in place at least partially by the first and second clamp components <NUM>, <NUM>.

In one variation, the docking interface <NUM> may include an over-center mechanism such as a lever <NUM> or other suitable locking component that mechanically cooperates with the clamp component <NUM>, for example, through a pin and slot arrangement or through another pivotable or movable connection, between the open and closed positions. The lever <NUM> can be movable, e.g., along a track or slot defined in a body or housing of the docking interface <NUM>, between a forward, locked position (e.g., a locked over-center position) and a rearward, unlocked position. When the lever <NUM> is moved toward the locked position, the lever <NUM> may urge the clamp component <NUM> downwardly toward the receiving space <NUM> and lock the clamp component <NUM> in the closed position such that a portion of the trocar <NUM> is securely held between the first and second clamp components <NUM>, <NUM>. In some variations, second clamp component <NUM> can be stationary or can be fixed. In one variation, the lever <NUM> can be controlled and/or driven with an electric motor or actuator under manual or processor control.

In some variations, the docking interface <NUM> may also provide a sterile barrier between sterile components such as the trocar <NUM> and non-sterile components such as the first and second clamp components <NUM>, <NUM> (or other non-sterile components of the surgical system). The sterile barrier may be provided, for example, by a sterile adapter formed of a surgical-grade polymer or other surgical-grade material that is interposed between the trocar <NUM> and the first and second clamp components <NUM>, <NUM> (not shown for clarity of illustration).

Referring additionally to <FIG>, the docking interface <NUM> also includes a sensor system <NUM> that includes at least a motherboard or first sensor board <NUM> at a first location of the docking interface <NUM> and a daughterboard or second sensor board <NUM> at second location of the docking interface <NUM> and in electrical communication with the first sensor board <NUM> via a cable <NUM> or other electrically conductive connection. In one variation, communication between the sensor boards <NUM>, <NUM> can employ a multi-slave and multi-master inter-integrated communication computer bus. One or both of the sensor boards <NUM>, <NUM> can include a microprocessor or other associated processor, for example, to control and/or read the sensors of the sensor boards <NUM>, <NUM> and to facilitate communication between the sensor boards <NUM>, <NUM>, e.g., to enable temporal synchronization between the sensor boards <NUM>, <NUM>. As shown, the first sensor board <NUM> and the second sensor board <NUM> are positioned spaced apart from but parallel to each other, e.g., facing each other, on opposite lateral sides of the chamber <NUM> of the docking interface <NUM>. The first sensor board <NUM> includes a first plurality of sensors <NUM> and the second sensor board <NUM> includes a second plurality of sensors <NUM>. In this regard, the sensors <NUM>, <NUM> are embedded in or otherwise coupled to the robotic arm <NUM> or the tool drive <NUM>. Each of the plurality of sensors <NUM>, <NUM> are arranged such that at least one sensor <NUM>, <NUM> is disposed rearward, e.g., at a depth measured from the frontal opening <NUM> of the docking interface <NUM>, with respect to another respective sensor <NUM>, <NUM>. As shown, sensors <NUM>, <NUM> are disposed at least at a first depth D1, a second depth D2, a third depth D3, and a fourth depth D4, with D4>D3>D2>D1. The depths D1, D2, D3, D4 can be spaced at uniform or non-uniform increments without departing from the disclosure. While the sensors <NUM>, <NUM> have been described in a grid-like configuration of rows R1-R4 and columns C1-C4, it will be understood that one or both of the pluralities of sensors <NUM>, <NUM> can have a different arrangement without departing from the disclosure.

As described further herein, the sensors <NUM>, <NUM> are operable to sense or measure a magnetic field associated with the trocar <NUM>, and produce respective corresponding electrical signals. In this regard, the sensors <NUM>, <NUM> can be configured as magnetometers, e.g., sensors that receive at least a portion of a magnetic field as an input and produce an output electrical signal corresponding to a strength or other characteristic of the magnetic field, and such that the sensors <NUM>, <NUM> can be transducers. Any of the sensors <NUM>, <NUM> can be configured to receive a different physical input and produce a corresponding electrical signal, for example, inertial measurement units, accelerometers, etc. In this regard, the sensors <NUM>, <NUM> produce an output electrical signal that can be electrically communicated to, for example, a processor or controller that is incorporated into the control tower <NUM> to provide force or velocity commands to guide (e.g., direct a movement of) the robotic arm <NUM> via the robotic arm actuators <NUM>, as described further herein. It will be understood that a processor can be incorporated into additional or alternative portions of the surgical robotic system <NUM>, and that the sensor system <NUM> can be in electrical communication with one or more different processors. A switch <NUM> or other control is mounted on or near the docking interface <NUM>, for example, behind the lever <NUM> at a position such that the lever <NUM> can be urged into contact with the switch <NUM>, as described further herein. The switch <NUM> can be in electrical communication with the processor in the control tower <NUM> to signal the processor to energize or activate one or both of the sensor boards <NUM>, <NUM> to activate the sensor system <NUM> to sense or measure magnetic fields, and to effect guidance of the robotic arm <NUM> toward the trocar <NUM> according to an algorithm, as described further herein. In one variation, the sensor system <NUM> can be activated by the processor prior to or independently of the action of the switch <NUM>, and the switch <NUM> can be used to signal the processor to begin calculations based on the signals received from the sensor system <NUM> to determine the estimated pose of the trocar and then affect guidance of the robotic arm <NUM> and its coupled tool drive <NUM>. The switch <NUM> can be have one of several different configurations, e.g., a mechanical button and mechanical switch combination may be preferred but another form of tactile interface or a touchscreen is also possible, that can be activated by a user. Such placement of the switch <NUM> on or near the docking interface <NUM> allows an operator to activate a docking process without the need to travel away from the robotic arm <NUM> to a separate control interface, for example, the user control <NUM> that is located away from the robotic arm <NUM>/tool drive <NUM>.

While the sensor boards <NUM>, <NUM> have been generally described as respective first and second printed circuit boards (PCBs) including the respective sensors <NUM>, <NUM> embedded therein or thereon, it will be understood that the sensor system <NUM> can be provided in a different arrangement, for example, as discrete components, without departing from the disclosure. Additionally, it will be understood that any of the components described herein can be in communication via wired and/or wireless links, using any suitable ones of a variety of data communication protocols.

Referring additionally to <FIG>, guidance and docking of the docking interface <NUM> of the tool drive <NUM> with a trocar <NUM> that is at least partially inserted into the patient <NUM> is illustrated according to one aspect of the disclosure. The trocar <NUM>, as shown, includes a generally tubular body <NUM> with a flanged upper portion or head <NUM> and an attachment portion <NUM> that protrudes from the head <NUM> for mating with the docking interface <NUM>. In one variation, the attachment portion <NUM> can be configured, for example, as having a nose or collar or pin-like arrangement, and can have one or more surface features, e.g., notches, ridges, protrusions, angles, hooks, etc., for interengaging the receiver <NUM> of the docking interface <NUM>.

The trocar <NUM> can have a different arrangement without departing from the disclosure. The trocar <NUM> includes a first magnet <NUM> and a second magnet <NUM> producing respective magnetic fields B1, B2 with known properties, e.g., known axes of polarization or angles therebetween, known dipole moments, known positions with respect to each other, etc. The first magnet <NUM> and the second magnet <NUM> each can have a different axis of polarization, e.g., an axis extending between opposite poles of the respective magnets <NUM>, <NUM>. In this regard, the first magnet <NUM> and the second magnet <NUM> may be obliquely arranged relative to one another, e.g., such that an angle is disposed between the respective axes of polarization. One or both of the magnets <NUM>, <NUM> can be embedded in or otherwise coupled to the trocar <NUM>, for example, by being integrally molded therein, by being inserted into a receiving portion thereof, or by being otherwise secured to the trocar <NUM>. In one variation, the magnets <NUM>, <NUM> are integrally formed in the attachment portion <NUM> of the trocar <NUM>. In other variations, the magnets <NUM>, <NUM> can be coupled to or embedded in a different portion of the trocar <NUM>. While the trocar <NUM> is described as having the pair of magnets <NUM>, <NUM>, it will be understood that the trocar <NUM> can have a different number of magnets, e.g., provided as multiple pairs or singly-arranged magnets, without departing from the disclosure. In one variation, the trocar <NUM> can include a single magnet.

Still referring to <FIG>, and with additional reference to the process flows of <FIG> and <FIG>, a method for docking the robotic arm <NUM> to the trocar <NUM> according to aspects of the disclosure will be described and shown. The robotic arm <NUM> and docking interface <NUM>, in a first or parked or unknown pose, is a pose in which the docking interface <NUM> is positioned a distance away from the magnets <NUM>, <NUM> in the attachment portion <NUM> of the trocar <NUM> and respective magnetic fields B1, B2 generated therefrom such that a closer distance between the docking interface <NUM> and the trocar <NUM> is desirable to facilitate effective receipt or sensing of the magnetic fields B1, B2 by the sensors <NUM>, <NUM>. The parked or unknown pose of the robotic arm <NUM> can be, for example, a stowed arrangement of the robotic arm <NUM>.

The docking interface <NUM> can be directed, guided, or driven to a second or entry position that is proximate, but physically separate from, the trocar <NUM>, for example, manually by an operator (e.g., such that the robotic arm <NUM> is manually forced or manually guided by the hand of the operator) or via the robotic arm actuators <NUM>. A suitable proximity of the docking interface <NUM> relative to the trocar <NUM> in which the sensors <NUM>, <NUM> of the sensor system <NUM> can effectively sense or measure the magnetic fields B1, B2 can be indicated, for example, with an audible beep or audible alarm, an indicator light or other visual indicia, or a tactile indicator such as haptic or vibratory feedback on a portion of the robotic arm <NUM> or tool drive <NUM>. In this regard, the sensors <NUM>, <NUM> can be activated by the processor, for example, upon an initial setup or preparation of the robotic arm <NUM> and the tool drive <NUM>, or via an input by an operator, prior to positioning of the robotic arm <NUM>/tool drive <NUM> at the entry position. As shown at block <NUM>, if the docking interface <NUM> is not in suitable proximity to the sensor system <NUM> to effectively sense the magnetic fields B1, B2, e.g., at the entry pose, the robotic arm <NUM> can be further guided toward the trocar <NUM>, for example, by manual forcing or guidance by the operator, automatically under control of the processor, or some combination thereof, until determination by the processor that the docking interface <NUM> is positioned to effectively sense the magnetic fields B1, B2.

In the entry position shown in <FIG>, the sensors <NUM>, <NUM> of the sensor system <NUM> can sense the magnetic fields B1, B2 emanating from the trocar <NUM> and produce corresponding electrical signals that are communicated to the processor in the control tower <NUM>. At such positioning of the robotic arm <NUM>/docking interface <NUM> at the entry position, the processor can begin to calculate a position and orientation of the trocar <NUM> relative to the docking interface <NUM> based upon signals received from the sensor system <NUM> according to an algorithm. The initialization or start of such algorithm can be prompted, for example, by activating the switch <NUM>. In one variation, the switch <NUM> can be activated by moving the lever <NUM> rearwardly into the unlocked (rearward) position such that the lever <NUM> contacts and actuates the switch <NUM>.

Accordingly, and with reference to block <NUM> in <FIG> and <FIG>, the processor in the control tower <NUM> is signaled by the switch <NUM> to apply an algorithm to determine the pose, e.g., spatial position and orientation, of the attachment portion <NUM> of the trocar <NUM> relative to the docking interface <NUM> to provide a transform, e.g., a transformation matrix, that can be used to guide or drive the robotic arm <NUM>, and the docking interface <NUM> of the tool drive <NUM> attached thereto, toward the trocar <NUM>. Such algorithm or set of algorithms can be a set of computer-implemented instructions, e.g., as part of a computer program product, firmware, etc., that can be stored on a non-transitory computer-readable medium for processing by a processor of the control tower <NUM>, and will be collectively referred to as an algorithm herein. The initialization of the algorithm by the processor can be considered a start of a docking procedure of the robotic arm <NUM>/tool drive <NUM>.

In one variation, and according to the algorithm, the processor in the control tower <NUM> measures a sensed pose of the attachment portion <NUM> of the trocar <NUM> with respect to a <NUM>-axis coordinate system, such as a system of X-, Y-, and Z-axes, by measuring and coordinating the electrical signals output by the sensors <NUM>, <NUM> of the sensor system <NUM> to determine the relative strength of the magnetic fields B1, B2 of the respective magnets <NUM>, <NUM> received at different locations, e.g., depths D1, D2, D3, D4, on the sensor boards <NUM>, <NUM>. For example, if the sensors <NUM>, <NUM> in the column C1 output electrical signals corresponding to the received magnetic fields B1, B2 that is greater than the output electrical signals of the sensors <NUM>, <NUM> in the column C2, a determination of a depth distance, e.g., an X-axis location, between the attachment portion <NUM> of the trocar <NUM> and the docking interface <NUM> can be calculated. Similarly, if the sensors <NUM>, <NUM> in the row R1 output electrical signals corresponding to the received magnetic fields B1, B2 that is greater than the output electrical signals of the sensors <NUM>, <NUM> in the columns R2, a determination of a vertical distance, e.g., a Z-axis location, between the attachment portion <NUM> of the trocar <NUM> and the docking interface <NUM> can be calculated. Furthermore, if the sensors <NUM> on the sensor board <NUM> output electrical signals corresponding to the received magnetic fields B1, B2 that is greater than the output electrical signals of the sensors <NUM> on the sensor board <NUM>, a determination of a horizontal distance, e.g., a Y-axis location, between the attachment portion <NUM> of the trocar <NUM> and the docking interface <NUM> can be calculated. In one example, as the docking interface <NUM> is guided or driven along the one or more of the X-axis, the Y-axis, and the Z-axis, the generation of electrical signals by the sensors <NUM>, <NUM> at the different depths D1, D2, D3, D4 can be used to determine when the trocar <NUM> becomes closer to the docking interface <NUM>. In this regard, relative saturation of one or more of the sensors <NUM>, <NUM> by the magnetic fields B1, B2, or degrees thereof, at different locations in the docking interface <NUM> can be used to determine the relative proximity of the docking interface <NUM> to the trocar <NUM>.

The generation of differential electrical signals of sensors <NUM>, <NUM> in different rows R1-R4 and different columns C1-C4 of the sensor boards <NUM>, <NUM> can also be used by the processor in the control tower <NUM> to determine rotation about two or more of the X-, Y-, and Z-axes, e.g., roll, pitch, and yaw. For example, in the case of an asymmetrical relative saturation of the sensors <NUM>, <NUM> by the magnetic fields B1, B2, e.g., such that the docking interface <NUM> is at least partially tilted with respect to the trocar <NUM>, an orientation of the attachment portion <NUM> of the trocar <NUM> with respect to at least two of the X-, Y-, and Z- axes can be determined. In addition, the generation of electrical signals by the sensors <NUM>, <NUM> can be compared by the processor to the known offset of the axes of polarization of the magnets <NUM>, <NUM> to determine the rotation of the orientation of the attachment portion <NUM> of the trocar <NUM> about another of the X-, Y-, and Z-axes. In this regard, the arrangement of the sensors <NUM>, <NUM> provides the processor in the control tower <NUM> with electrical signals corresponding to the magnetic fields B1, B2 according to the algorithm such that a real or sensed pose of the attachment portion <NUM> of the trocar <NUM> relative to the docking interface <NUM> can be determined with respect to six degrees of freedom (DOF): X-axis position, Y-axis position, Z-axis position, X-axis rotation, Y-axis rotation, and Z-axis rotation. In one variation, at least six measurements from the sensors <NUM>, <NUM> can be used to determine the pose of the trocar <NUM>. The accuracy and precision of the determination of the pose of the trocar <NUM> may correspond to a number of sensors <NUM>, <NUM> that are employed in the sensor system <NUM> such that a desired number of sensors can be selected for use in the sensor system <NUM>.

According to the algorithm, the processor in the control tower <NUM> can determine the sensed or measured pose of the trocar <NUM> based on the electrical signals produced by the sensors <NUM>, <NUM> as described above. It will also be understood that the sensors <NUM>, <NUM> on respective separate boards <NUM>, <NUM> can provide comparable electrical signals corresponding to the magnetic fields B1, B2, for example, to reduce error such as electromagnetic noise provided by components of the surgical robotic system <NUM>, for example, motors, actuators, displays, etc. Furthermore, one or more of the boards <NUM>, <NUM> can incorporate inertial measurement units, for example, to compensate for the magnetic field of the Earth or vibrations of the robotic arm <NUM>, such that associated motions of the robotic arm <NUM> that are not controlled by the algorithm can be minimized, inhibited, or prevented.

It will be understood that references to the pose of the trocar <NUM> herein are relative, specifically, to the sensor boards <NUM>, <NUM> of the sensor system <NUM> that are mounted in the docking interface <NUM> of the tool drive <NUM>. In this regard, an arrangement of the sensor boards <NUM>, <NUM> relative to the surrounding docking interface <NUM> may be taken into account in determinations of the pose of the docking interface <NUM> described herein.

In one embodiment, the algorithm applied by the processor in the control tower <NUM> to produce estimated sensor readings are output from a physical or deterministic model of the sensor system <NUM>, e.g., a deterministic model of a position and arrangement of the sensor boards <NUM>, <NUM> (see block <NUM> of <FIG>). Such deterministic model of the sensor system <NUM> can be provided by the processor in the presence of a virtual representation of the magnetic fields B1, B2 that is modeled on the known properties of the magnets <NUM>, <NUM>, and which include the known relative offset of the respective axes of polarization of the magnets <NUM>, <NUM>. Accordingly, the deterministic model can be obtained or otherwise available to the processor (block <NUM> in <FIG>) prior to the start of the algorithm described herein.

Such deterministic model can be a pre-defined function or set of functions applied by the processor that receive, as an input, an estimated pose of the trocar <NUM> relative to the modeled sensor system <NUM>, e.g., relative to the sensor boards <NUM>, <NUM>. Accordingly, the estimated pose of the trocar <NUM> that is input to the deterministic model can be considered a selected pose (or initially, a guessed pose) of the trocar <NUM>, and the deterministic model run by the processor produces, as an output, estimated sensor readings that correspond to this estimated pose of the trocar <NUM>. In one variation, the estimated pose of the trocar <NUM> that is initially run through the deterministic model by the processor can be a stored set of values, e.g., predefined values, that can be based on typical trocar placements or arrangements that are known from historical data.

The estimated sensor readings produced by the processor from the deterministic model may be different from the measured sensor readings received by the processor from the sensor system <NUM> such that it can be desirable to reconcile the measured sensor readings with the estimated sensor readings, for example, to account for variables that may affect the accuracy of the measured sensor readings, such as magnetic fields generated by other trocars or other surgical equipment in the vicinity of the robotic arm <NUM>, or other electromagnetic interference. Accordingly, the processor in the control tower <NUM> can compute a similarity measure in which the estimated sensor readings from the deterministic model are compared to the measured sensor readings from the sensor system <NUM>, and can be optimized by the processor, e.g., iteratively updated to approach one another within a predetermined range or tolerance of error (see block <NUM> of <FIG>).

At least blocks <NUM> through <NUM> of <FIG> illustrate the optimization algorithm of block <NUM> of <FIG>, according to one aspect of the disclosure. The optimization algorithm can incorporate an Interior-Point Algorithm with Analytic Hessian, a non-linear least-squares solver, or a different optimization algorithm. An initial estimated or guessed pose of the trocar <NUM> (block <NUM>) is run through the deterministic model by the processor to produce estimated sensor readings (block <NUM> in <FIG>). These are then compared by the processor to the measured sensor readings received from the sensor system <NUM> (block <NUM>), and the processor calculates whether the difference between the estimated sensor readings and the measured sensor readings is within an acceptable range or tolerance of error (block <NUM>). If the difference between the estimated sensor readings and the measured readings are not within the acceptable range or tolerance of error, the processor adjusts the guessed or estimated pose of the trocar <NUM> (block <NUM>) resulting in an updated estimated pose of the trocar <NUM> that is run through the deterministic model by the processor to produce updated estimated sensor readings (repeating block <NUM>). The difference between the updated estimated sensor readings and the measured sensor readings is then calculated by the processor (repeating block <NUM>) to determine whether the difference between the estimated sensor readings and the measured sensor readings are within the acceptable range or tolerance of error (repeating block <NUM>. ) If such difference is not within the acceptable range or tolerance of error, the estimated pose of the trocar <NUM> is iteratively updated again (repeating block <NUM>) and run through the deterministic model by the processor. This iterative optimization algorithm continues until a set of optimized or final updated estimated sensor readings are produced by the processor that are within the acceptable range or tolerance of error (the "yes" branch at the output of block <NUM>.

The final updated estimated sensor readings produced through the aforementioned optimization correspond to a "determined pose" of the attachment portion <NUM> of the trocar <NUM>, which, along with a pose of the docking interface <NUM>, provides a transform that can be associated with a target or planned trajectory for guiding or driving the robotic arm <NUM>, as described further herein. In this regard, via optimization by the processor of the estimated sensor readings produced through the deterministic model and the measured sensor readings received from the sensor system <NUM>, the surgical robotic system <NUM> is operable to discriminate between the magnetic fields B1, B2 that are representative of the pose of the trocar <NUM> and other magnetic fields or electromagnetic interference such as those produced by other trocars or other surgical equipment in the operating arena. In one variation, in the presence of multiple trocars, the surgical robotic system <NUM> can be configured to target and initiate magnetic sensing and docking of a given docking interface with a nearest trocar, and distinguish between the magnetic field produced by the nearest trocar and the magnetic fields produced by other trocars.

In a further operation performed by the processor, the final updated estimated sensor readings, which corresponds to the determined pose of the attachment portion <NUM> of the trocar <NUM>, are compared to the pose of the docking interface <NUM>, e.g., to provide a transform that is used to guide the docking interface <NUM> toward the trocar <NUM> (block <NUM> in <FIG>. ) In one variation, the pose of the docking interface <NUM> can be a known value, for example, as determined through a log of prior movements of the robotic arm <NUM> by the robotic arm actuators <NUM> or various other sensors of the surgical robotic system <NUM>, e.g., a gyroscope, accelerometer, position encoders, etc. In another variation, the pose of the docking interface <NUM> can be considered a geometric center from which the robotic arm <NUM> can be guided or driven to translate or rotate to approach the trocar <NUM>. Accordingly, and as shown in <FIG>, the processor in the control tower <NUM> can provide a set of guidance or driving control signals to the robotic arm actuators <NUM> based upon the final updated estimated sensor readings, to provide a tracking planned trajectory for the robotic arm <NUM> and to effect guidance or driving of robotic arm <NUM> to position and orient the docking interface <NUM> into docking facing relation with the attachment portion <NUM> of the trocar <NUM> such that the docking interface <NUM> matches or has substantially the same orientation as the orientation of the attachment portion <NUM> in a third or corrected entry position. It will be understood that, in the third or corrected entry position, the docking interface <NUM> is positioned proximate, but separate from, the trocar <NUM>, and that the docking interface <NUM> is oriented such that only a final translational guidance of the robotic arm <NUM>/docking interface <NUM> toward the trocar <NUM> will be sufficient to accomplish docking of the docking interface <NUM> with the trocar <NUM> (block <NUM>). Two approaches for guiding the robotic arm <NUM> and in particular its tool drive and docking interface <NUM>, toward the trocar <NUM> for docking are described further below in connection with <FIG> and <FIG>.

A physical/mathematical model to estimate the pose of one or more magnets in a target (such as a trocar) is difficult to determine and may yield incorrect results due to sensor or signal noise, imprecise modelling, or other/unknown magnetic fields. In order to offer an alternative, or improve upon the above-presented surgical robotic system, a further surgical robotic system is described below in which a programmed processor makes the prediction or estimate of the trocar pose relative to the robot arm, by means of a machine learning model, e.g., an artificial neural network, or simply "neural network. " The neural network is trained to predict the pose of the trocar, based on magnetic sensor readings such as in the embodiments described above. In one embodiment, this solution is deployed as the primary method for estimation of the trocar pose. In another embodiment, the neural network based solution is deployed in parallel with a deterministic model-based solution for pose estimation, which enables redundancy and therefore increases robustness of the docking procedure.

In various embodiments of a surgical robotic system described herein, a machine learning model is trained to perform regression analysis on the measured sensor readings (information obtained by the sensor readings) to thereby effectively estimate the pose of the trocar. The training process is performed offline, and may require a number of sensor measurements and corresponding known, ground truth pose data. The network is sufficiently trained when the change of the loss converges. To evaluate the performance of the machine learning model, pose is predicted (estimated) on a set of measurement data that has not been used for training and then the predicted pose is compared to the corresponding known ground truth pose to find out the error.

The machine learning model can be deployed either as a primary algorithm to estimate the pose of the trocar, or as a redundancy measure in parallel with a physical/deterministic model. The machine learning model may have, for example, a simple architecture of a combination of input layer, multiple convolutional layers (+multiple rectified linear unit layer), one or more fully connected layers, and a regression output layer that enables the estimation of the pose of the arrangement of magnets. Note that in contrast to many classification or segmentation problems, the neural network here performs regression analysis upon the magnetic sensor measurements to result in a pose (position and orientation) of a trocar. This allows the machine learning model to estimate poses that were not part of the training data. <FIG> described further below illustrates an example machine learning model that can be used to estimate the position or orientation of a set of one or more magnets in a trocar.

To improve accuracy, the machine learning model can be used in parallel with a physical model. A comparison of the poses of the trocar estimated by the physical model and by the machine learning model allows the system to confirm the two pose estimates, reject a current set of magnetic sensor measurements, or combine the two pose estimates into a single, final pose estimate. <FIG> illustrates a controller <NUM> that performs such a comparison. The controller <NUM> as a programmed processor executes a process that obtains magnetic sensor measurements from several magnetic field sensors, and computes a physical model estimate <NUM> of the pose based on a physical model <NUM> and (in a parallel path) a machine learning model estimate <NUM> of the pose based on an evaluation of the sensor measurements through a trained machine learning model <NUM>. After time synchronization is performed based on the timestamps of the sensor measurements (to ensure that the two estimates are referring to the same pose at a particular point in time), the two pose estimates are compared by an analyzer <NUM>, which allows the system to either reject the pose estimates, or accept them and thereby update a final estimate <NUM> (position and orientation). The updated final estimate <NUM> is then used by a robotic arm to trocar docking control algorithm (robot control <NUM>) that guides the robotic arm until the docking interface <NUM> of the arm <NUM> is ready to dock with the attachment portion trocar <NUM>.

Still referring to <FIG>, in one embodiment the controller <NUM> (as part of a surgical robotic system) receives digital output measurements from the magnetic field sensors <NUM>, <NUM> and has stored in microelectronic memory a physical model <NUM> (a stored data structure) and a machine learning model <NUM> (a stored data structure) which it uses to compute estimates of the pose of an object which is producing a magnetic field picked up by the sensor. The object, which can be for example the trocar <NUM> - see <FIG> - could have structural elements such as the flanged upper portion of head <NUM> and the attachment portion <NUM> that are made of magnetic material. As an alternative, or in addition, it could have two or more discretely or separately formed magnets that are attached to or embedded into a structural element. The magnetic field sensors <NUM>, <NUM> could be arranged in an array as shown in <FIG>, or in various further arrangements as readily devised in keeping with the teachings herein. The magnetic field sensors <NUM>, <NUM> are attached to the surgical robotic arm <NUM>. The arm <NUM> is attached to and is extending from the platform <NUM>, or in further embodiments is attached to another robotic arm or, more generally to any base or other apparatus. The platform <NUM> can be fixed or mobile.

The controller <NUM> has one or more processors ("a processor") that executes instructions stored in memory, which include a physical model <NUM> that produces an estimate of the pose as described above with reference to <FIG> and <FIG>. The memory also includes a machine learning model <NUM> such as an artificial neural network. Both the physical model and the machine learning model receive input from the magnetic field sensors <NUM>, <NUM> and produce position and orientation information as estimates <NUM>, <NUM> about the object, as a physical model estimate <NUM> and a machine learning model estimate <NUM> of the actual pose (position and orientation) of the object. The estimated position and orientation information from these two different paths are reconciled by an analyzer <NUM>, which produces reconciled position and orientation information as a final estimate <NUM> for use by a robot control <NUM> in controlling the surgical robotic arm <NUM>.

In one embodiment, the object is the trocar <NUM> with one or more magnets, and the magnetic field sensors <NUM>, <NUM> are installed in the end effector and more specifically in the docking interface <NUM> (see, e.g., <FIG>, <FIG> and <FIG>) attached to the surgical robotic arm <NUM>. The controller <NUM> automatically docks the docking interface of the surgical robotic arm to the trocar, using the position and orientation information of the estimates <NUM>, <NUM> and of the final estimate <NUM>. In one aspect, the surgical robotic system just has the machine learning model <NUM> to compute the estimate <NUM> which may be analyzed to result in the final estimate <NUM>, without relying on any estimate produced by the physical model <NUM>.

As examples of how the analyzer <NUM> reconciles position and orientation information of the estimates <NUM>, <NUM>, consider various possibilities. The analyzer <NUM> can monitor position and orientation information estimate <NUM> from the physical model <NUM> and position and orientation information estimate <NUM> from the machine learning model <NUM>, and compare the two. If one or the other path is producing an anomalous reading, this can be deduced by the analyzer <NUM>, which would then pass along what is considered the more accurate values as the position and/or orientation information, in the final estimate <NUM> of the pose, to the robot control <NUM>. The analyzer <NUM> could look for smoothly varying position and/or orientation information <NUM>, <NUM> from the two paths, and recognize when one set of values deviates sharply or erratically from the other. The analyzer <NUM> could perform averaging of the position and/or orientation information <NUM>, <NUM> from the two paths, or select one or the other set of position and/or orientation information <NUM>, <NUM>, rejecting the other or using the other as a cross check.

<FIG> illustrates the example where the machine learning model <NUM> is a convolutional neural network. Other types of neural networks may be suitable for further embodiments. The convolutional neural network has an input layer, which receives a two-dimensional array input <NUM> of measurements from the magnetic field sensors <NUM>, <NUM> (see <FIG> and <FIG>). One or more convolutional layers, one or more activation layers, and one or more fully connected layers propagate from one layer to another and produce classification scores. A regression layer analyzes these classification scores to produce position and/or orientation information as machine learning model estimate <NUM>.

The convolutional neural network undergoes training, followed by validation and then deployment into a functioning system, such as the surgical robotic system described here. Optionally, retraining can be performed. For example, the convolutional neural network could be retrained if there is a change in the magnetic field produced by the object. This could occur for instance if the trocar <NUM> is replaced with a different trocar, or it could occur if one or more magnets are added to the trocar <NUM>, or if one or magnets are moved (repositioned) or removed. In an embodiment where the trocar's structural element itself is magnetized (to be detected by the magnetic sensors on the robotic arm <NUM>), it could be that over time the magnetic field changes and therefore necessitates retraining of the convolutional neural network.

To train the machine learning model, training data is required that comprises sensor measurements and ground truth poses. In one embodiment described above with reference to <FIG>, magnets are attached to a trocar, magnetic field sensors are attached to a robot end-effector, and a large number of samples covering the entire workspace and entire range of rotations of each of many positions are obtained. This dataset is used to train the machine learning model, by optimizing the weights for each node in the network graph.

Experimental results for one embodiment indicate that the machine learning model performs with a similar accuracy as a physical model (e.g., mean norm error of <NUM>), while the duration of evaluation is significantly lower. The physical model converges within <NUM>-<NUM>, while the machine learning model provides results within <NUM>-<NUM> on the same computer.

<FIG> illustrates a two-dimensional physical array of sensors and a two-dimensional array of neural network input elements <NUM>, which are suitable for use in the convolutional neural network of <FIG> and the surgical robotic system of <FIG>. Various arrangements of sensor signals as input to a machine learning model are possible, and they are not limited to the specific arrangements described herein. It has been found advantageous to arrange output data from the magnetic field sensors <NUM>, <NUM> into a two-dimensional array of elements, for input to the convolutional neural network. The neural network may also be optimized as for image recognition for example as if the two-dimensional array of neural network input elements <NUM> were a digital image produced by a camera (e.g., a series of images forming video. ) It has been found as further advantageous to arrange the output data from the magnetic field sensors <NUM>, <NUM> in a non-adjacent manner, or even a random manner for input to the neural network <NUM>, as described below or in a variation thereof.

The embodiment depicted in <FIG> should be considered an example and not limiting as to numbers of sensors, type of sensor, number of output data elements per sensor, and arrangement of sensor output data in a two-dimensional physical array. See for example, the array of sensors <NUM>, <NUM> depicted in <FIG>. Each sensor, e.g., of magnetic field sensors <NUM>, <NUM> labeled sensor <NUM>, sensor <NUM>, etc., senses magnetic field in each of three orthogonal directions, and is of a type commonly known as an XYZ or three axis magnetic field sensor. Here, these magnetic field sensors are depicted as each sensing and producing signals for B0, B1 and B2 magnitudes of the magnetic field vector at the location of the sensor. The two-dimensional array of neural network inputs contains output data from the sensors (in the two-dimensional physical array) arranged at random, as a specific example of the two-dimensional array input for the convolutional neural network in <FIG>. Output data from adjacent magnetic field sensors in the two-dimensional physical array are non-adjacent elements in the two-dimensional array of neural network inputs. Or, adjacent elements in the two-dimensional array of neural network inputs <NUM> are from nonadjacent magnetic field sensors in the two-dimensional physical array of sensors. So, for example, the magnetic field sensors labeled sensor <NUM> and sensor <NUM> are adjacent in the two-dimensional physical array, but their output data are nonadjacent in the two-dimensional array of neural network inputs. And, the output data of sensor <NUM> and sensor <NUM> are adjacent in the two-dimensional array of neural network inputs <NUM>, but sensor <NUM> is not adjacent to sensor <NUM> in the two-dimensional physical array. Various further arrangements of sensors and inputs to the neural network with this principle of non-adjacency are readily devised. It should be appreciated that arrangements and rearrangements of sensors and their output data as inputs to the neural network <NUM> can be accomplished with sensor position changes, circuitry or wiring changes and/or with software or programming changes. Non-adjacency could even be extended to the individual signals from a given sensor, in arrangements for the neural network inputs.

<FIG> is a process flow of a method of sensing position and/or orientation of an object, which can be performed by embodiments of the surgical robotic system of <FIG> and variations thereof. In various embodiments, various objects that have a magnetic field can be used. The trocar <NUM> with a magnetic field can be used as the object. An objective of one version is to dock the docking interface <NUM> of the surgical robotic arm <NUM> to the trocar <NUM>, where the trocar has a magnetic field and the docking interface has magnetic field sensors.

In an action <NUM>, the magnetic field of the object is sensed through magnetic field sensors coupled to a machine learning model. The magnetic field sensors are attached to a surgical robotic arm, or more generally a robotic arm.

In an action <NUM>, the machine learning model is trained to output three-dimensional position and/or orientation of the object. Examples of machine learning models and training are described above.

In an action <NUM>, the machine learning model training is validated. Generally, this involves testing various positions and orientations of the object and verifying accuracy of the three-dimensional position and/or orientation information output by the machine learning model.

In an action <NUM>, the surgical robotic arm is guided, based on the three-dimensional position and/or orientation of the object as output by the machine learning model. In embodiments where the object is a trocar with a magnetic field, the surgical robotic arm is guided automatically to dock the surgical robotic arm to the trocar.

In an action <NUM>, it is determined whether there is a change in the object magnetic field. For example, the object could have aged and so its magnetic field is decreased, or one or more magnets of the object were added, removed, repositioned, etc. If the answer is no, there is no change to the magnetic field of the object, flow returns to the action <NUM> in order to continue moving the surgical robotic arm. If the answer is yes, in that the object magnetic field has changed, then flow proceeds to the action <NUM>, to retrain the machine learning model, and proceeds from there to the action <NUM> to validate the machine learning model training. Other branches are possible, such as other operations of the surgical robotic arm, or pausing or redirecting the surgical robotic system.

An aspect of the disclosure here is a method for training a machine learning model to output 3D position and 3D orientation of a trocar. The output 3D pose is to be used by an automated process that controls a surgical robotic arm for docking the arm to the trocar, for example while a tool drive is coupled to the arm and that has a docking interface in which there are a number of magnetic field sensors. The machine learning model may be trained to perform regression analysis on the measured sensor readings (information obtained by the sensor readings) to thereby effectively estimate the pose of the trocar. The training process is performed offline, and may require a number of sensor measurements and corresponding known, ground truth pose data. The ML model is sufficiently trained when the change of the loss converges. To evaluate the performance of the trained machine learning model, the model is asked to predict (estimate) a pose based on an input set of measurement data that has not been used for training. This predicted pose is then compared to the corresponding known ground truth pose to find out the error.

The machine learning model may be a convolutional neural network configured for propagating an input two-dimensional array of output data from the magnetic field sensors. In one aspect, the neural network comprises: an input layer arranged to receive the input two dimensional array of output data from the magnetic field sensors; a plurality of convolutional layers; one or more fully connected layers; and a regression output layer to output the three-dimensional position or three-dimensional orientation.

In another aspect, the machine learning model is a convolutional neural network arranged to receive the input as a two-dimensional array of output data from magnetic field sensors, and wherein output data of adjacent magnetic field sensors, of the plurality of magnetic field sensors, are arranged as nonadjacent elements in the two-dimensional array of output data.

Once the pose of the trocar <NUM> has been determined (estimated), the robotic arm <NUM> is guided to dock with the trocar <NUM>. Several approaches are possible for doing so in a way that makes it easier for a user or operator. In one aspect of the disclosure, as referred to by block <NUM> of <FIG>, the processor in the control tower <NUM> can activate the robotic arm actuators <NUM> to guide or drive the robotic arm <NUM> according to a transformation matrix that relates the final estimate of the trocar pose to the present pose of the docking interface <NUM>. Thus, the docking interface <NUM> is guided or driven toward the determined position and orientation of the trocar <NUM>. Such guidance may be fully automatic or it may assist an operator's manual forcing of the arm <NUM> along a planned trajectory by way of producing a virtual spring that urges the arm back to the planned trajectory. This driving or guidance of the robotic arm <NUM> may include re-positioning of the docking interface <NUM> (according to the transform), and in some instances re-orienting the docking interface <NUM> (according to the transform) to achieve a corrected entry pose so as to be ready to dock with the attachment portion <NUM> of the trocar <NUM>. Such guidance can be affected by the processor in the following two ways.

<FIG> (in conjunction with <FIG> and <FIG>) illustrates the situation where during guidance or driving of the robotic arm <NUM> by the robotic arm actuators <NUM> (once the pose of trocar has been determined in block <NUM>), the processor guides the arm <NUM> until the position of the docking interface <NUM> is close to, but not ready to dock with, the attachment portion <NUM> of the trocar <NUM>. During this guidance, the orientation of the docking interface <NUM> may remain fixed. When the processor detects that the position of the docking interface <NUM> is close enough to the attachment portion <NUM>, or has reached a so-called "entry pose", the processor then checks whether the orientation of the docking interface <NUM> matches that of the entry pose, i.e. whether the determined orientation of the docking interface <NUM> matches the estimated orientation of the trocar <NUM> (block <NUM>). If the processor determines that the docking interface <NUM> does not match the orientation of the trocar <NUM>, the processor controls the robotic arm actuators <NUM> to further drive or guide the robotic arm <NUM> so as to re-orient the docking interface <NUM> to match that of the trocar <NUM> (block <NUM>). Once the determined orientation of the docking interface <NUM> matches the estimated orientation of the trocar <NUM>, such that the docking interface <NUM> is now in its "corrected entry pose", the processor drives the robotic arm actuators <NUM> so as to move the docking interface <NUM> into the ready to dock position with the attachment portion <NUM> of the trocar, e.g., only in a translation movement and without having to now change the orientation of the docking interface <NUM> (block <NUM>. ) The tool drive <NUM> is now ready to dock with the trocar <NUM>.

<FIG> (in conjunction with <FIG> and <FIG>) serves to illustrate another docking process for the tool drive <NUM>, where here both the orientation and the position of the docking interface <NUM> are being adjusted automatically by the processor while the robotic arm <NUM> is guided towards the trocar (block <NUM>. ) The process is otherwise similar to <FIG> in that it too begins with block <NUM> in which the pose of the trocar is estimated (determined by the processor) as described above. In contrast to <FIG> however, here the processor is repeatedly or continually checking the full pose of the docking interface <NUM> against the estimated pose of the trocar <NUM> while the arm <NUM> is being guided toward the trocar, and in response adjusting as needed both the position and the orientation of the docking interface <NUM>. For example, the orientation of the docking interface <NUM> is thus maintained at all times (while the docking interface <NUM> moves toward the trocar) to match the estimated orientation of the attachment portion <NUM> of the trocar. This process continues or loops as shown, until block <NUM> reveals that the pose of the docking interface <NUM> matches the estimated pose of the trocar, at which the docking interface <NUM> is ready to dock.

Claim 1:
A surgical robotic system for sensing pose, as position and orientation of a trocar, the system comprising:
a surgical robotic arm;
a plurality of magnetic field sensors coupled to the surgical robotic arm; and characterised in that the surgical robotic system comprises
a processor that implements a machine learning model that receives signals from the plurality of magnetic field sensors and trained to output an estimate of a pose of a trocar that is producing a magnetic field, based on output data from the plurality of magnetic field sensors.