Patent Description:
<CIT>, <CIT>, <CIT> and <CIT> disclose preoperative planning systems for placing surgical ports.

Certain optional features of the invention are defined in the dependent claims. The methods described herein do not form part of the claimed subject-matter. The present disclosure relates to the provision of guidance for placing surgical ports on a patient's body, and more particularly, to systems, methods, and computer-readable media for generating and displaying virtual or augmented reality visual guidance to guide and/or assist clinicians during planning for and placing of surgical ports on a patient's body.

Provided in accordance with embodiments of the present disclosure are systems for guiding placement of surgical ports on a body. In an aspect of the present disclosure, the system includes an image capture device configured to capture an image of an operating environment including a body and generate image data based on the captured image, display device worn by a user, and a computing device including a processor and a memory storing instructions which, when executed by the processor, cause the computing device to obtain information regarding a location of a surgical site within the body, receive the image data from the image capture device, generate graphical guidance for placing a surgical port on the body based on the location of the surgical site within the body and the image data, and cause the display device to display the generated graphical guidance.

In another aspect of the present disclosure, the instructions, when executed by the processor, further cause the computing device to detect, based on the image data received from the image capture device, that the surgical port has been placed on the body, generate graphical guidance for inserting an endoscope through the surgical port into the body, and detect that the endoscope has been inserted through the surgical port.

In a further aspect of the present disclosure, the detecting that the surgical port has been placed on the body includes analyzing additional image data received from the image capture device to detect that the surgical port has been placed on the body.

In another aspect of the present disclosure, the detecting that the endoscope has been inserted through the surgical port includes analyzing additional image data received from the image capture device to detect that the endoscope has been inserted through the surgical port.

In yet another aspect of the present disclosure, the surgical port is a first surgical port, and the instructions, when executed by the processor, further cause the computing device to generate graphical guidance for placing a second surgical port on the body based on the location of the surgical site within the body and a detected location where the first surgical port has been placed on the body, and cause the display device to display the generated graphical guidance.

In still another aspect of the present disclosure, the generated graphical guidance includes a virtual surgical port displayed on the body, and the instructions, when executed by the processor, further cause the computing device to analyze additional image data received from the image capture device to detect movement of a hand of the user, determine, based on the detected movement of the user's hand, that the user is adjusting a pose of the virtual surgical port on the body, and update the graphical guidance based on the determined adjustment of the pose of the virtual surgical port on the body.

In a further aspect of the present disclosure, the instructions, when executed by the processor, further cause the computing device to provide an alert when the determined adjustment of the pose of the virtual surgical port on the body is more than a predetermined distance or change in orientation from the pose recommended by the graphical guidance.

In another aspect of the present disclosure, detecting movement of the user's hand includes detecting a gesture performed by the user's hand.

In a further aspect of the present disclosure, determining that the user is adjusting the pose of the virtual surgical port on the body is further based on the detected gesture performed by the user's hand.

In another aspect of the present disclosure, the image capture device is coupled to the display device.

In yet another aspect of the present disclosure, the graphical guidance includes textual instructions displayed by way of the display device.

In still another aspect of the present disclosure, the graphical guidance includes an augmented reality image of a recommended pose of the surgical port.

In a further aspect of the present disclosure, the recommended pose of the surgical port is determined based on a predetermined model of recommended surgical port placements.

In another aspect of the present disclosure, the recommended pose of the surgical port is determined based on a predetermined treatment plan.

In yet another aspect of the present disclosure, the recommended pose of the surgical port is determined based on a scan of the body.

In still another aspect of the present disclosure, the graphical guidance includes an augmented reality image of the surgical site within the body.

In a further aspect of the present disclosure, the computing device causes the display device to display the augmented reality image of the surgical site within the body as superimposed onto the body.

In another aspect of the present disclosure, the augmented reality image of the surgical site includes a position of at least one organ within the body.

In a further aspect of the present disclosure, the position of the at least one organ within the body is based on a predetermined model of positions of organs.

In another aspect of the present disclosure, the system further includes a surgical robot including one or more robotic assemblies including a base, a first joint, a first arm coupled to the first joint, a second joint coupled to the first arm and a second arm coupled to the second joint, and the base is coupled to the second arm by way of the first joint, first arm, and second joint.

In a further aspect of the present disclosure, the instructions, when executed by the processor, further cause the computing device to generate graphical guidance for placing the surgical robot based on the determined position of the surgical site within the body.

In yet another aspect of the present disclosure, the instructions, when executed by the processor, further cause the computing device to determine a range of motion of the surgical robot based on the placement of the surgical port on the body, and cause the display device to display an indication of the range of motion of the surgical robot.

In a further aspect of the present disclosure, the range of motion of the surgical robot is determined based on a characteristic of the surgical robot.

In yet a further aspect of the present disclosure, the characteristic of the surgical robot includes one or more of a configuration of the surgical robot, a pose of the surgical robot relative to the body, and a type of tool coupled to the surgical robot.

In another aspect of the present disclosure, the image capture device is a first image capture device, and the system further comprises a second image capture device, and the instructions, when executed by the processor, further cause the computing device to generate a three-dimensional scene of the operating environment based on image data received from the first and second image capture devices.

In a further aspect of the present disclosure, the first and second image capture devices are positioned about the operating environment.

In another aspect of the present disclosure, the image capture device is included in the computing device.

In yet another aspect of the present disclosure, the computing device is included in the display device.

In still another aspect of the present disclosure, the obtaining information regarding a location of the surgical site within the body includes receiving a selection of the surgical site via input from a user.

In yet another aspect of the present disclosure, the obtaining information regarding a location of the surgical site within the body includes determining the location of the surgical site based on image data acquired during a scan of the body.

In still another aspect of the present disclosure, the obtaining information regarding a location of the surgical site within the body includes determining the location of the surgical site based on a predetermined treatment plan.

In yet another aspect of the present disclosure, the instructions, when executed by the processor, further cause the computing device to determine a position of the surgical site within the body based on the received image data.

In still another aspect of the present disclosure, the body is at least a portion of a mannequin.

In yet another aspect of the present disclosure, the body is at least a portion of a patient.

Provided in accordance with embodiments of the present disclosure are methods for guiding placement of surgical ports on a body. In an aspect of the present disclosure, the method includes obtaining information regarding a location of a surgical site within the body, receiving image data, generated based on an image of the operating environment, from an image capture device, generating graphical guidance for placing a surgical port on the body based on the location of the surgical site within the body and the image data, and displaying the generated graphical guidance.

Provided in accordance with embodiments of the present disclosure are non-transitory computer-readable storage media including instructions which, when executed by a computer, cause the computer to obtain information regarding a location of a surgical site within the body, receive image data generated based on an image of an operating environment from an image capture device, generate guidance for placing a surgical port on the body based on the location of the surgical site within the body and the image data, and cause a display device to display the generated graphical guidance.

Any of the above aspects and embodiments of the present disclosure may be combined without departing from the scope of the present disclosure.

Various aspects and features of the present disclosure are described hereinbelow with references to the drawings, wherein:.

The present disclosure is directed to systems, methods, and computer-readable media for providing guidance for placing surgical ports on a patient's body (or, in some examples, a non-human body-such as a mannequin, a virtual body, and the like-that is modeled based upon the patient's body or of another test subject) (referred to collectively hereinafter as a "body"). More particularly, the present disclosure relates to instructing a user, such as a surgeon, nurse, technician, and/or other operating room staff (hereinafter referred to as a clinician or user), how and where to place surgical ports on the body to ensure suitable access for surgical tools coupled to a surgical robot to a surgical site within the body. Various forms of visual and/or auditory guidance may be displayed and/or emitted by one or more display devices, such as a head-mounted display, for example, an augmented reality headgear and/or virtual reality headgear. In some embodiments, the display devices may be one or more projectors or other light emitters coupled to one or more robotic arms of the surgical robot and/or other mount or assembly about the surgical environment, the display devices being configured to display guidance and/or illuminate one or more parts of the body. The guidance may include one or more commands directing the clinician to place surgical ports at particular locations on the body and enable the clinician to adjust the locations of the surgical ports. Thus, the systems, methods, and computer-readable media disclosed herein may be used during training for robotic surgery and/or during actual robotic surgical procedures to assist the clinician with preparing for robotic surgery.

With reference to <FIG>, there is shown a system <NUM> for providing guidance for placing surgical ports, according to an embodiment of the present disclosure. System <NUM> may include a table <NUM> capable of supporting a body B, a head-mounted display device (HMD) <NUM>, one or more image capture devices 125a, 125b, 125c, and 125d, one or more surgical ports 130a, 130b, 130c, and 130d, a surgical robot assembly <NUM>, a laparoscope <NUM>, and a computing device <NUM>. Image capture devices 125a, 125b, 125c, and 125d may be any image capture devices known to those skilled in the art, such as video cameras, still cameras, stereoscopic cameras, three-dimensional depth cameras, LIDAR sensors, and/or the like, and may be positioned about an operating environment. As used herein, the term "operating environment" refers to a setting, room, or scene where robotic surgery is performed. As such, the operating environment includes table <NUM>, HMD <NUM>, and one or more image capture devices 125a, 125b, 125c, and 125d. In some embodiments, the operating environment further includes body B on table <NUM>, one or more surgical ports 130a, 130b, 130c, and 130d, surgical robot assembly <NUM>, and/or laparoscope <NUM>. In other embodiments, system <NUM> may be used in training and/or planning procedures, some or all of body B, surgical ports 130a, 130b, 130c, and 130d, surgical robot assembly <NUM>, and/or laparoscope <NUM> may not be present in the operating environment. During such training and/or planning procedures, body B may be a virtual or non-human body, as further described below.

One or more images of the capture devices 125a, 125b, 125c, and 125d, for example image capture device 125d as shown in <FIG>, may be included in or coupled to HMD <NUM>. Additionally, or alternatively, image capture devices 125a, 125b, 125c, and 125d may be coupled or otherwise connected, whether by wireless or wired connection, to computing device <NUM>. Image capture devices 125a, 125b, 125c, and 125d are referred to collectively hereinafter as image capture devices <NUM>. Surgical ports 130a, 130b, 130c, and/or 130d may be any port or device usable for inserting surgical tools into body B, such as trocars. Surgical ports 130a, 130b, 130c, and 130d are referred to collectively hereinafter as surgical ports <NUM>.

Surgical robot assembly <NUM> may include a base <NUM>, a first joint <NUM> coupled to base <NUM>, a first robotic arm <NUM>, coupled to first joint <NUM>, a second joint <NUM> coupled to first robotic arm <NUM>, a second robotic arm <NUM> coupled to second joint <NUM>, and an instrument drive unit <NUM> coupled to second arm <NUM>. Laparoscope <NUM> may be attached to surgical robot assembly <NUM> via instrument drive unit <NUM>. In embodiments, surgical robot assemblies <NUM> may be used concurrently and may together form a surgical robot. While a single surgical robot assembly <NUM> is shown in <FIG>, multiple surgical robot assemblies <NUM> may be included in the surgical training environment, and those skilled in the art will recognize that the below-described methods may be applied to surgical robots having single and/or multiple surgical robot assemblies <NUM>, each including at least one base <NUM>, robotic arms <NUM> and <NUM>, joints <NUM> and <NUM>, and instrument drive unit <NUM>, without departing from the scope of the present disclosure.

Those skilled in the art will recognize that system <NUM> may also include other display devices in addition to or instead of HMD <NUM>, without departing from the scope of the present disclosure. For example, one or more projectors or other light-emitting devices may be coupled to robotic arms <NUM> and/or <NUM>, other equipment, lights, or mounts about the surgical environment, etc. Additionally or alternatively, various other augmented-reality enabled devices, such as tablet computer, smart phones, etc. may also be included in system <NUM>. As such, while HMD <NUM> is used herein as an illustrative embodiment, the present disclosure is not intended to be limited to having only HMD <NUM> as a display device.

Body B may be a human body, such as a body of a patient undergoing a surgical procedure, a non-human body, such as a mannequin, physical mock-up, or other simulated body, and/or a virtual body, which, in some cases is modeled based upon a human body of the patient or of a test subject. For example, in embodiments where body B is a virtual body, body B may be generated based on parameters of a particular patient and/or test subject, and may be stored in computing device <NUM> to be loaded during a surgical procedure and/or a training procedure. In embodiments where body B is a virtual body, body B is not physically present on table <NUM>, but is displayed as an augmented and/or virtual reality image via HMD <NUM>, as described further below. Further, in embodiments where body B is a virtual body, table <NUM> may also be a virtual table generated by computing device <NUM> and/or HMD <NUM> and displayed via HMD <NUM>. Thus, the systems and methods of the present disclosure may be used for training as well as real surgical purposes. In embodiments where a different type of display device, such as a tablet or other hand-held computing device, is used, body B and/or other components of system <NUM> may be displayed using "windowed augmented reality" - that is, the display device may display body B on table <NUM> when table <NUM> is within the view displayed by the display device. Further, those skilled in the art will recognize that various other embodiments of using virtual and/or augmented reality to display body B and/or other components of system <NUM> may be used in addition to or instead of the embodiments described herein without departing from the scope of the present disclosure.

Computing device <NUM> may be any computing device configurable for use during training for robotic surgery known to those skilled in the art. For example, computing device <NUM> may be a desktop computer, laptop computer, server and terminal configuration, and/or a control computer for a surgical robot including surgical robot assembly <NUM>, and/or the like. In some embodiments, computing device <NUM> may be included in HMD <NUM>. As described further below, system <NUM> may be used during training for robotic surgery, such as training on how to place surgical ports on body B.

Turning now to <FIG>, there is shown a schematic diagram of computing device <NUM> forming part of system <NUM> of <FIG>, according to an embodiment of the present disclosure. Computing device <NUM> includes a memory <NUM>, a processor <NUM>, an input interface <NUM>, a communications interface <NUM>, and an output interface <NUM>. Memory <NUM> stores a database <NUM> and an application <NUM>. Application <NUM> may include instructions which, when executed by processor <NUM>, cause computing device <NUM> to perform various functions, as described below. Application <NUM> further includes graphical user interface (GUI) instructions <NUM> which, when executed by processor <NUM>, cause computing device <NUM> to generate one or more GUIs (not shown in <FIG>), such as, for example, the example GUI <NUM> shown in <FIG> and <FIG>. Database <NUM> may store various scans and/or models of one or more patients' bodies and/or treatment plans for surgical procedures. For example, predetermined and/or patient specific treatment plans may be based on a location of a surgical site and various treatments to be performed at the surgical site.

Memory <NUM> may include any non-transitory computer-readable storage medium for storing data and/or software that is executable by processor <NUM> and which controls the operation of computing device <NUM>. In an embodiment, memory <NUM> may include one or more solid-state storage devices such as flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory <NUM> may include one or more mass storage devices connected to the processor <NUM> through a mass storage controller (not shown in <FIG>) and a communications bus (not shown in <FIG>). Although the description of computer-readable media included herein refers to a solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by processor <NUM>. That is, computer-readable storage media may include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device <NUM>.

Input interface <NUM> may be a mouse, keyboard, or other hand-held controller, foot pedal, touch screen, voice interface, and/or any other device or interface by means of which a user may interact with computing device <NUM>. Communications interface <NUM> may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and/or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and/or the internet. Output interface <NUM> may be a screen or other display device usable to output images or data by computing device <NUM>.

With reference to <FIG> and <FIG>, there is shown a flowchart of an exemplary method <NUM> for providing guidance for placing surgical ports, according to an embodiment of the present disclosure. In some embodiments, some or all of the steps of method <NUM> may be performed in a surgical environment, such as an operating room, prior to the start of a surgical procedure. In other embodiments, some or all of the steps of method <NUM> may be performed in an instructional environment, such as a training facility where a curriculum is taught. The exemplary embodiments described below are provided purely for illustrative purposes and should not be interpreted as limiting, as those skilled in the art will recognize that many other embodiments are also contemplated within the scope of the present disclosure.

Starting at step S302, one or more image capture devices <NUM> capture an image of the operating environment. As described above, the operating environment includes table <NUM> and surgical robot assembly <NUM>. In some embodiments, the operating environment may further include body B on table <NUM>. Image capture devices <NUM> then, at step S304, generates image data of the operating environment based on the image captured at step S302. Additionally, image capture device <NUM> may continuously capture images of the operating environment, and thus, even though steps S302 and S304 are shown at the start of method <NUM>, the capturing of images and generating of image data of the operating environment based on the captured images may be a continuous action performed throughout method <NUM>. As such, during some or all of the steps described below, additional images may be captured and additional image data of the operating environment generated and the images and/or image data provided to computing device <NUM>.

Thereafter, at step S306, the image data generated at step S304 is provided to computing device <NUM>. In some embodiments, image capture devices <NUM> do not generate the image data of the operating environment, but instead provide the captured image to computing device <NUM>, which in turn generates the image data of the operating environment based on the captured image. In such embodiments, it is the captured image that is provided to computing device <NUM> at step S306 and not the image data of the operating environment. Computing device <NUM> then processes the image data to generate a three-dimensional (3D) scene of the operating environment. The 3D scene of the operating environment is based on the image data, data received from other sources, and/or input provided by the user.

Computing device <NUM> then, at step S308, obtains information regarding a surgical site inside body B. The information regarding the surgical site may be provided by the user, such as via input interface <NUM>. In some embodiments, the information regarding the surgical site is preconfigured prior to the start of the surgical procedure. The information regarding the surgical site may include a location within body B where the surgical procedure will be performed, the position and/or type of organs and/or other structures involved in the surgical procedure, and/or the type of surgical procedure that will be performed. For example, the information regarding the surgical site may indicate that the surgical procedure will include removal of a portion of a patient's lung.

Thereafter, at step S310, computing device <NUM> determines a position of the surgical site within body B. The determination may be based on the information regarding the surgical site received at step S308, radiographic image data of body B, and/or the image data received at step S306. For example, radiographic image data, such as computed tomography and/or magnetic resonance image data of body B, that was acquired prior to the start of the procedure, may be loaded and warped or deformed to match the observed pose of body B as detected in the image data received at step S306 (which will likely not align with the pose of body B when the radiographic image data was acquired). Additionally or alternatively, in a statistical technique, various models of organ positions may be used to estimate locations of organs and other structures within body B, as described further below. The determination may further be based on a treatment plan. For example, in the example described above with reference to step S308, computing device <NUM> may process the image data to identify body B within the image data, and then identify the position of the surgical site, e.g. the patient's lungs, based on a pose of body B on table <NUM>. As used herein, the term "pose" is defined as a position and an orientation of an object. For example, the pose of body B refers to a position of body B on table <NUM>, an orientation of body B with respect to table <NUM> and/or the operating environment, as well as other objects in the operating environment. Similarly, a pose of other objects in the operating environment, such as table <NUM> and surgical robot assembly <NUM>, etc., may also be determined.

Identifying the position of the surgical site within the patient's body may be based on a model of organ locations and locations of other structures within a patient's body. The model of organ locations may be a general model based on data points relating to organ locations received from various patients, or the model of organ locations may be a model generated based on one or more scans and/or other radiographic imaging of the patient being treated.

Computing device <NUM> then, at step S312, generates guidance for placing a first surgical port <NUM> on body B. The guidance is generated based on the position of the surgical site within body B, as determined at step S310, and the 3D scene of the operating environment generated at step S306. The guidance may further be based on a model of recommended surgical port placement locations for a particular surgical site and/or a particular type of surgery. Additionally, the guidance may be based on recommended surgical port placement locations included in a predetermined treatment plan. For example, the recommended surgical port placements may be determined based on the dexterous workspace of surgical tools that will be inserted through the surgical ports-that is, the recommended surgical port placement locations may be determined such that the dexterous workspace of the surgical tools inserted through the surgical ports collectively includes the entire surgical site, or at least a predetermined portion thereof.

In embodiments, the dexterous workspace of the surgical tools is determined based on a bidexterous range of motion of surgical robot assembly <NUM> and/or robotic arms <NUM>, <NUM>. The determination of the bidexterous range of motion of surgical robot assembly <NUM> may be based on characteristics of surgical robot assembly <NUM>, including a configuration of surgical robot assembly <NUM>, a pose of surgical robot assembly <NUM> relative to body B, and/or a type of surgical tool coupled to surgical robot assembly <NUM>. The placement and/or pose of surgical robot assembly <NUM> about body B may be taken into account when determining the dexterous workspace of the surgical tools, and thus the recommended surgical port placement locations may be determined based on the placement and/or pose of surgical robot assembly <NUM>. Further still, the guidance may be based on recommended surgical port placement locations determined based on a scan of the patient's body. The guidance may be graphical guidance and may include one or more augmented reality images showing the position of the surgical site within body B (as shown in <FIG> and <FIG>, described below), and a virtual surgical port indicating where the first surgical port <NUM> should be placed on body B. In other embodiments, the graphical guidance may include images projected onto the patient's body by one or more display devices (e.g. projectors or light-emitting devices). The guidance may further include visual indicators, such as arrows, pointers, etc., as well as textual as well as auditory commands (as shown in, and further described with reference to, <FIG> and <FIG>) directing the user where to place the surgical ports <NUM>. Computing device <NUM> then, at step S314 displays the guidance generated at step S312 (as shown in <FIG>). In embodiments, computing device <NUM> may cause HMD <NUM> to display the guidance, and/or cause other display devices to project the guidance onto body B. For example, computing device <NUM> may cause HMD <NUM> to display a virtual representation of the surgical ports 410a, 410b, 410c, 410d, 410e, the surgical site <NUM>, and/or the dexterous workspace <NUM> of surgical tools inserted through one or more of surgical ports <NUM> as an overlay and/or superimposed onto body B, as shown in <FIG> and <FIG>. In some embodiments, computing device <NUM> may generate a virtual rendering of an expected view from an endoscope inserted through at least one surgical port <NUM>. The virtual rendering may be generated based on aforementioned models of the surgical site and/or body B. The virtual rendering may be displayed in conjunction with the guidance.

Next, at step S316, computing device <NUM> determines whether the user has moved, and particularly, whether a hand of the user has moved. The determination may be based on additional image data received by computing device <NUM> from image capture devices <NUM>. Computing device <NUM> may process and analyze the additional image data to identify the user and detect whether the user's hand has moved. If it is determined that the user's hand has not moved ("No" at step S316), processing skips ahead to step S326. Alternatively, if it is determined that the user's hand has moved ("Yes" at step S316), proceeds to step S318.

At step S318, computing device <NUM> determines whether the movement of the user's hand indicates that the user has adjusted a pose of the virtual surgical port displayed as part of the guidance. For example, computing device <NUM> may process and analyze the additional image data received during step S316 to detect a gesture performed by the user's hand. Various gestures may be performed, for example a pinching gesture, a spreading gesture, a poking gesture, and/or a moving gesture. Various gestures may indicate different actions performed by the user, for example a pinching gesture while the user's hand is directed at a particular virtual surgical port, as displayed by HMD <NUM>, may indicate that the user is selecting or "picking up" the particular virtual surgical port. Likewise, a spreading gesture performed while the user's hand is directed at a particular virtual surgical port, as displayed by HMD <NUM>, may indicate that the user is deselecting or "dropping" the particular virtual surgical port. Similarly, movement of the user's hand while a particular virtual surgical port is selected may indicate that the user is moving the particular virtual surgical port in the direction in which the user's hand is moved. In some embodiments, the user may move a virtual port based on a voice command and/or by eye tracking.

If it is determined that the movement of the user's hand does not indicate that the user has adjusted the pose of the virtual surgical port, or that the user has not given a voice and/or eye tracking command to adjust the pose of the virtual surgical port ("No" at step S318), processing skips ahead to step S326. Alternatively, if it is determined that the movement of the user's hand does indicate that the user has adjusted the pose of the virtual surgical port, or that the user has given a voice and/or eye tracking command to adjust the pose of the virtual surgical port ("Yes" at step S318), processing proceeds to step S320, where computing device <NUM> updates the guidance based on the adjustment. For example, computing device <NUM> may adjust the pose of the virtual surgical port from the pose recommended in the guidance generated at step S312 to the pose to which the user moved the virtual surgical port, as determined at step S318.

Thereafter, at step S322, computing device <NUM> determines whether the adjustment of the virtual surgical port is greater than a predetermined distance or change in orientation from the pose recommended in the guidance generated at step S312. The predetermined distance and orientation may be based on the location of the surgical site within body B and/or the type of surgical procedure being performed. For example, colorectal surgical procedures may require less precise placement of surgical ports, and thus allow the user greater flexibility in placing the surgical ports, while spinal and/or neuro surgical procedures may require more precise placement of surgical ports. If it is determined that the adjustment of the virtual surgical port is greater than the predetermined distance or change in orientation ("Yes" at step S322), processing proceeds to step S324, where an alert is provided to the user to indicate to the user that the pose of the virtual surgical port has been adjusted by more than the predetermined distance from the recommended pose. Thereafter, or if it is determined at step S322 that the adjustment of the virtual surgical port is not greater than the predetermined distance or change in orientation ("No" at step S322), processing returns to step S314, where the guidance, as updated at step S320, is displayed.

At step S326, computing device <NUM> determines whether the first surgical port <NUM> has been placed on body B. For example, as described above with reference to step S302, computing device <NUM> may receive additional image data from image capture devices <NUM>, and may process and analyze the additional image data to determine whether the first surgical port <NUM> has been placed on body B. If it is determined that the first surgical port <NUM> has not been placed on body B ("No" at step S326), processing returns to step S314. Alternatively, if it is determined that the first surgical port <NUM> has been placed on body B ("Yes" at step S326), processing proceeds to step S328, where it is determined whether there are additional surgical ports <NUM> to be placed on body B. For example, computing device <NUM> may determine based on the predetermined treatment plan and/or the model of surgical port placement locations, whether additional surgical ports <NUM> should be placed on body B. Computing device <NUM> may also receive input from the user indicating whether additional surgical ports <NUM> should be placed on body B.

If it is determined that there are additional surgical ports <NUM> to be placed on body B ("Yes" at step S328), processing proceeds to step S330, where computing device <NUM> generates guidance for placing another surgical port <NUM> on body B. The guidance may be generated based on the position of the surgical site within body B, as determined at step S310, the 3D scene of the operating environment generated at step S306, and/or the pose of the first surgical port <NUM> placed on body B. For example, the guidance may include one or more augmented reality images showing the position of the surgical site within body B, and a virtual surgical port indicating where the additional surgical port <NUM> should be placed on body B. Thereafter, processing returns to step S314, where the guidance is displayed.

Alternatively, if it is determined at step S328 that there are no additional surgical ports <NUM> to be placed on body B ("No" at step S328), processing proceeds to step S332, where computing device <NUM> generates guidance for inserting endoscope <NUM> via at least one surgical port <NUM> into body B. The guidance may be generated based on the position of the surgical site within body B, the locations where surgical ports <NUM> were placed on body B, and/or the 3D scene of the operating environment generated at step S306. The guidance may be graphical guidance and may include one or more augmented reality images showing the position of the surgical site within body B, the surgical ports <NUM> on body B, and an indication of the surgical port <NUM> into which endoscope <NUM> should be inserted. The guidance may further include visual indicators, such as arrows, pointers, etc., as well as textual as well as auditory commands. In some embodiments, endoscope <NUM> is inserted after the first surgical port <NUM> is place on the body, and thus in such embodiments steps S332, S334, and S336 may be performed prior to step S328.

Thereafter, at step S334, computing device <NUM> displays the guidance generated at step S332. For example, computing device <NUM> may cause HMD <NUM> to display the guidance. Then, at step S336, computing device <NUM> determines whether endoscope <NUM> has been inserted via a surgical port <NUM> into body B. For example, computing device <NUM> may receive additional image data from image capture devices <NUM>, as described above with reference to step S302, and may process and analyze the additional image data to determine whether endoscope <NUM> has been inserted via surgical port <NUM> into body B. If it is determined that endoscope <NUM> has not been inserted ("No" at step S336), processing returns to step S334. Alternatively, if it is determined that endoscope <NUM> has been inserted ("Yes" at step S336), processing proceeds to step S338.

At step S338, computing device <NUM> generates guidance for placing surgical robot assembly <NUM>. For example, computing device <NUM> may determine a recommended pose for placing surgical robot assembly <NUM> about body B based on the locations where surgical ports <NUM> were placed, and the position of the surgical site within body B. Generation of guidance for placing a surgical robot using augmented and/or virtual reality display devices is further described in commonly-owned co-pending <CIT>.

Thereafter, at step S340, computing device <NUM> displays the guidance generated at step S330. For example, computing device <NUM> may cause HMD <NUM> to display the guidance. Thereafter, processing ends.

While the description above relating to method <NUM> of <FIG> refers to functions being performed by computing device <NUM>, those skilled in the art will understand that such functions may be performed by computing device <NUM> based on execution of one or more applications, such as application <NUM>, and/or based on dedicated hardware and/or other software included in computing device <NUM>. Thus, while described as being performed by computing device <NUM>, the description of method <NUM> should not be interpreted as being limited to hardware embodiments.

Turning now to <FIG> and <FIG>, there is shown an exemplary graphical user interface (GUI) <NUM> that may be displayed by HMD <NUM> and/or by one or more other display devices, according to an embodiment of the present disclosure. GUI <NUM> may be displayed by HMD <NUM> and/or the other display devices at various points during a surgical procedure, such as the procedure described above with reference to method <NUM> of <FIG> and <FIG>. In the embodiment demonstrated in <FIG>, table <NUM> and surgical robot assembly <NUM> are physical objects in the operating environment shown to the user. Body B is shown lying on table <NUM>. Augmented reality images of virtual surgical ports 410a, 410b, 410c, 410d, and 410e and surgical site <NUM> may be overlaid onto body B. GUI <NUM> further includes guidance instructing the user to place surgical ports <NUM> as indicated by the locations of virtual surgical ports 410a, 410b, 410c, 410d, and 410e on body B. Augmented reality images of an indicator <NUM> of a bidexterous range of motion of a surgical tool inserted into a surgical port <NUM> may further be overlaid onto body B. GUI <NUM> may also include textual instructions <NUM> to display textual commands to the user.

If the user chooses to adjust the pose of one or more of virtual surgical ports 410a, 410b, 410c, 410d, and 410e, GUI <NUM> may be updated to indicate the new pose of the virtual surgical ports that have been moved. For example, as shown in <FIG>, an updated GUI <NUM> shows that virtual surgical port 410a has been moved, and that the movement has taken virtual surgical port 410a outside of a recommended range of placements for access to the surgical site. In such an embodiment, textual instructions <NUM> may provide an alert to the user indicating that virtual surgical port 410a has been adjusted more than a predetermined distance or change in orientation from the pose recommended by the guidance.

As one skilled in the art will appreciate, the systems, methods, and computer-readable media described in the present disclosure describes various aspects of instructing a clinician on how and where to place surgical ports on the body to ensure suitable access for surgical tools coupled to a surgical robot to a surgical site within the body. Various forms of visual and/or auditory guidance may be displayed and/or emitted by one or more display devices, including a head-mounted display and/or projectors or hand-held display devices, such as one or more commands directing the clinician to place surgical ports at particular locations on the body and enable the clinician to adjust the locations of the surgical ports. Thus, the systems, methods, and computer-readable media disclosed herein provide improvements to training and/or preparing for robotic surgical procedures.

Claim 1:
A system for guiding placement of surgical ports (<NUM>) on a body, the system comprising:
an image capture device (<NUM>) configured to:
capture an image of an operating environment including a body; and
generate image data based on the captured image;
a head mounted display device (<NUM>); and
a computing device (<NUM>) including a processor (<NUM>) and a memory (<NUM>) storing instructions which, when executed by the processor, cause the computing device to:
obtain information regarding a location of a surgical site within the body;
receive the image data from the image capture device; generate graphical guidance virtual placement for placing a virtual surgical port (<NUM>) on the body based on the location of the surgical site within the body and the image data;
cause the head mounted display device to display the generated graphical guidance virtual placement over the body, the system being characterized in that the instructions, when executed by the processor, further cause the the computing device to:
receive an adjusted placement for placing the virtual surgical port on the body from additional image data from the capture device capturing a movement of a user's hand; and, by determining that the user is adjusting a pose of the virtual surgical port,
providing an alert when a distance between the virtual placement and the adjusted placement is greater than a predetermined distance based on the location of the surgical site and a type of a surgical procedure.