AUGMENTED REALITY SYSTEM AND METHOD FOR TELE-PROCTORING A SURGICAL PROCEDURE

A system for tele-proctoring a surgical procedure includes an augmented reality head mounted display and a computer. The computer is configured to receive a visual experienced from the eyes of an onsite physician via the augmented reality head mounted display; receive additional content experienced by the onsite physician via the augmented reality head mounted display; integrate the visual experienced from the eyes of an onsite physician and the additional content into a single integrated view experienced by the onsite physician; communicate the integrated view to a remote computer for display on a remote display; receive an interaction with the integrated view from a remote physician via the remote computer; and present the interaction to the onsite physician via the augmented reality head mounted display.

FIELD OF DICSLOURE

The present disclosure relates to the field of surgical procedures and more specifically to the field of tele-proctoring a surgical procedure.

BACKGROUND

Surgical procedures may be complex, the success of which may be crucial to a patient's well-being. Thus, in order to perform surgical procedures, a physician is commonly required to undergo extensive training including performing or participating in surgical procedures under the guidance and supervision of a senior and more experienced physician. And even beyond the initial training and guidance, a senior physician may be required to proctor a surgical procedure being performed by a less senior physician and to confirm certain maneuvers, decisions, or techniques being selected or performed by the less senior physician. For example, a surgical procedure involving a craniotomy may require a senior physician to approve a marking made by a less senior physician indicative of where the procedure is to be performed before the actual procedure is initiated.

With advances in and increasing availability of various communication technologies, tele-proctoring or tele-assisting is becoming an increasingly popular option individuals or teams to remotely provide training, guidance, and support to other individuals or teams from a remote location. Moreover, augmented reality technologies are increasingly being used to facilitate remote interactions between two individuals by enabling remote individuals to overlay instructions on top of real-world views for local users. For example, augmented reality technology such as Microsoft's Dynamic365Remote Assist may enable such remote interaction. However, using such augmented reality technologies specifically for tele-proctoring a surgical procedure may not be possible or practical because of specific environmental conditions present within an operating room.

SUMMARY

A system for tele-proctoring a surgical procedure includes an augmented reality head mounted display and a computer, including one or more processors, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors. The program instructions are configured to receive a visual experienced from the eyes of an onsite physician via the augmented reality head mounted display; receive additional content experienced by the onsite physician via the augmented reality head mounted display; integrate the visual experienced from the eyes of an onsite physician and the additional content into a single integrated view experienced by the onsite physician; communicate the integrated view to a remote computer for display on a remote display; receive an interaction with the integrated view from a remote physician via the remote computer; and present the interaction to the onsite physician via the augmented reality head mounted display.

A method for tele-proctoring a surgical procedure includes the steps of receiving a visual experienced from the eyes of an onsite physician via an augmented reality head mounted display; receiving additional content experienced by the onsite physician via the augmented reality head mounted display; integrating the visual experienced from the eyes of an onsite physician and the additional content into a single integrated view experienced by the onsite physician; communicating the integrated view to a remote computer for display on a remote display; receiving an interaction with the integrated view from a remote physician via the remote computer; and presenting the interaction to the onsite physician via the augmented reality head mounted display.

DETAILED DESCRIPTION

The following acronyms and definitions will aid in understanding the detailed description:

AR—Augmented Reality—A live view of a physical, real-world environment whose elements have been enhanced by computer generated sensory elements such as sound, video, or graphics.

VR—Virtual Reality—A 3Dimensional computer generated environment which can be explored and interacted with by a person in varying degrees.

HMD—Head Mounted Display refers to a headset which can be used in AR or VR environments. It may be wired or wireless. It may also include one or more add-ons such as headphones, microphone, HD camera, infrared camera, hand trackers, positional trackers etc.

Controller—A device which includes buttons and a direction controller. It may be wired or wireless. Examples of this device are Xbox gamepad, PlayStation gamepad, Oculus touch, etc.

SNAP Model—A SNAP case refers to a 3D texture or 3D objects created using one or more scans of a patient (CT, MR, fMR, DTI, etc.) in DICOM file format. It also includes different presets of segmentation for filtering specific ranges and coloring others in the 3D texture. It may also include 3D objects placed in the scene including 3D shapes to mark specific points or anatomy of interest, 3D Labels, 3D Measurement markers, 3D Arrows for guidance, and 3D surgical tools. Surgical tools and devices have been modeled for education and patient specific rehearsal, particularly for appropriately sizing aneurysm clips.

Avatar—An avatar represents a user inside the virtual environment.

MD6DM—Multi Dimension full spherical virtual reality, 6 Degrees of Freedom Model. It provides a graphical simulation environment which enables the physician to experience, plan, perform, and navigate the intervention in full spherical virtual reality environment.

A surgery rehearsal and preparation tool previously described in U.S. patent application No. 8,311,791, incorporated in this application by reference, has been developed to convert static CT and MRI medical images into dynamic and interactive multi-dimensional full spherical virtual reality, six (6) degrees of freedom models (“MD6DM”) based on a prebuilt SNAP model that can be used by physicians to simulate medical procedures in real time. The MD6DM provides a graphical simulation environment which enables the physician to experience, plan, perform, and navigate the intervention in full spherical virtual reality environment. In particular, the MD6DM gives the surgeon the capability to navigate using a unique multidimensional model, built from traditional two-dimensional patient medical scans, that gives spherical virtual reality 6 degrees of freedom (i.e. linear; x, y, z, and angular, yaw, pitch, roll) in the entire volumetric spherical virtual reality model.

The MD6DM is rendered in real time using a SNAP model built from the patient's own data set of medical images including CT, MRI, DTI etc., and is patient specific. A representative brain model, such as Atlas data, can be integrated to create a partially patient specific model if the surgeon so desires. The model gives a 360° spherical view from any point on the MD6DM. Using the MD6DM, the viewer is positioned virtually inside the anatomy and can look and observe both anatomical and pathological structures as if he were standing inside the patient's body. The viewer can look up, down, over the shoulders etc., and will see native structures in relation to each other, exactly as they are found in the patient. Spatial relationships between internal structures are preserved and can be appreciated using the MD6DM.

The algorithm of the MD6DM takes the medical image information and builds it into a spherical model, a complete continuous real time model that can be viewed from any angle while “flying” inside the anatomical structure. In particular, after the CT, MRI, etc. takes a real organism and deconstructs it into hundreds of thin slices built from thousands of points, the MD6DM reverts it to a 3D model by representing a 360° view of each of those points from both the inside and outside.

Described herein is an augmented reality (“AR”) system, leveraging a MD6DM model, for tele-proctoring a surgical procedure. In particular, the AR system enables a remotely located physician to interact with and proctor a surgical procedure being performed on a patient by an onsite physician by providing the remote physician the same view as being experienced by the onsite physician via an augmented reality headset, the view including a visual experienced from the eyes of the onsite physician as well as additionally integrated content such as a prebuilt MD6DM model, and providing the remote physician with means for interacting with the view such that the onsite physician experiences the interactions. Thus, the patient is provided with the care and expertise that may not otherwise be available due to location and availability of healthcare professionals at the onsite location.

Integrating the additional content and features into the example AR systems as will be described herein in more detail allows for increased comfort for surgeons as well increased adoption since the AR HIVID may be worn during the entire surgical procedure without needing to take it off in order to view a microscope, to put on other loupe, and so on. The system described herein also enables better multitasking for a physician. Finally, it enables a remote attending physician to be more involved in a surgical procedure and thereby increase the safety of the procedure and reduce risk of error during the procedure.

It should be appreciated that example systems described herein may be used for pre-operative planning, preparing in the operating room, and during an actual surgical procedure. It should be further appreciated that, although an example application for use during a craniotomy may be described herein, the example systems may be sued for any suitable surgical procedure.

FIG. 1illustrates an AR tele-proctoring system100for enabling an onsite physician102located in a hospital104(or any similar suitable location) and performing a surgical procedure on a patient106to communicate with and interact with a remote physician108located in a remote location110. In particular, the AR system100enables the remote physician108to proctor and assist with the surgical procedure from the remote location110. Proctoring a surgical procedure can mean, for example, answering questions during the surgical procedure, making suggestions or providing instructions about how to perform the procedure, and confirming that the actions being taken by the onsite physician102are accurate and correct. It should be appreciated that, although the AR system100is described as being used during a surgical procedure, the AR system100can also be used for pre-operative planning and preparation.

The AR system100includes an AR head mounted display (“HMD”)112for providing the onsite physician102with an AR view including a live real life visual of the patient106in combination with additionally integrated content. For example, the AR system100includes an MD6DM computer114for retrieving a SNAP model from a SNAP database116, for rendering a MD6DM model118, and for providing the MD6DM model118to the AR HIVID112. The MD6DM computer114, in combination with the AR HIVID112, is configured to synchronize the MD6DM model with and overlay it on top of the live real life visual of the patient106in order to create an AR view (not shown) of the patient106via the AR HIVID112.

The AR system100further includes a tele-computer120configured to communicate to a remote computer122the AR view experienced by the onsite physician102. In particular, the tele-computer120is configured to receive a live video feed from a camera on the AR HIVID112that captures and represents the live real life visual of the patient106as seen by the onsite physician102. The tele-computer120is further configured to receive additionally integrated content and to synchronize the additionally integrated content with the live video feed from the AR HIVID112. For example, the tele-computer120is configured to receive from the MD6DM computer114the rendered MD6DM model118and to synchronize the MD6DM model118with the live video feed.

The remote computer122is configured to communicate the AR view including a live video feed124of the patient and a remote MD6DM model128synchronized with the live video feed124to a remote display126. It should be appreciated that the remote display126can be any suitable type of display, including a head mounted display (not shown). Thus, the remote physician108is able to experience in real time vie the remote display126the same view, including the live real life visual of the patient106and the additionally integrated content, as being experienced by the onsite physician102.

In one example, the remote location110includes a remote integrated content computer such as an MD6DM computer (not shown) for retrieving the additionally integrate content such as the SNAP model from a remote database (not shown). Thus, the tele-computer122does not need to synchronize or integrate any additional content with the live video feed received from the AR HMD112. Instead, the tele-computer122communicates the live video feed to the remote computer122without additional content, thereby conserving communication bandwidth. In such an example, the remote computer122retrieves the additional content from the remote integrated content computer and performs the integration and synchronization with the live video feed at the remote location110. For example, the remote computer122is configured to retrieve a SNAP model from a remote SNAP database (not shown) and render the remote MD6DM model128. The remote computer122is further configured to synchronize the remote MD6DM model128with the live video feed124and integrate the two onto the remote display126to form the view representative of the same view being experienced by the onsite physician102.

The remote computer122is further configured to receive, via either the display126or via additional peripheral input devices (not shown), interactions with the view from the remote physician108. For example, remote computer122may receive from the remote physician108markups, notes, and other suitable input interactions with both the live video feed124of the patient and the additionally integrated and synchronized content such as the remote MD6DM model128. The interactions may include, for example, the remote physician108manipulating the remote MD6DM model128or placing a mark on the remote MD6DM model128to indicate where to make an incision. The remote computer122is further able to distinguish between interactions with the live video feed124and interactions with the additionally integrated content such as the remote MD6DM model128.

The remote computer122is further configured to communicate the remote interactions of the remote physician108to tele-computer120, which in turn is configured to communicate and to appropriately render the received remote interactions to the AR HIVID112in connection with the corresponding content. The remote tele-computer120is configured to render received remote interactions with the respective content based on the distinctions identified between the interactions. For example, the tele-computer120may be configured to synchronize and integrate with the MD6DM model118the received remote interactions with the remote MD6DM model128such that the onsite physician102is able to experience the marked view in the MD6DM model118as provided by the remote physician108. In another example, the MD6DM computer114may be configured to receive the interactions from tele-computer120and to synchronize and integrate the remote interactions with the MD6DM model118. It should be appreciated that, although the MD6DM computer114and the tele-computer120are described as two distinct computers, the MD6DM computer114and the tele-computer120may be combined into a single computer (not illustrated).

By communicating markings and other suitable interactions from the remote physician108to the onsite physician102, the AR system100is able to facilitate proctoring of a craniotomy, for example, which requires a physician to mark an entry point for where the procedures should be initiated. Providing such a mark isn't always feasible or practical based on a real life live view alone and often requires the aid of additionally integrated content such a MD6DM model over-lay on top of the skull. Providing a remote view that includes the additionally integrated content enables improved proctoring capabilities and collaboration since the onsite physician102and the remote physician108are able to interact with each other and provide real time feedback with respect to both the real life live view as well as the additionally integrated content.

In one example AR system200, as illustrated inFIG. 2, the additional content integrated with the live real life visual of the patient106and included in the view experienced by the onsite physician includes video generated by an endoscope202. For example, an onsite physician102may use an endoscope to obtain a closeup inside view of the patient106. The closeup view from the endoscope202is incorporated into the view experienced by the onsite physician102via the AR HIVID112. For example, the closeup view from the endoscope may be presented to the onsite physician102in a portion of a lens of the AR HMD112such that the onsite physician102may easily look back and forth between the real life live view the patient or the closeup view form the endoscope, all within the same AR HMD112.

Thus, in order for the remote physician108to experience the same view as the onsite physician102, the tele-computer120is further configured to communicate to the remote computer122the same video generated by the endoscope202in addition to the live video feed captured by the camera on the AR HIVID112. In one example, the remote computer122is configured to integrate the additional content (e.g. the video generated by the endoscope202) with the live video feed from the camera on the AR HIVID112to produce on the display126the same integrated view for the remote physician108as experienced by the onsite physician102. Specifically, the remote computer122may present on the screen126simultaneously both the real life view received from the camera on the HIVID112in a first portion of the display126as well as the closeup view received form the endoscope202in a second portion of the display126.

In another example, the remote computer122may be configured to display one of either the real life view received from the camera on the HIVID112or the closeup view received form the endoscope202, depending on a selection of the remote physician108via an interface provided by the remote computer122. For example, the remote physician may selectively toggle between seeing the real life view received from the camera on the HIVID112or the closeup view received form the endoscope202and selectively interact with either one at any time. In another example, the remote computer122may be configured to automatically toggle the display between either the real life view received from the camera on the HIVID112or the closeup view received form the endoscope202depending on action taken by the onsite physician102. For example, the AR HIVID112may be configured to track the eye movement of the onsite physician102. In particular, the AR HIVID112may be configured to determine when the onsite physician's102eyes are focused on closeup view presented in the AR view and when the onsite physician's102eyes are focused anywhere else within the AR view. Thus, based on the onsite physician's102eye focus, the remote computer122may be configured to automatically present to the display126the same corresponding view.

In one example, the MD6DM computer114may be configured to receive the closeup video feed from the endoscope202and to synchronize and overlay a closeup view of the MD6DM model118over the closeup video feed before communicating the combined integrated closeup video feed to the AR HMD112. In such an example, the tele-computer122may be configured to provide the remote physician108with the same view experienced by the onsite physician102, including an integrated and synchronized closeup view with an MD6DM overlay generated from the endoscope202as well as the real life live view received from the camera on the AR HMD112.

As previously described, the remote computer114is configured to receive and distinguish different types of interactions with the view from the remote physician108and communicate the distinguished interactions to the tele-computer120, which in turn appropriately renders the interactions in connection with the corresponding content.

In one example AR system300, as illustrated inFIG. 3, the additional content integrated with the live real life visual of the patient106and included in the view experienced by the onsite physician includes video generated by a microscope302. The video generated by the microscope302may or may not be synchronized with an MD6DM model, as described in the previous example. Also as in the previous example, the video generated by the microscope302, either with or without MD6DM integration, may be presented to and experienced by the onsite physician102in an augmented view via the AR HMD112. Similarly as described for the previous example of the endoscope video, the additional content of the video from the microscope302may be presented to the remote physician108as part of the experienced view.

In one example, rather than providing a video feed from the microscope302into the AR view via the AR HIVID112, the onsite physician may choose to consume the microscope view by interacting directly with the microscope. For example, the onsite physician102may look through a viewer on the microscope302in order to see a closeup view of the patient106. However, the onsite physician102may still be wearing the AR HMD112and intend for the remote physician108to experience the same view. Thus, in such an example, a video feed from the microscope302may still be provided to by the tele-computer120to the remote computer122in order to enable the remote computer122to generate the same view for the remote physician108as experienced by the onsite physician102.

As described in the previous example, the remote computer122may be configured to display one of either the real life view received from the camera on the HIVID112or the closeup view received form the microscope302, depending on a selection of the remote physician108via an interface provided by the remote computer122.

In another example, the remote computer122may be configured to automatically toggle the display between either the real life view received from the camera on the HIVID112or the closeup view received form the microscope302depending on action taken by the onsite physician102. For example, the tele-computer120may be configured to determine, based on motion sensors or other suitable types of sensors on AR HIVID112and based on the video received from the AR HIVID112, the head position/location of the onsite physician102. In particular, when it is determined that the onsite physician's102head is tilted over top of or otherwise positioned at or near a viewer of the microscope302, the remote computer122may be configured to automatically present to the display126the view feed from the microscope and to present the real life video feed from AR HIVID112when the onsite physician's102head is positioned otherwise.

In one example, the MD6DM computer114may be configured to receive the closeup video feed from the microscope302and to synchronize and overlay a closeup view of the MD6DM model118over the closeup video feed before communicating the combined integrated closeup video feed to the AR HIVID112. In another example, the MD6DM computer114may be configured to inject, synchronize, and overlay a closeup view of the MD6DM model118directly into the view experienced by the looking into the view finder of the microscope302. In such examples, the tele-computer122may be configured to provide the remote physician108with the same view experienced by the onsite physician102, including an integrated and synchronized closeup view with an MD6DM overlay generated from the microscope302as well as the real life live view received from the camera on the AR HIVID112.

As previously described, the remote computer114is configured to receive and distinguish different types of interactions with the view from the remote physician108and communicate the distinguished interactions to the tele-computer120, which in turn appropriately renders the interactions in connection with the corresponding content.

In one example, as illustrated inFIG. 4, an AR HIVID402may have integrated (or removeable/detachable) loupes404for enabling the onsite physician102to get a closeup view of the patient106. For example, the onsite physician102may look straight through the AR HIVID402in order to experience a real life live view of the patient106. The onsite physician102may also choose to look through the loupes404at any time in order to get a closeup view. Thus, in order to provide the remote physician108with the same experience as viewed by the onsite physician, the zoom level of the live video received from the AR HIVID402and provided to the remote computer122is adjusted according to the onsite physician's102eye position relative to the loupes404determined by the AR HIVID402. In particular, if the AR HIVID402determines that the onsite physician's eyes are looking through the loupes404, then the live video received by the camera on the AR HIVID402and provided to the remote computer122is zoomed in to a closer view based on the magnification strength of the loupes404. In one example, the camera on the AR HIVID402is configured to automatically zoom in based on the determined eye position. In another example, the tele-computer120is configured to adjust the live video received from the camera of AR HIVID402based on the determined eye position.

In on example, the real life live view experienced via the AR HMD402may be augmented by a synchronized MD6DM model. In such an example, the MD6DM model may be adjusted and zoomed in as appropriate, depending on the eye position of the onsite physician102. Similarly, the real life live video received from the camera on the AR HIVID402and provided to the remote computer122may synchronized with an MD6DM model, either at the remote site110or at the hospital104before communicated to the remote site110. In addition, the MD6DM model synchronized with the real life live video for presentation at the remote site110may also be zoomed, depending on the onsite physician's102eye position.

As previously described, the remote computer114is configured to receive and distinguish different types of interactions with the view from the remote physician108and communicate the distinguished interactions to the tele-computer120, which in turn appropriately renders the interactions in connection with the corresponding content.

FIG. 5illustrates an example method for tele-proctoring a surgical procedure. At502, a visual experienced form the eyes of an onsite physician of a surgical procedure via an AR headset is received. At504, additional content experienced by the onsite physician via the AR headset is received. At506, the visual experienced form the eyes of an onsite physician and the additional content is integrated into a single view experienced by the onsite physician. At508, the view is provided to a remote physician. At510, an interaction from the remote physician is received. At512, the interaction is presented to the onsite physician via the AR headset.

FIG. 6is a schematic diagram of an example computer for implementing the tele-computer114, the MD6DM computer116, and the remote computer122ofFIG. 1. The example computer600is intended to represent various forms of digital computers, including laptops, desktops, handheld computers, tablet computers, smartphones, servers, and other similar types of computing devices. Computer600includes a processor602, memory604, a storage device606, and a communication port608, operably connected by an interface610via a bus612.

Processor602processes instructions, via memory604, for execution within computer600. In an example embodiment, multiple processors along with multiple memories may be used.

Memory604may be volatile memory or non-volatile memory. Memory604may be a computer-readable medium, such as a magnetic disk or optical disk. Storage device606may be a computer-readable medium, such as floppy disk devices, a hard disk device, optical disk device, a tape device, a flash memory, phase change memory, or other similar solid state memory device, or an array of devices, including devices in a storage area network of other configurations. A computer program product can be tangibly embodied in a computer readable medium such as memory604or storage device606.

Computer600can be coupled to one or more input and output devices such as a display614, a printer616, a scanner618, a mouse620, and a HMD624.

As will be appreciated by one of skill in the art, the example embodiments may be actualized as, or may generally utilize, a method, system, computer program product, or a combination of the foregoing. Accordingly, any of the embodiments may take the form of specialized software comprising executable instructions stored in a storage device for execution on computer hardware, where the software can be stored on a computer-usable storage medium having computer-usable program code embodied in the medium.

Databases may be implemented using commercially available computer applications, such as open source solutions such as MySQL, or closed solutions like Microsoft SQL that may operate on the disclosed servers or on additional computer servers. Databases may utilize relational or object oriented paradigms for storing data, models, and model parameters that are used for the example embodiments disclosed above. Such databases may be customized using known database programming techniques for specialized applicability as disclosed herein.

Any suitable computer usable (computer readable) medium may be utilized for storing the software comprising the executable instructions. The computer usable or computer readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer readable medium would include the following: an electrical connection having one or more wires; a tangible medium such as a portable computer diskette, a hard disk, a random access memory (RAM), a read -only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a compact disc read-only memory (CDROM), or other tangible optical or magnetic storage device; or transmission media such as those supporting the Internet or an intranet.

In the context of this document, a computer usable or computer readable medium may be any medium that can contain, store, communicate, propagate, or transport the program instructions for use by, or in connection with, the instruction execution system, platform, apparatus, or device, which can include any suitable computer (or computer system) including one or more programmable or dedicated processor/controller(s). The computer usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to the Internet, wireline, optical fiber cable, local communication busses, radio frequency (RF) or other means.

Computer program code having executable instructions for carrying out operations of the example embodiments may be written by conventional means using any computer language, including but not limited to, an interpreted or event driven language such as BASIC, Lisp, VBA, or VBScript, or a GUI embodiment such as visual basic, a compiled programming language such as FORTRAN, COBOL, or Pascal, an object oriented, scripted or unscripted programming language such as Java, JavaScript, Perl, Smalltalk, C++, C#, Object Pascal, or the like, artificial intelligence languages such as Prolog, a real-time embedded language such as Ada, or even more direct or simplified programming using ladder logic, an Assembler language, or directly programming using an appropriate machine language.