UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION

A system, method, and device for displaying relevant augmented reality (AR) content to a surgical staff members without over-saturating the augmented display with virtual elements. The surgical staff members (e.g., doctors, nurses, surgeons, technicians, etc.) require AR content that is delicately balanced between displaying helpful information without distracting the surgeon. A surgical hub receives a plurality of inputs related to surgical environment and displays only necessary information to allow the surgeon to provide effective care to the patient, based on a determined trigger event.

BACKGROUND

This disclosure relates to apparatuses, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure it would be desirable to provide an augmented reality interactive experience of a real-world environment where objects that reside in the real world are enhanced by overlaying computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. In the context of this disclosure, images of a surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, haptic, somatosensory, olfactory, or other sensory information onto the real world images of the surgical field and instruments or other objects appearing in the surgical field. The images may be streamed in real time or may be still images.

Real world surgical instruments include a variety of surgical devices including energy, staplers, or combined energy and stapler. Energy based medical devices include, without limitation, radio-frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combination RF electrosurgical and ultrasonic instruments, combination RF electrosurgical and mechanical staplers, among others. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including bariatric, thoracic, colorectal, gynecologic, urologic and general surgery.

SUMMARY

In various instances, the present disclosure provides a surgical system comprises: a remote server; augmented reality (AR) output device; one or more surgical devices; a surgical hub communicatively coupled to the remote server, the one or more surgical devices, and the AR output device, the surgical hub comprises a control circuit coupled to a memory, and wherein the control circuit is configured to: determine contextual parameters from a surgical environment; determine from the contextual parameter a specific surgical procedure being performed in an operating room; receive procedural data from the remote server, wherein the procedural data indicates steps and surgical instruments associated with the specific surgical procedure; determine that a trigger event is anticipated based on the procedural data and the contextual parameters; initiate a response to the anticipated trigger event according to the surgical instruments and procedural data.

In various instances, the present disclosure provides a method for managing surgical device interaction during a surgical procedure, the method comprising: determining, by a surgical hub, contextual parameters from a surgical environment; determining, by the surgical hub, a specific surgical procedure being performed in an operating room based on the contextual parameters; receiving, by the surgical hub, procedural data from a remote server, wherein the procedural data indicates steps and surgical instruments associated with the specific surgical procedure; determining, by the surgical hub, that a trigger event is anticipated based on the procedural data and the contextual parameters; and initiating, by the surgical hub, a response according to the surgical instruments and the procedural data.

Corresponding reference characters indicate corresponding parts throughout the several views. The exemplifications set out herein illustrate various disclosed embodiments, in one form, and such exemplifications are not to be construed as limiting the scope thereof in any manner.

DESCRIPTION

Applicant of the present application owns the following U.S. patent applications filed concurrently herewith, the disclosures of each of which is herein incorporated by reference in its entirety:U.S. patent application, titled METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS; Attorney Docket No. END9352USNP1/210120-1M;U.S. patent application, titled Utilization of surgical data values and situational awareness to control the overlay in surgical field view; Attorney Docket No. END9352USNP2/210120-2;U.S. patent application, titled SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW; Attorney Docket No. END9352USNP3/210120-3;U.S. patent application, titled RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW; Attorney Docket No. END9352USNP4/210120-4;U.S. patent application, titled SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY; Attorney Docket No. END9352USNP5/210120-5;U.S. patent application, titled SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS; Attorney Docket No. END9352USNP6/210120-6;U.S. patent application, titled CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION; Attorney Docket No. END9352USNP7/210120-7;U.S. patent application, titled INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS; Attorney Docket No. END9352USNP8/210120-8;U.S. patent application, titled COOPERATIVE OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING EFFECTED; Attorney Docket No. END9352USNP9/210120-9;U.S. patent application, titled ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS; Attorney Docket No. END9352USNP10/210120-10;U.S. patent application, titled MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN; Attorney Docket No. END9352USNP11/210120-11;U.S. patent application, titled SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT; Attorney Docket No. END9352USNP12/210120-12;U.S. patent application, titled COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED INFORMATION; Attorney Docket No. END9352USNP14/210120-14;U.S. patent application, titled INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS; Attorney Docket No. END9352USNP15/210120-15;U.S. patent application, titled ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS; Attorney Docket No. END9352USNP16/210120-16; andU.S. patent application, titled MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS; Attorney Docket No. END9352USNP17/210120-17.

Applicant of the present application owns the following U.S. patent applications, the disclosure of each of which is herein incorporated by reference in its entirety:U.S. patent application Ser. No. 16/209,423, titled METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS, now U.S. Patent Publication No. US-2019-0200981-A1;U.S. patent application Ser. No. 16/209,453, titled METHOD FOR CONTROLLING SMART ENERGY DEVICES, now U.S. Patent Publication No. US-2019-0201046-A1.

Before explaining various aspects of surgical devices and generators in detail, it should be noted that the illustrative examples are not limited in application or use to the details of construction and arrangement of parts illustrated in the accompanying drawings and description. The illustrative examples may be implemented or incorporated in other aspects, variations and modifications, and may be practiced or carried out in various ways. Further, unless otherwise indicated, the terms and expressions employed herein have been chosen for the purpose of describing the illustrative examples for the convenience of the reader and are not for the purpose of limitation thereof. Also, it will be appreciated that one or more of the following-described aspects, expressions of aspects, and/or examples, can be combined with any one or more of the other following-described aspects, expressions of aspects and/or examples.

Various aspects are directed to onscreen displays for surgical systems for a variety of energy and surgical stapler based medical devices. Energy based medical devices include, without limitation, radio-frequency (RF) based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combination RF electrosurgical and ultrasonic instruments, combination RF electrosurgical and mechanical staplers, among others. Surgical stapler devices include and combined surgical staplers with electrosurgical and/or ultrasonic devices. Aspects of the ultrasonic surgical devices can be configured for transecting and/or coagulating tissue during surgical procedures, for example. Aspects of the electrosurgical devices can be configured for transecting, coagulating, sealing, welding and/or desiccating tissue during surgical procedures, for example. Aspects of the surgical stapler devices can be configured for transecting and stapling tissue during surgical procedures and in some aspects, the surgical stapler devices may be configured to delivery RF energy to the tissue during surgical procedures. Electrosurgical devices are configured to deliver therapeutic and/or nontherapeutic RF energy to the tissue. Elements of surgical staplers, electrosurgical, and ultrasonic devices may be used in combination in a single surgical instrument.

In various aspects, the present disclosure provides onscreen displays of real time information to the OR team during a surgical procedure. In accordance with various aspects of the present disclosure, many new and unique onscreen displays are provided to display onscreen a variety of visual information feedback to the OR team. According to the present disclosure, visual information may comprise one or more than one of various visual media with or without sound. Generally, visual information comprises still photography, motion picture photography, video or audio recording, graphic arts, visual aids, models, display, visual presentation services, and the support processes. The visual information can be communicated on any number of display options such as the primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, among others, for example.

In various aspects, the present disclosure provides a large list of potential options to communicate visual information in real time to the OR team, without overwhelming the OR team with too much visual information. For example, in various aspects, the present disclosure provides onscreen displays of visual information to enable the surgeon, or other members of the OR team, to selectively activate onscreen displays such as icons surrounding the screen option to manage a wealth of visual information. One or a combination of factors can be used to determine the active display, these may include energy based (e.g., electrosurgical, ultrasonic) or mechanical based (e.g., staplers) surgical devices in use, the estimated risk associated with a given display, the experience level of the surgeon and the surgeons' choice among other things. In other aspect, the visual information may comprises rich data overlaid or superimposed into the surgical field of view to manage the visual information. In various aspects described hereinbelow, comprise superimposed imagery that requires video analysis and tracking to properly overlay the data. Visual information data communicated in this manner, as opposed to static icons, may provide additional useful visual information in a more concise and easy to understand way to the OR team.

In various aspects, the present disclosure provides techniques for selectively activating onscreen displays such as icons surrounding the screen to manage visual information during a surgical procedure. In other aspects, the present disclosure provides techniques for determining the active display using one or a combination of factors. In various aspects, the techniques according to the resent disclosure may comprise selecting the energy based or mechanical based surgical device in use as the active display, estimating risk associated with a given display, utilizing the experience level of the surgeon or OR team making the selection, among other things.

In other aspects, the techniques according to the present disclosure may comprise overlaying or superimposing rich data onto the surgical field of view to manage the visual information. A number of the display arrangements described by the present disclosure involve overlaying various visual representations of surgical data onto a livestream of a surgical field. As used herein the term overlay comprises a translucent overlay, a partial overlay, and/or a moving overlay. Graphical overlays may be in the form of a transparent graphic, semitransparent graphic, or opaque graphic, or a combination of transparent, semitransparent, and opaque elements or effects. Moreover, the overlay can be positioned on, or at least partially on, or near an object in the surgical field such as, for example, an end effector and/or a critical surgical structure. Certain display arrangements may comprise a change in one or more display elements of an overlay including a change in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, based on changes in display priority values. The graphical overlays are rendered on top of the active display monitor to convey important information quickly and efficiently to the OR team.

In other aspects, the techniques according to the present disclosure may comprise superimposing imagery that requires analyzing video and tracking for properly overlaying the visual information data. In other aspects, the techniques according to the present disclosure may comprise communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easy to understand manner. In other aspects, the visual overlays may be used in combination with audible and/or somatosensory overlays such as thermal, chemical, and mechanical devices, and combinations thereof.

The following description is directed generally to apparatuses, systems, and methods that provide an augmented reality (AR) interactive experience during a surgical procedure. In this context, images of a surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, haptic, somatosensory, olfactory, or other sensory information onto the real world images of the surgical field, instruments, and/or other objects appearing in the surgical field. The images may be streamed in real time or may be still images. Augmented reality is a technology for rendering and displaying virtual or “augmented” virtual objects, data, or visual effects overlaid on a real environment. The real environment may include a surgical field. The virtual objects overlaid on the real environment may be represented as anchored or in a set position relative to one or more aspects of the real environment. In a non-limiting example, if a real world object exits the real environment field of view, a virtual object anchored to the real world object would also exit the augmented reality field of view.

A number of the display arrangements described by the present disclosure involve overlaying various visual representations of surgical data onto a livestream of a surgical field. As used herein the term overlaying comprises a translucent overlay, a partial overlay, and/or a moving overlay. Moreover, the overlay can be positioned on, or at least partially on, or near an object in the surgical field such as, for example, an end effector and/or a critical surgical structure. Certain display arrangements may comprise a change in one or more display elements of an overlay including a change in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, based on changes in display priority values.

As described herein AR is an enhanced version of the real physical world that is achieved through the use of digital visual elements, sound, or other sensory stimuli delivered via technology. Virtual Reality (VR) is a computer-generated environment with scenes and objects that appear to be real, making the user feel they are immersed in their surroundings. This environment is perceived through a device known as a Virtual Reality headset or helmet. Mixed reality (MR) and AR are both considered immersive technologies, but they aren't the same. MR is an extension of Mixed reality that allows real and virtual elements to interact in an environment. While AR adds digital elements to a live view often by using a camera, an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

In an AR environment, one or more computer-generated virtual objects may be displayed along with one or more real (i.e., so-called “real world”) elements. For example, a real-time image or video of a surrounding environment may be shown on a computer screen display with one or more overlaying virtual objects. Such virtual objects may provide complementary information relating to the environment or generally enhance a user's perception and engagement with the environment. Conversely, the real-time image or video of the surrounding environment may additionally or alternatively enhance a user's engagement with the virtual objects shown on the display.

The apparatuses, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during a surgical procedure. The imaging devices may include a variety of scopes used during non-invasive and minimally invasive surgical procedures, an AR device, and/or a camera to provide images during open surgical procedures. The images may be streamed in real time or may be still images. The apparatuses, systems, and methods provide an augmented reality interactive experience by enhancing images of the real world surgical environment by overlaying virtual objects or representations of data and/or real objects onto the real surgical environment. The augmented reality experience may be viewed on a display and/or an AR device that allows a user to view the overlaid virtual objects onto the real world surgical environment. The display may be located in the operating room or remote from the operating room. AR devices are worn on the head of the surgeon or other operating room personnel and typically include two stereo-display lenses or screens, including one for each eye of the user. Natural light is permitted to pass through the two transparent or semi-transparent display lenses such that aspects of the real environment are visible while also projecting light to make virtual objects visible to the user of the AR device.

Two or more displays and AR devices may be used in a coordinated manner, for example with a first display or AR device controlling one or more additional displays or AR devices in a system with defined roles. For example, when activating display or an AR device, a user may select a role (e.g., surgeon, surgical assistant, nurse, etc., during a surgical procedure) and the display or AR device may display information relevant to that role. For example, a surgical assistant may have a virtual representation of an instrument displayed that the surgeon needs to perform for a next step of a surgical procedure. A surgeon's focus on the current step may see different information displayed than the surgical assistant.

Although there are many known onscreen displays and alerts, this disclosure provides many new and unique augmented reality interactive experiences during a surgical procedure. Such augmented reality interactive experiences include visual, auditory, haptic, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. The virtual feedback information overlaid onto the real world surgical environment may be provided to an operating room (OR) team, including personnel inside the OR including, without limitation, the operating surgeon, assistants to the surgeon, a scrub person, an anesthesiologist and a circulating nurse, among others, for example. The virtual feedback information can be communicated on any number of display options such as a primary OR screen display, an AR device, the energy or surgical stapler instrument, a tablet, augmented reality glasses, device etc.

FIG. 1depicts a computer-implemented interactive surgical system1that includes one or more surgical systems2and a cloud-based system4. The cloud-based system4may include a remote server13coupled to a storage device5. Each surgical system2includes at least one surgical hub6in communication with the cloud4. For example, the surgical system2may include a visualization system8, a robotic system10, and handheld intelligent surgical instruments12, each configured to communicate with one another and/or the hub6. In some aspects, a surgical system2may include an M number of hubs6, an N number of visualization systems8, an O number of robotic systems10, and a P number of handheld intelligent surgical instruments12, where M, N, O, and P are integers greater than or equal to one. The computer-implemented interactive surgical system1may be configured to provide an augmented reality interactive experience during a surgical procedure as described herein.

FIG. 2depicts an example of a surgical system2to perform a surgical procedure on a patient lying down on an operating table14in a surgical operating room16. A robotic system10is used in the surgical procedure as a part of the surgical system2. The robotic system10includes a surgeon's console18, a patient side cart20(surgical robot), and a surgical robotic hub22. The patient side cart20can manipulate at least one removably coupled surgical tool17through a minimally invasive incision in the body of the patient while the surgeon views the surgical site through the surgeon's console18or an augmented reality (AR) device66worn by the surgeon. An image (e.g., still or live streamed in real time) of the surgical site during a minimally invasive procedure can be obtained by a medical imaging device24. The patient side cart20can manipulate the imaging device24to orient the imaging device24. An image of an open surgical procedure can be obtained by a medical imaging device96. The robotic hub22processes the images of the surgical site for subsequent display on the surgeon's console18or the AR device66worn by the surgeon, or other person in the surgical operating room16.

The optical components of the imaging device24,96or AR device66may include one or more illumination sources and/or one or more lenses. The one or more illumination sources may be directed to illuminate portions of the surgical field. One or more image sensors may receive light reflected or refracted from tissue and instruments in the surgical field.

In various aspects, the imaging device24is configured for use in a minimally invasive surgical procedure. Examples of imaging devices suitable for use with this disclosure include, but not limited to, an arthroscope, angioscope, bronchoscope, choledochoscope, colonoscope, cytoscope, duodenoscope, enteroscope, esophagogastro-duodenoscope (gastroscope), endoscope, laryngoscope, nasopharyngo-neproscope, sigmoidoscope, thoracoscope, and ureteroscope. In various aspects, the imaging device96is configured for use in an open (invasive) surgical procedure.

In various aspects, the visualization system8includes one or more imaging sensors, one or more image-processing units, one or more storage arrays, and one or more displays that are strategically arranged with respect to the sterile field. In one aspect, the visualization system8includes an interface for HL7, PACS, and EMR. In one aspect, the imaging device24may employ multi-spectrum monitoring to discriminate topography and underlying structures. A multi-spectral image captures image data within specific wavelength ranges in the electromagnetic spectrum. Wavelengths are separated by filters or instruments sensitive to particular wavelengths, including light from frequencies beyond the visible light range, e.g., IR and ultraviolet. Spectral imaging can extract information not visible to the human eye. Multi-spectrum monitoring can relocate a surgical field after a surgical task is completed to perform tests on the treated tissue.

FIG. 2depicts a primary display19positioned in the sterile field to be visible to an operator at the operating table14. A visualization tower11is positioned outside the sterile field and includes a first non-sterile display7and a second non-sterile display9, which face away from each other. The visualization system8, guided by the hub6, is configured to utilize the displays7,9,19to coordinate information flow to operators inside and outside the sterile field. For example, the hub6may cause the visualization system8to display AR images of the surgical site, as recorded by an imaging device24,96on a non-sterile display7,9, or through the AR device66, while maintaining a live feed of the surgical site on the primary display19or the AR device66. The non-sterile display7,9can permit a non-sterile operator to perform a diagnostic step relevant to the surgical procedure, for example.

FIG. 3depicts a hub6in communication with a visualization system8, a robotic system10, and a handheld intelligent surgical instrument12. The hub6includes a hub display35, an imaging module38, a generator module40, a communication module30, a processor module32, a storage array34, and an operating room mapping module33. The hub6further includes a smoke evacuation module26and/or a suction/irrigation module28. In various aspects, the imaging module38comprises an AR device66and the processor module32comprises an integrated video processor and an augmented reality modeler (e.g., as shown inFIG. 10). A modular light source may be adapted for use with various imaging devices. In various examples, multiple imaging devices may be placed at different positions in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive or open surgical procedures). The imaging module38can be configured to switch between the imaging devices to provide an optimal view. In various aspects, the imaging module38can be configured to integrate the images from the different imaging devices and provide an augmented reality interactive experience during a surgical procedure as described herein.

FIG. 4shows a surgical data network51comprising a modular communication hub53configured to connect modular devices located in one or more operating theaters/rooms of a healthcare facility to a cloud-based system. The cloud54may include a remote server63(FIG. 5) coupled to a storage device55. The modular communication hub53comprises a network hub57and/or a network switch59in communication with a network router61. The modular communication hub53is coupled to a local computer system60to process data. Modular devices1a-1nin the operating theater may be coupled to the modular communication hub53. The network hub57and/or the network switch59may be coupled to a network router61to connect the devices1a-1nto the cloud54or the local computer system60. Data associated with the devices1a-1nmay be transferred to cloud-based computers via the router for remote data processing and manipulation. The operating theater devices1a-1nmay be connected to the modular communication hub53over a wired channel or a wireless channel. The surgical data network51environment may be employed to provide an augmented reality interactive experience during a surgical procedure as described herein and in particular providing augmented images if the surgical field to one or more than one remote display58.

FIG. 5illustrates a computer-implemented interactive surgical system50. The computer-implemented interactive surgical system50is similar in many respects to the computer-implemented interactive surgical system1. The computer-implemented interactive surgical system50includes one or more surgical systems52, which are similar in many respects to the surgical systems2. Each surgical system52includes at least one surgical hub56in communication with a cloud54that may include a remote server63. In one aspect, the computer-implemented interactive surgical system50comprises a modular control tower23connected to multiple operating theater devices such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating theater. As shown inFIG. 6, the modular control tower23comprises a modular communication hub53coupled to a computer system60.

Back toFIG. 5, the modular control tower23is coupled to an imaging module38that is coupled to an endoscope98, a generator module27that is coupled to an energy device99, a smoke evacuator module76, a suction/irrigation module78, a communication module13, a processor module15, a storage array16, a smart device/instrument21optionally coupled to a display39, and a sensor module29. The operating theater devices are coupled to cloud computing resources such as server63, data storage55, and displays58via the modular control tower23. A robot hub72also may be connected to the modular control tower23and to the servers63, data storage55, and displays58. The devices/instruments21, visualization systems58, among others, may be coupled to the modular control tower23via wired or wireless communication standards or protocols, as described herein. The modular control tower23may be coupled to a hub display65(e.g., monitor, screen) to display augmented images received comprising overlaid virtual objects on the real surgical field received from the imaging module38, device/instrument display39, and/or other visualization systems58. The hub display65also may display data received from devices connected to the modular control tower23in conjunction with images and overlaid images.

FIG. 6illustrates a surgical hub56comprising a plurality of modules coupled to the modular control tower23. The modular control tower23comprises a modular communication hub53, e.g., a network connectivity device, and a computer system60to provide local processing, visualization, and imaging of augmented surgical information, for example. The modular communication hub53may be connected in a tiered configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub53and transfer data associated with the modules to the computer system60, cloud computing resources, or both. Each of the network hubs/switches57,59in the modular communication hub53may include three downstream ports and one upstream port. The upstream network hub/switch57,59is connected to a processor31to provide a communication connection to the cloud computing resources and a local display67. Communication to the cloud54may be made either through a wired or a wireless communication channel.

The computer system60comprises a processor31and a network interface37. The processor31is coupled to a communication module41, storage45, memory46, non-volatile memory47, and input/output interface48via a system bus. The system bus can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures.

The processor31comprises an augmented reality modeler (e.g., as shown inFIG. 10) and may be implemented as a single-core or multicore processor such as those known under the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be an LM4F230H5QR ARM Cortex-M4F Processor Core, available from Texas Instruments, for example, comprising an on-chip memory of 256 KB single-cycle flash memory, or other non-volatile memory, up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, a 32 KB single-cycle serial random access memory (SRAM), an internal read-only memory (ROM) loaded with StellarisWare® software, a 2 KB electrically erasable programmable read-only memory (EEPROM), and/or one or more pulse width modulation (PWM) modules, one or more quadrature encoder inputs (QEI) analogs, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available for the product datasheet.

The system memory includes volatile memory and non-volatile memory. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer system, such as during start-up, is stored in non-volatile memory. For example, the non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random-access memory (RAM), which acts as external cache memory. Moreover, RAM is available in many forms such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

The computer system60also includes removable/non-removable, volatile/non-volatile computer storage media, such as for example disk storage. The disk storage includes, but is not limited to, devices like a magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-60 drive, flash memory card, or memory stick. In addition, the disk storage can include storage media separately or in combination with other storage media including, but not limited to, an optical disc drive such as a compact disc ROM device (CD-ROM), compact disc recordable drive (CD-R Drive), compact disc rewritable drive (CD-RW Drive), or a digital versatile disc ROM drive (DVD-ROM). To facilitate the connection of the disk storage devices to the system bus, a removable or non-removable interface may be employed.

In various aspects, the computer system60ofFIG. 6, the imaging module38and/or visualization system58, and/or the processor module15ofFIGS. 4-6, may comprise an image processor, image-processing engine, graphics processing unit (GPU), media processor, or any specialized digital signal processor (DSP) used for the processing of digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) technologies to increase speed and efficiency. The digital image-processing engine can perform a range of tasks. The image processor may be a system on a chip with multicore processor architecture.

FIG. 7illustrates an augmented reality system263comprising an intermediate signal combiner64positioned in the communication path between an imaging module38and a surgical hub display67. The signal combiner64combines audio and/or image data received from an imaging module38and/or an AR device66. The surgical hub56receives the combined data from the combiner64and overlays the data provided to the display67, where the overlaid data is displayed. The imaging device68may be a digital video camera and the audio device69may be a microphone. The signal combiner64may comprise a wireless heads-up display adapter to couple to the AR device66placed into the communication path of the display67to a console allowing the surgical hub56to overlay data on the display67.

FIG. 8illustrates an augmented reality (AR) system comprising an intermediate signal combiner positioned in the communication path between an imaging module and a surgical hub display.FIG. 8illustrates an AR device66worn by a surgeon73to communicate data to the surgical hub56. Peripheral information of the AR device66does not include active video. Rather, the peripheral information includes only device settings, or signals that do not have same demands of refresh rates. Interaction may augment the surgeon's73information based on linkage with preoperative computerized tomography (CT) or other data linked in the surgical hub56. The AR device66can identify structure—ask whether instrument is touching a nerve, vessel, or adhesion, for example. The AR device66may include pre-operative scan data, an optical view, tissue interrogation properties acquired throughout procedure, and/or processing in the surgical hub56used to provide an answer. The surgeon73can dictate notes to the AR device66to be saved with patient data in the hub storage45for later use in report or in follow up.

The AR device66worn by the surgeon73links to the surgical hub56with audio and visual information to avoid the need for overlays, and allows customization of displayed information around periphery of view. The AR device66provides signals from devices (e.g., instruments), answers queries about device settings, or positional information linked with video to identify quadrant or position. The AR device66has audio control and audio feedback from the AR device66. The AR device66is able to interact with other systems in the operating theater and have feedback and interaction available wherever the surgeon73is viewing. For example, the AR device66may receive voice or gesture initiated commands and queries from a surgeon, and the AR device66may provide feedback in the form of one or more modalities including audio, visual, or haptic touch.

FIG. 9illustrates a surgeon73wearing an AR device66, a patient74, and may include a camera96in an operating room75. The AR device66worn by the surgeon73may be used to present to the surgeon73a virtual object overlaid on a real time image of the surgical field through augmented reality display89or through the hub connected display67. The real time image may include a portion of a surgical instrument77. The virtual object may not be visible to others within the operating room75(e.g., surgical assistant or nurse), though they also may wear AR devices66. Even if another person is viewing the operating room75with an AR device66, the person may not be able to see the virtual object or may be able to see the virtual object in a shared augmented reality with the surgeon73, or may be able to see a modified version of the virtual object (e.g., according to customizations unique to the surgeon73) or may see different virtual objects.

A virtual object and/or data may be configured to appear on a portion of a surgical instrument77or in a surgical field of view captured by an imaging module38, an imaging device68during minimally invasive surgical procedures, and/or the camera96during open surgical procedures. In the illustrated example, the imaging module38is a laparoscopic camera that provides a live feed of a surgical area during a minimally invasive surgical procedure. An AR system may present virtual objects that are fixed to a real object without regard to a perspective of a viewer or viewers of the AR system (e.g., the surgeon73). For example, a virtual object may be visible to a viewer of the AR system inside the operating room75and not visible to a viewer of the AR system outside the operating room75. The virtual object may be displayed to the viewer outside the operating room75when the viewer enters the operating room75. The augmented image may be displayed on the surgical hub display67or the augmented reality display89.

The AR device66may include one or more screens or lens, such as a single screen or two screens (e.g., one per eye of a user). The screens may allow light to pass through the screens such that aspects of the real environment are visible while displaying the virtual object. The virtual object may be made visible to the surgeon73by projecting light. A virtual object may appear to have a degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

An AR system may be viewable to one or more viewers, and may include differences among views available for the one or more viewers while retaining some aspects as universal among the views. For example, a heads-up display may change between two views while virtual objects and/or data may be fixed to a real object or area in both views. Aspects such as a color of an object, lighting, or other changes may be made among the views without changing a fixed position of at least one virtual object.

A user may see a virtual object and/or data presented in an AR system as opaque or as including some level of transparency. In an example, the user may interact with the virtual object, such as by moving the virtual object from a first position to a second position. For example, the user may move an object with his or her hand. This may be done in the AR system virtually by determining that the hand has moved into a position coincident or adjacent to the object (e.g., using one or more cameras, which may be mounted on the AR device66, such as AR device camera79or separate96, and which may be static or may be controlled to move), and causing the object to move in response. Virtual aspects may include virtual representations of real world objects or may include visual effects, such as lighting effects, etc. The AR system may include rules to govern the behavior of virtual objects, such as subjecting a virtual object to gravity or friction, or may include other predefined rules that defy real world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device66may include a camera79on the AR device66(not to be confused with the camera96, separate from the AR device66). The AR device camera79or the camera96may include an infrared camera, an infrared filter, a visible light filter, a plurality of cameras, a depth camera, etc. The AR device66may project virtual items over a representation of a real environment, which may be viewed by a user.

The AR device66may be used in the operating room75during a surgical procedure, for example performed by the surgeon73on the patient74. The AR device66may project or display virtual objects, such as a virtual object during the surgical procedure to augment the surgeon's vision. The surgeon73may view a virtual object using the AR device66, a remote controller for the AR device66, or may interact with a virtual object, for example, using a hand to “interact” with a virtual object or a gesture recognized by the camera79of the AR device66. A virtual object may augment a surgical tool such as the surgical instrument77. For example, the virtual object may appear (to the surgeon73viewing the virtual object through the AR device66) to be coupled with or remain a fixed distance from the surgical instrument77. In another example, the virtual object may be used to guide the surgical instrument77, and may appear to be fixed to the patient74. In certain examples, a virtual object may react to movements of other virtual or real-world objects in the surgical field. For example, the virtual object may be altered when a surgeon is manipulating a surgical instrument in proximity to the virtual object.

The augmented reality display system imaging device38capture a real image of a surgical area during a surgical procedure. An augmented reality display89,67presents an overlay of an operational aspect of the surgical instrument77onto the real image of the surgical area. The surgical instrument77includes communications circuitry231to communicate operational aspects and functional data from the surgical instrument77to the AR device66via communication communications circuitry233on the AR device66. Although the surgical instrument77and the AR device66are shown in RF wireless communication between circuits231,233as indicated by arrows B, C, other communication techniques may employed (e.g., wired, ultrasonic, infrared, etc.). The overlay is related to the operational aspect of the surgical instrument77being actively visualized. The overlay combines aspects of tissue interaction in the surgical area with functional data from the surgical instrument77. A processor portion of the AR device66is configured to receive the operational aspects and functional data from the surgical instrument77, determine the overlay related to the operation of the surgical instrument77, and combine the aspect of the tissue in the surgical area with the functional data from the surgical instrument77. The augmented images indicate alerts relative to device performance considerations, alerts of incompatible usage, alerts on incomplete capture. Incompatible usage includes tissue out range conditions and tissue incorrectly balanced within the jaws of the end effector. Additional augmented images provide an indication of collateral events including indication of tissue tension and indication of foreign object detection. Other augmented images indicate device status overlays and instrument indication.

FIG. 10illustrates a system83for augmenting images of a surgical field with information using an AR display89, in accordance with at least one aspect of this disclosure. The system83may be used to perform the techniques described hereinbelow, for example, by using the processor85. The system83includes one aspect of an AR device66that may be in communication with a database93. The AR device66includes a processor85, memory87, an AR display89, and a camera79. The AR device66may include a sensor90, a speaker91, and/or a haptic controller92. The database93may include image storage94or preoperative plan storage95.

The processor85of the AR device66includes an augmented reality modeler86. The augmented reality modeler86may be used by the processor85to create the augmented reality environment. For example, the augmented reality modeler86may receive images of the instrument in a surgical field, such as from the camera79or sensor90, and create the augmented reality environment to fit within a display image of the surgical field of view. In another example, physical objects and/or date may be overlaid on the surgical field of view and/or the surgical instruments images and the augmented reality modeler86may use physical objects and data to present the augmented reality display of virtual object s and/or data in the augmented reality environment. For example, the augmented reality modeler86may use or detect an instrument at a surgical site of the patient and present a virtual object and/or data on the surgical instrument and/or an image of the surgical site in the surgical field of view captured by the camera79. The AR display89may display the AR environment overlaid on a real environment. The display89may show a virtual object and/or data, using the AR device66, such as in a fixed position in the AR environment.

The AR device66may include a sensor90, such as an infrared sensor. The camera79or the sensor90may be used to detect movement, such as a gesture by a surgeon or other user, that may be interpreted by the processor85as attempted or intended interaction by the user with the virtual target. The processor85may identify an object in a real environment, such as through processing information received using the camera79. In other aspects, the sensor90may be a tactile, audible, chemical, or thermal sensor to generate corresponding signals that may combined with various data feeds to create the augmented environment. The sensor90may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye tracking sensors, and voice command recognition functions.

The AR display89, for example during a surgical procedure, may present, such as within a surgical field while permitting the surgical field to be viewed through the AR display89, a virtual feature corresponding to a physical feature hidden by an anatomical aspect of a patient. The virtual feature may have a virtual position or orientation corresponding to a first physical position or orientation of the physical feature. In an example, the virtual position or orientation of the virtual feature may include an offset from the first physical position or orientation of the physical feature. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical aspect, or the like.

In one example, the AR device66may be an individual AR device. In one aspect, the AR device66may be a HoloLens 2 AR device manufactured by Microsoft of Redmond, Wash. This AR device66includes a visor with lenses and binaural audio features (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer), environmental sensors, depth camera, and video camera, hand and eye tracking, and voice command recognition functions. It provides an improved field of view with high resolution by using mirrors to direct waveguides in front of wearer's eyes. Images can be enlarged by changing angles of mirrors. It also provides eye tracking to recognize users and adjust lens widths for specific users.

In another example, the AR device66may be a Snapchat Spectacles 3 AR device. This AR device provides the ability to capture paired images and recreate 3D depth mapping, add in virtual effects, and replay 3D videos. The AR device includes two HD cameras to capture 3D photos and videos at 60 fps—while four built-in microphones record immersive, high-fidelity audio. Images from both cameras combine to build out a geometric map of the real world around the user to provide a new sense of depth perception. Photos and videos may be wirelessly synchronized to external display devices.

In yet another example, the AR device66may be a Glass 2 AR device by Google. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on lens (out of view) to supplement information.

In another example, the AR device66may be an Echo Frames AR device by Amazon. This AR device does not have cameras/displays. A microphone and speaker are linked to Alexa. This AR device provides less functionality than a heads-up display.

In yet another example, the AR device66may be a Focals AR device by North (Google). This AR device provides notification pusher/smartwatch analog; inertial measurement, screen overlay of information (weather, calendar, messages), voice control (Alexa) integration. This AR device provides basic heads-up display functionality.

In another example, the AR device66may be an Nreal AR device. This AR device includes spatial sound, two environmental cameras, a photo camera, IMU (accelerometer, gyroscope), ambient light sensor, proximity sensor functionality. A nebula projects application information on lenses.

In various other examples, the AR device66may be any one of the following commercially available AR devices: Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, EyeTap to create collinear light to that of the environment directly into the retina. A beam splitter makes the same light seen by the eye available to the computer to process and overlay information, for example. AR visualization systems include HUD, contact lenses, glasses, virtual reality (VR) headsets, virtual retinal display, on in operating room displays, and/or smart contact lenses (bionic lenses).

Multi-user interfaces for the AR device66include virtual retinal displays such as raster displays drawn directly on retinas instead of on a screen in front of the eye, smart televisions, smart phones, and/or spatial displays such as Sony spatial display systems.

Other AR technology may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, Apple Polycam app, Ubiquity 6 (Mirrorworld using Display.land app)—users can scan and get 3d image of real world (to create 3D model). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, objection recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, Magic Leap.

Situational awareness is the ability of some aspects of a surgical system to determine or infer information related to a surgical procedure from data received from databases and/or instruments. The information can include the type of procedure being undertaken, the type of tissue being operated on, or the body cavity that is the subject of the procedure. With the contextual information related to the surgical procedure, the surgical system can, for example, improve the manner in which it controls the modular devices (e.g., a robotic arm and/or robotic surgical tool) that are connected to it and provide contextualized information or suggestions to the surgeon during the course of the surgical procedure.

FIG. 11illustrates a timeline of a situational awareness surgical procedure.FIG. 11illustrates a timeline5200of an illustrative surgical procedure and the contextual information that a surgical hub5104can derive from the data received from the data sources5126at each step in the surgical procedure. The timeline5200depicts the typical steps that would be taken by the nurses, surgeons, and other medical personnel during the course of a lung segmentectomy procedure, beginning with setting up the operating theater and ending with transferring the patient to a post-operative recovery room. The situationally aware surgical hub5104receives data from the data sources5126throughout the course of the surgical procedure, including data generated each time medical personnel utilize a modular device5102that is paired with the surgical hub5104. The surgical hub5104can receive this data from the paired modular devices5102and other data sources5126and continually derive inferences (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub5104is able to, for example, record data pertaining to the procedure for generating reports, verify the steps being taken by the medical personnel, provide data or prompts (e.g., via a display screen) that may be pertinent for the particular procedural step, adjust modular devices5102based on the context (e.g., activate monitors, adjust the FOV of the medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and take any other such action described above.

First5202, the hospital staff members retrieve the patient's EMR from the hospital's EMR database. Based on select patient data in the EMR, the surgical hub5104determines that the procedure to be performed is a thoracic procedure.

Second5204, the staff members scan the incoming medical supplies for the procedure. The surgical hub5104cross-references the scanned supplies with a list of supplies that are utilized in various types of procedures and confirms that the mix of supplies corresponds to a thoracic procedure. Further, the surgical hub5104is also able to determine that the procedure is not a wedge procedure (because the incoming supplies either lack certain supplies that are necessary for a thoracic wedge procedure or do not otherwise correspond to a thoracic wedge procedure).

Third5206, the medical personnel scan the patient band via a scanner5128that is communicably connected to the surgical hub5104. The surgical hub5104can then confirm the patient's identity based on the scanned data.

Fourth5208, the medical staff turns on the auxiliary equipment. The auxiliary equipment being utilized can vary according to the type of surgical procedure and the techniques to be used by the surgeon, but in this illustrative case they include a smoke evacuator, insufflator, and medical imaging device. When activated, the auxiliary equipment that are modular devices5102can automatically pair with the surgical hub5104that is located within a particular vicinity of the modular devices5102as part of their initialization process. The surgical hub5104can then derive contextual information about the surgical procedure by detecting the types of modular devices5102that pair with it during this pre-operative or initialization phase. In this particular example, the surgical hub5104determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices5102. Based on the combination of the data from the patient's EMR, the list of medical supplies to be used in the procedure, and the type of modular devices5102that connect to the hub, the surgical hub5104can generally infer the specific procedure that the surgical team will be performing. Once the surgical hub5104knows what specific procedure is being performed, the surgical hub5104can then retrieve the steps of that procedure from a memory or from the cloud and then cross-reference the data it subsequently receives from the connected data sources5126(e.g., modular devices5102and patient monitoring devices5124) to infer what step of the surgical procedure the surgical team is performing.

Fifth5210, the staff members attach the EKG electrodes and other patient monitoring devices5124to the patient. The EKG electrodes and other patient monitoring devices5124are able to pair with the surgical hub5104. As the surgical hub5104begins receiving data from the patient monitoring devices5124, the surgical hub5104thus confirms that the patient is in the operating theater.

Sixth5212, the medical personnel induce anesthesia in the patient. The surgical hub5104can infer that the patient is under anesthesia based on data from the modular devices5102and/or patient monitoring devices5124, including EKG data, blood pressure data, ventilator data, or combinations. Upon completion of the sixth step5212, the pre-operative portion of the lung segmentectomy procedure is completed and the operative portion begins.

Seventh5214, the patient's lung that is being operated on is collapsed (while ventilation is switched to the contralateral lung). The surgical hub5104can infer from the ventilator data that the patient's lung has been collapsed. The surgical hub5104can infer that the operative portion of the procedure has commenced as it can compare the detection of the patient's lung collapsing to the expected steps of the procedure (which can be accessed or retrieved previously) and thereby determine that collapsing the lung is the first operative step in this particular procedure.

Eighth5216, the medical imaging device5108(e.g., a scope) is inserted and video from the medical imaging device is initiated. The surgical hub5104receives the medical imaging device data (i.e., still image data or live streamed video in real time) through its connection to the medical imaging device. Upon receipt of the medical imaging device data, the surgical hub5104can determine that the laparoscopic portion of the surgical procedure has commenced. Further, the surgical hub5104can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that a wedge procedure has already been discounted by the surgical hub5104based on data received at the second step5204of the procedure). The data from the medical imaging device124(FIG. 2) can be utilized to determine contextual information regarding the type of procedure being performed in a number of different ways, including by determining the angle at which the medical imaging device is oriented with respect to the visualization of the patient's anatomy, monitoring the number or medical imaging devices being utilized (i.e., that are activated and paired with the surgical hub5104), and monitoring the types of visualization devices utilized.

For example, one technique for performing a VATS lobectomy places the camera in the lower anterior corner of the patient's chest cavity above the diaphragm, whereas one technique for performing a VATS segmentectomy places the camera in an anterior intercostal position relative to the segmental fissure. Using pattern recognition or machine learning techniques, for example, the situational awareness system can be trained to recognize the positioning of the medical imaging device according to the visualization of the patient's anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, whereas another technique for performing a VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicably coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not utilized in a VATS lobectomy. By tracking any or all of this data from the medical imaging device5108, the surgical hub5104can thereby determine the specific type of surgical procedure being performed and/or the technique being used for a particular type of surgical procedure.

Ninth5218, the surgical team begins the dissection step of the procedure. The surgical hub5104can infer that the surgeon is in the process of dissecting to mobilize the patient's lung because it receives data from the RF or ultrasonic generator indicating that an energy instrument is being fired. The surgical hub5104can cross-reference the received data with the retrieved steps of the surgical procedure to determine that an energy instrument being fired at this point in the process (i.e., after the completion of the previously discussed steps of the procedure) corresponds to the dissection step.

Tenth5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub5104can infer that the surgeon is ligating arteries and veins because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similarly to the prior step, the surgical hub5104can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieved steps in the process.

Eleventh5222, the segmentectomy portion of the procedure is performed. The surgical hub5104infers that the surgeon is transecting the parenchyma based on data from the surgical instrument, including data from a staple cartridge. The cartridge data may correspond to size or type of staple being fired by the instrument. The cartridge data can indicate the type of tissue being stapled and/or transected for different types of staples utilized in different types of tissues. The type of staple being fired is utilized for parenchyma or other tissue types to allow the surgical hub5104to infer that the segmentectomy procedure is being performed.

Twelfth5224, the node dissection step is then performed. The surgical hub5104can infer that the surgical team is dissecting the node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, an RF or ultrasonic instrument being utilized after parenchyma was transected corresponds to the node dissection step, which allows the surgical hub5104to make this inference. It should be noted that surgeons regularly switch back and forth between surgical stapling/cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending upon the particular step in the procedure because different instruments are better adapted for particular tasks. Therefore, the particular sequence in which the stapling/cutting instruments and surgical energy instruments are used can indicate what step of the procedure the surgeon is performing. Upon completion of the twelfth step5224, the incisions and closed up and the post-operative portion of the procedure begins.

Thirteenth5226, the patient's anesthesia is reversed. The surgical hub5104can infer that the patient is emerging from the anesthesia based on the ventilator data (i.e., the patient's breathing rate begins increasing), for example.

Lastly, fourteenth5228, the medical personnel remove the various patient monitoring devices5124from the patient. The surgical hub5104can thus infer that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices5124. The surgical hub5104can determine or infer when each step of a given surgical procedure is taking place according to data received from the various data sources5126that are communicably coupled to the surgical hub5104.

In addition to utilizing the patient data from EMR database(s) to infer the type of surgical procedure that is to be performed, as illustrated in the first step5202of the timeline5200depicted inFIG. 11, the patient data can also be utilized by a situationally aware surgical hub5104to generate control adjustments for the paired modular devices5102.

The present disclosure provides a system, method, and device for displaying relevant augmented reality (AR) content to the surgical staff members without over-saturating the augmented display with virtual elements. The surgical staff members (e.g. doctors, nurses, surgeons, technicians, etc.) require AR content that is delicately balanced between displaying helpful information without distracting the surgeon. A surgical hub receives a plurality of inputs related to surgical environment and displays only necessary information to allow the surgeon to provide effective care to the patient.

In various aspects, the surgical hub generates virtual elements that are displayed in response to a triggered event or an evaluation of contextual parameter received by the situational awareness system (FIG. 11). The surgical hub may determine contextual information of the surgical procedure and environment including a current surgical procedure, an expected next step in the surgical procedure, and/or an active surgical instrument. The surgical hub evaluates the contextual information determine whether information is relevant, and provide necessary or emergency information to the surgeon. If the surgeon is inundated with information, they may begin to ignore notifications or have a difficult time distinguishing between monitored information and emergency notifications.

Additionally, the surgical hub may selectively display or update information at a refresh rate that is useful for the surgeon but not does create a lag, jitter, or delay. An essential component of AR content is that it is consumed by the user in real-time. Emergency notification must be immediately displayed and therefore processing lags and delays are unacceptable in a surgical environment. In order to prevent network and processing delays, certain traffic and virtual elements may be prioritized over others. In one aspect, certain parameters may be continuously monitored by the surgical hub, but only displayed in response to a predetermined threshold or a trigger event. The trigger event may include audible noise interference, low or high availability of bandwidth, and activation of a medical system, an unanticipated medical event, etc.

In various aspect, a trigger event may include an anticipated signal interruption that is the result of a surgical system, such as a mono-polar or bi-polar energy system. In response to the trigger event, the surgical hub may be configured to automatically display a warning on an AR device, or take remedial action and notify the surgical staff members through on an AR device.

In one aspect, the surgical hub may use redundant communication protocols for high resiliency communication. Redundant communication has a higher packet payload than a low weight packet like User Datagram Protocol (UDP), but comprises built-in redundancies to ensure packet receipt, such as checksum. The surgical hub may determine that a surgical device is connected to the surgical hub with an interference susceptible communication protocol, and suggests that the surgical device moves to a more resilient communication protocol, such as a low communication throughput Bluetooth. If the surgical device experiences interference during a communication transfer, the device can be rescanned after the procedure to initiate the transfer and confirm that the data received by the surgical hub was accurate.

In various aspects, the surgical hub evaluates a plurality of surgical systems and signal present during the surgical procedure to determine and prevent signal interference.FIG. 12shows a graphical representation17100of frequency shifting response by a surgical hub6,56(FIGS. 1-3, 5-8) to anticipated signal interference. Prior to the generation of signal17106by a first surgical device, the surgical hub may notify the surgical staff that a second surgical device produces signal17104that may experience interference or signal interruption during the use of the first surgical device. The surgical hub may prompt the second surgical device to shift communication signals to a different frequency, outside the frequency spectrum of the first surgical device. Additionally, if the surgical hub deems that the communication by the second surgical device is a critical communication, the surgical hub may automatically shift the frequency of signal17104to the frequency at17102. In one example, the 2.4 gHz frequency band is commonly used for under IEEE 802.11 for wireless communication, but is also used by electromechanical equipment. A bio-polar or monopolar energy system by generate signal noise in the 2.4 gHz frequency band that results in high packet loss and thus, communication delay. In order to avoid signal interference and communication delay, the surgical hub by seek another frequency band such as the 5.0 gHz frequency band. The second surgical device may then be enable to dynamically channel hop or channel hop based on a trigger communication sent by the surgical hub. Prior to the beginning of the surgical procedure, the surgical hub may recognize a potential communication interference between two devices and may initiate the prompt at the outset of surgery or automatically enable channel hoping as a precaution.

The surgical hub receives contextual data that indicates the specific procedure that is to be performed, the surgical instruments that are used during the procedure, situations when the surgical instruments are surgical instruments must be activated, flexible time intervals when the instruments may be activated, the potential for interference due to instrument activation, and the type of interference. Based on this information, the surgical hub can automatically schedule the activation of surgical systems and communication to avoid an anticipated interference.

FIG. 13shows a timeline17200of device activation and communication transmissions scheduled by a surgical hub6,56(FIGS. 1-3, 5-8). The surgical hub may delay communication signals17204auntil after the activation of a surgical system at17202a. The surgical hub repeats this sequence by delaying the communication of signals17204b, until after the activation of the surgical system at17202b. The surgical determines, based on a hierarchy of communication and device activation, that communication signal17204is a critical communication and all other devices are deactivated. Typically the surgical system17202would be activated, but is delayed for a critical communication. Upon the completion of the signal transmission17204, the surgical system resumes activation at17202c.

FIG. 14shows a flow diagram17300to evaluate a plurality of factors and determine a hierarchy of communication and device activation. The surgical hub6,56(FIGS. 1-3, 5-8) may determine17302a hierarchy based on the schedule of device activation in a surgical procedure and anticipated interference. The surgical hub also may prioritize certain communications, in real-time, based on critical factors (e.g., life and death situations). The surgical hub determines17304the type of interference that may inhibit the effectiveness of a device or system, and the degree for the interference to be perceptible. For example, the surgical hub6,56determines17304bif device is susceptible to the type of interference/noise that can be produced by the potential interference causing signal. The surgical hub6,56uses tracking and position information to determine17306whether the first device is located or positioned within the body of the patient. For example, the surgical hub6,56determines17306aif the device is located inside the body of the patient. A surgical device may comprise proximity markers that allow a tracking system to determine the location of various ends of the device in relation to the patient and other devices. The proximity to the patient can indicate the active use, or imminent use of a device. The surgical hub6,56may be configured to determine the proximal relationship between devices and system to evaluate the likelihood of interfering systems. The surgical hub6,56determines17308the location or proximity of the first devices to the potential interference causing device. For example, the surgical hub6,56determines17308aif the device is located within a predetermined distance from potential interference causing device. Further, the surgical hub6,56determines17308baffected proximity based on interference/noise type generated by the potential interference causing device. The surgical hub6,56can further evaluate a predetermine range that the inference can propagate. The surgical hub evaluates, based on the situation awareness system, the contextual parameters that indicate whether a communication or activation schedule is flexible. The surgical hub6,56determines17310whether the first device is in active usage/currently performing function. For example, the surgical hub6,56determines17310aif the device is in active operation mode. Further, the surgical hub6,56determines17310bif change in location, position, user inputs, power level, indicate device is in active operation or will be in inactive operation.

Prior to the activation of a first surgical device, the first surgical device sends a communication to the surgical hub that a potential noise/interference inducing event is about to occur. In response to the communication, the surgical hub changes one or more settings to mitigate the anticipated interference. In one example, bi-polar and mono-polar ablation systems are used during the same surgical procedure. The activation of the mono-polar system will interfere with the impedance control of the bipolar system. If the bi-polar is already in-cycle and the mono-polar is activated, the surgical hub changes the bi-polar control loop to continue operating in the manner just prior to the activation, rather than prohibiting the combined use of both devices. When the interface stops, the bi-polar goes back to its normal closed loop use.

In another aspect, the surgical hub is configured to takes a snap shot of setting or present operations for all the surgical devices. The snap shot of settings is used as a precaution in case one or more of the surgical devices is required to re-setup and reconnect to the surgical hub. Additionally, the snapshot may be used to reestablish a network communication setup (e.g. SSID, network domain, wireless communication authorization) and potential sub-device settings (by retransmitting the data back to the devices).

FIG. 15shows a graphical representation17400of an end effector signal and noise as the end effector clamps onto tissue and is in the process of firing. A mono-polar or microwave ablation system is activated and interferes with the internal function of the closed loop firing control of the end effector. Signal17406represents the interference signal. The microwave ablation system creates an RF overload or saturation event and can interfere with the control signal of the end effector. The mono-polar ablation system produces RF energy that could propagate noise up the metallic shaft into the contacts and cause a similar RF saturation event. Signal17402shows the desired result of the end effector and signal17404shows a failure in the control loop sensors and causes the end effector to stop. Once the interference event is finished, the surgical hub may repopulate the error term and rate cells of the PWM and enable the end effector to resume where it left off and finish the operation.

In various aspect, the surgical hub may determine that the repopulation of settings and error terms is prohibited. In one example, an ARM processor itself may be affected interference, not just the sub-function of a motor control, thus the surgical hub would prohibit repopulation. In another examples, the surgical hub determines that a predetermined amount of time has elapsed since the start of the interface, then the system could be reset rather than restarted. In other cases, the surgical hub may be required to initiate additional steps as part of the repopulation process. In the case of pulmonary artery or pulmonary vein transections, the system may require that the tissue is unclamped, checked, augmented, or reviewed prior to allows the end effector to complete the transection.

In various aspect, the surgical hub notifies sub-devices that could be affected directly by an anticipated interference event. The warning allows the sub-devices to prepare internal defenses for the anticipated interference. The internal defenses may include enabling noise filters that allow the device to continue to operate through periods of signal interference. The signal noise filters may be internal signal processing and built into the device.

In various aspect, the surgical hub comprises a noise receiver that monitors external noise and then communicates a warning to one or more surgical devices. External noise (e.g. audible, impedance, frequency, or other electrical signals) is sensed and generator/device signal (i.e., volume output, frequency output) may be adjusted according to a specific interference to compensate.

In various aspect, the surgical hub prohibits the combined activation or utilization of systems that have been identified to conflict or potentially interfere with one another. The surgical hub generates a warning and then inhibits any devices that could malfunction while being used while the noise event is in effect, according to a hierarchy of interference (FIG. 14). Additionally hierarchy factors include the prioritization of communication based on critical life support procedures, devices, and to maintain patient hemostasis (FIG. 14). The surgical hub prioritizes cooperative interaction of systems, such that the scheduling of communication and device activation is imperceptible to the surgical staff. Accordingly, critical communications and critical life support procedures may always be prioritized.

FIG. 16shows a flow diagram17500of surgical hub responses based on the anticipation or detection of a trigger event. The surgical hub notifies17502one or more devices of a detected or anticipated interference. The surgical hub takes a snapshot17504of all device settings, and determines whether the device(s) are may be reset and repopulated without issue. If the devices permit, the surgical hub initiates17508a repopulation event. Additionally or alternatively, the surgical hub determines17510the type of interference, and notifies17512a sub-device system of the anticipated interference type. The sub-device responds by initiating17514a defense to the interference (e.g. channel hopping). The surgical hub may evaluate and determine17516the communication and activation of systems according to a hierarchy of critical systems. The surgical hub determines17518whether the systems may interact in cooperation for concurrent operation or the systems need to be prioritized for sequential operation. In response to the interference type and schedule of systems, the surgical hub prevents17520the operation of one or more systems.

In various aspects, the surgical hub determines a trigger event based on a change in force or motion exerted on tissue during a retraction. In response the surgical hub generates a preemptive warning that displays tissue tension or similar metrics that provide real-time measurements of the retraction, as a virtual element, on an AV display. Additionally, the surgical hub may display measurements of displacement and force in relation to the patient or the ground. This information is useful for the surgeon to balance the intended forces of gravity, tissue tension, and grip force without creating undue collateral damage, or forces that were not intended for adjacent connections.

In various aspects, the surgical hub may continuously monitor a plurality of surgical parameters including force, direction, and magnitude created by a surgical instrument on tissue. Additionally, the surgical hub may determine that a force meets or exceeds a predetermined threshold at a specific location. In response, the surgical hub may initiate the display of a virtual element.

The surgical hub is configured to continuously monitor the force exerted on one or more organs as part of a surgical procedure, wherein the organs are retracted to aid the surgeon's vision in a laparoscopic procedure. If an organ fails to remain retracted, the surgeon's vision may be impeded or damage may result to the organ. The surgical hub may employ a predictive algorithm to evaluate changes in position, movement, and force of the organs. The predictive algorithm may identify a likely increase in tissue tension that exceeds a predetermined amount, at a specific location, and automatically displays a warning to the surgeon.

A warning may be provided in the form of an alert, notification, audio or visual cues, or message on a display. Tissue tension created by a surgical instrument may be display in real-time on an OR display. The system is configured to measure and monitor the force, tension and/or change in force or tension as related to the patient and/or the ground. The system may detection the initiation of force, the time that the force was initiated, and compare to an expected amount of tension or force. Real-time display is configured to provide warnings when tissue tension exceeds an intended level to prevent unintended tissue damage and undue collateral consequences.

With reference now toFIGS. 17 and 18, the disclosure turns to description of a method17260for managing surgical device21interaction during a surgical procedure. The method17260shown inFIG. 18may be practiced by the system17250shown inFIG. 17. According to the method17260, a surgical hub56determines17262contextual parameters from a surgical environment. As shown inFIG. 17, and in more detail inFIG. 6, the surgical hub56comprises a computer60including a control circuit31(e.g., a processor) coupled to a memory45, among other components. The surgical hub56determines17264a specific surgical procedure being performed in an operating room based on the contextual parameters. The surgical hub56receives17266procedural data from a remote server63. The procedural data indicates steps and surgical instruments21associated with the specific surgical procedure. The surgical hub56determines17268that a trigger event is anticipated based on the procedural data and the contextual parameters. The surgical hub56initiates17270a response according to the surgical instruments21and the procedural data.

In other aspects of the method17260, the surgical hub56may further determine a current procedural step of the surgical procedure. The current procedural step activates a first surgical device21and the surgical hub56determines that the first surgical device21is anticipated to interfere a communication of a second surgical device21. Accordingly, the surgical hub56notifies the second surgical device21of the anticipated interference and enables an interference defense on the second surgical device21. In one aspect, the interference defense enabled by the surgical hub56is a frequency shifting protocol configured to shift from a communication signal outside of an anticipated interference frequency band.

In other aspects of the method17260, the surgical hub56may further determine a current procedural step of the surgical procedure. The current procedural step activates a first surgical device21. The surgical hub56determines that the first surgical device21is anticipated to interfere a communication of a second surgical device21. The surgical hub56determines that the first surgical device21and the second surgical device21are operable in a cooperative schedule. The surgical hub56may then activate the first surgical device21and in response to completing a first surgical device activation, the surgical hub may initiate a communication of the second surgical device21.

In other aspects of the method17260, the trigger event is associated with a change in force exhibited on tissue while the tissue is being retracted. The change in force is determined by the surgical hub56based on readings from the surgical device21and is based on exceeding a predetermine tissue tension. Further, according to the method17260, the surgical hub56generates a virtual element to display on an augmented reality (AR) device66. The virtual element provides monitoring information for the retracted tissue.

Various additional aspects of the subject matter described herein are set out in the following numbered examples:

Example 1: A surgical system comprises: a remote server; augmented reality (AR) output device; one or more surgical devices; a surgical hub communicatively coupled to the remote server, the one or more surgical devices, and the AR output device, the surgical hub comprises a control circuit coupled to a memory, and wherein the control circuit is configured to: determine contextual parameters from a surgical environment; determine from the contextual parameter a specific surgical procedure being performed in an operating room; receive procedural data from the remote server, wherein the procedural data indicates steps and surgical instruments associated with the specific surgical procedure; determine that a trigger event is anticipated based on the procedural data and the contextual parameters; initiate a response to the anticipated trigger event according to the surgical instruments and procedural data.

Example 2: The system of Example 1, wherein the control circuit is further configured to: determine a current procedural step of the surgical procedure, wherein the current procedural step activates a first surgical device; determine that that the first surgical device is anticipated to interfere in a communication of a second surgical device; notify the second surgical device of the anticipated interference; enable an interference defense on the second surgical device.

Example 3: The system of Example 2, wherein the interference defense is a frequency shifting protocol configured to shift from a communication signal outside of an anticipated interference frequency band.

Example 4: The system of any one of Examples 1-3, wherein the control circuit is further configured to: determine a current procedural step of the surgical procedure, wherein the current procedural step activates a first surgical device; determine that that the first surgical device is anticipated to interfere the communication of a second surgical device; determine that the first surgical device and the second surgical device are operable in a cooperative schedule.

Example 5: The system of Example 4, wherein the control circuit is further configured to: active the first surgical device; and in response to a completed first surgical device activation, initiate the communication of the second surgical device.

Example 6: The system of anyone of Examples 1-5, wherein the control circuit is further configured to: evaluate a plurality of surgical devices associated with the surgical procedure according to a hierarchy of priority; determine a cooperative schedule based on the hierarchy of priority.

Example 7: The system of any one of Examples 1-6, wherein the control circuit is further configured to: determine that a first surgical device is associated with a critical health function; prohibit activation of all other devices during a communication transmission from the first surgical device.

Example 8: The system of any one of Examples 1-7, wherein the trigger event is associated with a change in force exhibited on tissue, and wherein the tissue is being retracted.

Example 9: The system of any one of Examples 1-8, wherein the response generates a virtual element to display on the AR device.

Example 10: The system of Example 9, wherein a change in force is determined based on exceeding a predetermine tissue tension.

Example 11: The system of Example 9, wherein the surgical hub generates a virtual element to display on the AR device, and wherein the virtual element provides monitoring information for retracted tissue.

Example 12: The system of Example 11, wherein the virtual element is anchored to the retracted tissue.

Example 13: A method for managing surgical device interaction during a surgical procedure, the method comprising: determining, by a surgical hub, contextual parameters from a surgical environment; determining, by the surgical hub, a specific surgical procedure being performed in an operating room based on the contextual parameters; receiving, by the surgical hub, procedural data from a remote server, wherein the procedural data indicates steps and surgical instruments associated with the specific surgical procedure; determining, by the surgical hub, that a trigger event is anticipated based on the procedural data and the contextual parameters; and initiating, by the surgical hub, a response according to the surgical instruments and the procedural data.

Example 14: The method of Example 13, further comprising: determining, by the surgical hub, a current procedural step of the surgical procedure, wherein the current procedural step activates a first surgical device; determining, by the surgical hub, that that the first surgical device is anticipated to interfere a communication of a second surgical device; notifying, by the surgical hub, the second surgical device of the anticipated interference; enabling, by the surgical hub, an interference defense on the second surgical device.

Example 15: The method of Example 14, wherein the interference defense is a frequency shifting protocol configured to shift from a communication signal outside of an anticipated interference frequency band.

Example 16: The method of any one of Examples 13-15, further comprising: determining, by the surgical hub, a current procedural step of the surgical procedure, wherein the current procedural step activates a first surgical device; determining, by the surgical hub, that that the first surgical device is anticipated to interfere a communication of a second surgical device; determining, by the surgical hub, that the first surgical device and the second surgical device are operable in a cooperative schedule.

Example 17: The method of Example 16, further comprising: activing, by the surgical hub, the first surgical device; and in response to completing first surgical device activation, initiating, by the surgical hub, a communication of the second surgical device.

Example 18: The method of any one of Examples 13-17, wherein the trigger event is associated with a change in force exhibited on tissue, and wherein the tissue is being retracted.

Example 19: The method of Example 18, wherein the change in force is determined based on exceeding a predetermine tissue tension.

Example 20: The method of Example 18, wherein the surgical hub generates a virtual element to display on an augmented reality (AR) device, and wherein the virtual element provides monitoring information for retracted tissue.

As used in any aspect herein, the terms “component,” “system,” “module” and the like can refer to a control circuit computer-related entity, either hardware, a combination of hardware and software, software, or software in execution.

A network may include a packet switched network. The communication devices may be capable of communicating with each other using a selected packet switched network communications protocol. One example communications protocol may include an Ethernet communications protocol which may be capable permitting communication using a Transmission Control Protocol/Internet Protocol (TCP/IP). The Ethernet protocol may comply or be compatible with the Ethernet standard published by the Institute of Electrical and Electronics Engineers (IEEE) titled “IEEE 802.3 Standard”, published in December, 2008 and/or later versions of this standard. Alternatively or additionally, the communication devices may be capable of communicating with each other using an X.25 communications protocol. The X.25 communications protocol may comply or be compatible with a standard promulgated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may be capable of communicating with each other using a frame relay communications protocol. The frame relay communications protocol may comply or be compatible with a standard promulgated by Consultative Committee for International Telegraph and Telephone (CCITT) and/or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using an Asynchronous Transfer Mode (ATM) communications protocol. The ATM communications protocol may comply or be compatible with an ATM standard published by the ATM Forum titled “ATM-MPLS Network Interworking 2.0” published August 2001, and/or later versions of this standard. Of course, different and/or after-developed connection-oriented network communication protocols are equally contemplated herein.