Surgical tracking and procedural map analysis tool

In some embodiments, methods and systems are provided for accessing a surgical dataset including surgical data collected during performance of a surgical procedure. The surgical data can include video data of the surgical procedure. Using the surgical data, a plurality of procedural states associated with the surgical procedure can be determined. For a procedural state of the plurality of procedural states, temporal information can be identified that identifies a part of the video data to be associated with the procedural state. For the procedural state of the plurality of procedural states, electronic data can be generated that characterizes the part of the video data and outputting the electronic data associated with the plurality of procedural states.

FIELD

Methods and systems disclosed herein relate generally to providing procedural guidance data during a surgical procedure. More specifically, procedural guidance data associated with a surgical procedure is represented by a pre-coded surgical procedural map that includes precise and accurate coded descriptions of patient states and surgical actions. Real-time data is processed to identify a particular position within the map, and corresponding procedural guidance data is identified and availed.

BACKGROUND

Variations in procedural methods between different surgeons to accomplish a particular surgical procedure can influence procedural outcomes. Procedural consistency of individual surgeons, in addition to variations in procedural methods, can also influence procedural outcomes. Presently, a number of surgeons achieve procedural outcomes that are below a minimum acceptable level. Further, there are surgeons that achieve outcomes above the minimum but below an average procedural outcome. A tool is needed to help standardize procedural methods between different surgeons and assist individual surgeons with improving procedural consistency.

SUMMARY

In some embodiments, methods and systems are provided for accessing a surgical dataset including surgical data collected during performance of a surgical procedure. The surgical data can include video data of the surgical procedure. Using the surgical data, a plurality of procedural states associated with the surgical procedure can be determined. For a procedural state of the plurality of procedural states, temporal information can be identified that identifies a part of the video data to be associated with the procedural state. For the procedural state of the plurality of procedural states, electronic data can be generated that characterizes the part of the video data and outputting the electronic data associated with the plurality of procedural states.

DETAILED DESCRIPTION

The structures and technologies described herein may function independently. In some embodiments, the structures and technologies described herein may function in combination to process live data to identify a real-time stage of a surgery, represented in a surgical data structure, which can be used to retrieve or generate pertinent information (e.g., corresponding to a current or subsequent action) for an entire operating room. The structures and technologies described herein may be integrated with existing operating room systems. Further, the structures and technologies described herein may include intelligent system that collects and analyzes operating room data and takes actions that maximize the chances of a desired procedural outcome. Fundamentally, the structures and technologies described herein use a surgical data structure as a reference during live surgical procedures such that appropriate guidance may be provided to the surgeon at the right moment of surgery. Structures and technologies described herein may describe an appropriate surgical route using the surgical data structure and a number of inputs including one or more user inputs, operating room video or audio streams, smart surgical tools, accelerometer data from a surgeon or other operating room personnel, and patient data. Current textual descriptions of surgery are often verbose, yet semantically constrained by the ambiguity of language they employ. Like GPS navigation relieved users of determining their position using a map or atlas, the structures and technologies described herein relieve surgeons and operating room personnel of tracking patient or procedural states and determining surgery progress from a reference. The structures and technologies described herein determine the procedural location and provide context to operating room personnel for the procedural location and provisions appropriate procedural guidance.

In some embodiments, systems and methods used to process live data to identify a real-time stage of a surgery, represented in a surgical data structure, which can be used to retrieve or generate pertinent information (e.g., corresponding to a current or subsequent action). A surgical data structure, that represents various stage in a surgery and corresponding information, can be locally retrieved or received. During a surgery, live data can be collected (e.g., from a sensor at a wearable device, smart surgical instrument, etc.). The live surgical data can be processed in accordance with the surgical data structure associated with the surgery to identify a specific procedural state (or stage). Procedural metadata associated with the state (e.g., which can include guidance information associated with an anticipated and/or recommended next procedural state) can be retrieved (e.g., from the surgical data structure or from a location identified by the surgical data structure.) and presented.

Accessing the surgical data structure can include (for example) retrieving the data structure from a remote or local data store and/or receiving the data structure from another device (e.g., a cloud server) in response to a request. The surgical data structure may include a plurality of nodes and a plurality of edges. Each edge of the plurality of edges may be configured to connect two nodes of the plurality of nodes. The nodes and edges may be associated with and/or arranged in an order that corresponds to a sequence in which various actions may be performed and/or states may be reached in a surgical procedure. Each of one more or all nodes in the surgical data structure and/or each of one, more or all edges in the surgical data structure may be associated with procedural metadata. The metadata can include description of and/or information corresponding an associated procedural state and/or surgical action. The surgical data structure may include a four dimensional surgical map, three dimensions corresponding to anatomy and the fourth dimension identifying a stage within a procedure.

Live surgical data may be collected by any device integrated with the surgical navigation system. In some embodiments, live surgical data can be collected at a wearable device worn by one or more surgical team members. Live surgical data can also be collected during a surgical procedure via (for example) captured images or video feeds, audio feeds, smart surgical tools, accelerometers on a wearable device on a surgeon, and/or team member inputs. The live surgical data may be encrypted, as necessary, in order to protect the privacy of individually identifiable health information in compliance with the Health Insurance Portability and Accountability Act of 1996.

Third, a processing device may receive the live surgical data. The processing device can be located in or near a surgery location or in the cloud. The processing device can be configured to identify the procedural state and procedural metadata associated with the procedural state. The live surgical data can be processed (e.g., using feature extraction) to identify a state of the patient/surgery. The state of the patient/surgery may be grounded in a physiological state (i.e., the condition or state or the patient's body or bodily functions). In some embodiments, a discrete procedural state (i.e., procedural location and/or procedural action) can be identified based on the state of the patient/surgery and procedural metadata corresponding to a node in the surgical data structure.

Metadata corresponding to the procedural state can be identified. The metadata may be (for example) included in the surgical data structure itself and/or identified by a part of the surgical data structure. For example, a node may include data that identifies surgical tools for an upcoming surgical action or it may point to a location in a table, array, data structure, data store, and so on that may include the particular information. The metadata can be used to generate real-time guidance information for output (e.g., identifying a surgical action to be taken, including an instruction to take a picture for documentation, identifying a tool to be used or prepped, presenting a vital sign for monitoring, paging for assistance, contacting a control center near an operating room with an assistance request, etc.). Guidance information can correspond to decision-tree type of structure. The guidance information may be configured to be presented as text data (e.g., to be presented on a display) and/or non-text data (e.g., audio signals, image overlays, artificial reality data). The type of output (e.g., information presentation versus control-center contact) may differ depending on (for example) a procedural state, patient state or action.

In some embodiments, live surgical data (or a processed version thereof, such as timestamped state identification) can be preserved for a post-hoc analysis. To improve the security corresponding to the live data, the data may be stored so as to anonymize the data and/or to obscure and/or encrypt particular (e.g., private or identifying) features. Live surgical data may be processed to identify current procedural states throughout the surgery. Edges corresponding to actions performed to transition between consecutive states can further be inferred, detected, or identified via an input or other data. Thus, for each surgery, trajectory data can identify a series of actions performed by a surgeon. This data may be aggregated, correlated with surgery outcomes and/or compared to guidance to, for example: (1) assess individual surgeons' compliance with guidelines or requirements; (2) generate audit logs; (3) update recommended actions/edges and/or guidance; and/or (4) identify overall preferences with regard to actions.

Further, live data that includes video content can be used to update a surgical data structure. For instance, a determination that particular frames of video of a surgery corresponds to step 4 of a carpal tunnel procedure can be used to annotate the video. The step identification can indicate what objects should be present in the video (e.g., a patient's hand, a surgeon's hand, and a scalpel blade with a handle masked by 1+ fingers). This knowledge can be used to automatically detect what a scalpel looks like with 1+ fingers covering the handle, which can be used to annotate the video.

Some techniques described herein can improve procedural consistency across entities involved in performance of a surgery (e.g., a surgeon, surgical team, hospital, etc.), improve procedural consistency for a given entity involved in performance of a surgery and/or improve outcomes of a surgery. Further, the real-time feedback can serve to supplement or partly replace training provided by a physically or remotely present procedural expert. Thus, the techniques can promote efficiency in this regard. Additionally, the feedback is provided at a critical time, such that mistakes can be avoided before any harm is cause to a patient.

Referring first toFIG. 1, an embodiment of a system100for processing live data to identify a real-time stage of a surgery, represented in a surgical data structure, which can be used to retrieve or generate pertinent information is illustrated. The system100can collect live data from a number of sources including (for example) a surgeon mounted headset110, a first additional headset120, a second additional headset122, surgical data150associated with a patient112, an operating room camera134, an operating room microphone136, and additional operating room tools not illustrated inFIG. 1. The live data is transmitted to a wireless hub160in communication with a local server170. The local server170receives the live data from the wireless hub160over a connection162and a surgical data structure from a remote server180. The local server170can process the live data to identify a state of the patient/surgery. The local server170can identify a node in the surgical data structure corresponding to the discrete procedural state. The local server170can process the metadata corresponding to the discrete procedural state and generate real time guidance information for output to the appropriate devices in the operating room102.

The local server170will be in contact with and synced with a remote server180. In some embodiments the remote server180can be located in the cloud106. In some embodiments processing performed by the local server170may be performed by the remote server180. A global bank of surgical procedures, described using surgical data structures, may be stored at the remote server180. Therefore, for any given surgical procedure, there is the option of running the system100as a local, or cloud based system. The local server170can create a surgical dataset that records data collected during the performance of a surgical procedure. The local server170can analyze the surgical dataset or forward the surgical dataset to the remote server180upon the completion of the procedure for inclusion in a global surgical dataset. In some embodiments, the local server can anonymize the surgical dataset. The proposed system100integrates data from the surgical data structure and sorts guidance data appropriately in the operating room using additional components.

In certain embodiments, surgical guidance, retrieved from the surgical data structure, may include more information than necessary to assist the surgeon with situational awareness. The system100may determine that the additional operating room information may be more pertinent to other members of the operating room and transmit the information to the appropriate team members. Therefore, in certain embodiments, the complete system100provides surgical guidance to more components than the surgeon mounted headset110.

In the illustrated embodiment, wearable devices such as a first additional headset120and a second additional headset122are included in the system100. Other members of the operating room team may benefit from receiving information and surgical guidance derived from the surgical data structure on the wearable devices. For example, a surgical nurse wearing the first additional headset120may benefit from guidance related to procedural steps and possible equipment needed for impending steps, seeFIG. 6B. An anesthetist wearing the second additional headset122may benefit from seeing the patient vital signs in the field of view. In addition, the anesthetist may be the most appropriate user to receive the real-time risk indication as one member of the operating room slightly removed from surgical action.

Distributing the real-time guidance information to team members can avoid work overload of an individual team member. For instance, a surgeon, wearing the surgeon's headset110, may be overloaded if presented with an equipment list to prepare for the next step in the surgical procedure. Therefore, to distribute workload, the system includes the provision of the first additional headset120and the second additional headset122to other members of the operating room team. The information presented to the nurse and the anesthetist illustrates how additional information included within the surgical data structure may guide other members of the operating room team besides the surgeon. The additional information and the integration of procedural guidance for operating room team members can benefit the execution of the medical procedure.

In addition to the first additional headset120and the second additional headset122there may be any number of peripheral information communicating devices including conventional displays130, transparent displays that may be held between the surgeon and patient, ambient lighting132, one or more operating room cameras134, one or more operating room microphones136, speakers140and procedural step notification screens placed outside the operating room to alert entrants of critical steps taking place. These peripheral components can function to provide information that may become critical during the procedure, yet the information displayed does not warrant constant distraction of the surgeon or another member of the operating room team.

Along with these peripheral communication devices, the system100can be integrated with any devices within the operating room providing live surgical data150concerning the state of the patient112, for example, heart rate and blood pressure monitors. In addition to vital signs, the live surgical data may include operating room characteristics such as lighting used, ambient temperatures, personnel identification/movement, and any other data that may indicate the progress of a surgical procedure. Sensors that can collect and provide live surgical data150may include (for example) electromagnetic sensors, (e.g. hall effect sensors to determine when surgical tools are lifted off an operating room table), optical sensors, in addition to video sensors, and near infrared sensors (to assist with mapping patient anatomy, tracking blood flow, etc.).

This fully integrated operating room system provides a structure through which to automatically coordinate operating room devices with procedural location. The wireless hub160may use one or more communications networks to communicate with operating room devices including various wireless protocols, such as IrDA, Bluetooth, Zigbee, Ultra-Wideband, and/or Wi-Fi. In some embodiments, existing operating room devices can be integrated with the system100. To illustrate, once a specific procedural location is reached automatic functions can be set to prepare or change the state of relevant and appropriate medical devices to assist with impending surgical steps. For example, operating room lighting132can be integrated into the system100and adjusted based on impending surgical steps.

In another embodiment, in addition to procedural guidance, warnings of hazards can be included within the system. The system100may use speaker140to warn the operating room of radioactive tracer excretion from the patient112and subsequent collection into a suction bag. A visual warning can be presented to operating room personnel using (for example) the surgeon headset110, the additional headsets, the conventional display130, and/or operating room lighting132.

In some embodiments, the system100may include a centralized hospital control center172. This center may be connected to one, more or all active surgeries and coordinate actions in critical situations as a level-headed, but skilled, bystander. In addition to having access to all live data from each operating room (including; video stream, real-time surgical risk, procedural step and other data), a user (e.g., healthcare professional) in the control center172may have access to data such as patient records and external personnel availability. In some embodiments, the data available to control center personnel could be packaged and provided as educational material to (for example) medical trainees, residents, and/or students. Control center personnel may have access to the surgical data structure using a surgical data structure explorer. The control center personnel may update the surgical data structure during surgery based on changing patient conditions.

In the event of a surgical complication, the control center manager may be able to directly page the most appropriate specialist on an integrated specialist pager174to mitigate the situation. In addition to identifying solutions to critical situations before complications can fully develop, the control center manager has access to information not easily integrated into the system100that may be requested by the operating room at times of use. The control center172has access to all the data within the operating room102and current procedural location, communication between the operating room102and control center172may take many forms including text communications and direct audio or visual links.

In some embodiments, a risk indication can be displayed within the system100to inform the progression and seriousness of surgery. To minimize distraction or stress, the system100may display the risk solely to a member of the operating room team removed from the present step of the surgery using (for example) the first additional headset120or the second additional headset122.

In certain embodiments, the indication of real-time surgical risk can be used to inform the operating room team of the risk or to trigger complication avoidance methods by the system. For example, when the risk reaches a pre-defined threshold or a level computed in real-time, the system may provide additional guidance directly to the surgeon using the surgeon headset110, an alert communication can be transmitted a local hospital control center172so the situation can be monitored more closely, and/or a relevant specialist pager174can be activated to contact a specialist necessary to resolve the situation. The risk level computed in real-time may be based on the surgical map and/or the probability of achieving optimal outcomes. In some embodiments, the system may dynamically flag an alert for procedural states, actions, or any other points in the surgical map.

In some embodiments, the system100calculates real-time surgical risk. The response of the system100based on this real-time risk can be flexible. The response may be adjusted by (for example) each hospital104or control center172. For example, the defined risk threshold upon which responses are actioned may be changed, or the individual parameter weighting for calculating risk may be manipulated. A user (e.g., surgeon, nurse or team member) may access the surgical data structure pre-surgery in the control center172using a surgical data structure explorer. The user may customize the risk thresholds or individual parameter weightings based on (for example) surgeon, team member, patient, and/or hospital specific characteristics.

The system100may adjust response actions based on other information, such as a surgeon's level of skill. The surgical data structure may include a variable corresponding to a surgeon's level of skill (e.g., corresponding to a particular occupational position, duration at a particular facility, duration practicing surgeries generally or of a particular type, complication probability based on past surgeries, and so on). The system100may determine the value for the variable corresponding to the surgeon's level of skill by analyzing pre-recorded surgical data associated with a surgeon or the variable may be entered by a user (e.g., surgeon, nurse or team member). For example, a trainee surgeon may have a lower threshold of risk before actions or insistent warnings are issued, a more experienced consultant may have to reach a higher level of risk before the same actions are issued. Using a tiered risk management system, the system100may reduce unnecessary risk by providing guidance to a user (e.g., surgeon, nurse or team member) and notifying specialists at the appropriate times. Reducing the unnecessary risk may decrease the occurrence of unfavorable patient outcomes.

A system100is not limited to the embodiments illustrated inFIG. 1. Some embodiments of a system100may remain the same with significant change in component organization. The embodiments herein relate to the provision of an intelligent system, which integrates with the whole operating room team, operating room data and procedural guidance; the embodiments herein do not relate to a specific medical device network structure.

Referring next toFIG. 2, one possible embodiment of a surgical data structure200is shown. In some implementations, the system for processing live data to identify a real-time stage of a surgery, which can be used to retrieve or generate pertinent information, is grounded on a surgical data structure200. The surgical data structure200allows concise descriptions of complex medical procedures. A discrete route through a surgical data structure200embodies a codified description of a specific surgical procedure. The surgical data structure200may include the information needed for the reproduction of an individual procedure and additional information to describe both subtly different and alternative procedural methods between individual users (e.g., surgeon, nurse or team member), yet, no more information than necessary to achieve both objectives.

In some embodiments, the precise route taken through a surgery is characterized by a series of transitions between discrete patient states. The surgical data structure200may describe medical procedures in full using graph theory or network structures. Within this graphical notation, a node210can represent a discrete physiological and procedural state of the patient during a procedure (e.g., initial preparation and sterilization complete, skin incision made, bone exposed, etc.). The node210may represent a pose, anatomical state, sterility, etc. The node210can be grounded in a 3D anatomical state. A 3D anatomical state characterizes a state of a patient, such as patient position, location of an organ, location of a surgical instrument, location of an incision, etc.

The node210and the associated discrete procedural state can be associated with a set of procedural metadata212that can include (for example) a description of and/or information corresponding to a procedural state. Procedural metadata212associated with a node210can include, for example, a description of a surgical action to be taken, an instruction to take a picture for documentation, a description of a surgical tool to be used or prepped, a description of a vital sign for monitoring; and/or a recommendation to page a control center for assistance. In some embodiments, the procedural metadata212can include a decision tree in machine readable format for use by a processing device or in natural language form for use by one or more system users.

An edge240can represent one or more surgical actions executed to transition between different nodes. In other embodiments, as shown inFIG. 3and described further below, edges may represent distinct states of the patient during a procedure and nodes may represent surgical actions taken to transition between different edges. Each edge may represent a medical action to be performed that will result in a transition between states. Each edge can be associated with information about the surgical action, such as an identification of the action, a time typically associated with the action, tools used for the action, a position of interactions relative to anatomy, etc.

In some implementations, for each individual procedure, the surgical data structure200is a multidimensional space representing the many routes that a single procedure may take. A single route through this multidimensional map may represent one method of executing the surgical procedure. The surgical data structure200may include a plurality of routes through the multidimensional map, including a preferred route230, indicated by a dashed line. The preferred route230may be linked to a number of factors including (for example) user (e.g., surgeon, nurse or team member) experience, operating room equipment, operating room personnel, minimizing risk, and/or surgical outcomes. The surgical data structure200allows the precise definition of important surgical landmarks using a constrained coded vocabulary.

Each node210may be associated with a set of procedural metadata212that can include a description of and/or information corresponding to the discrete procedural state. The description of and/or information corresponding to the discrete procedural state includes many different attributes that describe the condition of a patient at a given point during a medical procedure. The attributes describing patient state may include, but are certainly not limited to pose, anatomical state, and sterility. Any number of attributes may be needed in order to ensure a node is accurately described, and this may vary for different surgical procedures. The attributes associated with a discrete procedural state may also include variables not related directly to the patient but to the surgical procedure. For example, the personnel present in the operating room may relate to the state of the surgery but not to any particular patient state. In some embodiments, the fewest number of attributes is desirable. The description of and/or information corresponding to a discrete procedural state may describe a procedural state with sufficient detail to be used by a processing device in a feature extraction algorithm.

In some embodiments, each of one, more or all nodes is grounded in a 3D anatomical state (e.g., characterizing a state of a patient, such as a position, location of an organ, location of a surgical instrument, location of an incision, and so on). As such, the node210may include a detailed description of anatomical state. Surgical interventions aim to modify anatomy and their progress is constrained by patient topology. Thus, anatomical state and organization may present important checkpoints to split a complex procedure into discrete steps. These steps must be passed, regardless of method or prior event sequence. For example, to perform a total knee replacement, the skin must be cut and the bones at the knee joint exposed in order for the replacement knee to be applied.

In certain embodiments, using a surgical data structure200based on a graph model of surgery, edges240describe the transition between different nodes—each node representing a different patient state. As such, edges240represent actions taken by the users (e.g., surgeon, nurse or team member) during the procedure. Each edge may be associated with action specific metadata214, including, but not limited to: time taken, tools used, state of the tools used, and position of interactions relative to anatomy. In other embodiments, nodes may represent actions taken by the users (e.g., surgeon, nurse or team member) during the procedure and edges may provide a description of the patient state.

Node-to-node transitions in surgery may take multiple different routes between the same two destinations depending on factors entirely distinct from the two nodes. In some implementations, node-aware systems and methods facilitate flexible routes between a single destination node. Furthermore, different node-to-node transitions (i.e., surgical actions), or entirely different routes through the surgical data structure200, may be recommended for different age patients in the same procedure.

In some embodiments, for the provision of the structures and technologies described herein, the surgical data structure200must be translated from a generic coded anatomical model to real anatomy. For example, the surgical data structure200may need to be anatomically adjusted for success in specific procedures. Some embodiments may adjust the surgical data structure200for obese patients, patients with atypical anatomy or patients who may have undergone significant surgical procedures in the past.

The surgical data structure200may include anatomical models that are both malleable and include topologies that can be expected to remain consistent between many different patients. In some embodiments, the surgical data structure200may account for the anatomical distinction between the sexes, which may impact the description of certain surgeries. In certain embodiments, a selection of patient specific anatomic models may need to be implemented, or existing models adjusted in certain situations.

To further illustrate the use of a surgical data structure200, an embodiment describing a subset of steps performed during the surgical procedure for carpal tunnel release is provided herein. Carpal tunnel release surgery is a simple procedure to relieve the symptoms of carpal tunnel syndrome. The carpel tunnel is a narrow passage within the wrist capped by the carpal ligament. The syndrome is caused when the median nerve becomes compressed, often by swelling, within the carpal tunnel. The crucial step of carpal tunnel release surgery is to cut this ligament to release the median nerve and alleviate the symptoms of carpal tunnel syndrome. Above the ligament is a layer of tissue and a layer of skin respectively. Both must be divided in order to reach the carpal ligament.

FIG. 2details a simplified representation of a surgical data structure200for a carpal tunnel release procedure. To accomplish the carpal tunnel release procedure, at least four crucial procedural nodes210,216,218, and220may be defined by the anatomic states of four relevant pieces of anatomy: the skin, wrist tissue, carpal ligament and median nerve. Representative procedural metadata212is provided for each of the four crucial procedural nodes210,216,218, and220. During the carpal tunnel release procedure the skin, tissue and ligament may be intact or divided and the nerve trapped or released. The procedural metadata212may include additional data not illustrated inFIG. 2. The additional data may include attributes used to define more nodes along the route of surgery (such as the sterility of the surgical site), but these four anatomical states represent nodes through which the procedure must pass to progress.

The four key procedural nodes210,216,218, and220are connected by edges, which represent surgical interactions related to the associated node-to-node transition. For example, edge240may represent the action necessary to transition from the first key procedural node210to the second key procedural node216. The procedural metadata associated with the second key procedural node216indicates that the skin is incised. In order to facilitate the transition between procedural states, edge240represents the procedural action associated with the node-to-node transition. In this example, the action specific metadata214associated with edge240includes a description of the action, “incise the skin,” and the tools required, “scalpel.” The procedural action associated with edge240causes the associated physiological state of the skin to transition from intact to incised between node210and node216respectively.

The four key procedural nodes210,216,218, and220are illustrated on the preferred route230. The preferred route230passes through an intermediate node232to the third key procedural state218. The set of procedural metadata associated with the third procedural state218indicates that the tissue above the ligament has been divided. The final key procedural node220is associated with procedural metadata that indicates that the nerve has been released and the surgical preferred route230has reached a successful surgical outcome. The preferred route includes a final edge228that includes an instruction to “suture the skin” and a set of tools required to complete the procedural action.

In some embodiments, the surgical data structure200can account for surgical procedures that do not reach a procedural state associated with a successful surgical outcome.FIG. 2shows unsuccessful node224and its associated unsuccessful metadata226indicating that the nerve is still trapped after traversing the nodes in the surgical data structure200.

For simplicity, the complete list of possible attributes associated with the metadata for each node and edge has not been included in this stylized representation. A longer list of possible attributes may include, (for example) surface sterility, the interaction with specific instruments or physical characteristics such as the pressure or tension on specific anatomical structures, and/or patient vital signs.

To simplify the surgical data structure200of this surgery further, the route may be represented as distinct nodes without any metadata244. The four key procedural nodes210,216,218, and220and relevant edges shown inFIG. 2are only a small subsection (individual surgical route) of a large, multidimensional surgical data structure including many possible nodes and edges. A surgical data structure may represent all projected surgical paths possible within the procedure.

In some embodiments, nodes and edges may be defined as a list of contingent events. For example, the retraction of the skin is contingent on its initial incision. By defining each set of data (i.e., a node or an edge) in contingent terms, the network is able to self-construct, by forming all possible actions given the extended rules of contingency. In certain embodiments, the structures and technologies described herein may capture live surgical data to build descriptions of procedures, coded in the structure defined above. In some examples, the the structures and technologies described herein may harvest nodes and edges directly from surgical procedures themselves and construct a surgical data structure automatically.

In other embodiments, different successful routes may interconnect.FIG. 2, illustrates that more than one route in the surgical data structure200is possible. In certain embodiments, the preferred route230refers to a set of successful procedural routes which constitute a very small subsection of a much larger, multidimensional surgical data structure, where each node and edge are can include a description of and/or information corresponding to surgical states and actions respectively. In certain embodiments, there may be more than one successful procedural route. Graph theory concepts can then be applied to facilitate provision of real-time guidance (e.g., identifying one or more edges to cause a transition between two nodes defined for a surgery) or to facilitate post-hoc data analysis of one or more surgeries (e.g., to identify which paths are associated with particular consequences, an extent to which a predefined procedure was followed, etc.).

FIG. 3illustrates an embodiment of the surgical data structure300where surgical actions are associated with nodes310and procedural states are associated with edges330.FIG. 3uses the carpal tunnel procedure to illustrate an alternate embodiment of a surgical map.FIG. 3illustrates the four key procedural states associated with edges330,332,334, and336. Each key procedural state includes metadata to provide a description of and/or information corresponding to the procedural state.

In the embodiment illustrated byFIG. 3, the first node310is associated with a procedural action. The action metadata308includes instructions and tools necessary to perform the procedural action. The first node310includes instructions to position the patient. The first key procedural state is associated with a first edge330. The first edge330is associated with a set of procedural metadata312and includes a description of and/or information corresponding to the first key procedural state. The preferred route340, shown by the dashed line inFIG. 3, connects each of the four key procedural states. A second edge332is associated with a second set of procedural metadata314indicating physiological states associated with the second key procedural state. A third edge334is associated with a third set of procedural metadata316indicating physiological states associated with the third key procedural state. A fourth edge336is associated with a fourth set of procedural metadata318indicating physiological states associated with the fourth and final key procedural state. A plurality of nodes322are associated with procedural actions and action metadata connect the key procedural states associated with edges330,332,334, and336

FIG. 4illustrates a surgical data structure400that can recommend a preferred surgical route424based on weights assigned to one or more edges between the node-to-node transitions. In some embodiments, a weight may be assigned to each node in addition to, or in lieu of, the weight assigned to each edge. The weights may be associated with a plurality of factors including, (for example) surgical outcomes, risk, prevalence of use, current procedural state, patient characteristics, vital signs, procedure specific parameters, and/or user (e.g., surgeon, nurse or team member) experience. The risk may be based on maximum risk or a risk based on the probability of occurrence. The weights may be manually assigned when constructing the surgical data structure. In other embodiments, the surgical weights may be determined or updated using a machine learning algorithm based on collected surgical data, including surgical outcomes. In some embodiments, a user (e.g., surgeon, nurse or team member) may update the weights pre-surgery using a surgical data structure explorer.

The weight450assigned to an edge412may be based on a number of factors. In some embodiments, the weight450may be associated with the probability of a transition from a first node to a second node. InFIG. 4, the weight450is assigned to a first edge412and a second weight452is assigned to a second edge414. In this example, the weight450is lower than the second weight452and the lower weight may be associated with a preferred edge. The second weight452may be higher because in order to arrive at a preferred node420, a surgeon must perform two surgical actions and pass through an intermediate node416. In selecting the second edge414, the surgeon performs a surgical action associated with the second edge414to reach intermediate node416. Next, the surgeon must complete the surgical action associated with intermediate edge418to arrive at the preferred node420.

In some embodiments the systems and methods described herein can use the weights assigned to the edges of the surgical data structure to determine a preferred route424, illustrated as a dashed line inFIG. 4. The preferred route424may be determined by adding the weights of the associated edges to calculate a cost of traversing from an initial node410in the surgical data structure to a destination node440in the surgical data structure. In other embodiments, the preferred route may be recalculated using a current node and a target node that are a subset of nodes in the surgical data structure. The target node can be a node in the surgical data structure selected by (for example) a surgeon, an operating room team member, control room personnel, and/or one or more automated methods described herein.

The systems and methods described herein may calculate a cost for a route in the surgical data structure. Determining the cost may depend on whether weights are included in the surgical data structure. The cost may be calculated based on how values are assigned to the weights in the surgical data structure. In some embodiments, the lowest cost route may be the route selected. In other embodiments, the cost may be just one input into a processor that is configured to determine a route from an initial node to a destination node.

In other embodiments, the weights associated with the edges of a surgical data structure may be associated with an actual value and a maximum value. Weight454illustrates that edge426may be selected if actual value ‘a’ is less than the capacity ‘c’. In some embodiments, the capacity may represent patient vital signs. Edge428may be associated with a surgical action that may be performed if a set of vital signs are below a set of threshold values. Vital sign weight456can represent that edge428is available for selection if less than or equal to ‘2’ vital signs are below threshold values. For example, the ‘1’ in vital sign weight456can indicate that the patient's heart rate is above a threshold value but blood pressure and breathing rate are at acceptable levels. As a result, edge428is available for selection to transition from a present node422to a destination node440.

In some embodiments, the structures and technologies described herein include a global surgical graph402that may include a map to a plurality of pre-coded surgical data structures. During a surgical procedure using a fully developed pre-coded surgical data structure, there is a very low probability of a surgical complication that may cause a patient to transition to a procedural state that is not associated with a node. For example, the first pre-coded surgical data structure404may start at node410and represent the carpal tunnel procedure discussed above. Some rare carpal tunnel complications that may still need to be accounted for include heart attacks and pulmonary aspiration. During the carpal tunnel procedure, the patient may transition to a procedural state associated with a complication that is not represented by a node in the first pre-coded surgical data structure404.

To account for low probability complications, the systems and methods described herein may transition to another pre-coded surgical data structure in the global surgical graph402. For example, during the carpal tunnel procedure, the global surgical graph402can include a second pre-coded surgical data structure406associated with treating a patient experiencing a heart attack, and a third pre-coded surgical data structure408associated with treating a patient experiencing aspiration. The second pre-coded surgical data structure406begins at node460, which includes a description of and/or information corresponding to a procedural state and patient attributes associated with a heart attack. The third pre-coded surgical data structure408begins at node470, which includes a description of and/or information corresponding to a procedural state and patient attributes associated with pulmonary aspiration. The second pre-coded surgical data structure406and the third pre-coded surgical data structure408continue beyond line462to a node associated with a desired outcome for each procedure.

In some embodiments, the systems and methods described herein may automatically transition between the surgical data structures included in the global surgical graph402. In addition to the automatic transition, the systems and methods described herein may automatically transmit an alert to a specialist to assist with the associated complication. In alternate embodiments, the control center may monitor surgical data and determine that the current procedural state is not associated with a node in the present surgical data structure. The control center may select an appropriate surgical data structure to transition to based, for example, on the procedural state, surgical data, and/or other factors related to the patient state. The control center may page a specialist based on the severity of the complication.

In certain embodiments, the surgical data structure may be customized to direct surgery using the structures and technologies described herein. For example, a user (e.g., surgeon, nurse or team member) may access a surgical data structure prior to surgery and manually select any acceptable procedural route before surgical navigation is initiated. Furthermore, the whole procedure may be specifically planned, such that a tailored route (where a user chooses every node and edge) can be defined before the procedure. The structures and technologies described herein may provide a user (e.g., surgeon, nurse or team member) the capability to edit a surgical data structure using a software application such as a surgical data structure explorer. In alternate embodiments, the structures and technologies described herein may be configured to recommend the most appropriate surgical route, dependent on patient specific data (e.g., and/or live data). Further, the weights associated with the surgical data structure may be specifically set by the systems and methods described herein or by a user (e.g., surgeon, nurse or team member) before a specific surgical procedure.

In some embodiments, machine learning may be used to identify an action to recommend in a given circumstance (e.g., current state, patient characteristic, vital sign) and/or to a given operating room team member. Action recommendation may be based on assessing potential and/or expected risk associated with each of one or more actions. For example, a potential risk may be defined as a maximum risk impact or a weighted sum of risk impact (weighted based on probability of occurrence), and a selection may then be made to select an action with a lowest, low or below-threshold risk impact.

FIG. 5provides a schematic illustration of communication exchanges between the structures and technologies described herein. The structures and technologies described herein includes operating room interface devices510, patient data devices530, surgeon interface devices540, local server560, and cloud server580. The operating room interface devices510can include operating room cameras and microphones, non-surgeon operating room team member headsets, smart surgical tools, power usage, light source usage, suction devices, irrigation devices. The patient data devices530can include any patient monitoring equipment including vital signs, patient discharge measuring devices, and ventilation volume measurements. Surgeon interface devices540can include an interface device, such as a headset and additional surgeon monitoring devices. Additional surgeon monitoring devices may include galvanic skin response devices and accelerometers to determine surgeon stress levels. As illustrated, the local server560may generate and transmit a request for the current surgical data structure562from the cloud server580. The request may include version information including a version number, a date, or other indication of the local server version of the surgical data structure.

The cloud server580will perform a request evaluation502to determine if the local server560has the current version of the surgical data structure. In performing the request evaluation502, the cloud server will determine whether the local server560has the current surgical data structure. After evaluating the request, the cloud server580will generate and transmit a surgical data structure update584to the local server560. If the local server has the current version of the surgical data structure, the contents of the surgical data structure update584will be limited. If the surgical data structure is not current, the surgical data structure update584will include updates for the associated surgical data structure. In some embodiments, the request for the current surgical data structure562may be associated with a surgical data structure not present on the local server560. In this case, an entire surgical data structure may be included in the surgical data structure update584.

Upon receiving the surgical data structure update584the local server560may generate and transmit a start procedure message568to the surgeon interface devices, the patient data devices, and the operating room interface devices. The start procedure message568may include machine readable instructions that instruct the devices to begin collecting surgical data associated with the surgical procedure. Each device may generate and transmit surgical data to the local server.

Upon receipt of the start procedure message568, the surgeon interface devices540and the operating room interface devices510will present relevant procedural information to the surgeon and the operating room personnel respectively. Additionally, upon receipt of the start procedure message568, the surgeon interface devices540, the patient data devices530, and the operating room interface devices510will begin recording and transmitting data to the local server560. The data recorded and transmitted by the surgeon interface devices540, the patient data devices530, and the operating room interface devices510may be categorized as surgical data564.

Specific surgical data is generated by each of the devices in the structures and technologies described herein. The surgeon interface devices540generate audio and/or visual data (A/V data) associated with the surgeon's actions. The surgeon interface device540may be configured to receive inputs from a surgeon. Inputs may be auditory, visual, or haptic. In some embodiments, the surgeon's inputs may be in response to a stimulus. The surgeon A/V data542and the surgeon inputs544are transmitted to the local server560. The patient data devices530generate and transmit vital sign data532and patient state data534to the local server560. Data associated with operating room interface devices510includes anesthetist data512, operating room team member inputs516, operating room team member data514, operating room visual data520, and operating room surgical instrument data518. The data associated with operating room interface devices510is also transmitted to local hospital server560.

Next, the local server560performs feature extraction504and determines a current node506. Feature extraction504is performed by the local server560using the surgical data564generated and transmitted by the operating room interface devices510, the patient data devices530, and the surgeon interface devices540. The local server560may detect components related to the procedural metadata stored in the surgical data structure using feature extraction504. For example, components may include anatomical states, phonemes, and/or surgical tools. In some embodiments, the feature extraction504can include extracting visual or auditory components from the live surgical data.

Local server560determines a current node506using the surgical data structure and the components extracted from the surgical data564. In an alternate embodiment, the local server560forwards surgical data564bto the cloud server580. The cloud server may perform feature extraction504b, determine a current node506b, and transmit the current node596to local server560.

Based on the current node the local server560generates and transmits surgeon guidance566, and OR team member guidance564. The guidance may include information from the metadata associated with the current node and/or information related to a procedural action associated with an edge to transition from the current node to a target node. In some embodiments, the surgical data structure (or a version thereof) may be updated (e.g., at the cloud server580or at the local server560) during a surgical procedure or at another time. For example, the update may be based on the surgical data564breceived at the cloud server580. The cloud server580may generate and transmit a surgical data structure update588to the local server560during the surgical procedure. The Operating room interface devices510, patient monitoring devices530, and surgeon interface devices540continue generating and transmitting surgical data, and the local server560continues extracting features and determining current nodes until the local server560determines the surgical procedure is at a final node508. In some implementations, the local server560may transmit surgical data564bto the cloud server580, which determines the surgical procedure is at a final node508b.

Upon determining the surgical procedure is complete, the local server560may transmit a procedure complete message570to the operating room interface devices510, the patient data devices530, and the surgeon interface devices540. In response, the devices may present information to operating room team members and the surgeon that indicate the surgical procedure is complete. Further, the devices may stop transmitting surgical data to the local server upon receiving the procedure complete message570. In some implementations, the local server560may transmit a complete record of the surgical data572associated with the surgical procedure to the cloud server for analysis.

After receiving the complete record of surgical data572, the cloud server may analyze collected surgical data590. Analyzing the collected surgical data may include determining updates to the surgical data structure, determining user (e.g., surgeon, nurse or team member) compliance with procedure guidelines, and annotating the record of surgical data to indicate data associated with procedural states. Updates to the surgical data structure may include (for example) updating the metadata associated with current nodes and edges, generating new nodes, generating new edges to transition between current nodes and new nodes, updating the weights associated with edges, and/or deleting nodes/edges no longer present in a surgical procedure.

In some embodiments, analyzing the collected surgical data may include generating an annotated video file that includes video data and annotations associated with the nodes and edges present in the video data. In other embodiments, analyzing the collected surgical data may include generating compliance data that identifies entities involved in the surgical procedure and any departures from the preferred surgical guidelines. The compliance data may be stored in a compliance dataset. The compliance dataset can be transmitted to (for example) the surgeons that performed the procedure, medical schools, and/or medical review boards.

In some embodiments, analyzing the collected surgical data may include integrating the present set of surgical data with a global set of surgical data. The remote server can be configured to process the global set of surgical data and apply machine learning techniques to update the surgical data structure based on the global surgical data. For example, the global surgical dataset may contain distinct features and components that are not linked with a node in the associated surgical data structure. In response, the remote server may define a new discrete procedural state and update the associated surgical data structure with the appropriate nodes and edges.

FIGS. 6A, 6B, and 6Cillustrate embodiments of heads-up displays used to present surgical guidance to operating room personnel during a surgical procedure. The information presented may be included in metadata associated with a current node, a target node or an edge in the current surgical data structure.FIGS. 6A, 6B, and 6Cillustrate views of the operating room from a surgeon or operating room personnel-mounted headset. In addition to presenting information, the headset may be equipped to collect live surgical data. For example, data may be collected via a camera, speaker, haptic sensor, and/or receiver (e.g., receiving data from a smart surgical tool). Data may be collected continuously or periodically (e.g., at defined times, at defined time intervals or upon detecting an event triggering collection).

In some embodiments, a processing device, such as the local server, may determine and continuously update a trajectory through the surgical data structure. The trajectory may be based on (for example) live surgical data, weights associated with nodes and edges the surgical data structure, changes in patient/surgery states, and/or a pre-programmed trajectory entered by a user. A user may enter a pre-programmed trajectory using a software application, such as a surgical map. The local server may transmit the trajectory data to (for example) one of the heads up displays and/or the control center to increase situational awareness.

FIG. 6Aillustrates an augmented reality view600that can be used to present the steps associated with a surgical procedure from a surgeon's point-of-view (POV) heads-up display. The display can be semitransparent permitting the surgeon to see the patient690and surrounding operating room through the heads-up display. The graphics overlaid on the surgeon's POV include a primary display area610, a first supporting display area620, a second supporting display area630, a title area650, a procedural progress timeline652and indicator654, and an elapsed time indicator656. The display includes data retrieved from a surgical data structure. The local server can determine the relevant data based on live surgical data collected during a procedure.

The primary display area610may be used to present useful procedural information into the surgeon's field of view. Information in the primary display area may include (for example) identification of one or more actions, tools to be used, and/or expected results of an action. In some instances, data is presented in the primary display area610, the first supporting display area620, and/or the second supporting display area630only upon detecting a presentation-triggering event (e.g., that an orientation of a wearable device matches an orientation profile, that an acceleration of a wearable device matched an acceleration profile, or that a physical input of a predefined type was received).

InFIG. 6A, the primary display area includes a list of key steps associated with distal femoral preparation. The title612indicates that key steps are displayed. The steps are provided in a primary display list614with the current step highlighted616. The first supporting display620includes a list of phase objectives. First supporting display title622indicates that content provided in first supporting display620. The first supporting display list624is displayed to provide the surgeon with detailed phase objectives. The second supporting display630includes a video632of the current key step to be performed. The surgeon, operating room personnel, and/or the control center may remove any of the information from the field of view. The local server may automatically remove items from the surgeon's field of view based on surgical data received by the system.

FIG. 6Billustrates an augmented reality view600to present information associated with a surgical procedure to a non-surgeon operating room team member. The display can be semitransparent permitting the team member to see an operating room table692, operating room instruments626, and the surrounding operating room through the heads-up display. The graphics overlaid on the team member's POV include a primary display area610, a first supporting display area620, a second supporting display area630, a title area650, a procedural progress timeline652and indicator654, and an elapsed time indicator656.

The primary display area610may be used to present useful procedural information into the operating room team member's field of view. InFIG. 6B, the relevant operating room team member is a nurse. For a nurse, useful procedural information may include the required instruments for the current phase of a procedure. The title612on the nurse's primary display610reflects the difference in relevant information. the primary display list614lists the required instruments and the current instrument is highlighted616. The augmented reality view600also includes an instrument indicator618. The local server may use feature extraction from the associated headset's live video data to identify and highlight relevant instruments using an instrument indicator618. The instrument628associated with the instrument indicator618may correspond to the highlighted616instrument on the primary display610.

FIG. 6Cillustrates an augmented reality view600used to capture a snapshot during the surgical procedure. A snapshot is useful for post-surgery analysis of the surgical procedure. The snapshot view includes a title area650, a procedural progress timeline652and indicator654, an elapsed time indicator656, a set of framing elements680, and a visible timestamp682.

In addition to the visual data captured by the snapshot inFIG. 6C, the local server may record additional data associated with the snapshot. Additional sensors that can provide additional data associated with the snapshot can include at least one operating room video camera, a microphone, an infrared sensor and an accelerometer. The additional sensors can be used to capture real-time data from the operating room including video, pictures, audio annotation, manual procedural step logging and user (e.g., surgeon, nurse or team member) notes. For example, the user may use the head mounted camera to capture image stills of critical steps of surgery for documentation of their practice in the event of an unexpected complication.

In some embodiments the captured images, and any other data associated with additional sensors captured during surgery can be stored against the active surgical procedure at the precise procedural location at which it was measured. This data may be used as, for example, an audit record, or training resource. Further, these sensors may facilitate user-system interactions, which may include gestures, voice commands, other form of hands-free interaction or direct hardware interactions such as buttons, or pedals, which may be separate to the headset but connected to structures and technologies described herein to provide real-time surgical guidance.

The guidance presented to the surgeon and operating room personnel is not limited to the embodiments shown herein. Surgical guidance may take the form of text commands, images (separate and/or overlaid on real-time video), augmented reality (AR) or virtual reality (VR) surgical guides (such as paths of incisions), informative animations (both AR, VR and separate from real image of surgery), haptic feedback and audio or visual commands or stimulus (e.g., signals, alarms, music, and ambient light change).

With reference to now toFIG. 7, a block diagram of an illustrative surgical navigation local server700is shown. The local server700may correspond to the local server170described above. Local server700includes processing units704that communicate with a number of peripheral subsystems via a bus subsystem702. These peripheral subsystems include, for example, a storage subsystem710, an I/O subsystem726, and a communications subsystem732.

Bus subsystem702provides a mechanism for letting the various components and subsystems of local server700communicate with each other. Although bus subsystem702is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem702may be of any several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Such architectures may include, for example, an Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which may be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard.

Processing unit704, which may be implemented as one or more integrated circuits (e.g., a microprocessor or microcontroller), controls the operation of local server700. Processing unit704may be implemented as a special purpose processor, such an application-specific integrated circuit (ASIC), which may be customized for a particular use and not usable for general-purpose use. In some implementations, an ASIC may be used to increase the speed of feature extraction from live surgical data. In some embodiments, processing unit704may include one or more graphics processing units (GPUs). The GPUs may be configured to process live surgical data in order to perform feature extraction. One or more processors, including single core and/or multicore processors, may be included in processing unit704. As shown inFIG. 7, processing unit704may be implemented as one or more independent processing units706and/or708with single or multicore processors and processor caches included in each processing unit. In other embodiments, processing unit704may also be implemented as a quad-core processing unit or larger multi core designs (e.g., hexacore processors, octo-core processors, ten-core processors, or greater). In some embodiments, one or more independent processing units may be replaced with an ASIC optimized for feature extraction.

Processing unit704may execute a variety of software processes embodied in program code, and may maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may be resident in processor(s)704and/or in storage subsystem710. In some embodiments, local server700may include one or more specialized processors, such as digital signal processors (DSPs), outboard processors, graphics processors, application-specific processors, and/or similar specialized processors.

I/O subsystem726may include device controllers728for one or more user interface input devices and/or user interface output devices. I/O devices can include a user (e.g., surgeon, nurse or team member) I/O device730for use by a user pre, during or post-surgery. A surgical data structure explorer can be executed on the user I/O device730. The user I/O device730can provide a user the capability to adjust the preferred route, procedural metadata, assigned weights, and/or other data associated with a surgical data structure prior to surgery. Post-surgery a user may use the user I/O device730to review the analysis performed by a processing device. In some embodiments, the I/O devices can include control center I/O devices740.

User interface input and output devices may be integral with local server700(e.g., integrated audio/video systems, and/or touchscreen displays), or may be separate peripheral devices which are attachable/detachable from local server700. The I/O subsystem726may provide one or several outputs to a user by converting one or several electrical signals to user perceptible and/or interpretable form, and may receive one or several inputs from the user by generating one or several electrical signals based on one or several user-caused interactions with the I/O subsystem such as the depressing of a key or button, the moving of a mouse, the interaction with a touchscreen or trackpad, the interaction of a sound wave with a microphone, or similar user inputs.

Input devices may include a keyboard, pointing devices such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices with voice command recognition systems, microphones, and other types of input devices. Input devices730may also include three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio/visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, haptic devices, and eye gaze tracking devices. Additional input devices may include, for example, motion sensing and/or gesture recognition devices that enable users to control and interact with an input device through a natural user interface using gestures and spoken commands, eye gesture recognition devices that detect eye activity from users and transform the eye gestures as input into an input device, voice recognition sensing devices that enable users to interact with voice recognition systems through voice commands, medical imaging input devices, MIDI keyboards, digital musical instruments, electromagnetic sensors configured to detect movement of surgical tools, optical sensors, near infrared sensors, and similar devices.

Output devices may include one or more display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may include, for example, cathode ray tube (CRT) displays, flat-panel devices, such as those using a liquid crystal display (LCD) or plasma display, light-emitting diode (LED) displays, projection devices, touch screens, haptic devices, and the like. In general, use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from local server700to a user or other computer. For example, output devices may include, without limitation, a variety of display devices that visually convey text, graphics and audio/video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

Local server700may comprise one or more storage subsystems710, comprising hardware and software components used for storing data and program instructions, such as system memory718and computer-readable storage media716. The system memory718and/or computer readable storage media716may store program instructions that are loadable and executable on processing units704, as well as data generated during the execution of these programs. Program instructions may include instructions to perform one or more actions or part(s) or all of one or more methods or processes described herein. For example, program instructions may include instructions for identifying a discrete surgical procedural state. Program instructions may include instructions for generating, transmitting, and/or receiving communications. Program instructions may include instructions for automated processing. Program instructions may include instructions for generating automated processing and/or stage results. Program instructions may include instructions for performing a work flow iteration.

Depending on the configuration and type of local server700, system memory718may be stored in volatile memory (such as random access memory (RAM)712) and/or in non-volatile storage drives714(such as read-only memory (ROM), flash memory, etc.) The RAM712may include data and/or program modules that are immediately accessible to and/or presently being operated and executed by processing units704. In some implementations, system memory718may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input/output system (BIOS), including the basic routines that help to transfer information between elements within local server700, such as during start-up, may typically be stored in the non-volatile storage drives714. By way of example, and not limitation, system memory718may include surgical data structure storage720, recorded surgical data722, and application programs724such as feature extraction applications, procedural state identification applications, headset display applications, surgical data structure explorer applications, user applications, mid-tier applications, server applications, etc.

Storage subsystem710also may provide one or more tangible computer-readable storage media716for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that when executed by a processor provide the functionality described herein may be stored in storage subsystem710. These software modules or instructions may be executed by processing units704. Storage subsystem710may also provide a repository for storing data used in accordance with the embodiments illustrated herein.

Storage subsystem710may also include a computer-readable storage media reader that may further be connected to computer-readable storage media716. Together and, optionally, in combination with system memory718, computer-readable storage media716may comprehensively represent remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing, storing, transmitting, and retrieving computer-readable information.

Computer-readable storage media716containing surgical data structures, program code, or portions of program code, may include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to, volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage and/or transmission of information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer readable media. This may also include nontangible computer-readable media, such as data signals, data transmissions, or any other medium that may be used to transmit the desired information and that may be accessed by local server700.

Communications subsystem732may provide a communication interface from local server700and remote computing devices via one or more communication networks, including local area networks (LANs), wide area networks (WANs) (e.g., the Internet), and various wireless telecommunications networks. As illustrated inFIG. 7, the communications subsystem732may include, a remote server network interface controller (NIC)734. The NIC may be an Ethernet card, Asynchronous Transfer Mode NICs, Token Ring NICs or a similar interface. The communications subsystem732may include one or more wireless interfaces, including an operating room communications interface738. The operating room communications interface738may be implemented using a wireless network interface controller. Additionally and/or alternatively, the communications subsystem732may include one or more modems (telephone, satellite, cable, ISDN), synchronous or asynchronous digital subscriber line (DSL) units, FireWire interfaces, USB interfaces, and the like. Communications subsystem732also may include radio frequency (RF) transceiver components for accessing wireless voice and/or data networks (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution), Wi-Fi (IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and/or other components. A cellular enabled local server700may be used in austere conditions where users (e.g., surgeon, nurse or team member) may traditionally have limited access to traditional methods of surgical guidance.

The various physical components of the communications subsystem732may be detachable components coupled to the local server700via a computer network, a Fire Wire bus, a serial bus, or the like, and/or may be physically integrated onto a motherboard or circuit board of local server700. Communications subsystem732also may be implemented in whole or in part by software.

In some embodiments, communications subsystem732may also receive input communication in the form of structured and/or unstructured data feeds, event streams, event updates, and the like, on behalf of one or more devices that may use or access local server700. For example, communications subsystem732may be configured to receive data feeds associated with surgical data or surgical data structures in real-time from other communication services, web feeds such as Rich Site Summary (RSS) feeds, and/or real-time updates from one or more third party information sources. Additionally, communications subsystem732may be configured to receive data in the form of continuous data streams, which may include event streams of real-time events and/or event updates (e.g., surgical data, surgical data structure updates, preferred route updates, patient updates, etc.). Communications subsystem732may output such structured and/or unstructured data feeds, event streams, event updates, and the like to one or more data stores that may be in communication with device700.

Due to the ever-changing nature of computers and networks, the description of local server700depicted inFIG. 7is intended only as a specific example. Many other configurations having more or fewer components than the device depicted in the figure are possible. For example, customized hardware may also be used and/or particular elements may be implemented in hardware, firmware, software, or a combination. Further, connection to other computing devices, such as network input/output devices, may be employed. Based on the disclosure and teachings provided herein, it will be appreciated that there are other ways and/or methods to implement the various embodiments.

Referring next toFIG. 8, a flowchart of a process800for the automated provisioning of real-time custom procedural surgical guidance800is illustrated. The method for the automated provisioning of real time custom procedural guidance includes a plurality of steps connected with directional arrows. The directional arrows not only indicate that the elements are connected, but also indicate the direction that data may flow with respect to the various elements. For example, first, the method receives a surgical data structure802. For example, the surgical data structure may be received from another remote device (e.g., cloud server), stored and then retrieved prior to or during performance of a surgical procedure. The surgical data structure can provide precise and accurate coded descriptions of complete surgical procedures. The surgical data structure802can describe routes through surgical procedures and is described in detail above with reference toFIG. 2. For example, the surgical data structure802can include multiple nodes, each node representing a procedural state (e.g., corresponding to a state of a patient). The surgical data structure802can further include multiple edges, with each edge connecting multiple nodes and representing a surgical action.

After receiving the surgical data structure, live surgical data is received at block804. In certain embodiments, live surgical data804may include user inputs, a video stream, or operating room data. As one illustration, user inputs may include manual inputs directly provided by operating room personnel (e.g., via a wearable device or non-wearable device located inside or near an operating room). For example, operating room personnel may include the performing surgeon, a nurse, an anesthetist, or remote surgeon watching the procedure. Inputs may include gesture, voice, another form of hands-free sterile interaction and even direct contact with a computer user-interface. As another (additional or alternative) illustration, live surgical images may be recorded, such as those captured on video from a head-mounted surgeon camera. As yet another (additional or alternative) illustration, live surgical data804may include other operating room data such as; time elapsed in procedure, nodes already passed by the users (e.g., surgeon, nurse or team member), identified surgical instruments used or in use, personnel present in the operating room and user movements measured from accelerometers in a guidance system headset or other system-connected wearable devices.

At block806, a component is detected from the live surgical data. In some embodiments, the method detects a component from the live surgical data806using feature extraction. Anatomic feature extraction can be performed on the live surgical images to detect an anatomic state, anatomic pose, or any other attribute possible to derive from video that can be used later to determine a precise procedural state. Additional data such as user input or operating room data may assist with feature extraction. The feature extraction algorithm used to detect a component from the live surgical data may include machine learning iterations. In particular, if the first implementation of the systems and methods described herein requires manual navigation through the procedure by the user (e.g., surgeon, nurse or team member), the resulting dataset of live surgical images and associated navigation commands can be used as an effective algorithm training resource using machine learning software. In other embodiments, a component may be associated with a user input or operating room data.

At block808, a current node from the surgical data structure is identified. In some embodiments, the current node will correspond to the component detected from the live surgical data. For example, a component may identify a patient state, which may indicate which of multiple nodes corresponds to a current circumstance. In other embodiments, an edge may correspond to the component detected in the live surgical data. The current node and the component may correspond to a discrete procedural state. In other embodiments, the user input or the operating room data may be used to determine the current node from the surgical data structure.

At block810, procedural information for the current node is retrieved. For example, each of one, more or all nodes may have particular procedural information associated with the node, and some or all of the particular procedural information may be retrieved. As another example, a current node may be connected to each of one or more next-stage nodes via a corresponding edge. An edge may have particular procedural information associated with the edge, and some or all of the particular procedural information may be retrieved. Procedural information can include a description of and/or information corresponding to a patient state that fully describes the condition of a patient at a given point during a medical procedure. Procedural information may include (for example) pose, anatomical state, initial preparation and/or sterilization information. In some embodiments, the quantity of procedural information to be provided may be tailored e.g., to lower operating room personnel information workload.

At block812, the procedural information is transmitted. In some embodiments, the method may transmit the procedural information to a wearable device used by one or more operating room personnel. In some embodiments, the process800can return from block814to receiving live surgical data at block804and then repeat the blocks to detect components806, determine a new current node808, retrieve procedural information for the current node810, and again transmit the procedural information812for the current node. In some embodiments, the structures and technologies described herein may transmit the procedural information812based on a contingent event such as low point in operating room activity or an input form the surgeon or the operating room personnel.

The structures and technologies described herein may provide real time recommendations of surgical actions based on live data as described in process900, as depicted inFIG. 9. At block902, a procedural state is identified. For example, a node may be identified from amongst multiple nodes represented in a surgical data structure. The identified node may correspond to a discrete procedural state corresponding to a current circumstance (and/or recently received live data) of a surgical procedure.

At block906, one or more potential target nodes are identified. Each potential node can be connected (e.g., directly connected) to the current node by an edge, which can be identified at block908. Each of the one or more potential target nodes may include a node connected (e.g., directly connected) to the current node. Each of the one or more potential target nodes and/or each node connecting the current node to a potential node of the one or more potential target nodes may be associated with one or more variables, such as a weight, representing, for example, a potential surgical outcome, surgical risk, outcome probability and/or predicted time commitment. Surgical risk may represent the predicted probability of an unfavorable outcome occurring at any point during surgery. Surgical risk may represent the probability of complications in a node-to-node transition.

The variable(s) may have been generated based on (for example) an outcome of one or more real surgical procedures recorded by the guidance system, knowledge of outcome from annotated libraries of surgical videos and/or even appropriately tagged academic research relating to specific surgeries or surgical steps. For example, a plurality of intra-surgical indicators may be used to assess surgical risk and/or potential surgical outcomes. Direct surgical indicators may include, among other things, patient blood loss. The structures and technologies described herein may determine a patient who has undergone significant loss of blood is at more risk than one who has reached the same point in the procedure with less blood lost. Indirect surgical risk indicators may include, among other things, which instrument is being used, whether surgery is currently active near delicate anatomical structures and even simply the time spent by the surgeon on each individual surgical step.

Further, surgical risk may also be stratified by patient specific data. Patient information such as age, weight, prior surgical interventions and health history may further inform the calculation of surgical risk. Using collected metadata and other sources of risk indicators, such as academic research, associations and relevant ‘weightings’ of each of at least one of the one or more variables may be determined between patient specific data (e.g., age) and live surgical parameters at specific procedural-steps (e.g., blood loss during a specific phase of liver surgery).

The weighting of patient specific parameters may then be used to interpret live surgical data to perform a selection of a target node at block910. The target node need not be a final node of the data structure may correspond to a next (e.g., directly connected node). For example, a selection may be made to maximize (an outcome variable) or minimize a surgical risk or to maximize a score calculated based on a combination of multiple of the one or more variables.

At block914, procedural information associated with the target node912can be retrieved and transmitted (e.g., to a device present in or near an operating room or location associated with the surgical procedure).

FIG. 10shows a process1000for using live surgical data for tracking and procedural models. At block1002, surgical data is received. The surgical data can include data collected during a performance of a surgery. The surgical data can include digital video data, which may include a continuous signal that represents moving visual images. The video data may, but need not, include an audio signal that is aligned with the visual signal. The surgical data may (e.g., alternatively or additionally) include audio data, representations of one or more inputs received at a device (e.g., a voice command, a motion command or input received at a keyboard or other interface component), data from a smart surgical tool (e.g., indicative of a location, movement and/or usage), and so on. The surgical data can include data collected from a single device or from multiple devices. The surgical data can include data collected over a time period (e.g., a predefined time increment, since a previous state-transition time or procedure initiation time, or a time of an entire surgical procedure). The surgical data may include raw data collected during the surgery or a processed version thereof (e.g., to reduce noise and/or transform a signal, such as transforming accelerometer data collected at a headset to a movement command indicating that the surgical procedure has progressed to a next step).

At block1004, a procedural state is determined using all or part of the surgical data. In some instances, the surgical data is incrementally received (e.g., as it is collected in real-time), and surgical data can then be incrementally processed (e.g., by separately processing each data increment or by aggregating a recently received data increment with other data, such as data from a previous time increment or from another device, and processing the aggregated data). In these instances, a state may be determined for the new data (or aggregated data).

In some instances, the surgical data may include a larger data set that may represent multiple states. For example, a larger data set may include a continuous data set (e.g., video collected over the course of a surgical procedure). The surgical data may then be divided into multiple chunks (e.g., of a predefined duration), and a state may be determined for each chunk. Thus, for example, block1004may be performed multiple times.

The chunks may, or may not, be overlapping. When chunks are overlapping, an offset time by which the chunks advance may also be predefined. For example, in a 2-hour procedure, a video stream may be segmented into 60 two-minute, non-overlapping segments. As another example, in a 2-hour procedure, a video stream may be segmented to procedure119two-minute segments, where each segment overlaps with a previous segment by 1 minute. In some instances, surgical data includes a discrete data set. Discrete data points may be individually analyzed or binned into one of multiple time windows (e.g., which may, or may not be overlapping and may have a predefined duration), such that each discrete data point may be assessed in combination with other discrete data points and/or a chunk of each of one or more continuous data sets that correspond to a same time window. In some instances, surgical data includes deterministic indicators of state transitions, such as a command received from a user indicating transition to a new state. Time windows may then be defined as times between these transitions, and other surgical data may be binned in accordance with times associated with the data.

Determining the procedural state may include, for example, selecting a single node (or one or more nodes) from amongst a plurality of nodes in a surgical data structure. The surgical data structure can include one associated with a particular type of surgery being conducted (e.g., and may be identified based on input received before, during or after the surgery and/or based on a patient record or operating room schedule that identifies a type of surgical procedure being performed). The state may be identified as one that correspond with the selected node(s). In some instances, each node of a structure's plurality of nodes include one or more live-data signatures that include one or more characteristics of live data that are associated with the node. A characteristic may include (for example) a value identifier (e.g., identifying a particular word command), a definition of a range (e.g., for a distance expected between two particular devices), a threshold (e.g., a lower threshold for an acceleration value from a surgical tool), and so on. The determination at block1004may include determining to which node's signature the received data most closely corresponds (and/or corresponds with to at least a threshold degree). In some instances, the determination can be constrained, such that only a subset of the plurality of nodes are considered (e.g., those nodes of the plurality of nodes that are directly connected to a node associated with a current state). For example, the constraint may be such to not allow a determination that a “Surgery-Complete” state would immediately follow a “Surgery-Begins” state.

Various transformation, detection and/or extraction techniques may be used to identify to which node the surgical data corresponds. For example, a model generated based on performance of a machine-learning algorithm may be used to process the video data to identify the one or more procedural states. As another example, audio cues from the video may be assessed to identify the state(s). As yet another example, detecting that a particular tool is being used (e.g., based on a movement signal or activation signal received from the tool or based on detecting that the tool is being moved to a location overlapping with a patient location based on video data) may be used to identify the state(s).

It will be appreciated that one or both of blocks1002and1004may be performed during a surgical procedure (e.g., at a local surgery in an operating room or facility at which the procedure is being performed or at a remote server, such as a cloud server) or after the procedure using stored data collected during the procedure. For example, in one instances, surgical data is collected during a surgical procedure by devices located in an operating room, and subsequently a remote server may receive and process the surgical data. As another example, a local device may receive and process the surgery data during the procedure.

In some embodiments, block1004may determine that the surgical data structure does not include a procedural state that is associated with the surgical data received at block1002. Block1004may identify that the surgical data is associated with a new procedural state. In some embodiments, block1004may generate a new node using the process described inFIG. 11. An update to the nodes and procedural states identified in the surgical data structure may be initiated automatically by a local or remote server or by a user input. The user input may be entered by, for example, a user in the control center with access to a local server configured to execute surgical data structure explorer.

At block1006, temporal information is identified for a part of the video data associated with the procedural state. The temporal information may include (for example) a start time, end time, duration and/or range. The temporal information may be absolute (e.g., specifying one or more absolute times) or relative (e.g., specifying a time from a beginning of a surgery, from a beginning of a procedure initiation, etc.). The temporal information may include information defining a time period or time corresponding to the video data during which it is estimated that the surgery was in the procedural state.

The temporal information may be determined (for example) based on information associated with the state identification. For example, if surgery data is divided into data chunks, and each data chunk is associated with a procedural state, the temporal information may identify a start and end time that corresponds to (for example) a start time of a first data chunk associated with a given procedural state and an end time of a last data chunk associated with a given procedural state, respectively. As another example, if discrete data points are received that are indicative of state transitions, the temporal information may include times associated with two consecutive data points. As yet another example, a data set may be analyzed to detect state transitions (e.g., by detecting activation/inactivation of a particular smart tool, detecting a time period during which a particular visual characteristic is detected in video data, detecting times at which an audio command is detected in an audio signal that indicates that a state is transitioning, etc.). It will be appreciated that transitions between different states may have a same or different transition characteristic.

At block1008, electronic data is generated representing the part of the video data for the procedural state. The electronic data may include information included in or identified by (e.g., via a location identification) metadata of a node associated with the procedural state in a surgical data structure. The electronic data may (for example) identify the procedural state (e.g., by identifying a state of a patient, progress of a surgery, etc.), identify an action that is being performed or is about to be performed (e.g., as identified in an edge that connects a node corresponding to the procedural state with a node corresponding to a next procedural state), and/or identify one or more considerations (e.g., a risk, tools being used or that are about to be used, a warning, etc.). The electronic data may relate to a trajectory. For example, electronic data may indicate when an action is being performed that is contrary to (or, in other instances, in accordance with) a typical action, recommended action, or action prescribed by a guideline at a state of surgery.

The electronic data may include (for example) data that is to be concurrently presented during presentation of at least part of the part of the video data. For example, the electronic data may include text (e.g., an annotation) that is to be overlaid or presented next to a visual display of the part of the video data. As another example, the electronic data may include an audio signal (e.g., of spoken words) that is to be presented during a visual display of the part of the video data. Any audio data in the video signal may be, for example, eliminated, softened or maintained.

At block1010, it is determined whether electronic data has been generated for all procedural states. If electronic data does not exist for all the procedural states, process1000returns to block1004or1006, where a next procedural state is determined or where temporal information is identified for another part of the video data associated with a next procedural state.

If electronic data exists for all procedural states, process1000continues to block1012, where the electronic data (e.g., generated for each procedural state) is output. In some instances, the electronic data is aggregated across states. For example, modified video data may be generated that includes the electronic data generated for each procedural state. The modified video data may include all or part of original video data but with additional presentations of electronic data that represents a procedural state represented in a displayed part of the video signal. For example, if a one-hour surgery included six procedural states—each associated with successive, non-overlapping 10-minute time windows, modified surgery data may include a 60-minute visual signal accompanied by an annotation that changes every 10 minutes or that appears (with new text every ten minutes). As another example, for the same hypothetical surgery, an annotation may repeatedly change (e.g., every minute) during the video, but the text may always be text associated with a procedural state associated with a current image being displayed. As yet another example, a video signal may be parsed to identify a predefined number of images (e.g., 1 or a number greater than 1) associated with each procedural state, and a document may be generated that includes the images and text near the images that identify an action being performed.

The output may include transmitting the electronic data (e.g., in association with an identifier of the video data or surgical procedure) to storage or another device. In some instances, the temporal information is used to annotate and/or assess video data. For example, it may be determined whether a surgical team traversed through an approved set of actions. As another example, a video may be annotated with captions to identify what steps are being performed (e.g., to train another entity).

The electronic data generated using process1000may provide a base to describe surgical consensuses and assess individual procedural compliance. Contingent on the ability to define a discrete route through a surgical data structure for a given procedure is the possibility to compare two separate routes. As individual surgical data structures are coded, not textual descriptions, process1000provides data useful to directly quantify the differences between two sets of live surgical data. The comparison represents the difference in the defining attributes of nodes and actions coded within the two procedures. Further to these attributes, the comparison may also include metadata recorded against the map such as risk induced during the procedure.

In some embodiments, these data points may include, for example information relating the surgical data structure such as anatomical modifications made not within the original surgical route, time taken for each step and the types of instrument used, or data retrieved from patient measurements made during the procedure such as blood loss or loss of heart beat during the procedure.

In certain embodiments, a surgical route is identified based on live surgical data. The surgical route can then be compared to a pre-computed procedural profile (e.g., identifying one or more recommended or required procedure characteristics. A metric of procedural compliance may be generated. This metric of procedural guidance represents a quantitative analysis of how well a canonical procedure method was followed. In some embodiments, the metric may be used for training or education purposes. The metric may be generated using an “ideal” surgical data structure, which can be used to provide a procedural compliance score.

In some embodiments generating electronic data associated with procedural states enables auto-editing of surgical video and other material to streamline the design of accurate training tools. Using procedural step recognition1004, videos may be annotated, tagged or edited to become more easily navigable as a training tool. Furthermore, this functionality may be used to efficiently reduce the volume of data needed to be stored to contain all the crucial steps of surgery. Currently the process of surgical video editing is highly labor intensive and requires manual annotation or editing to produce an educational surgical video.

In additional embodiments, a defined surgical data structure route may be used to produce other training tools. By definition, a surgical data structure holds all the information needed to reproduce a surgical procedure, including a detailed anatomical reference model. Therefore, the data held in the specific sequence of particular actions and nodes referenced within this route may be reproduced or simulated in video, an AR format, or a VR format to produce comprehensive surgical simulations or animations. Further, these training simulation tools may be adjusted to provide a pre-operative training tool. By integrating patient specific data to a generic 4D surgical map or surgical data structure of a procedure, the resulting simulation may be used to refresh a surgeon or other entity involved in a surgical procedure of the preferred surgical method prior to a procedure.

FIG. 11shows a process1100for updating the surgical data structure. At block1102, a surgical data structure is accessed. In some embodiments, the surgical data structure is similar to the examples discussed in relation toFIGS. 2, 3, and 4.

At block1104, a new node is generated for the surgical data structure. A new node may be generated in response to detecting that a node-creation condition was satisfied. The node-creation condition may be configured to be satisfied upon detecting input that corresponds to an instruction to add a node to the surgical data structure (e.g., with an identification of a corresponding state and/or surgical action). An update to the node-creation condition can be initiated pre-surgery, during surgery, or post-surgery by (for example) an automated process and/or a manual input from a user accessing the surgical data structure.

In some embodiments, an update to the node-creation condition can be initiated by an automated process. A local server, a remote server, or another computing device can include instructions that configure the device to execute an automated process to update the node-creation condition. The automated process to update the node-creation condition can access surgical data from a single procedure or a dataset that includes surgical data from multiple procedures. The surgical data can include any of the types of data discussed in relation to block1002ofFIG. 10. The automated process to update the node-creation condition can process the surgical data into one or more discrete data points. The discrete data points can be associated with video data, audio data, representations of inputs received at a device (e.g., a voice command, a motion command or input received at a keyboard or other interface component), data from smart surgical tools, and so on. The discrete data points may be identified from the surgical data using a feature extraction algorithm. In some embodiments, the automated process to update the node-creation condition may receive the discrete data points from another process associated with providing surgical guidance.

After processing the surgical data into one or more discrete data points, the automated process to update the node-creation condition may determine that one or more of the discrete data points do not correspond to a procedural state associated with a node in the surgical data structure. If the discrete data points do not correspond to a node in the surgical data structure, the automated process to update the node-creation condition may configure the node-creation condition to be satisfied. In some embodiments, the process may be configured to transmit a request for approval to trigger the node-creation condition to an appropriate approval authority prior to updating the node-creation condition.

In some embodiments, the node-creation condition may be satisfied manually. A user (e.g., surgeon, nurse or team member) may access surgical data using a surgical data structure explorer. The surgical data structure explorer can be configured to provide a user access to surgical data associated with a surgical data structure. The user may provide inputs to the surgical data structure explorer that satisfy the node-creation condition. The user inputs can include, for example, a new procedural state, discrete data points associated with the surgical data, and/or additional information that the process can use to generate a new node.

After the node creation condition is satisfied by the automated process or manually, a new node can be generated for the surgical data structure. The new node will be associated with new procedural metadata. The procedural metadata can include any of the data discussed earlier in the specification. The process can identify procedural metadata to be associated with the new node from the one or more discrete data points extracted from the surgical data. The process can also receive procedural metadata from a user's manual inputs.

At block1106, the new node can be associated with a new procedural state. The generation of the new node and/or association with the new procedural state may be based on (for example) one or more of user inputs, sensor data from a device (e.g., wearable or non-wearable device) located in an operating room, data from a smart surgical tool, and/or data from a camera recording video data in an operating room. For example, a set of instructions, executable on the local server, remote server, and/or a computing device, can include instructions to update the surgical data structure to include a new procedural state associated with the one or more discrete data points extracted from the surgical data. Further instructions may update the surgical data structure to associate the new node generated in at block1104with the new procedural state. A computing device accessing the updated surgical data structure can identify the new node by determining received surgical data includes discrete data points that match discrete data points associated with the new procedural state.

The new node generated at block1104can be integrated into the surgical data structure. At block1108, a first node and second node of the surgical data structure1108that are related to the new node are identified. The first node can include one representing a procedural state preceding the new procedural state (e.g., corresponding to a time before a particular action was performed). The second node can include one representing a procedural state occurring after the new procedural state (e.g., corresponding to a time after a different particular action is performed). It will be appreciated that, in some instances, more than one first node and/or more than one second node is identified.

In some embodiments, the related nodes can be determined using the associated procedural metadata. In some instances, only a subset of the variables in the procedural metadata associated with a node may include values different from a second node in the surgical data structure. A relation threshold may be set to identify a minimum number of identical variables between a first set of procedural metadata and a second set of procedural metadata. If the number of identical variables exceeds the relation threshold, the first node may be related to the second node.

At block1110, a plurality of edges are generated to connect the new node to the first node and to the second node. In instances where more than one first node and/or more than one second node is identified, one or more additional edges can be generated. Each edge may correspond to a surgical action that results in a procedural state transitioning between states represented by the connected nodes. The discrete data points extracted from the surgical data can be used to identify a surgical action that is associated with transitioning from a first procedural state to a second procedural state. Also, procedural metadata associated with the new node, the first node, and the second node can be processed to identify surgical actions associated with transitioning from a first procedural state to a second procedural state.

At block1112, the surgical data structure with the new node and the new edges can be updated by the local server, the remote server, and/or a computing device. For example, an array, relational database or other type of data structure may be updated to include an element corresponding to the new node and multiple edge elements corresponding to the generated edges. The updated surgical data structure can be transmitted to another network connected device.

It will be appreciated that the steps described inFIG. 11may be performed during a surgical procedure (e.g., at a local surgery in an operating room or facility at which the procedure is being performed or at a remote server, such as a cloud server) or after the procedure using stored data collected during the procedure. For example, in one instance, surgical data is collected during a surgical procedure by devices located in an operating room, and subsequently a remote server may receive and process the surgical data. As another example, a local device may receive and process the surgery data during the procedure.

Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it is understood that the embodiments can be practiced without these specific details. For example, circuits can be shown in block diagrams in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary detail in order to avoid obscuring the embodiments.

Furthermore, embodiments can be implemented by (for example) hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and/or any combination thereof. When implemented in software, firmware, middleware, scripting language, and/or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine readable medium such as a storage medium.