Patent ID: 12220181

DETAILED DESCRIPTION

Camera control systems and methods for a computer-assisted surgical system are described herein. As will be explained in more detail below, an exemplary camera control system may access surgical session data for a surgical session, the surgical session including performance of one or more operations by a computer-assisted surgical system. The camera control system may identify, based on the surgical session data, an event associated with the surgical session, and may determine, based on the surgical session data, a location associated with the event. In response to the determination of the location of the event, the camera control system may direct an automatic adjustment of a view of a camera to capture a specific view of the location associated with the event. In some examples, imagery captured by the camera may be displayed by a display device associated with the computer-assisted surgical system.

To illustrate, during a minimally-invasive surgical procedure performed at a surgical facility with a computer-assisted surgical system, a camera control system may detect that a first manipulator arm to which a surgical instrument is attached is colliding with a second manipulator arm, thus preventing movement of a tip of a surgical instrument coupled to the first manipulator arm. As a result, the camera control system may determine that a location of the collision event is a location of the first manipulator arm and direct an automatic adjustment of a camera positioned within the surgical facility to capture a specific view of the first manipulator arm. Video from the camera may be displayed by a display device associated with a user control system used by a surgeon. Thus, the surgeon may quickly see why the tip of the surgical instrument is not moving and may quickly identify corrective actions, all without leaving his or her position at the user control system.

Various benefits may be provided by the camera control systems and methods described herein. For example, the systems and methods described herein may automatically direct an adjustment of a camera positioned within a surgical facility to capture imagery based on a detected context of a surgical session. The automatic adjustment of the camera may improve the efficiency of a surgical session by eliminating the interruption caused by manually adjusting a camera. Additionally, the systems and methods described herein may provide surgical team members with relevant contextual information in real-time during a surgical procedure, which may result in more effective and efficient collaboration and coordination among the surgical team members, and which may allow surgical team members to quickly and efficiently troubleshoot problems. Moreover, the systems and methods may predict events that may occur during the surgical session and direct the presentation of contextual visual content associated with such events, thus allowing surgical team members to prepare for and/or resolve such events before they occur.

Numerous technical computing benefits may also be realized by the systems and methods described herein. For example, the systems and methods described herein may be configured to access, transform, and process data from disparate computing systems in a manner that allows the systems and methods to provide timely (e.g., real-time) information to various users by way of various computing platforms. To this end, the systems and methods described herein may seamlessly integrate with one or more special purpose computing devices to process various types of data (e.g., by applying kinematics data, image data, sensor data, and/or surgical instrument data to one or more machine learning models) in order to identify events that occur, or that may occur, during a surgical session and/or identify contextual information associated with the events. In addition, the systems and methods described herein may utilize historical surgical session data generated during surgical sessions that precede a current surgical session to determine a context of the surgical session with reference to the other prior surgical sessions. In this manner, the systems and methods described herein may perform operations that are impossible to perform by a human alone. Moreover, the systems and methods described herein may improve the operation of a computer-assisted surgical system by improving efficiency, accuracy, and effectiveness of the computer-assisted surgical system.

Various embodiments will now be described in more detail with reference to the figures. The systems and methods described herein may provide one or more of the benefits mentioned above and/or various additional and/or alternative benefits that will be made apparent herein.

In some implementations, a camera control system may operate as part of or in conjunction with a computer-assisted surgical system. As such, an exemplary computer-assisted surgical system will now be described. The described exemplary computer-assisted surgical system is illustrative and not limiting. The camera control system may operate as part of or in conjunction with the computer-assisted surgical system described herein and/or with other suitable computer-assisted surgical systems.

FIG.1illustrates an exemplary computer-assisted surgical system100(“surgical system100”). As shown, surgical system100may include a manipulating system102, a user control system104, an auxiliary system106, and an auxiliary camera system108communicatively coupled one to another. In some examples, the camera control systems described herein may be implemented by one or more of these components.

Surgical system100may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient110positioned on an operating table112. As shown, the surgical team may include a surgeon114-1, a nurse114-2, an assistant114-3, and an anesthesiologist114-4, all of whom may be collectively referred to as “surgical team members114.” Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation. Surgical system100and surgical team members114are located at a surgical facility116. As used herein, a “surgical facility” may include any area where surgical system100and surgical team members114are located when they perform the computer-assisted surgical procedure. For example, a surgical facility may include an operating room, a training facility, a particular area within a room, and the like. In some examples, surgeon114-1may be positioned in a room or location separate from a room or location where manipulating system102, auxiliary system106, and/or other surgical team members114are located during the surgical procedure. In these examples, the surgical facility may include either or both locations.

WhileFIG.1illustrates an ongoing minimally invasive surgical procedure at surgical facility116, surgical system100may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system100. Additionally, it will be understood that the surgical session throughout which surgical system100may be employed may not only include an operative phase of a surgical procedure, as is illustrated inFIG.1, but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques are used on a patient to investigate, diagnose, or treat a physical condition of the patient. Additionally, a surgical procedure may include any procedure that is not performed on a live patient, such as a calibration procedure, a training procedure, and an experimental or research procedure.

As shown inFIG.1, manipulating system102may include a plurality of manipulator arms118(e.g., manipulator arms118-1through118-4) to which a plurality of surgical instruments (not shown) may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue-interaction functions), medical tool, monitoring instrument (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure (e.g., by being at least partially inserted into patient110and manipulated to perform a computer-assisted surgical procedure on patient110). While manipulating system102is depicted and described herein as including four manipulator arms118, it will be recognized that manipulating system102may include only a single manipulator arm118or any other number of manipulator arms as may serve a particular implementation.

Surgical instruments coupled to manipulator arms118may each be positioned at a surgical area associated with a patient. A “surgical area” may, in certain examples, be entirely disposed within a patient and may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure being performed on tissue internal to a patient, the surgical area may include the tissue, anatomy underlying the tissue, as well as space around the tissue where, for example, surgical instruments being used to perform the surgical procedure are located. In other examples, a surgical area may be at least partially disposed external to the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed on the patient. For instance, surgical system100may be used to perform an open surgical procedure such that part of the surgical area (e.g., tissue being operated on) is internal to the patient while another part of the surgical area (e.g., a space around the tissue where one or more surgical instruments may be disposed) is external to the patient. A surgical instrument may be referred to as being positioned or located at or within a surgical area when at least a portion of the surgical instrument (e.g., a distal portion of the surgical instrument) is located within the surgical area.

User control system104may be configured to facilitate control by surgeon114-1of manipulator arms118and surgical instruments coupled to manipulator arms118. For example, surgeon114-1may interact with user control system104to remotely move or manipulate manipulator arms118and surgical instruments. To this end, user control system104may provide surgeon114-1with imagery (e.g., high-definition stereoscopic imagery) of a surgical area associated with patient110as captured by an imaging device (e.g., an endoscope). In certain examples, user control system104may include a stereo viewer having two displays where stereoscopic images of a surgical area associated with patient110and generated by a stereoscopic imaging system may be viewed by surgeon114-1. Surgeon114-1may utilize the imagery to perform one or more procedures with one or more surgical instruments coupled to manipulator arms118.

To facilitate control of surgical instruments, user control system104may include a set of master controls (not shown). These master controls may be manipulated by surgeon114-1to control movement of surgical instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon114-1. In this manner, surgeon114-1may intuitively perform a surgical procedure using one or more surgical instruments.

User control system104may further be configured to facilitate control by surgeon114-1of other components of surgical system100. For example, surgeon114-1may interact with user control system104to change a configuration or operating mode of surgical system100, to change a display mode of surgical system100, to generate additional control signals used to control surgical instruments attached to manipulator arms118, to facilitate switching control from one surgical instrument to another, or to perform any other suitable operation. To this end, user control system104may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from surgeon114-1.

Auxiliary system106may include one or more computing devices configured to perform primary processing operations of surgical system100. The one or more computing devices included in auxiliary system106may control and/or coordinate operations performed by various other components (e.g., manipulating system102, surgical instruments attached to manipulator arms118, and/or user control system104) of surgical system100. For example, a computing device included in user control system104may transmit instructions to manipulating system102by way of the one or more computing devices included in auxiliary system106. As another example, auxiliary system106may receive, from manipulating system102and process image data representative of imagery captured by an imaging device attached to one of manipulator arms118.

In some examples, auxiliary system106may be configured to present visual content to surgical team members114who may not have access to the images provided to surgeon114-1at user control system104. To this end, auxiliary system106may include a display monitor120configured to display one or more user interfaces, such as images (e.g., 2D images) of the surgical area, information associated with patient110and/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitor120may display images of the surgical area together with additional content (e.g., graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitor120is implemented by a touchscreen display with which surgical team members114may interact (e.g., by way of touch gestures) to provide user input to surgical system100.

Auxiliary camera system108may be configured to capture and provide, on one or more display devices of surgical system100, visual content based on a detected context of the surgical procedure. As shown inFIG.1, auxiliary camera system108may include an auxiliary camera122(“camera122”) coupled to a camera manipulating system124. Camera122may be implemented by any suitable camera. In some examples, camera122may be implemented by a plurality of imaging devices configured to provide stereoscopic imagery. While auxiliary camera system108is depicted and described herein as including one camera122, it will be recognized that auxiliary camera system108may include multiple cameras122as may serve a particular implementation. For instance, if user control system104and surgeon114-1are located remotely from other components of surgical system100, an additional camera may be positioned to capture and provide imagery of user control system104and surgeon114-1.

Imagery (e.g., one or more still images or video images) captured by camera122may be transmitted to and/or displayed by any suitable display device associated with surgical system100. For example, imagery captured by camera122may be transmitted to and displayed by display monitor120, a stereo viewer of user control system104, a mobile device communicatively paired with surgical system100during the surgical session (e.g., a mobile device used by assistant114-3), and/or a display device associated with a remote computing device. In some examples in which surgical system100includes dual user control systems104, such as for training a surgeon to use surgical system100, imagery captured by camera122may be displayed by a display device associated with a second user control system104for viewing by another user (e.g., a remote proctor monitoring the surgical session and/or training the surgeon). Additionally or alternatively, image data representative of the imagery captured by camera122may be streamed and/or recorded by a remote computing device, thereby enabling playback of the imagery at a later time (e.g., after the surgical session).

Camera manipulating system124may be configured to automatically adjust a position and/or orientation of camera122to capture a specific view126within surgical facility116. As used herein, a “position” of camera122within surgical facility116may refer to a particular location of camera122within a three-dimensional (“3D”) space, and an “orientation” of camera122may refer to an imaging direction (e.g., view126) of camera122, which may be any direction within the 3D space.

Camera manipulating system124may be implemented by any suitable mechanism(s) configured to adjust a position and/or orientation of camera122. For example, as shown inFIG.1, camera manipulating system124may include a camera manipulator arm128to which camera122is coupled by way of camera pivot130. Camera manipulator arm128may be configured to adjust a position and/or an orientation of camera122by movement of camera manipulator arm128. Camera pivot130may be configured to adjust an orientation of camera122. For example, camera pivot130may rotate camera122within a horizontal plane, such as by panning from left to right (as viewed in the imaging direction of camera122). Additionally or alternatively, camera pivot130may tilt camera122up or down within a vertical plane, such as by adjusting the imaging direction of camera122up or down (i.e., at an angle relative to a horizontal plane). Camera manipulating system124may include additional or alternative components as may serve a particular implementation. For example, camera manipulating system124may include various gears, motors, arms, pivots, joints, bearings, and any other electrical and/or mechanical components that may facilitate adjustment of the position and/or orientation of camera122.

Camera manipulating system124of auxiliary camera system108may be configured to adjust a position and/or orientation of camera122in any suitable manner. In some examples, the position and/or orientation of camera manipulating system124may be adjusted manually, such as by a user pushing or moving camera122to a desired position or operating one or more electronic controls of camera manipulating system124. Additionally or alternatively, the position and/or orientation of camera122may be adjusted automatically (e.g., without user input) based on a context of the surgical session (e.g., based on an event associated with the surgical session), as will be explained below in more detail.

As shown inFIG.1, auxiliary camera system108(e.g., camera manipulating system124) is coupled to manipulating system102. In alternative examples, auxiliary camera system108may be coupled to any other component of surgical system100as may suit a particular implementation, such as user control system104, auxiliary system106, or a cart132(e.g., instrument cart132-2. In additional examples, auxiliary camera system108may be separate from manipulating system102, user control system104, and auxiliary system106. For example, camera manipulating system124may be mounted to a wall of an operating room. Alternatively, auxiliary camera system108may be a standalone system within surgical facility116, as shown inFIG.2.

FIG.2is the same asFIG.1except that auxiliary camera system108includes a movable camera cart202. Camera manipulating system124is supported on camera cart202, thereby allowing a surgical team member114to easily position camera122at a desired location within surgical facility116by moving camera cart202. Additionally, a surgical team member114may move camera cart202as necessary during the surgical procedure.

Referring again toFIG.1, surgical system100may also include one or more movable carts132for holding certain components of surgical system100and/or supplies to be used during the surgical procedure. For example, one or more computing devices included in auxiliary system106may be housed within auxiliary cart132-1. Additionally, surgical instruments that are not coupled to a manipulator arm118may be stowed on an instrument cart132-2for easy access (e.g., by assistant114-3) when coupling surgical instruments to a manipulator arm118and/or swapping surgical instruments during the surgical procedure. Supply cart132-3may be used, for example, by anesthesiologist114-4to store medications and/or other agents (e.g., fluorescence imaging agents) that may be administered to patient110during the surgical procedure.

Various components of surgical system100may include one or more displacement transducers, orientational sensors, and/or positional sensors used to generate raw (i.e., uncorrected) kinematics information (hereinafter “surgical system sensors”). For example, user control system104, auxiliary system106, auxiliary camera system108, operating table112, manipulator arms118, surgical instruments attached to manipulator arms118, and carts132may include one or more surgical system sensors. Surgical system100(e.g., auxiliary system106) may be configured to use the kinematics information to track (e.g., determine positions of) and/or control the various components of surgical system100.

Components of surgical system100may be communicatively coupled one to another in any suitable manner. For example, as shown inFIG.1, manipulating system102, user control system104, auxiliary system106, auxiliary camera system108, and carts118may be communicatively coupled by way of control lines134, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system102, user control system104, auxiliary system106, auxiliary camera system108, and carts132may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

As mentioned, the position and/or orientation of camera122may be adjusted automatically based on a context of the surgical session to capture a specific view (e.g., view126) within surgical facility116.FIG.3illustrates an exemplary camera control system300configured to automatically control certain operations of a camera system associated with a computer-assisted surgical system (e.g., auxiliary camera system108, an endoscope coupled to a manipulator arm118, a plurality of cameras associated with the computer-assisted surgical system, etc.) to provide contextual visual content associated with an event associated with a computer-assisted surgical session. As shown, camera control system300may include, without limitation, a storage facility302and a processing facility304selectively and communicatively coupled to one another. Facilities302and304may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). In some examples, facilities302and304may be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

Storage facility302may maintain (e.g., store) executable data used by processing facility304to perform any of the operations described herein. For example, storage facility302may store instructions306that may be executed by processing facility304to perform any of the operations described herein. Instructions306may be implemented by any suitable application, software, code, and/or other executable data instance.

Storage facility302may also maintain any data received, generated, managed, used, and/or transmitted by processing facility304. For example, as will be described below in more detail, storage facility302may maintain surgical session data, user profile data, user input data, and the like.

Processing facility304may be configured to perform (e.g., execute instructions306stored in storage facility302to perform) various processing operations associated with automatically adjusting a view of a camera to capture a specific view of a location associated with an event associated with a surgical session. For example, processing facility304may access surgical session data for a surgical session. The surgical session may include performance of one or more operations by a computer-assisted surgical system (e.g., surgical system100). Processing facility304may identify, based on the surgical session data, an event associated with the surgical session, such as an event that has occurred or an event that is likely to occur. Based on the surgical session data, processing facility304may determine a location associated with the event and may direct, in response to the determination of the location of the event, an automatic adjustment of a view of a camera (e.g., camera122) to capture a specific view of the location associated with the event. These and other operations that may be performed by processing facility304are described herein in more detail.

In some examples, camera control system300is implemented entirely by the computer-assisted surgical system itself. For example, camera control system300may be implemented by one or more computing devices included in surgical system100(e.g., in one or more computing devices included within manipulating system102, user control system104, auxiliary system106, and/or auxiliary camera system108).

FIG.4illustrates another exemplary implementation400of camera control system300. In implementation400, a remote computing system402may be communicatively coupled to surgical system100by way of a network404. Remote computing system402may include one or more computing devices (e.g., servers) configured to perform any of the operations described herein. In some examples, camera control system300may be entirely implemented by remote computing system402. Alternatively, camera control system300may be implemented by both remote computing system402and surgical system100.

Network404may be a local area network, a wireless network (e.g., Wi-Fi), a wide area network, the Internet, a cellular data network, and/or any other suitable network. Data may flow between components connected to network404using any communication technologies, devices, media, and protocols as may serve a particular implementation.

FIG.5illustrates another exemplary implementation500of camera control system300.FIG.5is similar toFIG.4except that a user device502may be communicatively paired with surgical system100and connected to network404. As shown, user device502may communicate with remote computing system402and/or surgical system100by way of network404. Additionally or alternatively, user device502may communicate with surgical system100by way of a direct connection504(e.g., a direct wired connection and/or a direct wireless connection, such as a Bluetooth connection, a near field communication connection, and the like). In some examples, camera control system300may be implemented by remote computing system402, surgical system100, and/or user device502.

User device502may be any suitable computing device configured to receive and transmit user input and/or present visual content. For example, user device502may be, but is not limited to, a mobile device (e.g., a mobile phone, a handheld device, a tablet computing device, a laptop computer, a personal computer, etc.), a wearable device (e.g., a smartwatch device, an activity tracker, a head-mounted display device, a virtual or augmented reality device, etc.), and/or a display device (e.g., a television, a projector, a monitor, a touch screen display device, etc.).

As shown, a user506(e.g., a surgical team member114) may use or otherwise have access to user device502. In some examples, user506may have to be logged in to user device502or to an application executed by user device502in order to use and/or interact with user device502. In some examples, the application executed by user device502may be used by user506to provide user input regarding workflow segmentation of the surgical session and to view imagery captured by a camera (e.g., camera122). For instance, user506may provide input regarding a task completed by user506or another surgical team member (e.g., one or more operations performed by a surgical team member or the computer-assisted surgical system), a stage of a surgical procedure, a condition of a patient, a malfunction of a component of the surgical system, a change in personnel on the surgical team, and any other information as may suit a particular implementation. As will be explained below in more detail, such user input may be utilized by camera control system300to more accurately identify surgical session events that occur, or that may occur, during a surgical session.

AlthoughFIG.5shows only one user device502communicatively paired with surgical system100and connected to network404, any number of user devices502may be paired with surgical system100and/or connected to network404. For example, each surgical team member114may utilize, during a surgical session, a user device502to provide input regarding a workflow of the surgical session, as well as to view imagery captured by a camera (e.g., camera122).

Various operations that may be performed by camera control system300(e.g., by processing facility304of camera control system300), and examples of these operations, will now be described. It will be recognized that the operations and examples described herein are merely illustrative of the many different types of operations that may be performed by camera control system300.

Camera control system300may direct an automatic adjustment of a view of a camera (e.g., view126of camera122) to capture a specific view of a location associated with a surgical session event. To this end, camera control system300may access surgical session data for a surgical session and, based on the surgical session data, identify an event associated with the surgical session. Various examples of these operations will now be provided.

In some examples, surgical session data accessed by camera control system300may be generated during the surgical session and may be based on one or more operations performed by a computer-assisted surgical system (e.g., surgical system100) during the surgical session. The operations performed by the computer-assisted surgical system may include any mechanical, electrical, hardware, and/or software-based operations as may serve a particular implementation. The surgical session data may be generated by the computer-assisted surgical system (e.g., by one or more components within surgical system100), by one or more components coupled to the computer-assisted surgical system during the surgical session (e.g., one or more surgical instruments coupled to a manipulator arm118), by a user device (e.g., user device502) communicatively paired with the computer-assisted surgical system during the surgical session, and/or by any other device associated with the computer-assisted surgical system as may serve a particular implementation. In scenarios in which camera control system300is implemented entirely by remote computing system402, surgical session data may additionally or alternatively be generated by remote computing system402while, for example, remote computing system402tracks operations performed by surgical system100.

Surgical session data generated during a surgical session may include various types of data. For example, surgical session data generated during a surgical session may include kinematic data, image data, sensor data, surgical instrument data, and/or any other type of data as may serve a particular implementation.

Kinematic data may be representative of a position, a pose, and/or an orientation of a component within the computer-assisted surgical system and/or a component coupled to the computer-assisted surgical system. For example, kinematic data may be representative of a position, a pose, and/or an orientation of a manipulator arm118and/or a surgical instrument coupled to a manipulator arm118. As another example, kinematic data may be representative of a position of manipulating system102, user control system104, auxiliary camera system108, operating table112, and/or carts132.

Image data may be representative of one or more images captured by an imaging device coupled to the computer-assisted surgical system. For example, image data may be representative of one or more images captured by an imaging device (e.g., a stereoscopic endoscope) coupled to a manipulator arm118. The one or more images may constitute one or more still images and/or video captured by the imaging device. In some examples, camera control system300may access image data by receiving (e.g., by way of a network) images output by the imaging device. In additional or alternative examples, image data may include image data generated by an imaging device that is external to a patient, such as camera122.

Sensor data may include any data generated by surgical system sensors included in or associated with a computer-assisted surgical system. Sensor data may be representative of any sensed parameter as may serve a particular implementation. For example, sensor data may be indicative of whether operating table112is moving, or whether surgeon114-1is actively interacting with user control system104.

Surgical instrument data may include any data generated by a surgical instrument, and may be representative of an identification (“ID”) of the surgical instrument, an operational state of the surgical instrument (e.g., open, closed, electrically charged, idle, etc.), a fault code of the surgical instrument, etc.

In some examples, camera control system300may additionally or alternatively access surgical session data generated by the computer-assisted surgical system during one or more other surgical sessions that, for example, precede the current surgical session. For example, camera control system300may generate surgical session data during a first surgical session in which the computer-assisted surgical system is used to perform a first surgical procedure with respect to a first patient. Camera control system300may also generate additional surgical session data during a second surgical session in which the computer-assisted surgical system is used to perform a second surgical procedure with respect to the patient or another patient. During the second surgical session, camera control system300may access both the surgical session data and the additional surgical session data. Surgical session data that is generated prior to a current surgical session may be referred to herein as “historical surgical session data.” As will be described below, historical surgical session data may allow camera control system300to more effectively identify and/or predict an event that may occur during the second surgical session.

Camera control system300may additionally or alternatively access surgical session data based on operations performed by one or more computer-assisted surgical systems other than the computer-assisted surgical system being used during a particular surgical session. For example, camera control system300may access surgical session data generated during a plurality of distinct computer-assisted surgical sessions located within a particular medical center, a network of hospitals, and/or any other grouping. This type of surgical session data may be referred to herein as “global surgical session data” and, as will be described below, may allow camera control system300to more effectively identify and/or predict an event that may occur during a particular surgical session in which a particular computer-assisted surgical system included in the grouping is used to perform a surgical procedure.

In some examples, camera control system300may provide a user interface configured to allow a user to define a particular grouping of computer-assisted surgical sessions and/or computer-assisted surgical systems from which surgical session data may be accessed by camera control system300.

Camera control system300may identify an event associated with a computer-assisted surgical session (a “surgical session event’) based on surgical session data for the surgical session, historical surgical session data, and/or global surgical session data. A surgical session event may include any distinct operation or action that occurs, or that may occur, with respect to the computer-assisted surgical system during the surgical session. A surgical session event may occur during a preoperative phase, an operative phase, and/or a postoperative phase of a surgical procedure.

For example, a surgical session event may include any operation or action associated with various preoperative phase operations. Exemplary preoperative phase operations may include, but are not limited to, patient intake (e.g., admitting the patient to a medical facility, receiving patient documentation, etc.), preparing an operating room, sterilizing surgical instruments, testing the computer-assisted surgical system and equipment, draping the computer-assisted surgical system (i.e., covering one or more components of computer-assisted surgical system, such as manipulator arms118, with a sterile or protective covering), preparing the patient for the surgical procedure (e.g., checking patient vital signs, providing intravenous fluids, administering anesthesia to the patient, bringing the patient into the operating room), and targeting the computer-assisted surgical system with respect to the patient (e.g., positioning manipulating system102at the patient bedside and positioning or configuring one or more manipulator arms118).

A surgical session event may additionally or alternatively include any operation or action associated with various operative phase operations. Exemplary operative phase operations may include, but are not limited to, opening a surgical area associated with a patient (e.g., by making an incision on external patient tissue), inserting a surgical instrument into the patient, performing surgical operations on patient tissue (e.g., by cutting tissue, repairing tissue, suturing tissue, cauterizing tissue, etc.), and closing the surgical area associated with the patient (e.g., removing surgical instruments from the patient, suturing closed the incision point, dressing any wounds, etc.).

A surgical session event may additionally or alternatively include any operation or action associated with various postoperative phase operations. Exemplary postoperative phase operations may include, but are not limited to, removing the computer-assisted surgical system from the patient (e.g., removing manipulating system102from the patient bedside), patient care and recovery operations (e.g., removing the patient from the operating room, monitoring the patient as the patient recovers from the surgical procedure, etc.), cleaning the operating room, cleaning the computer-assisted surgical system and/or surgical instruments, receiving reporting documentation by surgical team members, and patient discharge operations.

Camera control system300may identify a surgical session event based on surgical session data in any suitable manner.FIG.6shows an exemplary manner in which camera control system300may identify a surgical session event based on surgical session data. As shown, camera control system300may apply surgical session data602as an input to an event detection heuristic604. Event detection heuristic604may analyze the surgical session data602and output various instances of event data606(e.g., event data606-1through event data606-N). Each instance of event data606may represent a particular surgical session event identified by event detection heuristic604.

Event detection heuristic604may include any suitable heuristic, process, and/or operation that may be performed or executed by camera control system300and that may be configured to identify events based on surgical session data602. To illustrate, event detection heuristic604may detect an indicator and/or pattern in surgical session data that is indicative of an occurrence of a particular surgical session event.

For example, kinematic data generated during a particular portion of a surgical session may indicate movement of a surgical instrument in a suturing pattern. Additionally, surgical instrument data may indicate that the surgical instrument used during the same portion of the surgical session is a needle driver. Based on this kinematic data and surgical instrument data, camera control system300may determine that a suturing event is occurring or has occurred.

As another example, image data representative of images generated by an endoscope may indicate that a particular surgical instrument has remained out of a view of the endoscope for a predetermined period of time. Such image data may be indicative of an idle state event (e.g., that the surgical instrument is in an idle state).

In some examples, surgical session data602may include historical surgical session data, as described above. In these examples, one of the event data instances606output by event detection heuristic604may be representative of a surgical session event that camera control system300predicts will occur based on the historical surgical session data. For example, the historical surgical session data may include surgical session data generated during multiple surgical sessions in which the same type of surgical procedure is performed with the computer-assisted surgical system. Based on this historical surgical session data, event detection heuristic604may predict that a certain second event will occur following the occurrence of a certain first event.

In some examples, surgical session data602may include global surgical session data, as described above. In these examples, one of the surgical event data instances606output by event detection heuristic604may be representative of a surgical session event that is determined to occur based on the global surgical session data. For example, the global surgical session data may indicate that a particular kinematic data value for a particular surgical tool indicates that the surgical tool is located within a predetermined distance from patient tissue. When the actual kinematic data for the surgical tool being used during the surgical session is equal to or less than this value, event detection heuristic604may detect a surgical session event that indicates that the surgical tool is actually located within the predetermined distance from patient tissue.

Event detection heuristic604may receive additional or alternative types of input as may serve a particular implementation. For example,FIG.7is similar toFIG.6, but shows that event detection heuristic604may accept user profile data702(e.g., data representative of a user profile of one or more surgical team members involved with a surgical session) as an additional input. In this configuration, event detection heuristic604may detect surgical session events based on both surgical session data602and user profile data702.

To illustrate, user profile data702may include data representative of a user profile of a surgeon (e.g., surgeon114-1) involved with a surgical session. The user profile for the surgeon, combined with the surgical session data, may indicate that the surgeon performs various operations in a certain order unique to the surgeon. Accordingly, event detection heuristic604may detect that a particular surgical session event is going to occur in accordance with the certain order.

FIG.8illustrates another example of receiving additional or alternative types of input.FIG.8is similar toFIG.6, but shows that event detection heuristic604may accept user input data802as an additional input. User input data802may be representative of information input by a user by way of a computing device included in the computer-assisted surgical system (e.g., by way of user control system104or auxiliary system106) or communicatively paired with the computer-assisted surgical system (e.g., user device502). In this configuration, event detection heuristic604may detect surgical session events based on both surgical session data602and user input data802. User input data802may include, for example, information input by way of an application executed by a user device associated with a surgical team member.

To illustrate, user506(e.g., anesthesiologist114-4) may input, via an application currently being executed by user device502, information indicating that patient110is fully sedated. This information, combined with the surgical session data, may indicate that a preoperative phase of the surgical session has ended. Accordingly, event detection heuristic604may detect that a particular surgical session event is likely to occur, such as opening of an incision site on patient110. Thus, camera control system300may direct an automatic adjustment of camera122to capture a specific view of a location associated with the next event, such as instrument cart132-2.

In some examples, event detection heuristic604may implement a machine learning model. The machine learning model may use historical surgical session data, global surgical session data, user profile data, user input data, or any combination or sub-combination thereof, to identify one or more unique patterns of surgical system operations and associate surgical session events with the detected patterns of surgical system operations. As camera control system300collects more data, event data606output by event detection heuristic604may be updated or corrected as necessary.

When camera control system300identifies a surgical session event, camera control system300may determine a location, within the surgical facility, associated with the identified surgical session event (the “event location”). Camera control system300may then direct an automatic adjustment by a camera system associated with the computer-assisted surgical system (e.g., auxiliary camera system108) to automatically adjust a view of a camera (e.g., view126of camera122) to capture a specific view of the event location.

Camera control system300may determine the event location in any suitable way. In some examples, camera control system300may determine the event location by determining a location of the component associated with the identified surgical session event (the “event component”). To this end, in certain implementations camera control system300may access an event location table to identify a component associated with the identified surgical session event. To illustrate,FIG.9shows an exemplary event location table900that may be maintained or otherwise accessed by camera control system300. As shown in column902, table900may include a plurality of entries representative of various surgical session events that may occur during a surgical session (e.g., manipulator arm collision events, instrument exchange events, a bed motion event, a manipulator arm error event, a master controller collision event, and a manipulator arm no signal event). As shown in column904, table900may also list a component associated with each surgical session event (e.g., a particular manipulator arm, an operation table, a user control system, and an auxiliary cart).

Once the event component has been identified, camera control system300may determine the location of the event component (the “component location”) in any suitable way. In some examples, camera control system300may determine the component location based on surgical session data generated during the surgical session. For example, camera control system300may be configured to determine the component location based on kinematic data generated during the surgical session for the event component. Additionally or alternatively, the event location table may list component location data representative of a current location of each component listed in the event location table. The component location data may be populated when the surgical session begins and updated throughout the surgical session based on surgical session data (e.g., kinematic data) generated during the surgical session.

Additionally or alternatively to accessing kinematic data, camera control system300may determine the component location based on component tracking. In some examples, camera control system300may utilize marker-based computer vision tracking. To this end, unique markers (e.g., bar codes, colors, patterns, shapes, etc.) may be integrated on or otherwise attached to various components of the computer-assisted surgical system (e.g., manipulator arms118, surgical instruments, carts132, etc.), and the event location table may list marker data representative of a unique marker associated with each component. Camera control system300may access the event location table to identify a particular marker data instance associated with the event component. Based on the identified marker data instance, camera control system300may then determine the component location by detecting, in imagery captured by the camera system, the marker associated with the event component.

In additional or alternative examples, camera control system300may utilize signal-based tracking to determine the component location. To this end, emitters may be integrated on or otherwise attached to various components of the computer-assisted surgical system (e.g., manipulator arms118, surgical instruments, carts132, etc.), and the camera system may include a sensor configured to detect the signals emitted from the components. The emitters may be configured to emit any suitable signal, such as an infrared (“IR”) signal. Each emitter may be configured emit a unique signal based on, for example, a unique wavelength, a unique flashing pattern, a unique frequency, etc. The event location table may list signal data representative of a unique signal associated with each component. Camera control system300may access the event location table to identify a particular signal data instance associated with the event component. Camera control system300may then direct the event component to emit its particular signal. Alternatively, the components may periodically emit their associated signals. Camera control system300may then identify the component location as the source of an emitted signal that matches the identified signal data instance associated with the event component.

Once camera control system300has determined the event location, camera control system300may direct an automatic adjustment of a view of a camera to capture a specific view of the event location. In some examples, camera control system300may cause a camera manipulating system to which a camera is coupled to adjust a position and/or an orientation of the camera to capture the specific view of the event location. In alternative examples in which the camera system is implemented by a standalone computing device, camera control system300may transmit, to the camera system, data representative of the event location and a command to adjust the position and/or the orientation of the camera to capture the specific view of the event location.

In some examples in which camera control system determines the event location based on marker-based tracking or signal-based tracking, camera control system300may direct an adjustment of the camera toward the marker or the emitted signal associated with the event location.

In additional or alternative examples, the amount and direction of the adjustment of the position and/or orientation of the camera may be determined based on a spatial relationship between the camera and the event location. In some examples in which the camera system is physically coupled (e.g., mounted) directly on a component of the surgical system, a spatial relationship between the camera and the event location may be determined based on surgical session data (e.g., kinematic data, sensor data, etc.). Accordingly, adjustment of the view of the camera may be based on surgical session data. For example, camera control system300may determine, based on kinematic data and sensor data associated with the camera system, a position and an orientation of the camera relative to the event location. Based on the determined position and orientation of the camera, camera control system300may direct an automatic adjustment of the camera to capture the specific view of the event location.

In other examples in which the camera system is not physically coupled to a component of the surgical system, the spatial relationship between the camera and the event location may not be readily available or determinable from surgical session data alone. In such examples, to facilitate automatic adjustment of the view of the camera to capture the specific view of the event location, imagery captured by the camera of the camera system and 3D positional data tracked by the computer-assisted surgical system may be registered to a common 3D space. Additionally or alternatively, the camera system (e.g., camera122and/or camera manipulating system124) may be registered to the common 3D space. As mentioned above, the computer-assisted surgical system may use kinematic information generated from surgical system sensors to track (e.g., determine positions of) various components of the surgical system within a 3D space. Registration of the imagery captured by the camera to the same 3D space used by the computer-assisted surgical system may allow camera control system300to use kinematic information associated with the computer-assisted surgical system, including the camera system, to determine a spatial relationship between the camera and the event location. Such spatial relationship information may be utilized by camera control system300to determine a direction and amount of adjustment of the position and/or orientation of the camera to capture a view of the event location. Even where tracking-based techniques are used to determine the component location (and hence the event location), registration may enable estimation of the position and/or orientation of the camera based on surgical session data and thereby speed up the tracker-based determination of the component location.

Camera control system300may be configured to perform the registration process once (e.g., as part of a calibration, setup procedure, or other initial registration) and/or periodically or continuously after initial registration to refine the initial registration (e.g., to account for changes in physical positions, which positions may have various kinematic errors) and/or to account for positional adjustments of surgical system components. Registration may be performed by any suitable registration technique, including but not limited to vision-based registration, model-based registration, and marker-based registration.

To illustrate, imagery captured by the camera may be used for the registration process. For example, camera control system300may determine, based on imagery captured by the camera and using a trained neural network, a location of a component included in the computer-assisted surgical system. In certain examples, this may include camera control system300using a trained neural network to identify an object of interest depicted in the imagery, associating the object of interest to a component of the surgical system in any suitable way (e.g., object recognition), and determining a location (e.g., an image-based location) of the object.

Camera control system300may then perform an optimization to fit a model of the surgical system component to the determined location of the object (e.g., to the location of the object in the images). For components that are positioned on a floor, such as carts132, camera control system300may be configured to constrain the optimization to search only for solutions that are rotations and translations on the plane of the floor on which the components are placed. This constrained optimization may provide faster and/or more accurate results than a traditional optimization performed in all six degrees of freedom.

The registration process may be used by camera control system300to determine a missing link in a kinematic chain connecting the camera that is not physically coupled to a component of the computer-assisted surgical system. Camera control system300may represent the missing link as a transform that defines an estimated spatial relationship between the camera and the components of the surgical system. The transform may define a rotation and a translation that may be applied to convert data points from a reference frame of the camera to a reference frame of the computer-assisted surgical system (e.g., to convert coordinate points from a coordinate system of the camera to a coordinate system of the computer-assisted surgical system) and vice versa.

Camera control system300may use the missing link to complete a kinematic chain connecting a camera and the surgical system such that the complete kinematic chain is known and accessible to camera control system300. Camera control system300may then use kinematics of the camera system to track the position of the camera and determine the appropriate positional and/or orientational adjustment of the camera to capture the view of the event location.

As mentioned, the imagery captured by the camera may be transmitted to and displayed by a display device associated with the computer-assisted surgical system and/or a display device associated with a remote computing device. In this way a surgical team member, a remote proctor, a remote technician, and the like may view contextual visual content associated with a surgical session event and use the contextual visual content, for example, to perform one or more tasks during the surgical session (e.g., to perform a surgical procedure, to monitor or train a surgical team member, to troubleshoot technical issues, etc.). Examples of this processing will now be described.

For example, if camera control system300detects that manipulator arm118-1is colliding with manipulator arm118-2(labeled “MA1_Collide_MA2” in table900), camera control system300may determine that a location of the detected event is a location of manipulator arm118-1(labeled “MA1” in table900). Based on this determination, camera control system300may utilize the surgical session data (e.g., kinematic data) to determine a location, within the surgical facility, of manipulator arm118-1. Camera control system300may then direct camera manipulating system124to adjust a position and/or an orientation of camera122to capture a view of manipulator arm118-1. The imagery captured by camera122may then be displayed on a display device associated with a surgical team member114(e.g., surgeon114-1), thus allowing the surgical team member114to quickly identify why manipulator arm118-1is not moving.

As another example, if camera control system300determines that no signal is detected from a surgical instrument coupled to fourth manipulator arm118-4(labeled “MA4_NoSignal” in table900), camera control system300may determine that a location of the detected event is a location on fourth manipulator arm118-4where the surgical instrument should be connected (labeled “MA4” in table900). Based on this determination, camera control system300may utilize the surgical session data (e.g., kinematic data) to determine a location, within the surgical facility, of fourth manipulator arm118-4. Camera control system300may then direct camera manipulating system124to adjust a position and/or an orientation of camera122to capture a view of fourth manipulator arm118-4. The imagery captured by camera122may then be displayed on a display device associated with a surgical team member114(e.g., a user device used by assistant114-3), thus allowing the surgical team member114to quickly determine if the surgical instrument is properly connected to fourth manipulator arm118-4.

In some examples, a plurality of surgical system components may be associated with a particular event. For example,FIG.10shows an exemplary event location table1000that may be maintained or otherwise accessed by camera control system300. As shown in column1002, table100may include a plurality of entries representative of various events that may occur during a surgical session. As shown in column1004, table1000may list a primary component associated with each event. As shown in column1006, table1000may list a secondary component associated with each event. Camera control system300may determine the event location as the location of the primary component associated with the detected event. Camera control system300may then switch, in response to a location switch event, the event location to a location of the secondary component.

In some examples, the location switch event may include the passage of a predetermined period of time (e.g., 20 seconds) since a specific view of the primary component was first displayed. In some examples a user may specify the duration of the predetermined period of time.

Alternatively, the location switch event may be the receipt, by camera control system300, of user input directing camera control system300to switch to the secondary location. The user input may be received by way of any suitable computing device associated with the computer-assisted surgical system (e.g., user device502, remote computing system402, or a computing device included in user control system104, auxiliary system106, or camera system108).

To illustrate, if camera control system300detects that a surgical instrument coupled to manipulator arm118-3does not move when operated by surgeon114-1(labeled “MA3_NoMovement” in table1000), camera control system300may determine that a primary location of the detected event is a location of manipulator arm118-3(labeled “MA3” in table1000). Based on this determination, camera control system300may utilize the surgical session data (e.g., kinematic data) to determine a location, within the surgical facility, of manipulator arm118-3. Camera control system300may then direct auxiliary camera system108to adjust a view of camera122to capture a specific view of manipulator arm118-3. When the image captured by camera122is displayed on a display associated with user control system (e.g., on a touchscreen or on user device502), a surgeon can quickly determine if the surgical instrument is not properly coupled to manipulator arm118-3or if manipulator arm118-3is colliding with another manipulator arm. The surgeon can then provide input via user control system104(e.g., via a touchpad or foot pedal on user control system104) or user device502to view the secondary location. In response to this user input, camera control system300may direct auxiliary camera system108to adjust a view of camera122to capture a specific view of auxiliary cart132-1. This may allow the surgeon to determine if a cable of the surgical instrument is properly connected to auxiliary cart132-1.

In some examples, a user may be able to manually adjust the view of the camera after it has been adjusted to capture the view of the event location. Manual adjustments may include positional and/or orientational adjustments, as well as optical adjustments (e.g., zoom and focus adjustments).

In some examples, camera control system300may be configured to provide, for display by a display device associated with the computer-assisted surgical session, contextual information in addition to imagery captured by the camera. For example, camera control system300may identify contextual content associated with the identified surgical session event and/or the identified event location. The contextual content may be stored, for example, in table900or table1000. The contextual content may include, for example, a message to be displayed with the imagery captured by the camera.

To illustrate, camera control system300may determine, based on surgical session data, user profile data, and/or user input data, that a surgical procedure has progressed to a stapling stage. As a result, camera control system300may direct an automatic adjustment of a camera to capture a specific view of a manipulator arm, and may direct a display device to display, along with the imagery of the manipulator arm, a message such as: “Replace this surgical instrument with a stapler instrument.”

Additionally or alternatively, camera control system300may be configured to provide contextual information other than by way of a display device, such as by directing an illumination device (e.g., a task light, a laser light, etc.) to illuminate the event location. For instance, camera control system300may direct an adjustment of a laser device in the computer-assisted surgical session to illuminate the manipulator arm to which a surgical instrument should be coupled. Adjustment of the illumination device may be accomplished in a manner similar to the adjustment of the camera described herein.

FIG.11shows an exemplary camera control method1100. WhileFIG.11illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, combine, and/or modify any of the steps shown inFIG.11. One or more of the operations shown in inFIG.11may be performed by camera control system300, any components included therein, and/or any implementation thereof.

In step1102, the camera control system accesses surgical session data for a surgical session. The surgical session includes performance of one or more operations by a computer-assisted surgical system. Step1102may be performed in any of the ways described herein.

In step1104, the camera control system identifies, based on the surgical session data, an event associated with the surgical session. Step1104may be performed in any of the ways described herein.

In step1106, the camera control system determines, based on the surgical session data, a location associated with the event. Step1106may be performed in any of the ways described herein.

In step1108, the camera control system directs, in response to the determination of the location of the event, an automatic adjustment of a view of a camera to capture a specific view of the location associated with the event. Step1108may be performed in any of the ways described herein.

In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.

A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g. a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (“RAM”), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

FIG.12illustrates an exemplary computing device1200that may be specifically configured to perform one or more of the processes described herein. As shown inFIG.12, computing device1200may include a communication interface1202, a processor1204, a storage device1206, and an input/output (“/O”) module1208communicatively connected one to another via a communication infrastructure1210. While an exemplary computing device1200is shown inFIG.12, the components illustrated inFIG.12are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device1200shown inFIG.12will now be described in additional detail.

Communication interface1202may be configured to communicate with one or more computing devices. Examples of communication interface1202include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.

Processor1204generally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein. Processor1204may perform operations by executing computer-executable instructions1212(e.g., an application, software, code, and/or other executable data instance) stored in storage device1206.

Storage device1206may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example, storage device1206may include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device1206. For example, data representative of computer-executable instructions1212configured to direct processor1204to perform any of the operations described herein may be stored within storage device1206. In some examples, data may be arranged in one or more databases residing within storage device1206.

I/O module1208may include one or more I/O modules configured to receive user input and provide user output. One or more I/O modules may be used to receive input for a single virtual experience. I/O module1208may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O module1208may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.

I/O module1208may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, VO module1208is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

In some examples, any of the systems, computing devices, and/or other components described herein may be implemented by computing device1200. For example, processing facility304may be implemented by processor1204and storage facility302may be implemented by storage device1206.

In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.