SYSTEM FOR AUDIO PLAYBACK CONTROL IN A SURGICAL ROBOTIC SYSTEM

An audio player subsystem of a robotic surgical system is configured to play an audio file through an audio output device and control the playback of audio based on error handling and detections of the notification events. The surgical robotic system includes a robotic arm, a surgical console, a control tower, an error handler, and the audio player subsystem.

FIELD

The present disclosure generally relates to a surgical robotic system that controls audio playback based on a surgical procedure and notification events.

BACKGROUND

Surgical robotic systems are currently being used in minimally invasive medical procedures. Some surgical robotic systems include a surgical console controlling a surgical robotic arm and a surgical instrument having an end effector (e.g., forceps or grasping instrument) coupled to and actuated by the robotic arm.

Users in the operating room may desire to play music prior to the procedure, during the procedure, and after the procedure. The systems generate notifications to inform the users of the surgical system of the existence of any errors and other alarming or non-alarming conditions within the system or the surgical setting. In certain settings, and under certain conditions, such music playback in the operating room can be distracting, thereby diverting the attention of the users in the surgical setting away from errors and other alarming or non-alarming conditions occurring within the operating room. Therefore, it is possible for notifications to be delivered to users without the user being made aware that such notifications have been delivered.

SUMMARY

The present disclosure provides for a surgical robotic system that controls audio playback based on a surgical procedure and notification events.

According to one aspect of the present disclosure, a surgical robotic system is disclosed and includes a robotic arm including a surgical instrument coupled thereto, a surgical console, a control tower, an error handler, and an audio player subsystem. The surgical console includes a handle communicatively coupled to the robotic arm, and a display device configured to display a user interface. The control tower is communicatively coupled to the robotic arm and the surgical console. The error handler is communicatively coupled to at least one of the robotic arm, the surgical console, or the control tower and is configured to detect a notification event. The audio player subsystem is communicatively coupled to the error handler and is configured to play an audio file through an audio output device operably coupled to the audio player subsystem, receive a detection of a notification event from the error handler, and control playback of the audio file through the audio output device based on the detection of the notification event.

In an aspect, the audio player subsystem is configured to control playback of the audio file through the audio output device based on the detection of the notification event by at least one of stopping, pausing, or muting the playback of the audio file.

In an aspect, the audio player subsystem is configured to receive a user acknowledgment of the detection of the notification event and at least one of restart, un-pause, or un-mute the playback of the audio file upon receipt of the user acknowledgment.

In an aspect, the audio player subsystem is configured to control playback of the audio file through the audio output device based on the detection of the notification event by decreasing a volume level of the playback of the audio file.

In an aspect, the audio player subsystem is configured to receive a user acknowledgment of the detection of the notification event, and increase the volume level of the playback of the audio file upon receipt of the user acknowledgment.

In an aspect, the audio output device is a directional audio output device configured to generate directional sound waves configured to be targeted in a direction of a specific user associated with the notification event.

In an aspect, the surgical robotic system includes a second audio output device operably coupled to the audio player subsystem. The audio player subsystem may be configured to control playback of the audio file to the audio output device and the second audio output device based on the detection of the notification event. Additionally, or alternatively, the audio player subsystem may be configured to control playback of the audio file through the audio output device and the second audio output device based on the detection of the notification event by adjusting the playback of the audio file through the audio output device in a first manner and adjusting the playback of the audio file through the second audio output device in a second manner different from the first manner.

In an aspect, the surgical robotic system includes an alarm and notification subsystem coupled to the error handler. The alarm and notification subsystem may be configured to display a notification on the display device based on the detection of the notification event from the error handler.

According to another aspect of the present disclosure, a surgical robotic system is disclosed and includes video analysis subsystem and an audio player subsystem. The video analysis subsystem is configured to detect a notification event within a surgical setting and detect a type of surgical procedure or a stage of the surgical procedure. The audio player subsystem is communicatively coupled to the video analysis subsystem and is configured to receive the detected type of surgical procedure or stage of the surgical procedure from the video analysis subsystem and select an audio file from a plurality of audio files to play based on at least one of the detected type of surgical procedure or stage of surgical procedure as detected by the video analysis subsystem. Additionally, the audio player subsystem is configured to play the selected audio file through an audio output device operably coupled to the audio player subsystem, receive a detection of a notification event from the video analysis subsystem, and control playback of the audio file through the audio output device based on the detection of the notification event. The audio player subsystem is configured to control playback of the audio file through the audio output device by at least one of stopping the playback of the audio file, pausing the playback of the audio file, muting a volume level of the playback of the audio file, or decreasing the volume level of the playback of the audio file.

In an aspect, the audio player subsystem is configured to receive a user acknowledgment of the detection of the notification event and control playback of the audio file through the audio output device based on the received user acknowledgment by at least one of restarting the playback of the audio file, un-pausing the playback of the audio file, un-muting the playback of the audio file, or increasing the volume level of the playback of the audio file.

In an aspect, the audio output device is a directional audio output device configured to generate directional sound waves configured to be targeted in a direction of a specific user associated with the notification event.

In an aspect, the surgical robotic system includes a second audio output device operably coupled to the audio player subsystem. The audio player subsystem may be configured to control playback of the audio file to the audio output device and the second audio output device based on the detection of the notification event. Additionally, or alternatively, the audio player subsystem may be configured to control playback of the audio file through the audio output device and the second audio output device based on the detection of the notification event by adjusting the playback of the audio file through the audio output device in a first manner and adjusting the playback of the audio file through the second audio output device in a second manner different from the first manner.

In an aspect, the surgical robotic system includes an alarm and notification subsystem configured to cause a display device to display a notification based on the detection of the notification event from the video analysis subsystem.

According to another aspect of the present disclosure, an audio player subsystem of a surgical robotic system is disclosed. The audio player subsystem is configured to play an audio file through an audio output device operably coupled to the audio player subsystem, receive priority alarm information, and control playback of the audio file through the audio output device based on the received priority alarm information. The audio player subsystem is configured to control playback of the audio file through the audio output device by at least one of stopping the playback of the audio file, pausing the playback of the audio file, muting a volume level of the playback of the audio file, or decreasing the volume level of the playback of the audio file.

In an aspect, the audio player subsystem is configured to at least one of restart, un-pause, or un-mute the playback of the audio file upon at least one of receipt of a user acknowledgment of the received priority alarm information or expiration of a preset period of time. Additionally, or alternatively the audio player subsystem may be configured to increase the volume level of the playback of the audio file upon at least one of receipt of a user acknowledgment of the received priority alarm information or expiration of a preset period of time.

In an aspect, the audio output device is a directional audio output device configured to generate directional sound waves configured to be targeted in a direction of a specific user associated with the received priority alarm information.

In an aspect, the audio player subsystem includes a second audio output device. The audio player subsystem may be configured to control playback of the audio file to the audio output device and the second audio output device based on the received priority alarm information. Additionally, or alternatively, the audio player subsystem may be configured to control playback of the audio file through the audio output device and the second audio output device based on the received priority alarm information by adjusting the playback of the audio file through the audio output device in a first manner and adjusting the playback of the audio file through the second audio output device in a second manner different from the first manner.

In an aspect, the surgical robotic system includes a video analysis subsystem configured to detect a stage of the surgical procedure. In an aspect, the audio player subsystem is configured to control playback of audio based on the detected stage of the surgical procedure.

DETAILED DESCRIPTION

The term “application” may include a computer program designed to perform functions, tasks, or activities for the benefit of a user. Application may refer to, for example, software running locally or remotely, as a standalone program or in a web browser, or other software which would be understood by one skilled in the art to be an application. An application may run on a controller, or on a user device, including, for example, a mobile device, an IOT device, or a server system.

As will be described in detail below, the present disclosure is directed to a surgical robotic system, which includes a surgical console, a control tower, and one or more movable carts having a surgical robotic arm coupled to a setup arm. The surgical console receives user input through one or more interface devices, which are interpreted by the control tower as movement commands for moving the surgical robotic arm. The surgical robotic arm includes a controller, which is configured to process the movement command and to generate a torque command for activating one or more actuators of the robotic arm, which would, in turn, move the robotic arm in response to the movement command.

With reference toFIG.1, a surgical robotic system10includes a control tower20, which is connected to all of the components of the surgical robotic system10including a surgical console30and one or more robotic arms40. Each of the robotic arms40includes a surgical instrument50removably coupled thereto. Each of the robotic arms40is also coupled to a movable cart60.

The surgical instrument50is configured for use during minimally invasive surgical procedures. In embodiments, the surgical instrument50may be configured for open surgical procedures. In embodiments, the surgical instrument50may be an endoscope configured to provide a video feed for the user. In further embodiments, the surgical instrument50may be an electrosurgical forceps configured to seal tissue by compression tissue between jaw members and applying electrosurgical current thereto. In yet further embodiments, the surgical instrument50may be a surgical stapler including a pair of jaws configured to grasp and clamp tissue whilst deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue.

Each of the robotic arms40may include a camera51configured to capture video of the surgical site. The camera51may be a stereoscopic camera and may be disposed along with the surgical instrument50on the robotic arm40. The surgical console30includes a first display32, which displays a video feed of the surgical site provided by camera51of the surgical instrument50disposed on the robotic arms40, and a second display device34, which displays a user interface for controlling the surgical robotic system10. The surgical console30also includes a plurality of user interface devices, such as foot pedals36and a pair of handle controllers38aand38bwhich are used by a user to remotely control robotic arms40.

The control tower20includes a display23, which may be a touchscreen, and outputs on the graphical user interfaces (GUIs). The control tower20also acts as an interface between the surgical console30and one or more robotic arms40. In particular, the control tower20is configured to control the robotic arms40, such as to move the robotic arms40and the corresponding surgical instrument50, based on a set of programmable instructions and/or input commands from the surgical console30, in such a way that robotic arms40and the surgical instrument50execute a desired movement sequence in response to input from the foot pedals36and the handle controllers38aand38b.

Each of the control tower20, the surgical console30, and the robotic arm40includes a respective computer21,31,41. The computers21,31,41are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area networks, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are not limited to, transmission control protocol/internet protocol (TCP/IP), datagram protocol/internet protocol (UDP/IP), and/or datagram congestion control protocol (DCCP). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 802.15.4-2003 standard for wireless personal area networks (WPANs)).

The computers21,31,41may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically-erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and/or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted for by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and/or set of instructions described herein.

With reference toFIG.2, each of the robotic arms40may include a plurality of links42a,42b,42c, which are interconnected at joints44a,44b,44c, respectively. The joint44ais configured to secure the robotic arm40to the movable cart60and defines a first longitudinal axis. With reference toFIG.3, the movable cart60includes a lift61and a setup arm62, which provides a base for mounting of the robotic arm40. The lift61allows for vertical movement of the setup arm62. The movable cart60also includes a display69for displaying information pertaining to the robotic arm40.

The setup arm62includes a first link62a, a second link62b, and a third link62c, which provide for lateral maneuverability of the robotic arm40. The links62a,62b,62care interconnected at joints63aand63b, each of which may include an actuator (not shown) for rotating the links62band62brelative to each other and the link62c. In particular, the links62a,62b,62care movable in their corresponding lateral planes that are parallel to each other, thereby allowing for extension of the robotic arm40relative to the patient (e.g., surgical table). In embodiments, the robotic arm40may be coupled to the surgical table (not shown). The setup arm62includes controls65for adjusting movement of the links62a,62b,62cas well as the lift61.

The third link62cincludes a rotatable base64having two degrees of freedom. In particular, the rotatable base64includes a first actuator64aand a second actuator64b. The first actuator64ais rotatable about a first stationary arm axis which is perpendicular to a plane defined by the third link62cand the second actuator64bis rotatable about a second stationary arm axis which is transverse to the first stationary arm axis. The first and second actuators64aand64ballow for full three-dimensional orientation of the robotic arm40.

The robotic arm40also includes a plurality of manual override buttons53disposed on instrument drive unit52and the setup arm62, which may be used in a manual mode. The user may press one or the buttons53to move the component associated with the button53.

With reference toFIG.2, the robotic arm40also includes a holder46defining a second longitudinal axis and configured to receive the instrument drive unit52(FIG.1) of the surgical instrument50, which is configured to couple to an actuation mechanism of the surgical instrument50. Instrument drive unit52transfers actuation forces from its actuators to the surgical instrument50to actuate components (e.g., end effectors) of the surgical instrument50. The holder46includes a sliding mechanism46a, which is configured to move the instrument drive unit52along the second longitudinal axis defined by the holder46. The holder46also includes a joint46b, which rotates the holder46relative to the link42c.

The robotic arm40also includes a plurality of manual override buttons53disposed on the IDU52and the setup arm62, which may be used in a manual mode. The clinician may press one or the buttons53to move the component associated with the button53.

The joints44aand44binclude an actuator48aand48bconfigured to drive the joints44a,44b,44crelative to each other through a series of belts45aand45bor other mechanical linkages such as a drive rod, a cable, or a lever and the like. In particular, the actuator48ais configured to rotate the robotic arm40about a longitudinal axis defined by the link42a.

The actuator48bof the joint44bis coupled to the joint44cvia the belt45a, and the joint44cis in turn coupled to the joint46cvia the belt45b. Joint44cmay include a transfer case coupling the belts45aand45b, such that the actuator48bis configured to rotate each of the links42b,42cand the holder46relative to each other. More specifically, links42b,42c, and the holder46are passively coupled to the actuator48bwhich enforces rotation about a pivot point “P” which lies at an intersection of the first axis defined by the link42aand the second axis defined by the holder46. Thus, the actuator48bcontrols the angle θ between the first and second axes allowing for orientation of the surgical instrument50. Due to the interlinking of the links42a,42b,42c, and the holder46via the belts45aand45b, the angles between the links42a,42b,42c, and the holder46are also adjusted in order to achieve the desired angle θ. In embodiments, some or all of the joints44a,44b,44cmay include an actuator to obviate the need for mechanical linkages.

With reference toFIG.4, each of the computers21,31,41of the surgical robotic system10may include a plurality of controllers, which may be embodied in hardware and/or software. The computer21of the control tower20includes a controller21aand safety observer21b. The controller21areceives data from the computer31of the surgical console30about the current position and/or orientation of the handle controllers38aand38band the state of the foot pedals36and other buttons. The controller21aprocesses these input positions to determine desired drive commands for each joint of the robotic arm40and/or the instrument drive unit52and communicates these to the computer41of the robotic arm40. The controller21aalso receives back the actual joint angles and uses this information to determine force feedback commands that are transmitted back to the computer31of the surgical console30to provide haptic feedback through the handle controllers38aand38b. The safety observer21bperforms validity checks on the data going into and out of the controller21aand notifies a system fault handler if errors in the data transmission are detected to place the computer21and/or the surgical robotic system10into a safe state.

The computer41includes a plurality of controllers, namely, a main cart controller41a, a setup arm controller41b, a robotic arm controller41c, and an instrument drive unit (IDU) controller41d. The main cart controller41areceives and processes joint commands from the controller21aof the computer21and communicates them to the setup arm controller41b, the robotic arm controller41c, and the IDU controller41d. The main cart controller41aalso manages instrument exchanges and the overall state of the movable cart60, the robotic arm40, and the instrument drive unit52. The main cart controller41aalso communicates actual joint angles back to the controller21a.

The setup arm controller41bcontrols each of joints63aand63b, and the rotatable base64of the setup arm62and calculates desired motor movement commands (e.g., motor torque) for the pitch axis and controls the brakes. The robotic arm controller41ccontrols each joint44aand44bof the robotic arm40and calculates desired motor torques required for gravity compensation, friction compensation, and closed loop position control of the robotic arm40. The robotic arm controller41ccalculates a movement command based on the calculated torque. The calculated motor commands are then communicated to one or more of the actuators48aand48bin the robotic arm40. The actual joint positions are then transmitted by the actuators48aand48bback to the robotic arm controller41c.

The IDU controller41dreceives desired joint angles for the surgical instrument50, such as wrist and jaw angles, and computes desired currents for the motors in the instrument drive unit52. The IDU controller41dcalculates actual angles based on the motor positions and transmits the actual angles back to the main cart controller41a.

The robotic arm40is controlled as follows. Initially, a pose of the handle controller controlling the robotic arm40, e.g., the handle controller38a, is transformed into a desired pose of the robotic arm40through a hand eye transform function executed by the controller21a. The hand eye function, as well as other functions described herein, is/are embodied in software executable by the controller21aor any other suitable controller described herein. The pose of one of the handle controller38amay be embodied as a coordinate position and role-pitch-yaw (“RPY”) orientation relative to a coordinate reference frame, which is fixed to the surgical console30. The desired pose of the surgical instrument50is relative to a fixed frame on the robotic arm40. The pose of the handle controller38ais then scaled by a scaling function executed by the controller21a. In embodiments, the coordinate position is scaled down and the orientation is scaled up by the scaling function. In addition, the controller21aalso executes a clutching function, which disengages the handle controller38afrom the robotic arm40. In particular, the controller21astops transmitting movement commands from the handle controller38ato the robotic arm40if certain movement limits or other thresholds are exceeded and in essence acts like a virtual clutch mechanism, e.g., limits mechanical input from effecting mechanical output.

The desired pose of the robotic arm40is based on the pose of the handle controller38aand is then passed by an inverse kinematics function executed by the controller21a. The inverse kinematics function calculates angles for the joints44a,44b,44cof the robotic arm40that achieve the scaled and adjusted pose input by the handle controller38a. The calculated angles are then passed to the robotic arm controller41c, which includes a joint axis controller having a proportional-derivative (PD) controller, the friction estimator module, the gravity compensator module, and a two-sided saturation block, which is configured to limit the commanded torque of the motors of the joints44a,44b,44c.

With reference toFIG.5, the present disclosure provides an audio player subsystem200which operates in conjunction with an alarm and notification subsystem150and an error handler network100. The audio player subsystem200may be part of the control tower20, surgical console30, and/or robotic arms40, or may be a standalone component that is communicatively coupled to the surgical robotic system10. The audio player subsystem200may be hard wired or communicably coupled to one or more audio output devices250(e.g., speakers) within the surgical setting to play audio (e.g., music) from any source including but not limited to a podcast, music player, show, etc. The source of the audio files/selection of audio files to play may live on a user device (e.g., smartphone or other external audio player) that is wirelessly connected to the audio player subsystem200. The output devices may be part of the surgical robotic system10or standalone devices communicatively coupled to the audio player subsystem200of the surgical robotic system10. The output devices250may also include wearable devices, such as headphones, or portable speakers. The audio output devices250may be a component of one or more of the control tower20, surgical console30, robotic arms40, or may be a separate component such as standalone speakers, headsets, or the like, and may be configured to output directional soundwaves directed at one or more specific users in the operating room. In one aspect, one audio output device250may be dedicated to one user or one group of users while a different audio output device250may be dedicated to another user or another group of users.

The audio player subsystem200controls the playback of audio (e.g., music) during the surgical procedure, for example, by adjusting or modifying playback of audio (controlling the playback speed, volume, tone, pausing, or stopping, etc.) and/or by selecting which audio file to play based on the user that is operating the system, errors detected within the surgical robotic system10(also referred to herein as “notification events”), delivery of notifications, the type of surgical procedure, and/or progression of the surgical procedure. The audio player subsystem200may automatically select the audio to play based on different inputs or factors or the audio play subsystem200may play audio based on received user input or selection. In one embodiment, the audio play subsystem200is configured to select which audio file to play and may adjust the playback depending on a variety of factors selected by a user or automatically selected by the surgical robotic system10, for example, based on the type of procedure, the current stage of the procedure, and/or other factors. For example, for procedures, or during a certain stage of a procedure that requires relatively higher concentration by a user, the audio player subsystem200may select to play a song, or a portion of a song, that promotes concentration and/or may adjust the playback of the audio in a manner that prevents the audio playback from distracting the user (e.g., slow the playback speed, reduce the volume, pause or stop the playback, etc.). In contrast, during another procedure, or a different stage of the procedure that requires relatively lower concentration by a user (e.g., during setup or clean-up), the audio player subsystem200may select to play a song or a portion of a song that promotes rapid movement by the users and/or may adjust the playback of the audio in a manner that promotes faster movement by the users (e.g., increase the playback speed, increase the volume, etc.). In an aspect, the audio player subsystem200monitors the progress and phase of the procedure during the course of the procedure and adjusts the tone of the audio playback (e.g., the music) based on the current stage or type of the procedure. The audio player subsystem200may automatically adjust or modify the playback of audio based on different inputs or factor or the audio play subsystem200may adjust or modify the playback of audio based on received user input or selection.

The audio player subsystem200may include a memory which stores a plurality of audio files for playback. In some embodiments, the audio files may be stored remotely on a server, a cloud service, or a music streaming service.

In an aspect, the audio player subsystem200may include input devices (e.g., microphones, image devices, laparoscopic cameras, sensors, etc.) configured to monitor stress levels of users in the operating room and upon detecting elevations or changes in stress levels within the operating room, may adjust the audio playback based on the elevated stress levels of the users or participants. In one particular example, the audio player subsystem200monitors the voices of participants on the operating room and detects elevated stress levels based on the voices. In further embodiments, the audio player subsystem200may receive surgical phases, procedures warnings, notification events, or surgical events from a surgical video analysis subsystem300. In this aspect, the audio play subsystem200may adjust or select the audio playback based on the received surgical events, surgical warnings, or surgical phases. For example, the tempo or type of the music selected may vary based on if it is the beginning, middle, or ending of the procedure. The surgical video analysis subsystem300may be part of the control tower20, surgical console30, and/or robotic arms40, or may be a standalone component.

In addition to operating based on detected stages of the surgical procedure and other signals received from the surgical video analysis subsystem300, the audio player subsystem200may additionally or alternatively operate in conjunction with the alarm and notification subsystem150and/or the error handler network100to modify the audio playback based on detected notification events. A notification event, for example, may be a detected surgical event, a detected surgical phase, a detected surgical error, a detected issue that requires an alert or alarm, or any detected surgical or robotic event that may benefit from a notification to the user. For example, the notification may alert any relevant users (or all users) to the existence of an error or any alarm-type or non-alarm-type condition. The modification of the audio playback by the audio player subsystem200, originating from detected notification events, may correspond to the type (e.g., the priority) of error or notification event detected and may be associated with (e.g., be relevant to) only select users within the surgical setting. Therefore, depending on the type of error or notification event detected, while audio player subsystem200may adjust the audio playback for one specific relevant user in a first manner (e.g., mute playback volume for the user) to indicate to the relevant user that the corrective action is required of the relevant user, the audio player subsystem200may adjust the audio playback for other non-relevant users in a second, different, manner (e.g., reduce playback volume) to make the non-relevant users aware that another individual within the surgical setting is responsible for administering a corrective action. For example, for a high priority event (e.g., one that may require the attention of all of the individuals within the surgical setting), the audio player subsystem200may be configured to mute, decrease the playback volume, or stop the playback of all of the audio to all of the users within the surgical setting, and in contrast, for a low priority event (e.g., one that may require the attention of only one or more users within the surgical setting), the audio player subsystem200may be configured to mute, decrease the playback volume, or stop the audio playback to only those relevant users.

The error handler network100may include any number of error handlers, which may be embodied as software executable by one or more corresponding controllers. In one example embodiment, the error handler network100includes error handlers121a,121b,141a,141b,141c,141d, which are embodied as software executable by each of the corresponding controller: the controller21a, the safety observer21b, the main cart controller41a, the setup arm controller41b, the robotic arm controller41c, and the IDU controller41d, respectively. The error handler network100is a virtual network for transmitting signals communicating between multiple subsystems (e.g., controllers) to transmit information about errors and desired system reactions. Thus, if an error occurs at one component, then a controller (e.g., the main cart controller41a) associated with that component reacts to the error with a preprogrammed response (e.g., preprogrammed action that places the component in a safe state). The controller (e.g., main cart controller41a) also reports the error to other subsystems such as the safety observer21b, the main cart controller41a, the setup arm controller41b, and the alarm and notification subsystem150which operates in conjunction with the audio player subsystem200to alert or otherwise notify the applicable user(s) of an event, when appropriate. These subsystems then respond by initiating related safe behaviors. Safety behavior may also define an error response that includes a hard stop, e.g., complete shutdown of the surgical robotic system10, in which all modes (including manual control mode) are disabled for the remainder of the procedure until deactivation of the surgical robotic system10.

The error handlers121a-band141a-dare networked together to coordinate a response across the surgical robotic system10and subsystem level in upstream/downstream and parent/child manner. Child subsystems (or child processes) refer to downstream, lower-level subsystems such as the robotic arm controller41c, the setup arm controller41b, and IDU controller41d. Mid-level subsystems refer to the main cart controller41aof each of the movable carts60. The top-level subsystem is the controller21a. Thus, all other error handlers141a-141dare downstream from the error handler121aand error handler141ais the parent of the error handlers141b-141d. The computer31of the surgical console30may also include a controller having an error handler, which is downstream of the error handler121aand operates in the similar manner as the error handlers141a-141d.

With reference toFIG.6, initially, the error signals reported by the error handlers121a-band141a-141dare categorized as an operable error or an inoperable error. As used herein, an operable error is an error during which normal function of the controllers (e.g., position control of the robotic arm controller41c) is not affected and the user is simply notified that an error was detected. Inoperable error is an error in response to which position control of the robotic arm40and the surgical instrument50is disabled and manual control is also interrupted. Each of the controllers21aand41a-dreacts to the error signal based on the type of error signal. The reaction may have a predetermined duration for displaying a notification and/or a predetermined duration of how long an operational mode is disabled.

Responses to inoperable errors cause manual control of the robotic arms40and/or the surgical instruments50to be interrupted without permanently disabling the robotic arms40and the surgical instruments50. Thus, if the user is actively using manual control when an inoperable error occurs then that instance of manual control is ended, and the user can let go of the button53and then reactivate the button53to re-enter manual control.

The error signals reported by the error handlers121a-band141a-141dmay further be classified as recoverable or nonrecoverable and as relevant to one or more specific users within the surgical setting such that actions by the audio player subsystem200(e.g., volume reduction, volume mute, stop or pause audio play, modify speed or tone of audio play, etc.) and actions by the alarm and notification subsystem150(e.g., notifications delivery) associated with the error are only imparted upon the specific relevant user or users. Recoverability for a given error is defined at both the system level and the subsystem level. In embodiments, the error is defined as affecting the entire surgical robotic system10and one of the components, e.g., the movable cart60. A system nonrecoverable error denotes that the entire surgical robotic system10cannot be recovered back to a usable state. A subsystem nonrecoverable error denotes that only one or more of the components where the error occurred cannot be recovered back to a usable state.

When a nonrecoverable error is encountered, manual control is temporarily interrupted and position control is disabled until the surgical robotic system10or specific component thereof is restarted. In one example implementation, when a nonrecoverable error is encountered, the audio player subsystem200stops playback of the audio until the surgical robotic system10or specific component thereof is restarted. When a recoverable error is encountered, manual control is interrupted and position control is disabled only for a period of time, which is based on whether the error is transient or persistent, and whether the error is dismissible. In one example implementation, when a recoverable error is encountered, the audio player subsystem200reduces the volume of the audio playback for a period of time, which is based on whether the error is transient or persistent, and whether the error is dismissible.

Inoperable recoverable errors further include two categories: transient-type errors and persistent-type errors. Transient-type recoverable errors are further segregated into transient and dismissible errors. Persistent-type errors include nonrecoverable, persistent recoverable, and persistent-to-dismissible recoverable errors. Additionally, any of the transient-type errors or persistent-type errors, whether recoverable or non-recoverable, may be categorized as user-specific, that is, as relevant to one or more specific users within the surgical setting. Thus, the errors may be classified into the following types:Nonrecoverable inoperable error—the affected subsystem notifies the user, adjusts the playing audio, disables position control, and interrupts manual control. Position control is disabled until the surgical robotic system10or specific component thereof is restarted. In one example implementation, when a nonrecoverable inoperable error is encountered, the audio player subsystem200stops playback of the audio until the surgical robotic system10is restarted.Transient recoverable inoperable error—the affected subsystem notifies the user, adjusts the playing audio, interrupts position control, and interrupts manual control. All modes are available immediately after the error has occurred. In one example implementation, when a transient recoverable inoperable error is encountered, the audio player subsystem200reduces the playback volume of the audio for a period of time.Persistent recoverable inoperable error—the affected subsystem notifies the user, adjusts the playing audio, disables position control, and interrupts manual control. Position control is disabled until the reason for the error is resolved (e.g., by user intervention). In one example implementation, when a persistent recoverable inoperable error is encountered, the audio player subsystem200reduces the playback volume of the audio until the reasons for the error is resolved (e.g., by user intervention).Transient dismissible recoverable inoperable error—the affected subsystem notifies the user, adjusts the playing audio, disables position control, and interrupts manual control. Position control is disabled until the user acknowledges the notification, regardless of whether the reason for the error disappears. In one example implementation, when a transient dismissible recoverable inoperable error is encountered, the audio player subsystem200reduces the playback volume of the audio until the user acknowledges the notification, regardless of whether the reason for the error disappears, and then increases the playback volume once the user acknowledges the notification.Persistent dismissible recoverable inoperable error—the affected subsystem notifies the user, adjusts the playing audio, disables position control, and interrupts manual control. Position control is disabled until the reason for the error disappears and the user acknowledges the notification. In one example implementation, when a persistent dismissible recoverable inoperable error is encountered, the audio player subsystem200reduces the playback volume of the audio until the user acknowledges the notification and the reason for the error disappears, and then increases the playback volume once the user acknowledges the notification and the reason for the error disappears.

With reference toFIG.5, each controller21aand41a-duses a corresponding error handler121a-band141a-dto communicate with other error handlers of the other controllers as well as with the state machines in its own controller. When an error occurs in a subsystem, that subsystem reacts by disabling/interrupting control modes and then sending messages to a neighboring subsystem via the error handler. The robotic arm controller41c, the setup arm controller41b, and IDU controller41dcommunicate error signals to the main cart controller41a, which then replies with confirmation of receipt and control commands. The main cart controller41aalso reports error information up to the controller21a, which then replies with confirmation of receipt and nominal control commands as well. This communication combined with the designated recovery type affects the timing of when the controllers41b-dcan resume nominal functionality.

After detecting a problem, the error handler141b, disables position control, and sends an alarm to an alarm and notification subsystem (ANS)150, which may be embodied as a software application and is executed by the controller21a. The ANS150is coupled to various inputs and outputs of the surgical robotic system10and displays various audio and visual alarm and notifications on one or more of the displays23,32,34,69. The ANS150operates in conjunction with the audio player subsystem200to control the audio (e.g., music) being played by the audio player subsystem200based on the type (e.g., priority) of the error or notification event detected. The alarm and notification subsystem150also uses audio to indicate/display notifications. In one example, instead of just lowering the volume or stopping the playback, the audio player subsystem200plays the notification-specific audio. Additionally, or alternatively, when the playback of the user audio is not stopped, it could mix in (with increasing volume) the playback of the navigation or other audio.

The error handler141bthen sends the error signal to the error handler141a, which then sends a confirmation of receiving the error signal and initiates a hold protocol and stops position control. Error signals are also sent to the controller21a, which recognizes the system-wide inoperable error and interrupts any robotic arms40in position control or manual control. The controller21aalso sends hold commands to all robotic arms40and sends a confirmation of receiving an error signal to the robotic arm40. Additionally, the ANS150and the audio player subsystem200are also independently aware of the type of the error or other notification event (e.g., that the error is a system persistent dismissible type) and deliver notifications and adjust audio playback, respectively, to the relevant user(s) based on the type of error or other notification event. Depending on the type of error or other notification event, the controller21acan stop position control on any robotic arms40until the user dismisses the notification, and the user cannot dismiss the notification until the original problem is resolved.

Thus, all robotic arms40are locked out of position control as well as manual control. A notification appears on GUIs one or more of the displays23,32,34,69with a graphical “dismiss” button to dismiss the error and the audio playing by the audio player subsystem200may be adjusted based on the type and priority of the error or other notification event, for example to bring the user's attention to the error, the notification, or the state of the error. A bedside assistant can re-activate the button53to get back into manual control. However, the user is not able to get back into position control because the error was a persistent dismissible type. After the problem responsible for the error independently disappears, the user can then dismiss the notification, and then the user is able to resume position control and the audio playback by the audio player subsystem200resumes to the preadjusted levels.

The error handler network100includes the following conditions, which are mapped to specific error states: manual mode for one robotic arm40is unavailable, position control for one robotic arm40is unavailable, position, manual, and hold modes for one robotic arm40are unavailable, and position control for all robotic arms40is unavailable. Conditions are mapped to the set of error states based on whether the error affects a subsystem, the surgical robotic system10, or both. Mapping also depends on error operability and recoverability. Some error states are set while the condition is true, while other error states are set as true when the condition occurs.

As described above, inoperable recoverable errors are categorized into two categories: transient-type and persistent-type. Transient-type recoverability includes transient and dismissible errors. Persistent-type errors include nonrecoverable, persistent recoverable, and persistent to dismissible recoverable errors. In the transient-type inoperable errors, manual control and position control, as well as position control of all robotic arms40, are interrupted but not disabled. In the persistent-type inoperable errors only manual control is interrupted and position control on all robotic arms40is disabled. The error state mapping for the system and subsystem persistent dismissible recoverable inoperable errors is the same as for persistent-type inoperable errors.

Regarding nonrecoverable errors, certain conditions are mapped to system and/or subsystem nonrecoverable errors. Nonrecoverable signals indicate when a nonrecoverable error has occurred on the subsystem. Each error signal generated by an error handler has its own signal pair, where the first error signal indicates subsystem (local) position control is nonrecoverable and the second error signal indicates system (global) position control is nonrecoverable. Nonrecoverable signals are set to being true during error state mapping and are transmitted upwards to parent subsystems. In the parent error handler (e.g., error handler121aor141a), nonrecoverable signals are analyzed by combining all of the nonrecoverable signals using an “or” operator (e.g., error handlers141b-das well as parent's nonrecoverable signals). If activated, these signals are latched until system deactivation.

The error handler network100also stores interrupt counters which when exceeded interrupt operation of the surgical robotic system10and/or any of the robotic arms40. This allows the error handler network100to interrupt operation of the surgical robotic system10even when error states are dismissed. The error handlers141a-dhave a unique subsystem interrupt counter signal and a system interrupt counter. The counters represent how many errors have occurred in that subsystem. The counters are increased while the specific controller41a-dis running continuously. The counters are reset once the specific controller41a-dis shut down.

When an error occurs, that subsystem's error handler141b-dincrements the counter(s) and sends the signal upwards to the error handler141aand/or the error handler121a. The error handlers121aand141adetect when the counters are incremented and send interrupt signals to their state machines. The incremented counters are then sent to the child subsystems, e.g., the error handlers141b-d. This confirmation transmission forms a cycle, which enables the child error handlers141b-dto confirm that the messages were received by the upstream error handlers121aand141a.

The error confirmation cycle is used to end an active interruption of position control and/or manual control of the robotic arm40. When any control mode is interrupted, error handlers121aand141alatch the error states as true until the downward counter matches the upwards counter. Error states may remain true for another reason (e.g., a persistent error), but the interrupt is ended.

The error handlers121amay also verify the system interrupt counters. If a condition in a child error handler141b-dmaps to a system level inoperable error, the system interrupt counter is incremented and sent upwards to the controller21a. The error handler121achecks for increments, sends an interrupt signal to all controllers41aof the movable cart60, and sends the increment counter back down to the error handler141aof the problematic robotic arm40. The child error handlers141b-dmay not receive a direct confirmation that the system-level message was received. Rather, since the child process is receiving commanded states from the controller41a, the controller41akeeps track of the counters and releases system-level error states appropriately. In embodiments, the error handlers141b-dmay also receive confirmation that the system-level messages were received.

The error handler141achecks for subsystem interrupt counter increments, whereas the error handler121achecks for system interrupt counter increments. The child error handler141b-d, which detected an error continually marks error states as true for several ticks until the subsystem and system counters match. This system of counters forms a cycle which clears the pipeline of nominal commands. It also prevents a user from being able to command unavailable functionality in the window of time between an error being detected and the interrupt/error state signals reaching their destinations.

Error handlers121a-band141a-din all subsystems send messages to ANS150and audio player subsystem200about what conditions rise and fall. Each condition or other notification event has a unique alarm ID and ANS150and audio player subsystem200knows which movable cart60and/or the robotic arm40originates the alarm. The ANS150includes a database storing, for each alarm ID, a notification type, priority, and message and duration of the notification along with an identification of any users being particularly relevant to the specific notification event if applicable. Likewise, the audio player subsystem200includes a database storing, for each alarm ID, a corresponding action to take against the audio being played (e.g., adjust playback speed, change song being played, adjust playback volume, adjust playback tone, stop or pause playback, etc.) and a list of relevant users associated with the alarm ID of the notification event. The subsystem error handlers141a-dinclude a database storing for each alarm ID corresponding controller reaction types and durations for the reaction.

With respect to dismissible errors, since the error handlers121a-band141a-dare not aware of notification dismissals, the ANS150outputs a position control blocking signal which is sent to the controller21a. The position control blocking signal is false by default. When a dismissible error occurs, ANS150sets the blocking signal to true and the error handler121auses this signal to make position control unavailable for all robotic arms40. Even if the dismissible error did not occur in the controller21a, the blocking signal from the ANS150to the error handler121acauses the controller21ato command nominal hold signals to all lower-level state machines.

Alarm IDs are set and cleared through ANS150. Alarms are set upon occurrence of certain conditions (e.g., the existence of an error or a notification event) and alarms may be cleared upon the condition ending or other circumstances. If the alarm is related to recoverable dismissible errors, then those alarms are cleared immediately after being set, because the ANS150utilizes the clear signal to enable a “dismiss” button (not shown) on a notification user interface displayed on one or more of the displays23,32,34,69and the user's attention is brought to the “dismiss” button (not shown) by virtue of the adjustment to the playing audio (e.g., music) being played by the audio player subsystem200. Without clearing the alarm through ANS150, the user will not be able to dismiss the notification, and the error will not be transient.

For persistent dismissible errors, the alarms clear upon condition ending. Thus, the ANS150does not enable the “dismiss” button on the notification user interface until the condition ends, at which point the user is allowed to dismiss the notification and the ANS150stops blocking teleoperation by the controller21a. An error signal mapped to a transient dismissible error may still be present, but the controller21ais able to command position control again as soon as the user acknowledges the notification. To bring the user's attention to the ability to select the “dismiss” button (e.g., upon enablement of the “dismiss” button when the condition ends), the audio player subsystem200may adjust the audio being played when the “dismiss” button becomes enabled and selectable.

Each of the controller21aand the controller41ahave an error aggregator221aand241arespectively, which aggregate errors from non-controller software in their respective domains, and forward those errors to the corresponding controller21aor41a. Error aggregators221aand241amay be embodied as software applications executable by each of the corresponding controllers21aand41a, respectively. The error aggregator221aaggregates errors from all nodes of the control tower20and the surgical console30and forwards those errors to the controller21a. The error aggregator241aaggregates errors from non-controller software on each movable cart60and forwards those errors to the main cart controller41a. The error aggregators221aand241aforward the errors to the error handlers141b-din the form of an array, which includes a counter of all active errors of a specific system and subsystem including their recoverability type. Upon receiving this array, the error handlers141b-didentify any new errors that have occurred in the form of an increase in any of the counters, and also for any active errors in the form of non-zero counter values. These two conditions are used by the error handlers141a-dto create the appropriate controller responses. The error handler response is the same as the response as if the error occurred in the respective controller that received the error report.

With reference toFIG.7, a method for controlling audio in a surgical robotic system10will now be described as method700. Method700may be embodied as software instructions executed by the computers21,31,41or any other computing device and may be carried out by one or more components of the surgical robotic system10. Additionally, although method700is described as including particular steps and as being carried out in a particular order, it is contemplated that method700may include more or less steps than described and may be carried out in any order.

Method700begins in step701where particular music is selected for playback. As described above, audio player subsystem200(FIG.5) may store, or may be communicatively coupled to a storage device that stores, a plurality of music audio files. In embodiments, the audio files may be stored remotely on a server, a cloud service, or may be provided by a music streaming service. In step701, audio player subsystem200selects a particular audio file to play based on the type of procedure to be performed or based on a particular stage of the procedure. For example, if the procedure stage is at the start of the procedure (e.g., during preparation and set up), then audio player subsystem200may select a song that is suitable for the setup stages of a procedure.

As the procedure progresses and the media is being played back during the procedure, the surgical robotic system10constantly monitors its status and the status of the surgical setting and in step702determines whether a notification event is detected (e.g., error detection). If in step702, a notification event is not detected, then method700proceeds to step703where the progression of the stages of the procedure are monitored. In step704, the audio player subsystem200modifies the tone of the music and/or another setting of the music playback based on the current stage of the procedure (as monitored in step703).

If in step702, a notification event is detected, then method700proceeds to step705where a determination is made whether the notification event warrants a stop of the music playback. In particular, the audio player subsystem200looks up the alarm ID data corresponding to the notification event detected in step702and determines whether the notification event warrants a stop of the music play for any of the users based on the data associated with the alarm ID. This may be accomplished by stopping the music playback to audio output devices250associated with specific users or by redirecting the directional soundwaves emitted by a directional audio output device250toward the specific users only and away from the nonrelevant users. If the notification event does warrant a stop of music play for any of the users, then method700proceeds to step706, where a determination is made whether the notification event is relevant to any specific users within the surgical setting. In particular, the audio player subsystem200looks up the alarm ID data corresponding to the notification event detected in step702at determines whether the notification event is relevant to all of the users or only a select number of users based on the data associated with the alarm ID.

If in step706it is determined that the notification event is relevant to all of the users, then, in step707, the audio player subsystem200stops the music playing to all of the users. Stopping the music may include pausing or muting the playback of the music. Alternatively, if in step706it is determined that the notification event is relevant to only a select number of the users, then, in step708, the audio player subsystem200stops the music playing to only the relevant users and continues playing music to the other users within the surgical setting, for example by stopping the music playback to audio output devices250associated with specific users or by redirecting the directional soundwaves emitted by a directional audio output device250toward the specific users only and away from the nonrelevant users. The audio player subsystem200may reduce stop the music play for a preset period of time, and then continue the music play, based on the specific notification event and the alarm ID data and whether the notification event requires user acknowledgement or intervention.

If in step705, it is determined that the notification event does not warrant a stop of the music playback, then method700proceeds to step709, where a determination is made whether the notification event warrants a volume reduction. In particular, the audio player subsystem200looks up the alarm ID data corresponding to the notification event detected in step702at determines whether the notification event warrants a volume reduction of the music play for any of the users based on the data associated with the alarm ID. If the notification event does warrant a volume reduction of music play for any of the users, then method700proceeds to step710, where a determination is made whether the notification event is relevant to any specific users within the surgical setting. In particular, the audio player subsystem200looks up the alarm ID data corresponding to the notification event detected in step702at determines whether the notification event is relevant to all of the users or only a select number of users based on the data associated with the alarm ID. If in step710it is determined that the notification event is relevant to all of the users, then, in step711, the audio player subsystem200reduces the volume of the music playing to all of the users. Alternatively, if in step710it is determined that the notification event is relevant to only a select number of the users, then, in step712, the audio player subsystem200reduces the volume of the music playing to only the relevant users and continues playing music without a reduction in volume to the other users within the surgical setting, for example by reducing the volume of the music playback to audio output devices250associated with specific users or by redirecting the directional soundwaves emitted by a directional audio output device250toward the specific users only and away from the nonrelevant users. The audio player subsystem200may reduce the volume for a preset period of time, and then raise to the original volume level, based on the specific notification event and the alarm ID data and whether the notification event requires user acknowledgement or intervention.

After any of steps707,708,711, or712, method700proceeds to step713where a notification is sent to either all of the users or only relevant users depending on the type of notification event and the alarm ID data. The notification delivered may be tactile, audible, vibratory, visual, and/or any other type of notification. In step714, a determination is made whether user intervention is required to address the notification event based on the alarm ID data of the notification event. User intervention may include acknowledging the notification event (for example, by selecting a button displayed on a user interface such as a “dismiss” button) and/or requiring a correction of the issue that has caused the notification event. If in step714it is determined that the notification event requires user intervention, then method700proceeds to step715where it is determined whether the required user intervention is received. If after a preset period of time (which may be different based on the type of notification event and the alarm ID data associated therewith), the required user intervention is not received, then method700reverts back to step713where the delivery of the notification is continued. However, if in step714it is determined that user intervention is not required, or in step715it is determined that the required user intervention is received, then method700proceeds to step716where the music playback is resumed to original volume levels.

It will be understood that various modifications may be made to the embodiments disclosed herein. In embodiments, the sensors may be disposed on any suitable portion of the robotic arm. Therefore, the above description should not be construed as limiting, but merely as exemplifications of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended thereto.