Patent Description:
A conventional computer-assisted surgical system may detect when a head of a user (e.g., a surgeon) is located within a vicinity of a viewer console included in the user control system (e.g., by sensing whether a series of transverse infrared beams are blocked in front of eyepieces of the viewer console). Based on this detection, the computer-assisted surgical system may set an appropriate operating mode for the user control system. For example, the computer-assisted surgical system may only allow control of the teleoperated surgical instruments when the user's head is located within the vicinity of the viewer console. This may prevent unintentional and therefore uncontrolled movement of the teleoperated surgical instruments. However, there remains room to improve selection and control of an operating mode of the user control system.

The present invention provides a system and a method in accordance with the independent claims. Optional features for the system and the method are defined by the dependent claims. An exemplary system may comprise a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to access head presence data generated by a head sensor included in a user control system of a computer-assisted surgical system, the head presence data indicating a presence or an absence of a head of a user within a vicinity of a viewer console included in the user control system; access eye tracking data generated by an eye sensor included in the user control system, the eye tracking data indicating whether an eye of the user is gazing through an eyepiece included in the user control system; direct, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is gazing through the eyepiece, the user control system to operate in a first operating mode; and direct, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is not gazing through the eyepiece, the user control system to operate in a second operating mode different from the first operating mode.

An exemplary system may comprise a head sensor configured to detect a presence or an absence of a head of a user within a vicinity of a viewer console included in a user control system of a computer-assisted surgical system and generate head presence data indicating the presence or the absence of the head of the user within the vicinity of the viewer console; an eye sensor configured to detect whether an eye of the user is gazing through an eyepiece included in the user control system and generate eye tracking data indicating whether the eye of the user is gazing through the eyepiece; and a processor communicatively coupled to the memory and configured to execute instructions to access the head presence data and the eye tracking data; direct, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is gazing through the eyepiece, the user control system to operate in a first operating mode; and direct, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is not gazing through the eyepiece, the user control system to operate in a second operating mode different from the first operating mode.

An exemplary method may comprise accessing head presence data generated by a head sensor included in a user control system of a computer-assisted surgical system, the head presence data indicating a presence or an absence of a head of a user within a vicinity of a viewer console included in the user control system; accessing eye tracking data generated by an eye sensor included in the user control system, the eye tracking data indicating whether an eye of the user is gazing through an eyepiece included in the user control system; directing, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is gazing through the eyepiece, the user control system to operate in a first operating mode; and directing, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is not gazing through the eyepiece, the user control system to operate in a second operating mode different from the first operating mode.

An exemplary system may comprise a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to access, over time during a surgical session, head presence data indicating a presence or an absence of a head of a user within a vicinity of a viewer console included in a user control system of a computer-assisted surgical system; access, over time during the surgical session, eye tracking data indicating whether an eye of a user is gazing through an eyepiece included in the user control system; access, over time during the surgical session, hand presence data indicating a presence or an absence of a hand of the user within a vicinity of a master control included in the user control system; determine, based on a combination of the head presence data, the eye tracking data, and the hand presence data, an intent of the user to interact with the user control system; and direct the user control system to facilitate user interaction, during the surgical session, in accordance with determined intent of the user.

Exemplary operating mode control systems and methods are described herein. An exemplary operating mode control system may access head presence data generated by a head sensor included in a user control system of a computer-assisted surgical system and access eye tracking data generated by an eye sensor included in the user control system. The head presence data may indicate a presence or an absence of a head of a user within a vicinity of a viewer console included in the user control system. The eye tracking data may indicate whether an eye of the user is gazing through an eyepiece included in the user control system. If the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is gazing through the eyepiece, the operating mode control system may direct the user control system to operate in a first operating mode. If the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is not gazing through the eyepiece, the operating mode control system may direct the user control system to operate in a second operating mode different from the first operating mode.

To illustrate, while a surgeon is positioned at a user control system and is viewing, through a set of eyepieces of the user control system, stereoscopic imagery of a surgical area associated with a patient as generated by a stereoscopic endoscope, the head presence data may indicate that a head of the surgeon is within a vicinity of a viewer console included in the user control system and the eye tracking data may indicate that the eyes of the surgeon are gazing through the eyepieces. The operating mode control system may accordingly direct the user control system to operate in an active operating mode. In the active operating mode the surgeon may manipulate a set of master controls to teleoperate surgical instruments (e.g., to perform a minimally-invasive surgical procedure).

With his or her head still positioned at the user control system, the surgeon may look down and away from the stereoscopic imagery (e.g., at one or more foot pedals or other input devices on the user control system and/or or at his or her hands). In this case, the eye tracking data may indicate that the user's eyes are not gazing through the eyepieces (e.g., that the surgeon is not looking at the stereoscopic imagery). In response, the operating mode control system may direct the user control system to switch to operate in a suspended operating mode. In the suspended operating mode the surgical instruments cannot be controlled by way of the master controls. To regain control of the surgical instruments, the surgeon may return to looking at the stereoscopic imagery by gazing through the eyepieces. In response, the operating mode control system may direct the user control system to switch back to operating in the active operating mode.

Various benefits may be provided by the operating mode control systems and methods described herein. For example, the operating mode control systems and methods described herein combine head proximity sensing with eye sensing in order to distinguish between the intent of a user to interact with imagery generated by a computer-assisted surgical system and the intent of the user to interact with an environment outside of the viewer console. The operating mode control systems and methods may automatically infer the intent of a user and set an operating mode thereof accordingly. Moreover, the operating mode control systems and methods described herein may enable and adjust features of the user control system based on a state of a user's interaction with the user control system. As a result, the user control system may implement safety features while at the same time enabling additional features that may be useful to a surgeon while performing a surgical procedure.

Various embodiments will now be described in more detail with reference to the figures. The systems and methods described herein may provide one or more of the benefits mentioned above and/or various additional and/or alternative benefits that will be made apparent herein.

The operating mode control systems and methods described herein may be implemented as part of or in conjunction with a computer-assisted surgical system. As such, an exemplary computer-assisted surgical system will now be described. The following exemplary computer-assisted surgical system is illustrative and not limiting, as the operating mode control systems and methods described herein may be implemented as part of or in conjunction with other suitable surgical systems.

<FIG> illustrates an exemplary computer-assisted surgical system <NUM> ("surgical system <NUM>"). As shown, surgical system <NUM> may include a manipulating system <NUM>, a user control system <NUM>, and an auxiliary system <NUM> communicatively coupled one to another. In some examples, surgical system <NUM> may be implemented by one or more of these components.

Surgical system <NUM> may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient <NUM>. As shown, the surgical team may include a surgeon <NUM>-<NUM>, an assistant <NUM>-<NUM>, a nurse <NUM>-<NUM>, and an anesthesiologist <NUM>-<NUM>, all of whom may be collectively referred to as "surgical team members <NUM>. " Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation.

While <FIG> illustrates an ongoing minimally invasive surgical procedure, surgical system <NUM> may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system <NUM>. Additionally, it will be understood that the surgical session throughout which surgical system <NUM> may be employed may not only include an operative phase of a surgical procedure, as is illustrated in <FIG>, but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques are used on a patient to investigate, diagnose, or treat a physical condition of the patient. Additionally, a surgical procedure may include any procedure that is not performed on a live patient, such as a calibration procedure, a training procedure, and an experimental or research procedure.

As shown in <FIG>, manipulating system <NUM> may include a plurality of manipulator arms <NUM> (e.g., manipulator arm <NUM>-<NUM> through <NUM>-<NUM>) to which a plurality of surgical instruments (not shown) may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue-interaction functions), medical tool, monitoring instrument (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure (e.g., by being at least partially inserted into patient <NUM> and manipulated to perform a computer-assisted surgical procedure on patient <NUM>). While manipulating system <NUM> is depicted and described herein as including four manipulator arms <NUM>, it will be recognized that manipulating system <NUM> may include only a single manipulator arm <NUM> or any other number of manipulator arms as may serve a particular implementation.

Manipulator arms <NUM> and/or surgical instruments attached to manipulator arms <NUM> may include one or more displacement transducers, orientational sensors, and/or positional sensors used to generate raw (i.e., uncorrected) kinematics information (hereinafter "surgical system sensors"). One or more components of surgical system <NUM> may be configured to use the kinematics information to track (e.g., determine positions of) and/or control the surgical instruments.

Surgical instruments attached to manipulator arms <NUM> may each be positioned at a surgical area associated with a patient. A "surgical area" may, in certain examples, be entirely disposed within a patient and may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure being performed on tissue internal to a patient, the surgical area may include the tissue, anatomy underlying the tissue, as well as space around the tissue where, for example, surgical instruments being used to perform the surgical procedure are located. In other examples, a surgical area may be at least partially disposed external to the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed on the patient. For instance, surgical system <NUM> may be used to perform an open surgical procedure such that part of the surgical area (e.g., tissue being operated on) is internal to the patient while another part of the surgical area (e.g., a space around the tissue where one or more surgical instruments may be disposed) is external to the patient. A surgical instrument may be referred to as being positioned or located at or within a surgical area when at least a portion of the surgical instrument (e.g., a distal portion of the surgical instrument) is located within the surgical area.

User control system <NUM> may be configured to facilitate control by surgeon <NUM>-<NUM> of manipulator arms <NUM> and surgical instruments attached to manipulator arms <NUM>. For example, surgeon <NUM>-<NUM> may interact with user control system <NUM> to remotely move or manipulate manipulator arms <NUM> and the surgical instruments coupled to manipulator arms <NUM>. To this end, user control system <NUM> may provide surgeon <NUM>-<NUM> with imagery (e.g., high-definition stereoscopic imagery) of a surgical area associated with patient <NUM> as captured by an imaging device (e.g., a stereoscopic endoscope). Surgeon <NUM>-<NUM> may utilize the imagery to perform one or more procedures with one or more surgical instruments coupled to manipulator arms <NUM>.

To facilitate control of surgical instruments, user control system <NUM> may include a set of master controls (not shown). These master controls may be manipulated by surgeon <NUM>-<NUM> to control movement of surgical instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon <NUM>-<NUM>. In this manner, surgeon <NUM>-<NUM> may intuitively perform a surgical procedure using one or more surgical instruments.

User control system <NUM> may further be configured to facilitate control by surgeon <NUM>-<NUM> of other components of surgical system <NUM>. For example, surgeon <NUM>-<NUM> may interact with user control system <NUM> to change a configuration or operating mode of surgical system <NUM>, to change a display mode of surgical system <NUM>, to generate additional control signals used to control surgical instruments attached to manipulator arms <NUM>, to facilitate switching control from one surgical instrument to another, or to perform any other suitable operation. To this end, user control system <NUM> may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from surgeon <NUM>-<NUM>.

Auxiliary system <NUM> may include one or more computing devices configured to perform primary processing operations of surgical system <NUM>. The one or more computing devices included in auxiliary system <NUM> may control and/or coordinate operations performed by various other components (e.g., manipulating system <NUM> and/or user control system <NUM>) of surgical system <NUM>. For example, a computing device included in user control system <NUM> may transmit instructions to manipulating system <NUM> by way of the one or more computing devices included in auxiliary system <NUM>. As another example, auxiliary system <NUM> may receive, from manipulating system <NUM> (e.g., from an imaging device), and process image data representative of imagery captured by an endoscope attached to a manipulator arm <NUM>.

In some examples, auxiliary system <NUM> may be configured to present visual content to surgical team members <NUM> who may not have access to the imagery provided to surgeon <NUM>-<NUM> at user control system <NUM>. To this end, auxiliary system <NUM> may include a display monitor <NUM> configured to display one or more user interfaces, such as images (e.g., 2D images) of the surgical area, information associated with patient <NUM> and/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitor <NUM> may display images of the surgical area together with additional content (e.g., graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitor <NUM> is implemented by a touchscreen display with which surgical team members <NUM> may interact (e.g., by way of touch gestures) to provide user input to surgical system <NUM>.

While auxiliary system <NUM> is shown in <FIG> as a separate system from manipulating system <NUM> and user control system <NUM>, auxiliary system <NUM> may be included in, or may be distributed across, manipulating system <NUM> and/or user control system <NUM>.

Manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may be communicatively coupled one to another in any suitable manner. For example, as shown in <FIG>, manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may be communicatively coupled by way of control lines <NUM>, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc..

<FIG> illustrates an exemplary user control system <NUM> that may be used in accordance with the systems and methods described herein to facilitate control of various operations of a computer-assisted surgical system (e.g., surgical system <NUM>). In some examples, user control system <NUM> implements user control system <NUM>.

As shown, user control system <NUM> includes a display module <NUM>, a set of master controls <NUM> (e.g., master control <NUM>-L and master control <NUM>-R), and a set of foot pedals <NUM> (e.g., foot pedals <NUM>-<NUM> through <NUM>-<NUM>). User control system <NUM> may include additional or alternative components as may serve a particular implementation. For example, user control system <NUM> may include various computing components (e.g., processors, memory, etc.), support structures (e.g., a base, a column, etc.), adjustment mechanisms (e.g., pivots, motors, etc.), and the like.

As shown, display module <NUM> includes an image display system <NUM>, a viewer console <NUM>, and eyepieces <NUM> (e.g., eyepiece <NUM>-L and eyepiece <NUM>-R). Display module <NUM> may also include one or more head sensors (not shown in <FIG>) configured to detect a presence of a head of a user within a vicinity of viewer console <NUM>, and one or more eye sensors (not shown in <FIG>) configured to track an eye of the user (e.g., detect a presence of the user's eye within a viewing range of an eyepiece <NUM> and/or detect a gaze direction of the user's eye).

Image display system <NUM> is configured to present imagery generated by a surgical system (e.g., surgical system <NUM>), such as imagery of a surgical area associated with a patient (e.g., patient <NUM>). <FIG> shows an exemplary image display system <NUM> that may be included in display module <NUM> to provide a user with stereoscopic imagery of a surgical area associated with a patient as generated by a stereoscopic endoscope. As shown in <FIG>, image display system <NUM> includes display devices <NUM> (e.g., left display device <NUM>-L and right display device <NUM>-R), mirrors <NUM> (e.g., left mirror <NUM>-L and right mirror <NUM>-R), eyepieces <NUM> (e.g., left eyepiece <NUM>-L and right eyepiece <NUM>-R), and eye sensor <NUM>. Image display system <NUM> may also include additional or alternative components, such as one or more optics (e.g., lenses, filters, polarizers, light guides, etc.), as may suit a particular implementation.

Display devices <NUM> may display imagery generated by surgical system <NUM>, such as imagery of a surgical area associated with a patient. In some examples, display devices <NUM> may also display supplemental visual content concurrently with the imagery of the surgical area associated with the patient. Such supplemental visual content may include, for example, other medical imagery (e.g., imagery generated by ultrasound imaging, computed tomography (CT), optical coherence tomography (OCT), magnetic resonance imaging (MRI), and the like), contextual information about surgical system <NUM> and/or the surgical procedure, patient information, and the like. Imagery of the surgical area associated with the patient may be presented in a main area of display devices <NUM>, and the supplemental visual content may be displayed, for example, in a peripheral area of display devices <NUM>.

Display devices <NUM> may be implemented by any suitable display devices configured to emit visible light representative of imagery generated by surgical system <NUM> and/or supplemental visual content. For example, display devices <NUM> may be implemented by a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED display, a digital micromirror display (DMD), and the like. Display device <NUM>-L emits visible light <NUM>-L representative of a left image toward mirror <NUM>-L, and display device <NUM>-R emits visible light <NUM>-R representative of a right image toward mirror <NUM>-R. Mirror <NUM>-L and mirror <NUM>-R reflect visible light <NUM>-L and visible light <NUM>-R, respectively, toward eyepiece <NUM>-L and eyepiece <NUM>-R.

Eyepieces <NUM> are configured to enable a user to view the imagery presented by display devices <NUM>. Eyepieces <NUM> receive visible light <NUM> from mirrors <NUM> and direct visible light <NUM> to distal ends of eyepieces <NUM>. When a user is gazing into eyepieces <NUM>, the user may view the imagery presented by display devices <NUM>. Each eyepiece <NUM> may include a housing (e.g., a lens barrel) and one or more optics (e.g., lenses, filters, polarizers, light guides, etc.) within the housing as may suit a particular implementation. In some examples, image display system <NUM> may include an interpupillary adjustment mechanism configured to adjust an interpupillary distance between eyepiece <NUM>-L and eyepiece <NUM>-R. In this way, eyepieces <NUM> may be adjusted to fit the shape and size of the user's face. The interpupillary adjustment mechanism may be implemented by any suitable mechanism(s).

Eye sensor <NUM> is configured to track a user's eye (e.g., detect a presence of a user's eye within a viewing range of an eyepiece <NUM> and/or detect a gaze direction of the user's eye). Eye sensor <NUM> may track the user's eye in any suitable manner and using any suitable eye detection and/or gaze tracking technologies, including but not limited to iris detection, pupil detection, pupil and glint detection, and the like.

For example, as shown in <FIG>, eye sensor <NUM> may include non-visible light sources <NUM> (e.g., non-visible light source <NUM>-L and non-visible light source <NUM>-R) and an imaging device <NUM>. Eye sensor <NUM> may also include or be implemented by any additional or alternative components as may suit a particular implementation, such as an image processing facility, a memory, a computing device (e.g., a computing device included in auxiliary system <NUM>), and the like. Non-visible light sources <NUM> may be configured to emit non-visible light <NUM> (e.g., non-visible light <NUM>-L and non-visible light <NUM>-R), such as infrared (IR) light, toward mirrors <NUM>. Non-visible light <NUM> passes through mirrors <NUM> toward eyepieces <NUM>. As mentioned, mirrors <NUM> are configured to reflect visible light <NUM> from display devices <NUM> toward eyepieces <NUM> and transmit non-visible light <NUM> from eye sensor <NUM> toward eyepieces <NUM>. Mirrors <NUM> may be implemented by any suitable mirror.

Eyepieces <NUM> receive non-visible light <NUM> and direct non-visible light <NUM> to the distal ends of eyepieces <NUM>. If a user is positioned in front of eyepieces <NUM>, non-visible light <NUM> may be reflected by the user's head or by the user's eyes <NUM> (e.g., left eye <NUM>-L and/or right eye <NUM>-R) toward imaging device <NUM>. Imaging device <NUM> is configured to detect non-visible light <NUM> reflected by the user. Imaging device <NUM> may be implemented by any suitable type of camera or other type of image capture device capable of capturing non-visible light <NUM>. In some examples, imaging device <NUM> may include an IR filter configured to transmit only the narrow range of non-visible light <NUM> emitted by non-visible light sources <NUM>.

An eye sensor image may be generated (e.g., by imaging device <NUM> or another computing device communicatively coupled to imaging device <NUM>) based on the non-visible light captured by imaging device <NUM>. The eye sensor image may be used to generate eye tracking data, which indicates either that an eye of the user is gazing through an eyepiece <NUM> or that an eye of the user is not gazing through an eyepiece <NUM>. As will be explained below in more detail, eye tracking data may be used to determine and set an operating mode in which user control system <NUM> is to operate.

In some examples, eye tracking data is generated based on tracking (e.g., detecting) a presence or absence of an eye of a user within a viewing range of an eyepiece <NUM>. The viewing range of an eyepiece <NUM> is a region in front of an eyepiece <NUM> from which the user's eye <NUM> may view, via the eyepiece <NUM>, the imagery presented by display devices <NUM>. Eye <NUM> is determined to be present within the viewing range of an eyepiece <NUM> if, in the eye sensor image, eye <NUM> (e.g., the pupil) is detected within the eyepiece <NUM>. If eye <NUM> is determined to be present, then it is inferred that eye <NUM> is gazing through eyepiece <NUM> to view imagery presented by display device <NUM>. In such case, the eye tracking data indicates that an eye of a user is gazing through an eyepiece. On the other hand, an eye is determined to be absent from within the viewing range of eyepiece <NUM> if, in the eye sensor image, no eye is detected within eyepiece <NUM>. If an eye is determined to be absent, then it is inferred that an eye of a user is not gazing through eyepiece <NUM> to view imagery presented by display device <NUM>. In such case, the eye tracking data indicates that an eye of a user is not gazing through an eyepiece.

Additionally or alternatively, eye tracking data is generated based on tracking a gaze direction of a user's eye. For example, the gaze direction of an eye <NUM>, as detected by eye sensor <NUM>, may indicate that the eye <NUM> is gazing through eyepiece <NUM>. In such case, the eye tracking data indicates that an eye of a user is gazing through an eyepiece. Alternatively, the gaze direction of eye <NUM>, as detected by eye sensor <NUM>, may indicate that the eye <NUM> is not gazing through eyepiece <NUM>, such as when the user is looking downward at master controls <NUM> or foot pedals <NUM>. In such case, the eye tracking data indicates that an eye of a user is not gazing through an eyepiece.

In some embodiments, the gaze direction of the eye <NUM>, as detected by eye sensor <NUM>, may further indicate that the user's eye is gazing at a particular region of imagery presented by display devices <NUM> (e.g., a peripheral area where supplemental visual content is presented rather than a main area where imagery of a surgical area associated with a patient is presented). Accordingly, eye tracking data may additionally indicate a particular region of imagery presented by display devices <NUM> to which the user's gaze is directed.

Exemplary eye sensor images, and eye tracking data that may be generated from such eye sensor images, will now be described with reference to <FIG> illustrates an exemplary eye sensor image 400A as captured by imaging device <NUM>. As shown in <FIG>, eyes <NUM> (e.g., the pupils) are detected within eyepieces <NUM>, but a gaze of eyes <NUM> is directed toward a peripheral region of the imagery presented by the image display system. Accordingly, eye tracking data generated based on eye sensor image 400A indicates that an eye of a user is gazing through an eyepiece. In some examples, the eye tracking data may also specify a particular region of the imagery to which eyes <NUM> are directed (e.g., a peripheral region where supplemental visual content is presented).

<FIG> illustrates another exemplary eye sensor image 400B captured by imaging device <NUM>. As shown, eyes <NUM> are looking away from eyepieces <NUM> (e.g., looking down at foot pedals <NUM>), and eyes <NUM> (e.g., the pupils) are not detected within eyepieces <NUM>. Accordingly, eye tracking data generated based on eye sensor image 400B indicates that an eye of a user is not gazing through an eyepiece.

<FIG> illustrates another exemplary eye sensor image 400C captured by imaging device <NUM>. As shown, no eyes are detected within eyepieces <NUM>. Accordingly, eye tracking data generated based on eye sensor image 400C indicates that no eye of a user is gazing through an eyepiece.

<FIG> illustrates another exemplary eye sensor image 400D captured by imaging device <NUM>. As shown, eyes <NUM> are detected within eyepieces <NUM>, but eyes <NUM> are looking away from eyepieces <NUM> (e.g., looking down at foot pedals <NUM>). If the eye tracking data generated from eye sensor image 400D is based on the presence of eyes <NUM> within eyepieces <NUM>, the eye tracking data indicates that an eye of a user is gazing through an eyepiece. However, if the eye tracking data generated from eye sensor image 400D is additionally or alternatively based on the detected gaze direction of eyes <NUM>, the eye tracking data indicates that an eye of a user is not gazing through an eyepiece.

In some examples, eye tracking data may indicate that an eye of a user is gazing through an eyepiece only when the eye is determined to be gazing through the eyepiece for at least a predetermined period of time (e.g., <NUM> seconds). Additionally or alternatively, eye tracking data may indicate that an eye of a user is not gazing through an eyepiece only when the eye is determined to be not gazing through the eyepiece for at least another predetermined period of time (e.g., <NUM> seconds).

In some examples, eye sensor <NUM> may implement a temporal filter configured to filter out temporary, intermittent loss of eye presence or gaze direction tracking caused by blinking of the eye. Any suitable temporal filter or temporal filtering technique may be used. With this configuration, blinking of an eye will not trigger a change in operating mode of user control system <NUM>.

Referring again to <FIG>, viewer console <NUM> facilitates viewing of the imagery presented by image display system <NUM>. <FIG> illustrates an exemplary viewer console <NUM> that may be included in display module <NUM>. As shown, viewer console <NUM> includes eyepieces <NUM> (e.g., left eyepiece <NUM>-L and right eyepiece <NUM>-R), a headrest <NUM>, and a plurality of head sensors <NUM> (e.g., left head sensor <NUM>-L, center head sensor <NUM>-C, and right head sensor <NUM>-R). While <FIG> shows three head sensors <NUM>, viewer console <NUM> may include any number of head sensors as may suit a particular implementation.

In some examples, eyepieces <NUM> are separate from the eyepieces of image display system <NUM> (e.g., eyepieces <NUM> of image display system <NUM>) but optically aligned with the eyepieces of the image display system. Alternatively, eyepieces <NUM> are a portion of image display system <NUM>. For example, eyepieces <NUM> may be distal end portions of eyepieces <NUM> that protrude from an exterior surface of display module <NUM> (e.g., an exterior surface of viewer console <NUM>).

Headrest <NUM> is located above eyepieces <NUM> such that, when a user is positioned at viewer console <NUM>, the user may rest the user's head on headrest <NUM> while looking into eyepieces <NUM>. In some examples, headrest <NUM> may include a headrest sensor (not shown) configured to detect when a head of a user is resting on headrest <NUM>. A headrest sensor may be implemented by any suitable sensor, such as a force-torque (FT) sensor.

Head sensors <NUM> may be configured to detect a proximity (e.g., distance) of a user's head from head sensors <NUM>. Head sensors <NUM> may be implemented by any suitable sensor. Any suitable range or proximity sensors may be used, including those that operate based on range imaging, triangulation (e.g., stereo triangulation, IR triangulation, etc.), interferometry, ultrasound, laser (e.g., LIDAR), structured light, and time-of-flight (TOF). Head sensors <NUM> may sample at any regular interval as may suit a particular implementation (e.g., <NUM>). Additionally or alternatively to head sensors, a depth camera may be utilized to determine a proximity of the user's head from viewer console <NUM>. Head sensors <NUM> may include or be implemented by any components as may suit a particular implementation, such as an emitter, a receiver, a processing facility, a memory, a computing device (e.g., a computing device included in auxiliary system <NUM>), and the like.

Head sensors <NUM> may be located in viewer console <NUM> at any suitable location. In some examples, head sensors <NUM> are positioned to point toward a user's skin (e.g., temples or cheek bones) when the user's head is positioned within a vicinity of viewer console <NUM>. As shown in <FIG>, head sensor <NUM>-L is positioned on a left side of viewer console <NUM> (e.g., at a position pointing to a left temple or left cheek bone of the user's head when the user's head is positioned in viewer console <NUM>), head sensor <NUM>-C is positioned at a center of viewer console <NUM> (e.g., at a position pointing to a forehead of the user when the user's head is positioned in viewer console <NUM>), and head sensor <NUM>-R is positioned on a right side of viewer console <NUM> (e.g., at a position pointing to a right temple or right cheek bone of the user's head when the user's head is positioned in viewer console <NUM>).

The detection result of head sensors <NUM> may be used to generate head presence data. Head presence data may be representative of a real-time head presence state of user control system <NUM>. Head presence data may indicate a proximity and/or a position of a head of a user relative to viewer console <NUM>. Head presence data may also indicate a presence of a head of a user within a vicinity of viewer console <NUM> or an absence of a head of a user within the vicinity of viewer console <NUM>. In some examples, a head of a user is present within the vicinity of viewer console <NUM> when the head (e.g., a surface of the head, such as the forehead, a temple, or a cheek) is determined to be located within a predetermined distance (e.g., <NUM>) of one or more of head sensors <NUM>. In some examples, to prevent false positive determinations of head presence, a head of a user is determined to be present when the head is determined to be located within a predetermined distance of each of a plurality of head sensors. In additional examples, since a human head is generally laterally symmetric when oriented toward viewer console <NUM>, a head of a user is determined to be present when proximity measurements by a plurality of head sensors are comparable (e.g., the proximity measurements differ by no more than a predetermined tolerance, e.g., <NUM>, <NUM>%, etc.).

In some examples, head presence may also be determined based on a rate of change of proximity as measured by the head sensor(s). For example, a head may be determined to be present only after the rate of change of proximity is at or below a predetermined threshold value. In this way, a head may be determined as not present while the user is moving into position, and then the head may be determined to be present only after the user has settled into position.

In some examples, head presence data may indicate that a head of a user is present within the vicinity of viewer console <NUM> only when the head is determined to be located within the predetermined distance of one or more of head sensors <NUM> for at least a predetermined period of time (e.g., <NUM> seconds). Additionally or alternatively, head presence data may indicate that a head of a user is absent within the vicinity of viewer console <NUM> only when the head is determined to be outside of the predetermined distance of one or more of head sensors <NUM> for at least another predetermined period of time (e.g., <NUM> seconds).

In some examples, the predetermined distance used to determine head presence may be different depending on the operating mode in which the user control system is operating. To illustrate, when the user control system is operating in an inactive operating mode, the predetermined distance may be a first distance (e.g., <NUM>), and when the user control system is operating in an active operating mode, the predetermined distance may be a second distance longer than the first distance (e.g., <NUM>). In this way, a head of the user must be closer to the viewer console in the inactive operating mode than in the active operating mode to trigger a head presence detection. This allows the user to slightly relax his or her posture in the active operating mode after the user has entered into the active operating mode.

Various head positions that may be detected by head sensors <NUM>, and head presence data that may be generated based on such head positions, will now be described with reference to <FIG>. As shown in <FIG>, a head <NUM> of a user <NUM> is positioned in front of a viewer console <NUM> of a user control system. Head <NUM> is resting on headrest <NUM>, and a distance d<NUM> between head sensor <NUM> and head <NUM>, as measured by head sensor <NUM>, is less than a predetermined distance. An eye <NUM> of user <NUM> is gazing into eyepiece <NUM> to view imagery presented by an image presentation system of the user control system. <FIG> is similar to <FIG> except that head <NUM> is in a hovering state above viewer console <NUM>, i.e., head <NUM> is not in contact with headrest <NUM> or any other portion of viewer console <NUM>. However, a distance d<NUM> between head sensor <NUM> and head <NUM> is less than a predetermined distance. In the examples of <FIG>, head presence data generated based on a detection result of head sensor <NUM> indicates that a head of a user is present within a vicinity of the viewer console because head <NUM> is in contact with headrest <NUM> or because head <NUM> is located within the predetermined distance from head sensor <NUM>. In both scenarios, eye tracking data also indicates that an eye of a user is gazing through an eyepiece because eye <NUM> is detected within the viewing range of eyepiece <NUM> and/or is detected to be gazing at imagery presented by the image presentation system.

As shown in <FIG>, a head <NUM> of a user <NUM> is positioned in front of a viewer console <NUM> of a user control system. Head <NUM> is resting on headrest <NUM>, and a distance d<NUM> between head sensor <NUM> and head <NUM> is less than a predetermined distance. An eye <NUM> of user <NUM> is looking away from eyepiece <NUM>. For example, user <NUM> may be looking at foot pedals (e.g., foot pedals <NUM>) of the user control system or at his or her hands. <FIG> is similar to <FIG> except that head <NUM> is in a hovering state above viewer console <NUM>, e.g., head <NUM> is not in contact with headrest <NUM> or any other portion of viewer console <NUM>. However, a distance d<NUM> between head sensor <NUM> and head <NUM> is less than the predetermined distance. In the examples of <FIG>, the head presence data indicates that a head of a user is present within the vicinity of the viewer console <NUM> because head <NUM> is in contact with headrest <NUM> or because head <NUM> is located within the predetermined distance from head sensor <NUM>. In both scenarios, eye tracking data indicates that an eye of a user is not gazing through an eyepiece because eye <NUM> is not detected within the viewing range of eyepiece <NUM> and/or is detected to not be gazing at imagery presented by the image presentation system.

As shown in <FIG>, a head <NUM> of a user <NUM> is positioned in front of a viewer console <NUM> of a user control system. Head <NUM> is in a hovering state above viewer console <NUM>, e.g., head <NUM> is not in contact with headrest <NUM> or any other portion of viewer console <NUM>, and a distance d<NUM> between head sensor <NUM> and head <NUM> is less than the predetermined distance. An eye <NUM> of user <NUM> is looking toward eyepiece <NUM>. In the example of <FIG>, the head presence data indicates that a head of a user is absent within the vicinity of viewer console <NUM> because distance d<NUM> is greater than the predetermined distance. Depending on the sensitivity and/or range of an eye sensor, the eye tracking data could indicate that an eye of a user is either gazing through an eyepiece or not gazing through the eyepiece.

The predetermined distance for determining head presence may be set in any suitable way. In some examples, the predetermined distance may be a fixed value that is set in advance (e.g., prior to shipment or delivery of the computer-assisted surgical system). Additionally or alternatively, the predetermined distance may be manually set by a user. In further examples, because a user will self-regulate the viewing distance so that most or all of the imagery presented by the image display system is visible and not occluded by the exit pupil of eyepieces <NUM>, the predetermined distance may be determined and adjusted automatically (e.g., by user control system <NUM>) based on head presence data tracked over time. For instance, a particular proximity or proximity range for long periods of time, or during periods of time during which surgical procedures are performed, may be indicative of a head present state. Accordingly, the predetermined distance may be set based on tracked head proximity indicative of a head present state.

Referring again to <FIG>, master controls <NUM> (e.g., a left master control <NUM>-L and a right master control <NUM>-R) may be manipulated by surgeon <NUM>-<NUM> to control movement of surgical instruments (e.g., by utilizing robotic and/or teleoperation technology). <FIG> illustrates an exemplary master control <NUM> that may be included in user control system <NUM>. As shown, master control <NUM> is configured to be manipulated by a right hand <NUM> of a surgeon (e.g., surgeon <NUM>-<NUM>). User control system <NUM> may also include a left hand master control configured to be manipulated by a left hand of the surgeon. The left hand master control may be similar to master control <NUM> and therefore discussion of the left hand master control is omitted. As shown, master control <NUM> includes finger loops <NUM> configured to receive a finger and/or thumb of the surgeon. Master control <NUM> may also include a variety of mechanisms (e.g., buttons, levers, joints, pivot points, etc.) as may suit a particular implementation. Master control <NUM> may be configured to detect a variety of hand, wrist, and finger movements by the surgeon to control movement of surgical instruments. Accordingly, the surgeon may manipulate master control <NUM> in various ways and with multiple degrees of freedom in order to telemanipulate a surgical instrument.

In some examples surgical system <NUM> (e.g., manipulating system <NUM>, user control system <NUM>, and/or auxiliary system <NUM>) may receive from master control <NUM> information regarding position, pose, orientation, movement, state, etc. of master control <NUM> and/or information regarding user interaction with master control <NUM>. Based on the information received from master control <NUM>, surgical system <NUM> may track the position, pose, orientation, movement, state, and/or other attributes of master control <NUM>.

In some examples, user control system <NUM> may also include a hand sensor configured to detect a presence of a hand of a user within a vicinity of master control <NUM>. The hand sensor may be implemented by any suitable sensor configured to detect a proximity of a hand to master control <NUM> and/or detect physical contact of a hand of a user with master control <NUM>. Suitable hand sensors may include, but are not limited to, range or proximity sensors, IR beam-break sensor, capacitive touch sensors, force-torque sensors, and the like. Additionally or alternatively to hand sensors, a depth camera may be positioned on user control system <NUM> to determine a proximity of the user's hand from master control <NUM>.

As shown in <FIG>, user control system <NUM> includes a hand sensor <NUM> configured to detect a proximity of hand <NUM> to master control <NUM>. Hand sensor <NUM> may be implemented, for example, by a TOF proximity sensor positioned to face the palm of hand <NUM> when hand <NUM> is gripping or approaching master control <NUM>. Hand sensor <NUM> may be positioned in any suitable location configured to detect a proximity of hand <NUM> (e.g., a palm, fingers, thumb, etc.).

The detection result of a hand sensor may be used to generate hand presence data. Hand presence data may be representative of a real-time hand presence state of user control system <NUM>. Hand presence data may indicate a presence of a hand of a user within a vicinity of master control <NUM> or an absence of a hand of a user within a vicinity of master control <NUM>. In some examples, a hand is present within a vicinity of master control <NUM> only when the hand is detected to be in physical contact with master control <NUM> (e.g., in contact with finger loops <NUM>). In another example, a hand is present within a vicinity of master control <NUM> when the hand is detected to be located within a predetermined distance of master control <NUM>.

In some examples, hand presence data may additionally or alternatively be generated based on kinematic information generated by master control <NUM> regarding a position, pose, orientation, movement, state, etc. of master control <NUM>. For example, a deliberate gesture provided by a user by way of master control <NUM> (e.g., a pinch of finger loops <NUM>) may indicate that a hand of a user is in physical contact with master control <NUM>.

As will be explained below in more detail, when user control system <NUM> is operating in an active operating mode, user control system <NUM> may process the information received from master controls <NUM> to generate information and/or signals to send to manipulator arms <NUM> to cause manipulator arms <NUM> and/or surgical instruments to follow master controls <NUM>, e.g., to operate in accordance with the information received from master controls <NUM>. In this or a similar manner, surgical system <NUM> may translate attributes of master controls <NUM> into corresponding operations of manipulator arms <NUM> and surgical instruments, such as by translating movement of master controls <NUM> into corresponding movement of manipulator arms <NUM> and surgical instruments. In this way, surgical system <NUM> couples master controls <NUM> to manipulator arms <NUM> such that a surgeon may telemanipulate surgical instruments attached to manipulator arms using master controls <NUM>.

In some examples, surgical system <NUM> may require that the user provide user input via master controls <NUM> before the user may operate user control system <NUM> and/or interact with features of user control system <NUM>, such as interact with master controls <NUM> to control surgical instruments, etc. Accordingly, in some examples surgical system <NUM> may require the user to perform a deliberate movement of a master control <NUM> (e.g., a finger pinch, a gesture, a movement in a particular direction or in a particular pattern, etc.) in order to initiate control of a surgical instrument associated with the master control <NUM>. The deliberate movement confirms that the user's hand is present within a vicinity of master control <NUM> (e.g., that the hand is grasping the master control <NUM> and/or the user's fingers are coupled within finger loops of the master control <NUM>). As will be explained below in more detail, upon confirmation of hand presence, user control system <NUM> may operate in an active control state, and surgical instrument control may be suspended and resumed during master clutch and camera control operations without requiring additional deliberate input steps by the user. As will be explained below, the deliberate user input by way of master controls <NUM> may be a form of user validation that confirms that the user is allowed to operate user control system <NUM>.

Foot pedals <NUM> (e.g., foot pedals <NUM>-<NUM> through <NUM>-<NUM>) facilitate control of surgical instruments. While <FIG> shows five foot pedals <NUM>, user control system <NUM> may have fewer or more foot pedals as may suit a particular implementation. Foot pedals <NUM> enable surgeon <NUM>-<NUM> to perform various operations, such as swapping control of surgical instruments, controlling features of an imaging system (e.g., endoscope), and activating surgical instrument features (e.g., energizing a cautery instrument, firing a stapling instrument, etc.).

As shown in <FIG>, user control system <NUM> also includes an armrest <NUM> to support the arms of the user while the user is operating master controls <NUM>.

In some examples, user control system <NUM> may also include one or more auxiliary controls configured to allow a user to control various components or settings of user control system <NUM> and/or surgical system <NUM> other than surgical instruments and/or manipulator arms <NUM>. For example, as shown in <FIG> user control system <NUM> includes a set of controls <NUM> (e.g., soft buttons, hard buttons, knobs, dials, joysticks, etc.) that may be manually operated by the user to effectuate a positional adjustment of one or more components of user control system <NUM>. To illustrate, user control system <NUM> may be configured to adjust a position (e.g., height, extension, tilt, etc.) of one or more components of user control system <NUM> (e.g., display module <NUM>, master controls <NUM>, foot pedals <NUM>, eyepieces <NUM>, armrest <NUM>, etc.) to optimize ergonomics for the user. As shown in <FIG>, controls <NUM> are located on armrest <NUM>. However, controls <NUM> are not limited to this location, and may be located on user control system <NUM> at any other suitable location(s).

Additionally, as shown in <FIG>, user control system <NUM> includes a touchscreen display <NUM> with which a user of user control system <NUM> may view content and interact (e.g., by way of touch gestures) to provide user input to surgical system <NUM>. Touchscreen display <NUM> may present content such as user login information, surgical team member information, settings information (surgical system settings, user control system settings, ergonomic position settings, etc.) and/or any other visual content as may serve a particular implementation. Additionally or alternatively, touchscreen display <NUM> may include an operation panel (e.g., a number pad, a keypad, a set of buttons, etc.) configured to receive user input (e.g., a username, a password, user profile information, user preference information, system settings information, etc.). As shown in <FIG>, touchscreen display <NUM> is positioned at a center portion of armrest <NUM>. However, touchscreen display <NUM> may be positioned on user control system <NUM> at any other location as may suit a particular implementation.

In some examples, surgical system <NUM> may require user authentication before the user may operate user control system <NUM> and/or interact with features of user control system <NUM>, such as interact with controls <NUM>, interact with touchscreen display <NUM>, etc. Accordingly, touchscreen display <NUM> may display an authentication interface, and the user may provide, by way of touchscreen display <NUM>, authentication information (e.g., login name, password, personal identification number (PIN), biometric information (e.g., a fingerprint), etc.). Upon successful authentication of the user, the user may be permitted to operate user control system <NUM>. As will be explained below, user authentication may be an additional or alternative form of user validation that confirms that the user is allowed to operate user control system <NUM>.

To facilitate user interaction with the various input devices included in user control system <NUM> (e.g., master controls <NUM>, foot pedals <NUM>, controls <NUM>, and/or touchscreen display <NUM>), user control system <NUM> may include an illumination system configured to provide task lighting for any one or more of the input devices and/or any other components of user control system <NUM>. The illumination system may include, for example, one or more lights (e.g., LEDs) positioned (e.g., on an underside of display module <NUM>, on armrest <NUM>, etc.) to illuminate each input device. As an example, user control system <NUM> may include a first task light configured to illuminate left master control <NUM>-L, a second task light configured to illuminate right master control <NUM>-R, and a third task light configured to illuminate foot pedals <NUM>.

As will be explained below in more detail, an illumination state of each of the various task lights of the illumination system may be responsive to a detected user presence state and user intent to interact with user control system <NUM>. Accordingly, the illumination state of the illumination system may be adjusted in accordance with a current operating mode of user control system <NUM>.

As mentioned, user control system <NUM> may be configured to operate in a plurality of different operating modes. <FIG> illustrates an exemplary operating mode control system <NUM> ("operating mode system <NUM>") configured to control an operating mode in which a user control system (e.g., user control system <NUM>) is to operate. As shown, operating mode system <NUM> may include, without limitation, a storage facility <NUM> and a processing facility <NUM> selectively and communicatively coupled to one another. Facilities <NUM> and <NUM> may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). In some examples, facilities <NUM> and <NUM> may be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

Storage facility <NUM> may maintain (e.g., store) executable data used by processing facility <NUM> to perform any of the operations described herein. For example, storage facility <NUM> may store instructions <NUM> that may be executed by processing facility <NUM> to perform any of the operations described herein. Instructions <NUM> may be implemented by any suitable application, software, code, and/or other executable data instance.

Storage facility <NUM> may also maintain any data received, generated, managed, used, and/or transmitted by processing facility <NUM>. For example, as will be described below in more detail, storage facility <NUM> may maintain head presence data, eye tracking data, image data, operating mode data, user profile data, and the like.

Processing facility <NUM> may be configured to perform (e.g., execute instructions <NUM> stored in storage facility <NUM> to perform) various processing operations associated with selecting and activating an operating mode of user control system. For example, processing facility <NUM> may access head presence data generated by a head sensor included in a user control system of a computer-assisted surgical system, the head presence data indicating a presence or an absence of a head of a user within a vicinity of a viewer console included in the user control system. Processing facility <NUM> may also access eye tracking data generated by an eye sensor included in the user control system, the eye tracking data indicating whether an eye of a user is gazing through an eyepiece included in the user control system. Processing facility <NUM> may also access hand presence data generated by a hand sensor included in the user control system, the hand presence data indicating a presence or an absence of a hand of the user within a vicinity of a master control included in the user control system. In some examples, processing facility <NUM> may also implement a part of the head sensor, the eye sensor, and/or the hand sensor by generating the head presence data, the eye tracking data, and/or the hand presence data based on the detected signals from the respective sensor.

Based on the head presence data, the eye tracking data, and/or the hand presence data, processing facility <NUM> may select a particular operating mode from among various available operating modes and direct the user control system to operate in accordance with the selected operating mode. These and other operations that may be performed by processing facility <NUM> are described herein.

In some examples, operating mode system <NUM> is implemented entirely by the computer-assisted surgical system itself. For example, operating mode system <NUM> may be implemented by one or more computing devices included in surgical system <NUM> (e.g., in one or more computing devices included within manipulating system <NUM>, user control system <NUM>, and/or auxiliary system <NUM>).

<FIG> illustrates another exemplary implementation <NUM> of operating mode system <NUM>. In implementation <NUM>, a remote computing system <NUM> may be communicatively coupled to surgical system <NUM> by way of a network <NUM>. Remote computing system <NUM> may include one or more computing devices (e.g., servers) configured to perform any of the operations described herein. In some examples, operating mode system <NUM> may be entirely implemented by remote computing system <NUM>. Alternatively operating mode system <NUM> may be implemented by both remote computing system <NUM> and surgical system <NUM>.

Network <NUM> may be a local area network, a wireless network (e.g., Wi-Fi), a wide area network, the Internet, a cellular data network, and/or any other suitable network. Data may flow between components connected to network <NUM> using any communication technologies, devices, media, and protocols as may serve a particular implementation.

Various operations that may be performed by operating mode system <NUM> (e.g., by processing facility <NUM> of operating mode system <NUM>), and examples of these operations, will now be described. It will be recognized that the operations and examples described herein are merely illustrative of the many different types of operations that may be performed by operating mode system <NUM>.

Operating mode system <NUM> may access head presence data. As explained above, head presence data may indicate either a presence of a head of a user within a vicinity of a viewer console included in the user control system or an absence of a head of a user within the vicinity of the viewer console. Head presence data may additionally or alternatively indicate a position or proximity of a user's head with respect to the viewer console. Head presence data may be generated by a head sensor (e.g., one of head sensors <NUM>) included in a user control system of a computer-assisted surgical system, by one or more computing components coupled to the head sensor and included in the computer-assisted surgical system (e.g., auxiliary system <NUM>, processing facility <NUM>, etc.), by a remote computing device (e.g., remote computing system <NUM>), and/or by any other device associated with the computer-assisted surgical system as may serve a particular implementation. In some examples, head presence data is stored in and/or accessed from storage facility <NUM>.

In some examples, head presence data may additionally or alternatively be generated based on the detection result of the eye sensor. In these examples, the presence or absence of a head of the user may be inferred from the presence or absence of an eye of the user or a gaze direction of the eye of the user since, in nearly all cases, the head of the user will be present within the vicinity of the viewer console when the eye of the user is detected to be gazing through the eyepiece. Accordingly, in some examples the user control system may not include any head sensors.

Operating mode system <NUM> may also access eye tracking data. As explained above, eye tracking data may indicate either that an eye of a user is gazing through an eyepiece included in the user control system or that an eye of a user is not gazing through the eyepiece. Eye tracking data may additionally or alternatively indicate a direction of gaze of the user's eye. Eye tracking data may be generated based on a detection result by an eye sensor (e.g., eye sensor <NUM>) included in the user control system. Such eye tracking data may be generated by the eye sensor (e.g. by imaging device <NUM>), by one or more computing components coupled to the eye sensor and included in the computer-assisted surgical system (e.g., auxiliary system <NUM>, processing facility <NUM>, etc.), by a remote computing device (e.g., remote computing system <NUM>), and/or by any other device associated with the computer-assisted surgical system as may serve a particular implementation. In some examples, eye tracking data is stored in and/or accessed from storage facility <NUM>.

In some examples, operating mode system <NUM> may access hand presence data. As explained above, hand presence data may indicate a presence of a hand of a user within a vicinity of a master control included in the user control system or an absence of a hand of a user within the vicinity of the master control. Hand presence data may additionally or alternatively indicate a position, proximity, or contact of a hand with respect to a master control. Hand presence data may be generated by a hand sensor included in a user control system of a computer-assisted surgical system, by one or more computing components coupled to the hand sensor and included in the computer-assisted surgical system (e.g., auxiliary system <NUM>, processing facility <NUM>, etc.), by a remote computing device (e.g., remote computing system <NUM>), and/or by any other device associated with the computer-assisted surgical system as may serve a particular implementation. In some examples, hand presence data is stored in and/or accessed from storage facility <NUM>.

Operating mode system <NUM> may select, based on the accessed head presence data, the accessed eye tracking data, and/or the accessed hand presence data, an operating mode from among various available operating modes for a user control system of a computer-assisted surgical system (e.g., surgical system <NUM>) and direct the user control system to operate in accordance with the selected operating mode. As will be described below, each different operating mode may provide a distinct combination of settings relating to the response of user control system to user input (e.g., manual user input via master controls <NUM>, foot pedals <NUM>, controls <NUM>, and/or touchscreen display <NUM>; voice input via one or more microphones; etc.), the sensitivity of the user control system to user input, information presented to the user by way of the user control system (e.g., visual, audio, and/or haptic information), the availability of user control system features, levels for certain user control system outputs (e.g., illumination levels of illumination system lighting and speaker volume levels), and the like. Some of these settings may be user defined and thus maintained with a user profile.

In some examples, operating mode system <NUM> may be configured to select an operating mode from among an active operating mode, a suspended operating mode, and an inactive operating mode. Examples of various settings and configurations of the active operating mode, suspended operating mode, and inactive operating mode will now be explained. These examples are merely illustrative and are not limiting.

While operating in the active operating mode, the user control system may be configured to enable control of operations performed by the surgical system based on input provided by a user by way of the user control system. For example, surgical instruments coupled to manipulator arms <NUM> may be configured to follow (e.g., mimic) movement of master controls <NUM> and respond to operation of foot pedals <NUM>. To illustrate, when a user moves left master control <NUM>-L to the left and then pinches or squeezes left master control <NUM>-L, a surgical instrument controlled by left master control <NUM>-L (e.g., a grasping instrument) likewise moves to the left and then an end effector of the surgical instrument closes (e.g., grasps). As another example, when a user presses foot pedal <NUM>-<NUM>, an operation of a surgical instrument feature controlled by foot pedal <NUM>-<NUM> is activated (e.g., a stapling instrument is fired or a cautery instrument is energized).

To further enable control of operations performed by the surgical system while user control system is operating in the active operating mode, an image display system of the user control system may be configured to present imagery generated by the surgical system (e.g., imagery of a surgical area associated with a patient).

Additionally, while operating in the active operating mode the user control system may be configured to apply a first configuration for an illumination system of the user control system. The first configuration may specify an illumination level for each task light included in the illumination system. For example, the user control system may set the illumination level of task lights for master controls <NUM> and foot pedals <NUM> to a first level (e.g., a minimum level or a user-specified predetermined level) and turn off turn touchscreen display <NUM>. In this way illumination provided by the illumination system during the active operating mode does not distract the user while the user is viewing imagery presented by the image display system and/or controlling operations performed by the surgical system.

Additionally, while operating in the active operating mode the user control system may be configured to prevent or slow down adjustment of ergonomic adjustments of the user control system, thereby limiting distraction to the user.

While operating in the suspended operating mode, the surgical system may be configured to respond differently, as compared with the active operating mode, to user input provided by way of the user control system (e.g., by way of master controls <NUM> and/or foot pedals <NUM>). For example, user control system <NUM> may be configured to suspend control of operations performed by the surgical system based on input provided by the user by way of user control system <NUM>. For instance, surgical instruments coupled to manipulator arms <NUM> may be configured to not follow movement of master controls <NUM> or respond to operation of foot pedals <NUM>. Rather, the surgical instruments may be configured to remain static and/or inoperable even if the user manipulates master controls <NUM> or operates foot pedals <NUM>. In some examples, user control system <NUM> may suspend the output of information received from master controls <NUM> to manipulating system <NUM> and/or auxiliary system <NUM>.

Additionally or alternatively to suspending control of operations performed by the surgical system, the user control system (e.g., master controls <NUM>, foot pedals <NUM>, etc.) may provide visual, audio, or haptic feedback to the user to indicate to the user that the user control system is operating in the suspended operating mode. For instance, touchscreen display <NUM> may display a notification or message to the user. As another example, a speaker included in user control system <NUM> may output a notification tone or a spoken message. As another example, when a user manipulates master control <NUM>-L while user control system <NUM> is operating in the suspended operating mode, master control <NUM>-L may vibrate and/or remain locked in its present position so that the user cannot move or manipulate master control <NUM>-L.

While operating in the suspended operating mode, the user control system may additionally or alternatively be configured to apply a second configuration for the illumination system of the user control system. For example, the user control system may set the illumination level of master controls <NUM> and foot pedals <NUM> to a second level different than the first level (e.g., a maximum level or another user-specified predetermined level) and turn on touchscreen display <NUM>. To illustrate, when the user looks away from eyepieces <NUM> and toward foot pedals <NUM> to position the user's foot on the correct foot pedal <NUM>, user control system <NUM> may operate in the suspended operating mode and illuminate task lighting for foot pedals <NUM> to aid the user in correctly positioning the user's foot. Similarly, when a head of a user is detected to be present but eyes of the user are not detected to be gazing through an eyepiece and a hand of the user is not detected to be present, user control system <NUM> may operate in the suspended operating mode and illuminate task lighting for master controls <NUM> to aid the user in correctly locating and gripping master controls <NUM>. As another example, the user control system may set the brightness of touchscreen display <NUM> so as to facilitate user interaction with touchscreen display <NUM>.

During the suspended operating mode touchscreen display <NUM> may be configured to display visual content as may suit a particular implementation. For example, touchscreen display <NUM> may display supplemental visual content such as medical imaging, surgical team information, patient information, information associated with the surgical procedure, notifications or messages, instructional content, and the like.

Additionally, while operating in the suspended operating mode the user control system may be configured to allow adjustment of ergonomic adjustments of the user control system. In some examples the speed of ergonomic adjustments may be increased as compared with the speed of ergonomic adjustments made during the active operating mode.

While operating in the suspended operating mode the user control system may also be configured to seamlessly transition to operating in the active operating mode. For example, the user control system may be configured to switch to operating in the active operating mode without requiring -re-validation of the user, as will be explained below in more detail. Additionally or alternatively, the image display system of the user control system may continue presenting imagery generated by the surgical system (e.g., imagery of a surgical area associated with a patient) while operating in the suspended operating mode. Thus, transitioning to operating in the active operating mode does not require re-initiation of the image display system or any associated surgical instrument (e.g., an endoscope).

While operating in the inactive operating mode, the user control system may cease control of operations performed by the surgical system based on input provided by the user by way of the user control system. For instance, surgical instruments coupled to manipulator arms <NUM> may cease following movement of master controls <NUM> and responding to operation of foot pedals <NUM>. The surgical instruments may remain static and/or inoperable even if the user attempts to manipulate master controls <NUM> or operate foot pedals <NUM>. In some examples, master controls <NUM> may be locked so they cannot be moved or otherwise manipulated. Additionally, in some examples the image display system of the user control system may also cease presenting imagery generated by the surgical system.

In some examples, while operating in the inactive operating mode the user control system may additionally or alternatively be configured to apply a third configuration for the illumination system of the user control system. The third configuration may be different from the first and second configurations. For example, the user control system may set the illumination level of task lights for master controls <NUM> and foot pedals <NUM> to a third level (e.g., off) but turn on turn touchscreen display <NUM> and set a screen brightness to a minimum level until a user input operation is received by way of touchscreen display <NUM>.

In additional examples, while operating in the inactive operating mode the user control system may be locked out from automatically switching to operating in the active operating mode or the suspended operating mode. For example, as will be explained below in more detail, switching from operating in the inactive operating mode to operating in the active operating mode and/or the suspended operating mode may be conditioned on successful validation (or re-validation) of the user of the user control system.

While the user control system has been described as being configured to operate in an active operating mode, a suspended operating mode, and an inactive operating mode, the user control system may be configured to operate in any additional operating modes. Other operating modes may be based at least in part on eye gaze information included in eye tracking data, hand presence information (e.g., whether a hand of a user is in physical contact with each master control <NUM>), master control information (e.g., a position, pose, orientation, or state of master controls <NUM>), and the like. For example, operating mode system <NUM> may direct the user control system to operate in an additional suspended operating mode when a head of a user is present and an eye of the user is gazing through an eyepiece but a hand of the user is not present within a vicinity of master controls <NUM>. In such operating mode the user control system may be configured to re-center master controls <NUM>, e.g., reposition each master control <NUM> to an optimal position based on a location of surgical instruments within the imagery presented by image display system <NUM>. Additionally or alternatively, the user control system may be configured to reposition each master control <NUM> to improve ergonomics (e.g., to reposition each master control <NUM> for easier access and control by a user) and/or to prevent collision with the other master control <NUM> or other components of the user control system (e.g., a system enclosure, armrest <NUM>, etc.).

In some examples a computer-assisted surgical system (e.g., surgical system <NUM>) may include multiple user control systems. For example, a first user control system may be used by a student to perform a surgical procedure, and a second user control system may be used by a proctor to monitor and assist with the surgical procedure. In such surgical systems, operating mode system <NUM> may be configured to set the operating mode of the first user control system based on the state of the second user control system. For example, operating mode system <NUM> may direct the first control system to operate in a suspended operating mode when eye tracking data generated by the second user control system indicates that an eye of the user of the second user control system is not gazing through an eyepiece included in the second user control system. In this way, when control of surgical instruments by the student may be suspended while the proctor is not viewing the imagery of the surgical area associated with the patient.

In some examples, operating mode system <NUM> may implement a machine learning model configured to classify a state of a user of the user control system based on head presence data, eye tracking data, and/or hand presence data. The state of the user may then be used to select and set an operating mode of the user control system.

<FIG> illustrates an exemplary configuration <NUM> in which a supervised machine learning model <NUM> is maintained or otherwise accessed by operating mode system <NUM>. Supervised machine learning model <NUM> is supervised in that it is specifically trained with pre-classified data prior to being used by operating mode system <NUM> to determine a state of a user of the user control system.

Supervised machine learning model <NUM> may be maintained by operating mode system <NUM> itself (e.g., by storage facility <NUM>). Alternatively, supervised machine learning model <NUM> may be maintained by a system remote from operating mode system <NUM> and accessed by way of a network.

As shown, supervised machine learning model <NUM> receives head presence data <NUM>, eye tracking data <NUM>, and/or hand presence data <NUM> as input. Head presence data <NUM> may represent a real-time state of a head of a user (e.g., a presence, position, and/or proximity of a head of a user relative to a viewer console included in the user control system). Eye tracking data <NUM> may represent a real-time state of one or more eyes of a user (e.g., a presence and/or direction of gaze of the eye(s)). Hand presence data <NUM> may represent a real-time state of one or more hands of a user (e.g., a presence, position, proximity, and/or contact of the hand(s) relative to a master control device and/or another user input device).

Supervised machine learning model <NUM> may analyze the head state, eye state, and hand state represented by head presence data <NUM>, eye tracking data <NUM>, and hand presence data <NUM> in any suitable manner. For example, supervised machine learning model <NUM> may analyze head presence data <NUM>, eye tracking data <NUM>, and hand presence data <NUM> in accordance with one or more decision tree learning algorithms, association rule learning algorithms, artificial neural network learning algorithms, deep learning algorithms, bitmap algorithms, and/or any other suitable data analysis technique as may serve a particular implementation.

In some examples, supervised machine learning model <NUM> is configured to classify a user state <NUM> based on the head state, eye state, and hand state represented by head presence data <NUM>, eye tracking data <NUM>, and hand presence data <NUM>. The user state <NUM> may be indicative of the user's intent to interact with the user control system. For the example, the user intent may be an intent to control surgical instruments by manipulating master controls <NUM>, intent to adjust ergonomic settings of user control system <NUM>, intent to temporarily pause interaction with user control system <NUM> to converse with other surgical team members, intent to review information provided by a touchscreen display (e.g., touchscreen display <NUM>), intent to interact with foot pedals <NUM>, intent to terminate interaction with user control system <NUM>, and the like. Operating mode system <NUM> may then set an appropriate operating mode of the user control system for the classified user state <NUM>.

Supervised machine learning model <NUM> may be trained in any suitable manner. For example, supervised machine learning model <NUM> may be trained by providing data representative of known user states and/or data representative of known transitions between user states as training inputs to supervised machine learning model <NUM>. Additionally or alternatively, supervised machine learning model <NUM> may be trained based on historical changes in head presence data, eye tracking data, and hand presence data mapped to historical changes in user intent. The training may be performed prior to operating mode system <NUM> using supervised machine learning model <NUM> to classify a user state based on head presence data <NUM>, eye tracking data <NUM>, and hand presence data <NUM>.

Additionally or alternatively, supervised machine learning model <NUM> may be trained while operating mode system <NUM> is using supervised machine learning model <NUM> to classify a user state. For example, in response to a user state classification for a particular combination of head state, eye state, and hand state, operating mode system <NUM> may provide a notification to a user (e.g., to surgeon <NUM>-<NUM> or another surgical team member <NUM>). The user may provide user input (e.g., by selecting an option included in the notification) confirming or refuting the user state classification. Operating mode system <NUM> may receive the user input and provide the user input as a training input to supervised machine learning model <NUM>.

As mentioned, operating mode system <NUM> may select, based on the accessed head presence data and/or the accessed eye tracking data, an operating mode from among various available operating modes and direct the user control system to operate in accordance with the selected operating mode.

<FIG> illustrates an exemplary method <NUM> of setting an operating mode for a user control system of a computer-assisted surgical system. While <FIG> illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in <FIG>.

In operation <NUM>, operating mode system <NUM> accesses head presence data and eye tracking data.

In operation <NUM>, operating mode system <NUM> determines whether the head presence data indicates a presence of a head of a user within a vicinity of a viewer console of the user control system or indicates an absence of a head of a user within the vicinity of the viewer console. If the head presence data indicates an absence of a head of a user within the vicinity of the viewer console, operating mode system <NUM> proceeds to operation <NUM>. In operation <NUM>, operating mode system <NUM> directs the user control system to operate in accordance with an inactive operating mode, and processing then returns to operation <NUM>. However, if the head presence data indicates a presence of a head of a user within the vicinity of the viewer console, operating mode system <NUM> proceeds to operation <NUM>.

In operation <NUM>, operating mode system <NUM> determines whether the eye tracking data indicates that an eye of the user is gazing through an eyepiece included in the user control system or indicates that an eye of the user is not gazing through the eyepiece. If the eye tracking data indicates that an eye of the user is not gazing through the eyepiece, operating mode system <NUM> proceeds to operation <NUM>. In operation <NUM>, operating mode system <NUM> directs the user control system to operate in accordance with a suspended operating mode, and processing then returns to operation <NUM>. However, if the eye tracking data indicates that an eye of the user is gazing through the eyepiece, operating mode system <NUM> proceeds to operation <NUM>.

In operation <NUM>, operating mode system <NUM> directs the user control system to operate in accordance with an active operating mode, and processing then returns to operation <NUM>.

If the head presence data and/or eye tracking data changes while the user control system is operating in a particular operating mode, operating mode system <NUM> may direct the user control system to switch operating modes. <FIG> illustrates an exemplary manner in which the user control system may switch operating modes. As shown, the user control system may operate in an active operating mode <NUM>, a suspended operating mode <NUM>, and an inactive operating mode <NUM>. The user control system may switch between any two operating modes based on a change in the head presence data, the eye tracking data, and/or the hand presence data.

For example, while the user control system is operating in active operating mode <NUM>, a user of the user control system may look down at touchscreen display <NUM> to see which surgical team members have logged in to surgical system <NUM>. Accordingly, operating mode system <NUM> may detect that the head presence data remains the same (the head remains present) but that the eye tracking data switches from indicating that the eye of the user is gazing through the eyepiece to indicating that the eye of the user is not gazing through the eyepiece. In response to the change in eye tracking data, operating mode system <NUM> may direct the user control system to switch from operating in active operating mode <NUM> to operating in suspended operating mode <NUM>. In suspended operating mode <NUM>, a brightness of touchscreen display <NUM> may be increased as compared with the brightness in active operating mode <NUM>.

While the user control system is operating in suspended operating mode <NUM>, operating mode system <NUM> may detect that, while the head presence data remains unchanged, the eye tracking data switches from indicating that the eye of the user is not gazing through the eyepiece to indicating that the eye of the user is gazing through the eyepiece. For example, the user may return to looking into eyepieces <NUM> to view imagery presented by image display system <NUM>. In response to the change in eye tracking data, operating mode system <NUM> may direct the user control system to transition from operating in suspended operating mode <NUM> to operating in active operating mode <NUM>.

While user control system is operating in active operating mode <NUM> or suspended operating mode <NUM>, operating mode system <NUM> may detect that the head presence data switches from indicating that the head of the user is present within the vicinity of the viewer console to indicating that the head of the user is absent within the vicinity of the viewer console. For example, the user may leave the user control system to talk to the new surgical team member. In response to the change in the head presence data, operating mode system <NUM> may direct the user control system to switch from operating in active operating mode <NUM> or suspended operating mode <NUM> to operating in inactive operating mode <NUM>. In inactive operating mode <NUM>, control of surgical instruments by the user control system is terminated, presentation of imagery by the image display system is terminated, and system illumination of the user control system may be set to a standby level.

While user control system is operating in inactive operating mode <NUM>, operating mode system <NUM> may detect that the head presence data switches from indicating that the head of the user is absent within the vicinity of the viewer console to indicating that the head of the user is present within the vicinity of the viewer console. For example, the user may return to the user control system to recommence the surgical procedure. In response to the change in the head presence data, operating mode system <NUM> may direct the user control system to switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> or suspended operating mode <NUM>, in accordance with an eye state.

In some examples, the user control system may switch between any two operating modes only if a change in the head presence data and/or the eye tracking data persists for a predetermined time period. For example, the user control system may switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> or suspended operating mode <NUM> only after the head presence data indicates that the head of the user is present within the vicinity of the viewer console for at least a predetermined time period (e.g., <NUM> seconds). As another example, the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> only after the eye tracking data indicates that the eye of the user is gazing through the eyepiece for at least another predetermined time period (e.g., <NUM> seconds).

In some examples, operating mode system <NUM> may use eye tracking data to determine head presence. As explained above, the presence or absence of a head of a user may be inferred from the presence or absence and/or gaze direction of an eye of the user. For example, while the user control system is operating in inactive operating mode <NUM>, operating mode system <NUM> may detect that the eye tracking data switches from indicating that the eye of the user is not gazing through the eyepiece to indicating that the eye of the user is gazing through the eyepiece and thereby determine that a head of a user is present within the vicinity of the viewer console. In response to the change in the eye tracking data, operating mode system <NUM> may direct the user control system to switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM>.

In some examples, operating mode system <NUM> may require head presence data to validate an initial head presence determination (e.g., when switching from operating in inactive operating mode <NUM> to operating in active operating mode <NUM>). Once head presence has been successfully determined and validated based on both eye tracking data and head presence data, operating mode system <NUM> may determine head presence based only on eye tracking data such that a change in head presence data does not result in a change in the head presence state. In this way, operating mode system <NUM> may prevent undesired changes in operating mode and interruption of control due to false negative determinations of loss of head presence based on head presence data. Such false negatives may arise, for example when the user slightly relaxes his or her head position in the viewer console while still looking into the eyepieces, or when hair or skin color result in a loss of head presence.

In some examples, as shown in <FIG>, the user control system may switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> only after first switching to operating in suspended operating mode <NUM>. In some examples, the user control system must operate in the suspended operating mode <NUM> for at least a predetermined period of time before it may switch to operating in active operating mode <NUM>. In this way, operation in active operating mode <NUM> may be delayed until the user has had a sufficient amount of time to become situated with the user control system while the user control system is operating in suspended operating mode <NUM>. For example, while in the suspended operating mode the user may adjust settings of user control system (e.g., ergonomic position settings, etc.) and interact with touchscreen display <NUM> to view information about surgical team members currently logged in to surgical system <NUM>.

In some examples, a user must be validated prior to the user control system switching from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> and/or suspended operating mode <NUM>. <FIG> illustrates an exemplary manner of implementing user validation to switch operating modes. <FIG> is similar to <FIG>, except that the user control system may only switch from operating in inactive operating mode <NUM> to operating in suspended operating mode <NUM> upon a successful validation <NUM> of the user of the user control system. As explained above, user validation <NUM> may include receiving a deliberate user input provided by way of a master control and/or authentication of the user. For example, in response to a change in head presence data while the user control system is operating in inactive operating mode <NUM>, touchscreen display <NUM> may present a user login interface by which the user may provide user authentication information (e.g., login name and password). Upon successful authentication of the information provided by the user, the user control system may switch to operating in suspended operating mode <NUM>. As another example, while the user control system is operating in inactive operating mode <NUM>, a user may make a deliberate gesture (movement) via master controls <NUM>. In response to this deliberate gesture, the user control system may switch to operating in suspended operating mode <NUM>. In some examples, switching to operating in suspended operating mode <NUM> from inactive operating mode <NUM> may also be conditioned on a determination that a head of the user is present.

As shown in <FIG>, the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> without further user validation <NUM>. For example, while operating in suspended operating mode <NUM>, the user control system may keep the user logged in, thereby enabling user interaction with certain features of the user control system <NUM>. In this way, operating mode system <NUM> may direct the user control system to seamlessly transition from suspended operating mode <NUM> to active operating mode <NUM> while the head of the user is present within a vicinity of a viewer console of the user control system.

<FIG> illustrates another exemplary manner of implementing user validation to switch operating modes. <FIG> is similar to <FIG>, except that the user control system may switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> only with successful validation <NUM> of the user of the user control system. In addition, the user control system is not configured to switch from operating in inactive operating mode <NUM> to operating in suspended operating mode <NUM>, thereby ensuring that the user control system operates in active operating mode <NUM> only with successful user validation <NUM>. However, the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> without user validation <NUM>. In this way, the user control system may seamlessly transition between active operating mode <NUM> and suspended operating mode <NUM> while the head of the user is in a present state.

<FIG> illustrates another exemplary manner of implementing user validation to switch operating modes. <FIG> is similar to <FIG> and <FIG>, except that the user control system may switch from operating in inactive operating mode <NUM> to operating in either active operating mode <NUM> or suspended operating mode <NUM> only with successful validation <NUM> of the user of the user control system. However, the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> without user validation <NUM>, thereby facilitating a seamless transition from suspended operating mode <NUM> to active operating mode <NUM>.

<FIG> illustrates another exemplary manner of implementing user validation to switch operating modes. As shown, the user control system may switch from operating in inactive operating mode <NUM> to operating in active operating mode <NUM> (either directly or by way of suspended operating mode <NUM>) only with successful user validation <NUM>. Additionally, if the user control system entered suspended operating mode <NUM> directly from inactive operating mode <NUM>, then the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> only with successful validation <NUM> of the user. However, if the user control system entered suspended operating mode <NUM> from active operating mode <NUM>, then the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> without user validation <NUM>. With this configuration, the user control system may enter active operating mode <NUM> only after successful user validation <NUM>, but after user validation <NUM> the user control system may seamlessly transition from suspended operating mode <NUM> to active operating mode <NUM> while the head of the user is present.

While operating in the suspended operating mode, the user control system may also keep the user logged in (or may automatically re-authenticate the user), thereby enabling user interaction with certain features of user control system <NUM>. For example, the user may adjust settings of user control system (e.g., ergonomic position settings, etc.) or interact with touchscreen display <NUM> to view information about surgical team members <NUM> currently logged in to surgical system <NUM>.

In any of the examples described above, the operating mode may be set based on hand presence data and/or master control data indicative of a position, pose, or orientation of master controls (e.g., master controls <NUM>) in addition to head presence data and/or eye tracking data. For example, the user control system may switch from operating in suspended operating mode <NUM> to operating in active operating mode <NUM> only if hand presence data indicates that a hand of a user is present within a vicinity of master controls (e.g., master controls <NUM>) of the user control system. In another example, the user control system may switch from operating in active operating mode <NUM> to operating in suspended operating mode <NUM> when head presence and eye presence (or eye gaze through eyepieces) are lost so long as hand presence persists. For example, if the user moves his or her head and eyes away from viewer console <NUM> in order to converse with another surgical team member but keeps his or her hands in physical contact with master control <NUM>, the user control system would switch to operate in suspended operating mode <NUM> instead of switching to operating in inactive operating mode <NUM>. If, however, the user also removes his or her hand from master controls <NUM> such that hand presence is also lost, the user control system would switch to operating in inactive operating mode <NUM>.

As a further example, the user control system may switch to operating in active operating mode <NUM> only if a state (e.g., a position, pose, orientation, etc.) of the set of master controls matches a state (e.g., a position, pose, orientation, etc.) of the surgical instrument. To illustrate, while user control system <NUM> is operating in active operating mode <NUM>, the user may manipulate (e.g., pinch) left master control <NUM>-L to close a stapling instrument on tissue to be stapled. While the stapling instrument is closed on the tissue, the user may look down at foot pedals <NUM> to locate the appropriate foot pedal <NUM> to fire the stapling instrument. While the user is looking down, user control system <NUM> may switch to operating in suspended operating mode <NUM>. While in suspended operating mode <NUM>, the stapling instrument cannot move but remains in a closed position on the tissue to be stapled. After locating the appropriate foot pedal <NUM>, the user may then return to looking into eyepieces <NUM> to view the imagery of the surgical area. If the master control data indicates that the user's hand is not in physical contact with left master control <NUM>-L, the user control system does not switch to operating in active operating mode <NUM> until the master control data indicates that the user's hand is in physical contact with left master control <NUM>-L. In additional examples, the user control system does not switch to operating in active operating mode <NUM> until the master control data indicates that a state of left master control <NUM>-L matches a state of the stapling instrument (e.g., left master control <NUM>-L is in a closed state).

<FIG> illustrates an exemplary method <NUM> of setting an operating mode for a user control system of a computer-assisted surgical system. While <FIG> illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in <FIG>. One or more of the operations shown in <FIG> may be performed by operating mode system <NUM>, any components included therein, and/or any implementation thereof.

In operation <NUM>, an operating mode control system accesses head presence data generated by a head sensor included in a user control system of a computer-assisted surgical system. The head presence data indicates a presence or an absence of a head of a user within a vicinity of a viewer console included in the user control system. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the operating mode control system accesses eye tracking data generated by an eye sensor included in the user control system. The eye tracking data indicates whether an eye of a user is gazing through an eyepiece included in the user control system. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the operating mode control system directs, if the head presence data indicates that the head of the user is present within a vicinity of the viewer console and the eye tracking data indicates that the eye of the user is gazing through the eyepiece, the user control system to operate in a first operating mode (e.g., an active operating mode). Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the operating mode control system directs, if the head presence data indicates that the head of the user is present within the vicinity of the viewer console and the eye tracking data indicates that the eye of the user is not gazing through the eyepiece, the user control system to operate in a second operating mode (e.g., a suspended operating mode) different from the first operating mode. Operation <NUM> may be performed in any of the ways described herein.

<FIG> illustrates an exemplary computing device <NUM> that may be specifically configured to perform one or more of the processes described herein. As shown in <FIG>, computing device <NUM> may include a communication interface <NUM>, a processor <NUM>, a storage device <NUM>, and an input/output ("I/O") module <NUM> communicatively connected one to another via a communication infrastructure <NUM>. While an exemplary computing device <NUM> is shown in <FIG>, the components illustrated in <FIG> are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device <NUM> shown in <FIG> will now be described in additional detail.

One or more I/O modules may be used to receive input for a single virtual experience. For example, I/O module <NUM> may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or IR receiver), motion sensors, and/or one or more input buttons.

In some examples, any of the systems, computing devices, and/or other components described herein may be implemented by computing device <NUM>. For example, processing facility <NUM> may be implemented by processor <NUM> and storage facility <NUM> may be implemented by storage device <NUM>.

Claim 1:
A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to perform a process comprising:
accessing (<NUM>) head presence data indicating a presence or an absence of a head of a user within a vicinity of a viewer console included in a user control system of a computer-assisted surgical system;
accessing (<NUM>) eye tracking data indicating whether an eye of the user is gazing through an eyepiece at a display device included in the user control system;
selecting (<NUM>, <NUM>), based on a combination of the head presence data and the eye tracking data, a first operating mode from among a plurality of operating modes for the user control system; and
directing (<NUM>, <NUM>) the user control system to operate in the first operating mode.