Patent Description:
A surgical system has been devised in which a surgical system controls a surgical apparatus by inputting such contactless information as voices, gestures, and lines of sight (refer to PTL <NUM>, for example). With such a surgical system, an operator for whom the practicing of sterilization measures is essential is able to control a surgical apparatus without touching the manipulation buttons and other controls.

However, as compared with inputs by touch, contactless inputs may cause the erroneous recognition of inputs, thereby making a surgical apparatus operate in an erroneous manner. With a surgical system, any erroneous operation of a surgical apparatus affects the life of a patient, so that it is essential to prevent any erroneous operation of the surgical apparatus.

Therefore, in controlling surgical apparatuses by contactless inputs, the realization of fail-safe is demanded so as to prevent erroneous operations of the surgical apparatuses.

The present disclosure, executed in consideration of the above-mentioned situations, is intended to prevent erroneous operations of surgical apparatuses when surgical apparatuses are controlled by contactless inputs.

The invention is defined by the independent claims, with various optional features being set out in the dependent claims.

According to one aspect of the present disclosure, a surgical apparatus can be controlled. In addition, according to another aspect of the present disclosure, an erroneous operation of a surgical apparatus can be prevented when the surgical apparatus is controlled with contactless inputs.

It should be noted that the effects described here are not necessarily restricted; namely, any of the effects described in the present disclosure may be effects denoted here.

The following describes modes (hereafter referred to as embodiments) of executing the present disclosure. It should be noted that the description will be done in the following sequence.

<FIG> is a block diagram illustrating one example of a configuration of a surgical system practiced as a first embodiment to which the present disclosure is applied.

A surgical system <NUM> has a surgical camera <NUM>, a camera arm <NUM>, an action recognition camera <NUM>, a display <NUM>, a control apparatus <NUM>, an operating table <NUM>, surgical glasses <NUM>, a microphone <NUM>, a marker <NUM>, and a foot switch <NUM>. The surgical system <NUM> is arranged in an operating room or the like and enables such treatments as surgical operations and the like that reference images taken with the surgical camera <NUM>.

To be more specific, the surgical camera <NUM> (a surgical imaging apparatus) of the surgical system <NUM> is a modality device such as a 3D camera or the like held by the camera arm <NUM>. The surgical camera <NUM> takes an image of the surgical field of a patient <NUM> lying on the operating table <NUM> and transmits a resultant 3D image to the control apparatus <NUM> as a surgical field image. The camera arm <NUM> holds the surgical camera <NUM> so as to control the position and the angle of the surgical camera <NUM>.

The action recognition camera <NUM> is a 2D camera, for example, and arranged on top of the display <NUM>. The action recognition camera <NUM> takes an image of an operator <NUM> who wears the surgical glasses <NUM>, the microphone <NUM>, and the marker <NUM> on the head 22A. The action recognition camera <NUM> transmits a 2D image obtained as a result of imaging to the control apparatus <NUM> as an operator image.

The display <NUM> is a 3D display having a comparatively large screen and arranged at a position (in the example depicted in <FIG>, a position directly opposite to the operator <NUM> with the operating table <NUM> in between) comparatively far from the operator <NUM>. Surgical field images and the like sent from the control apparatus <NUM> are displayed.

The control apparatus <NUM> sets an operation mode to a manual mode or a hands-free mode. In the manual mode, the surgical system <NUM> is controlled on the basis of the input (force application to the camera arm <NUM> and an operation of manipulation buttons and the other controls, not depicted, installed on each of the component blocks, for example) by the hands of the operator <NUM>. In the hands-free mode, the surgical system <NUM> is controlled on the basis of the contactless input of voice, line of sight, movement and direction of the head 22A, and gesture that are independent of the hands of the operator <NUM> and on the basis of the input by the contact of a leg 22B onto the foot switch <NUM>.

The control apparatus <NUM> receives an operator image sent from the action recognition camera <NUM> and detects the position of the marker <NUM> worn on the head 22A of the operator <NUM> within the operator image, thereby detecting the movement of the head 22A (head tracking) and recognizing the direction of the head 22A. Further, the control apparatus <NUM> recognizes a gesture done by the operator <NUM> from the operator image.

The control apparatus <NUM> receives the information indicative of the direction of the line of sight of the operator <NUM> sent from the surgical glasses <NUM> and, the basis of this information and the position and direction of the head 22A, recognizes the position of the line of sight on the screen of the display <NUM>. The control apparatus <NUM> receives a voice sent from the microphone <NUM> so as to execute voice recognition on that voice. The control apparatus <NUM> receives, from the foot switch <NUM>, a manipulation signal indicative of a manipulation done on the foot switch <NUM> and, on the basis of that manipulation signal, recognizes the contents of the manipulation done on the foot switch <NUM>.

Further, if the operation mode is the hands-free mode, the control apparatus <NUM> uses, as input information, the movement and direction of the head 22A, a gesture of the operator <NUM>, the line-of-sight positional information indicative of the position of a line of sight on the screen of the display <NUM>, voice recognition results, sound volume, and the manipulation information indicative of the contents of a manipulation done on the foot switch <NUM>. On the basis of the input information, the control apparatus <NUM> recognizes a command from the operator <NUM> and a state of the operator <NUM>.

In accordance with a state of the operator <NUM>, the control apparatus <NUM> permits a command from the operator <NUM>. In accordance with the permitted command, the control apparatus <NUM> controls the imaging by the surgical camera <NUM>, controls the driving of the camera arm <NUM>, controls the displaying of the display <NUM>, and changes operation modes.

The surgical glasses <NUM> are worn around the head 22A of the operator <NUM> and include a 3D polarized glasses and a line-of-sight detection device. The operator <NUM> can look at the display <NUM> through the 3D polarized glasses of the surgical glasses <NUM>, thereby recognizing a surgical field image displayed on the display <NUM> as a 3D image.

Further, by seeing surroundings through the surgical glasses <NUM>, the operator <NUM> enters the line of sight into the surgical glasses <NUM>. A line-of-sight device of the surgical glasses <NUM> detects the line of sight of the operator <NUM> and transmits the information indicative of the direction of the line of sight to the control apparatus <NUM>.

The microphone <NUM> is worn on the head 22A of the operator <NUM>. The microphone picks up a surrounding voice including a voice and so on of the operator <NUM> and transmits the picked-up voice to the control apparatus <NUM>. The marker <NUM> is worn on the head 22A of the operator <NUM>.

The foot switch <NUM> is arranged around the operator <NUM> and manipulated by the contact of the leg 22B of the operator <NUM>. The foot switch <NUM> transmits a manipulation signal indicative of a manipulation done by the leg 22B of the operator <NUM> to the control apparatus <NUM>.

With the surgical system <NUM> as described above, the operator <NUM> lays the patient <NUM> on the operating table <NUM> and executes treatment such as a surgical operation while looking through the surgical glasses <NUM> at a surgical field image and so on displayed on the display <NUM>.

In addition, when the operation modes, the imaging conditions of the surgical camera <NUM>, the positions and angles of the surgical camera <NUM>, the displays of the display <NUM> or the like are changed, the operator <NUM> executes a contactless input operation or contact foot input operation. Therefore, the operator <NUM> is able to executes an input operation with a surgical tool, not depicted, held in the hand. It should be noted that the operator <NUM> need not execute sterilization processing every time the operator <NUM> executes an input operation.

It should also be noted that, for a line-of-sight detection method, a method of detecting the movement and direction of the head 22A and a gesture of the operator <NUM>, and a method of obtaining voice, any method may be employed. For example, the line-of-sight detection device or the microphone <NUM> may be not a wearable device.

In the present description, the horizontal direction of the display <NUM> is referred to as x direction, the vertical direction is referred to as y direction, and the direction perpendicular to the screen of the display <NUM> is referred to as z direction.

<FIG> is a diagram illustrating the driving of the surgical camera <NUM> by the camera arm <NUM> of <FIG>.

As depicted in A of <FIG>, the camera arm <NUM> can make the surgical camera <NUM> execute a pivot movement for changing the imaging angles without changing the imaging center. To be more specific, the camera arm <NUM> can move the surgical camera <NUM> so as to always keep constant the distance from center P of a surgical field that is a target of the imaging by the surgical camera <NUM>. This setup allows the surgical camera <NUM> to take surgical field images that are same in center P of the surgical field but different in the imaging angle.

Further, as depicted in B of <FIG>, the camera arm <NUM> is capable of making the surgical camera <NUM> execute a slide movement in the x direction in which the imaging center is moved in the x direction. Specifically, the camera arm <NUM> is capable of moving the surgical camera <NUM> in the x direction. This setup allows the surgical camera <NUM> to move center P of the surgical field that is a target of imaging along the x direction.

Further, although not depicted, the camera arm <NUM> is capable of making the surgical camera <NUM> execute a slide movement in the y direction or the z direction. If the surgical camera <NUM> executes a slide movement in the y direction, the surgical camera <NUM> can zoom in or zoom out an imaging range. In addition, if the surgical camera <NUM> executes a slide movement in the z direction, the surgical camera <NUM> can move center P of the surgical field along the z direction.

It should be noted that, in the present description, it is assumed that a slide movement of the surgical camera <NUM> be executed by the movement of the surgical camera <NUM> by the camera arm <NUM>; however, it is also practicable to execute a slide movement by changing the imaging angles of the surgical camera <NUM> by the camera arm <NUM>.

<FIG> is a block diagram illustrating one example of the configuration of the control apparatus <NUM> depicted in <FIG>.

The control apparatus <NUM> depicted in <FIG> has a recognition block <NUM>, a command block <NUM>, a mode setting block <NUM>, a state estimation block <NUM>, a control block <NUM>, and an image processing block <NUM>.

The recognition block <NUM> of the control apparatus <NUM> has a voice recognition block <NUM>, a line-of-sight recognition block <NUM>, a head recognition block <NUM>, a gesture recognition block <NUM>, and a manipulation recognition block <NUM>.

The voice recognition block <NUM> (a contactless input recognition block) executes voice recognition on a voice sent from the microphone <NUM> so as to recognize a speech as the contactless input by the operator <NUM> (the user). In addition, the voice recognition block <NUM> recognizes the volume of a voice sent from the microphone <NUM> as the contactless input by the operator <NUM>. The voice recognition block <NUM> supplies the speech and volume that are results of the voice recognition to the command block <NUM> as input information.

The line-of-sight recognition block <NUM> (a contactless input recognition block) recognizes the position of the line of sight on the screen of the display <NUM> as the contactless input by the operator <NUM> on the basis of the information indicative of the direction of line of sight sent from the surgical glasses <NUM> and the position and direction of the head 22A recognized by the head recognition block <NUM>. The line-of-sight recognition block <NUM> supplies the line-of-sight positional information indicative of the position thereof to the command block <NUM>, the state estimation block <NUM>, and the image processing block <NUM> as input information.

The head recognition block <NUM> (a contactless input recognition block) detects the position of the marker <NUM> inside an operator image from the operator image sent from the action recognition camera <NUM> so as to recognize the position, movement and direction of the head 22A of the operator <NUM> as the contactless input by the operator <NUM>. The head recognition block <NUM> supplies the movement and direction of the head 22A to the command block <NUM> and the state estimation block <NUM> as input information. In addition, the head recognition block <NUM> supplies the position and direction of the head 22A to the line-of-sight recognition block <NUM>.

The gesture recognition block <NUM> (a contactless input recognition block) recognizes, as the contactless input from the operator <NUM>, the input of a gesture done by the operator <NUM> from an operator image sent from the action recognition camera <NUM>. The gesture recognition block <NUM> supplies the gesture done by the operator <NUM> to the command block <NUM> as input information.

The manipulation recognition block <NUM> (a contact input recognition block) receives a manipulation signal sent from the foot switch <NUM> and recognizes the contents of the manipulation done on the foot switch <NUM> as the contact input from the operator <NUM>. The manipulation recognition block <NUM> supplies the manipulation information indicative of the contents of that manipulation to the command block <NUM> as input information.

On the basis of the input information supplied from the recognition block <NUM>, the command block <NUM> recognizes a command issued from the operator <NUM>. If the recognized command is a command for changing operation modes, then the command block <NUM> notifies the mode setting block <NUM> of that command.

On the other hand, if the recognized command issued from the operator <NUM> is not a command for changing operation modes, then the command block <NUM> (a restriction block) restricts the command issued from the operator <NUM> in accordance with a state supplied from the state estimation block <NUM>. That is, in accordance with the state supplied from the state estimation block <NUM>, the command block <NUM> permits a predetermined command issued from the operator <NUM>. The command block <NUM> supplies the permitted command to the control block <NUM>.

In accordance with a command supplied from the command block <NUM>, the mode setting block <NUM> sets the operation mode to the manual mode or the hands-free mode. The mode setting block <NUM> supplies the set mode to the state estimation block <NUM>.

If the operation mode supplied from the mode setting block <NUM> is the hands-free mode, the state estimation block <NUM> estimates a state of the operator <NUM> on the basis of the input information supplied from the recognition block <NUM>. The state estimation block <NUM> notifies the command block <NUM> of the estimated state.

The control block <NUM> executes the command supplied from the command block <NUM>. To be more specific, if the command supplied from the command block <NUM> is a command associated with the imaging control of the surgical camera <NUM>, then the control block <NUM> executes imaging control of the surgical camera <NUM> (the surgical apparatus) in accordance with that command.

If the command supplied from the command block <NUM> is a command associated with the driving control of the camera arm <NUM>, then control block <NUM> executes driving control of the camera arm <NUM> (the surgical apparatus) in accordance with that command. Further, if the command supplied from the command block <NUM> is a command associated with the display control of the display <NUM>, then the control block <NUM> controls the image processing block <NUM> (the surgical apparatus) by supplying that command to the image processing block <NUM>.

The image processing block <NUM> processes a surgical field image sent from the surgical camera <NUM>. To be more specific, the image processing block <NUM> supplies a surgical field image sent from to the surgical camera <NUM> to the display <NUM> without change, thereby displaying that surgical field image.

Further, if the command supplied from the control block <NUM> is an annotation display command, then the image processing block <NUM> superimposes a mark (a predetermined image) on the position corresponding to the line of sight of the operator <NUM> inside the surgical field image sent from the surgical camera <NUM> on the basis of the line-of-sight positional information supplied from the line-of-sight recognition block <NUM>. Next, the image processing block <NUM> supplies the surgical field imaged superimposed with the mark to the display <NUM>, thereby displaying that surgical field image.

In addition, if the command supplied from the control block <NUM> is a menu display command for displaying GUI (Graphical User Interface) such as menu buttons onto the display <NUM>, then the image processing block <NUM> superimposes a surgical field image sent from the surgical camera <NUM> with a GUI image. The image processing block <NUM> supplies the surgical field image superimposed with the GUI to the display <NUM>, thereby displaying that surgical field image.

<FIG> is a diagram illustrating one example of a relation between input information and commands.

As depicted in <FIG>, if the voice recognition result in the input information is "zoom-in" and the line-of-sight positional information is indicative of a position inside the screen of the display <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a zoom-in imaging command) for having the surgical camera <NUM> zoom in around a subject corresponding to the line-of-sight position indicated in the line-of-sight positional information.

Likewise, if the voice recognition result in the input information is "zoom-out" and the line-of-sight positional information is indicative of a position inside the screen of the display <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a zoom-out imaging command) for having the surgical camera <NUM> zoom out around a subject corresponding to the line-of-sight position indicated in the line-of-sight positional information.

If the voice recognition result in the input information is "focus" and the line-of-sight positional information is indicative of a position inside the screen of the display <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a focus control command) for executing focus control of the surgical camera <NUM> such that the subject corresponding to the line-of-sight position indicated by the line-of-sight positional information is focused.

It should be noted that a zoom-in imaging command, a zoom-out imaging command, and a focus control command are commands associated with the imaging control of the surgical camera <NUM>, so that these types of command are classified into "imaging control.

As described above, the operator <NUM> is able to enter the contents of imaging control with a voice suited for command input and enter a position necessary for imaging control with the line of sight suited for positional input. Therefore, the operator <NUM> can easily execute commands associated with imaging control.

Further, if the voice recognition result in the input information is "pivot," the line-of-sight positional information is indicative of a position inside the screen of the display <NUM>, the line-of-sight positional information does not change with time, the movement of the head 22A is travel, and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a pivot movement command) for controlling the camera arm <NUM> such that the surgical camera <NUM> pivotally moves in accordance with the movement of the head 22A.

If the voice recognition result in the input information is "slide," the movement of the head 22A is rotation, the line-of-sight positional information is indicative of a position inside the screen of the display <NUM>, the direction in the temporal change of the position indicated by the line-of-sight positional information is the same as the rotational direction of the head 22A, and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a slide movement command) for controlling the camera arm <NUM> such that the surgical camera <NUM> slides in accordance with the position of the line of sight.

It should be noted that a pivot movement command and a slide movement command are commands associated with the driving control of the camera arm <NUM>, so that these types of commands are classified into "camera arm control.

As described above, if a combination of two or more pieces of input information does not satisfy the conditions, then the command block <NUM> does not recognize any such commands that for changing surgical field images as of "imaging control" type or "camera arm control" type as commands issued from the operator <NUM>.

For example, even if the voice recognition result in the input information is "zoom-in" ("zoom-out" or "focus"), but the line-of-sight positional information is not indicative of a position inside the screen, the command block <NUM> determines that the recognition done is erroneous, thereby not recognizing that the command from the operator <NUM> is a zoom-in imaging command (a zoom-out imaging command or a focus control command).

Conversely, even if the line-of-sight positional information in the input information is indicative of a position inside the screen, but the voice recognition result is not "zoom-in" ("zoom-out" or "focus"), the command block <NUM> determines the recognition done is erroneous, thereby not recognizing that the command from the operator <NUM> is a zoom-in imaging command (a zoom-out imaging command or a focus control command).

Even if the voice recognition result in the input information is "focus," the line-of-sight positional information is indicative of a position inside the screen, the movement of the head 22A is travel, and the manipulation information is indicative of the pressing of the foot switch <NUM>, but the line-of-sight positional information is indicative of temporal change, then the command block <NUM> determines that the recognition done is erroneous, thereby not recognizing that the command from the operator <NUM> is a pivot movement command.

Further, even if the voice recognition result in the input information is "focus," the line-of-sight positional information is indicative of a position inside the screen, the movement of the head 22A is travel, and the line-of-sight positional information does not temporarily change, but the manipulation information does not indicate that the foot switch <NUM> is pressed, then the command block <NUM> determines that the recognition done is erroneous, thereby not recognizing that the command from the operator <NUM> is a pivot movement command.

Therefore, a recognition hit ratio of commands that change surgical field images and therefore greatly affect surgical operations can be enhanced. Consequently, the safety of surgery can be enhanced.

In addition, a command of which type is "camera arm control" that greatly changes the contents of a surgical field image affects the surgery more than a command of which type is "imaging control. " Therefore, in the example depicted in <FIG>, the number of pieces of input information under recognition conditions of a command of which type is "camera arm control" is greater than the number of pieces of input information under recognition conditions of a command of which type is "imaging control" by <NUM> to <NUM>.

It should be noted that a condition that manipulation information is indicative of the pressing of the foot switch <NUM> may be added to the recognition conditions of a command of which type is "imaging control," thereby increasing the number of pieces of input information under the recognition information to <NUM>.

If the voice recognition result in the input information is "menu" and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a menu display command. It should be noted that a menu display command is a command associated with the display control of GUI such as menu buttons and other controls of the image processing block <NUM> (the display control apparatus), so that the type of a menu display command is classified into "menu display control.

Further, if the voice recognition result in the input information is "annotation" or "pointer" and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is an annotation display command for displaying, as an annotation, a mark at a position corresponding to the line of sight of the operator <NUM> inside the screen of the display <NUM>. It should be noted that an annotation display command is a command associated with the display control of an annotation of the image processing block <NUM>, so that the type of an annotation command is classified into "annotation display control.

In addition, if the voice recognition result in the input information is "hands-free" and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a hands-free mode command) for setting the operation mode to the hands-free mode.

If the voice recognition result in the input information is "stop" and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a command (hereafter referred to as a manual mode command) for setting the operation mode in a normal state to the manual mode.

As described above, when the operator <NUM> enters a speech associated with a menu display command, an annotation display command, or a manual mode command into the microphone <NUM> and executes an enter manipulation by pressing the foot switch <NUM>, the command block <NUM> recognizes the entered command.

Further, if the manipulation information in the input information is indicative of the long pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a manual mode command in a normal state. If the position indicated by the line-of-sight positional information in the input information is outside the screen of the display <NUM> and the manipulation information is indicative of the pressing of the foot switch <NUM>, then the command block <NUM> recognizes that the command from the operator <NUM> is a manual mode command in a normal state.

In addition, if the gesture of the operator <NUM> in the input information is other than a registered gesture or the sound volume in the input information is greater than a predetermined value, then the command block <NUM> recognizes that the command is a manual mode command in an emergency state. An emergency state denotes a state in which the hands-free mode must be stopped in emergency due to an erroneous operation or the like.

It should be noted that conditions of recognizing a manual mode command in an emergency state may be other than that the gesture of the operator <NUM> is a registered gesture or other than that the sound volume is greater than a predetermined value if these recognition conditions are other than the recognition conditions of the other commands.

A hands-free mode command and a manual mode command are commands associated with the control of the operation mode of the control apparatus <NUM>, so that the types of these commands are classified into "mode control.

It should be noted that the relation between input information and commands is not restricted to the above-mentioned example depicted in <FIG>.

That is, if the operator <NUM> can enter the input contents necessary for command recognition by use of a voice and a sound volume, a line of sight, a movement and direction of the head 22A, a gesture, or the manipulation of the foot switch <NUM> that are suited for the type of these input contents, the recognition conditions are not restricted to particular ones. In the example depicted in <FIG>, for example, the number of types of input information for contactless input in the case of command recognition conditions that the types are "menu display control," "annotation display control," and "mode control" is one; however, the number of input information types may be more than one.

Further, a command to be recognized by the command block <NUM> may be any command as far as the command is for controlling each block of the surgical system <NUM>. For example, the command block <NUM> may recognize a command for setting a various types of parameters of the surgical camera <NUM>.

<FIG> is a diagram illustrating the execution of a pivot movement command by the control block <NUM> depicted in <FIG>.

It should be noted that A of <FIG> is a diagram illustrating the head 22A and the display <NUM> as viewed in the y direction. B of <FIG> is a diagram illustrating the surgical camera <NUM> as viewed in a direction between the z direction and the y direction.

As depicted in A of <FIG>, when the operator <NUM> utters a voice "pivot" and the line of sight of the operator <NUM> is positioned at position R inside the screen of the display <NUM>, if the operator <NUM> shifts only the head 22A in the x direction without moving the line-of-sight position on the screen while pressing the foot switch <NUM>, then the command block <NUM> recognizes a pivot movement command.

If a pivot movement command is supplied from the command block <NUM>, the control block <NUM> driving-controls the camera arm <NUM> to cause the surgical camera <NUM> to do a pivot movement in the x direction by an amount corresponding to a travel amount of the head 22A. Consequently, as depicted in B of <FIG>, the surgical camera <NUM> travels by an amount corresponding to a travel amount of the head 22A in the x direction without changing a distance from center P.

<FIG> is a diagram illustrating the execution of a slide movement command by the control block <NUM> depicted in <FIG>.

It should be noted that A of <FIG> is a diagram illustrating the head 22A and the display <NUM> as viewed in the y direction while B of <FIG> is a diagram illustrating the surgical camera <NUM> as viewed from the z direction.

As depicted in A of <FIG>, when the operator <NUM> utters a voice "pivot" and the line of sight of the operator <NUM> is positioned at position R inside the screen of the display <NUM>, if the operator <NUM> causes the head 22A to rotate by angle α in a right direction so as to move the line-of-sight position on the screen in the x direction while pressing the foot switch <NUM>, then the command block <NUM> recognizes a slide movement command.

If a slide movement command is supplied from the command block <NUM>, the control block <NUM> driving-controls the camera arm <NUM> so as to cause the surgical camera <NUM> to slide in the x direction, thereby placing a subject corresponding to position R' of the line of sight on the screen after the movement to the center of imaging. Consequently, center P of the surgical field that is an imaging target of the surgical camera <NUM> travels in the x direction.

It should be noted that the control block <NUM> may control the speed of a slide movement in accordance with a rotational speed of the head 22A.

<FIG> is a diagram illustrating an example of states of the operator <NUM> that are estimated by the state estimation block <NUM> of <FIG>.

As depicted in <FIG>, the state estimation block <NUM> estimates that the operator <NUM> is in an action-other-than-surgical-procedure state, a downward viewing state, a close watching state, or an observing state.

The action-other-than-surgical-procedure state denotes a state in which the operator <NUM> is executing an action other than a surgical procedure (for example, checking the hand holding forceps or understanding a situation of assistants and staffs around). In the action-other-than-surgical-procedure state, it is assumed that the operator <NUM> be not directly opposite to the display <NUM>. Therefore, there is no need for changing surgical field images. Consequently, if the state of the operator <NUM> is estimated to be the action-other-than-surgical-procedure state, the command block <NUM> restricts the command from the operator <NUM> other than a command of which type is "mode control" that changes operation modes to the command of which type is "menu display control" that does not change surgical field images.

The downward viewing state denotes a state in which the operator <NUM> is overlooking the surgical field in order to check for a tissue damage or bleeding, for example. In the downward viewing state, it is assumed that the line of sight of the operator <NUM> be frequently moving inside the screen of the display <NUM>. In addition, in the downward viewing state, it is possible for the operator <NUM> to indicate a predetermined position within the surgical field image to surrounding assistants or staffs. Therefore, if the state of the operator <NUM> is estimated to be the downward viewing state, the command from the operator <NUM> other than a command of which type is "mode control" to a command of which type is "menu display control" and a command of which type is "annotation display control" that superimposes an annotation on a surgical field image.

The close watching state is a state in which the operator <NUM> is executing a surgical operation while closely watching a single point inside a surgical field image. In the close watching state, the line of sight of the operator <NUM> is inside the screen of the display <NUM> and the movement of the line of sight of the operator <NUM> is less frequent, but the operator <NUM> is assumed to be moving. In the close watching state, it is not necessary for the operator <NUM> to change the contents of a surgical field image but the operator <NUM> must look at the surgical field image taken under the imaging conditions suited for the surgical procedure. Therefore, if the state of the operator <NUM> is estimated to the close watching state, then the command block <NUM> restricts the command from the operator <NUM> other than a command of which type is "mode control" to the commands of which types are "menu display control" and "annotation display control" and a command of which type is "imaging control" that changes imaging conditions.

The observation state is a state in which the operator <NUM> temporarily interrupts the surgical procedure so as to observe the patient <NUM> for an important treatment. In the observation state, it is assumed that the line of sight of the operator <NUM> be inside the screen of the display <NUM> and the movement of the line of sight of the operator <NUM> and the movement of the operator <NUM> be less frequent. In the observation state, it is necessary for the operator <NUM> to observe a surgical field from various directions, so that the contents of a surgical field image must be changed.

Consequently, if the state of the operator <NUM> is assumed to be the observation state, the command block <NUM> permits all of the commands from the operator <NUM> other than the commands of which type is "mode control. " That is, the command block <NUM> permits only the commands of which types are "menu display control," "annotation display control," and "imaging control" but also the commands of which type is "camera arm control" that changes the positions of the surgical camera <NUM>.

As described above, the degree of the necessity for changing surgical field images increases from the action other than surgical procedure state to the downward viewing state to the close watching state to the observation state in this order.

It should be noted here that it is assumed that, in a state higher in the necessity for changing surgical field images, all of the commands that are permitted in the lower states be permitted.

For example, if the state of the operator <NUM> is the action-other-than-surgical-procedure state, the downward viewing state, the close watching state, or the observation state, then only the commands of which type is "menu display control," "annotation display control," "imaging control," or "camera arm control" may be permitted.

<FIG> is a diagram illustrating a program for causing a computer to carry out a method of estimating a state of the operator <NUM> in the state estimation block <NUM> depicted in <FIG>.

On the basis of the direction of the head 22A or the line-of-sight positional information in the input information, the state estimation block <NUM> determines whether the operator <NUM> is directly opposite to the display <NUM>.

To be more specific, if the direction of the head 22A is in the direction of the display <NUM>, the state estimation block <NUM> determines that the operator <NUM> is directly opposite to the display <NUM>; if the direction of the head 22A is not in the direction of the display <NUM>, the state estimation block <NUM> determines that the operator <NUM> is not directly opposite to the display <NUM>.

Alternatively, if the position indicated by the line-of-sight positional information is inside the screen of the display <NUM>, the state estimation block <NUM> determines that the operator <NUM> is directly opposite to the display <NUM>; if the position indicated by the line-of-sight positional information is outside the screen of the display <NUM>, the state estimation block <NUM> determines that the operator <NUM> is not directly opposite to the display <NUM>.

If the travel amount is greater than a predetermined value on the basis of the travel amount within a predetermined time of a position indicated by the line-of-sight positional information, the state estimation block <NUM> determines the that travel amount of the line of sight is high; if the travel amount is less than the predetermined value, the state estimation block <NUM> determines that the travel amount of the line of sight is low.

Further, if the amount of movement of the head 22A is greater than a predetermined value within a predetermined time on the basis of the movement of the head 22A, the state estimation block <NUM> determines that the operator <NUM> is moving; if the amount of movement of the head 22A is less than the predetermined value, the state estimation block <NUM> determines that the operator <NUM> is not moving.

It should be noted that it is also practicable that the recognition block <NUM> recognizes the movement of a part other than the head 22A of the operator <NUM> and, on the basis of the movement of the part other than the head 22A of the operator <NUM>, the state estimation block <NUM> determines whether the operator <NUM> is moving or not. In this case, if the amount of movement of the part other than the head 22A of the operator <NUM> within a predetermined time is greater than a predetermined value, then the recognition block <NUM> determines that the operator <NUM> is moving; if the amount of movement of the part other than the head 22A of the operator <NUM> is less than the predetermined value, the recognition block <NUM> determines that the operator <NUM> is not moving.

As depicted in <FIG>, if the operator <NUM> is found to be not directly opposite to the display <NUM>, the state estimation block <NUM> estimates that the state of the operator <NUM> is the action-other-than-surgical-procedure state. In this case, the type other than "mode control" of commands from the operator <NUM> that are permitted is "menu display control.

Also, if the operator <NUM> is found to be directly opposite to the display <NUM> and the travel amount of the line of sight is high, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the downward viewing state. In this case, the types of permitted commands other than "mode control" are "menu display control" and "annotation display control.

Further, if the operator <NUM> is found to be directly opposite to the display <NUM>, the travel amount of the line of sight is found to be low, and the operator <NUM> is found to be moving, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the close watching state. In this case, the types of permitted commands other than "mode control" are "menu display control," "annotation display control," and "imaging control.

In addition, if the operator <NUM> is found to be not directly opposite to the display <NUM>, the travel amount of the line of sight is found to be low, and the operator <NUM> is found to be not moving, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the observation state. In this case, the types of permitted commands other than "mode control" are "menu display control," "annotation display control," "imaging control," and "camera arm control.

It should be noted that, since the operator <NUM> executes a surgical procedure by use of forceps and the like while looking at the display <NUM>, the frequency of the movement of the head 22A of the operator <NUM> during a surgical procedure is very low, but the frequency of the movement of the hands is high. Therefore, if not in the case where the amount of the movement of the head 22A is greater than a predetermined value but the amount of the movement of the head 22A is less than the predetermined value and the amount of the movement of other than the head 22A is greater than the predetermined value, the state estimation block <NUM> may determine that the state of the operator <NUM> is the close watching state.

<FIG> is a flowchart indicative of the control processing to be executed by the control apparatus <NUM> of the surgical system <NUM> depicted in <FIG>. This control processing starts when the power to the control apparatus <NUM> is turned on, for example.

In step S11 depicted in <FIG>, the mode setting block <NUM> sets the processing mode to the manual mode and supplies this information to the state estimation block <NUM>.

In step S12, the recognition block <NUM> recognizes the input information. Of the input information, the recognition block <NUM> supplies voice recognition result information, sound volume information, gesture information, and manipulation information to the command block <NUM>. In addition, the recognition block <NUM> supplies line-of-sight positional information to the command block <NUM>, the state estimation block <NUM>, and the image processing block <NUM>. The recognition block <NUM> supplies the movement and direction of the head 22A to the command block <NUM> and the state estimation block <NUM> as the input information.

In step S13, on the basis of the input information supplied from the recognition block <NUM>, the command block <NUM> recognizes a command from the operator <NUM>. In step S14, the command block <NUM> determines whether the type of the recognized command is "mode control" or not.

If the type of the command recognized in step S14 is "mode control," then the command block <NUM> notifies the mode setting block <NUM> of that command, upon which the processing goes to step S15. In step S15, in accordance with the command supplied from the command block <NUM>, the mode setting block <NUM> changes operation modes. The mode setting block <NUM> supplies the changed mode to the state estimation block <NUM>, upon which the processing goes to step S16.

On the other hand, if the type of the command recognized in step S14 is not found to be "mode control," then the processing goes to step S16.

In step S16, the state estimation block <NUM> determines the operation mode supplied from the mode setting block <NUM> is the hands-free mode or not. If the operation mode is found to be the hands-free mode in step S16, then the processing goes to step S17.

In step S17, the control apparatus <NUM> executes state estimation processing for estimating a state of the operator <NUM> on the basis of the input information supplied from the recognition block <NUM>. Details of this state estimation processing will be described later with reference to <FIG>.

In step S18, the command block <NUM> determines whether the type of the command recognized in step S13 from the operator <NUM> other than commands of which type is "mode control" is permitted or not. If the type of that command is found to be permitted in step S18, then the command block <NUM> supplies that command to the control block <NUM>.

Then, in step S19, the control block <NUM> executes the command supplied from the command block <NUM>, upon which the processing goes to step S20.

On the other hand, if the operation mode is found to be not the hands-free mode in step S16 or if the type of the command from the operator <NUM> other than commands of which type is "mode control" is found to be not permitted in step S18, then the processing goes to step S20.

In step S20, the control apparatus <NUM> determines whether or not to turn off the power to the control apparatus <NUM>; for example, the control apparatus <NUM> determines whether or not a command of powering off the control apparatus <NUM> has been issued by the operator <NUM>. If the power to the control apparatus <NUM> is found to be not turned off in step s20, then the processing returns to step S12 so as to repeat the processing of steps S12 through S20.

On the other hand, if the power to the control apparatus <NUM> is found to be turned off in step S20, the processing is terminated.

<FIG> is a flowchart indicative of details of the state estimation processing in step S17 depicted in <FIG>.

In step S41 depicted in <FIG>, on the basis of the direction of the head 22A or the line-of-sight positional information in the input information, the state estimation block <NUM> determines whether the operator <NUM> is directly opposite to the display <NUM> or not.

If the operator <NUM> is found to be not directly opposite to the display <NUM> in step S41, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the action-other-than-surgical-procedure state in step S42, notifying the command block <NUM> thereof.

In step S43, the command block <NUM> sets the type of the command from the operator <NUM> to be permitted other than "mode control" to "menu display control. " Then, the processing returns to step S17 depicted in <FIG>, upon which the processing of step S18 is executed.

On the other hand, if the operator <NUM> is found to be directly opposite to the display <NUM> in step S41, the state estimation block <NUM> determines whether the travel amount of the line of sight is high or not on the basis of the travel amount within a predetermined time of the position indicated by the line-of-sight positional information in step S44.

If the travel amount of the line of sight is found high in step S44, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the downward viewing state in step S45, thereby notifying the command block <NUM> thereof.

In step S46, the command block <NUM> sets the types of the commands from the operator <NUM> to be permitted other than "mode control" to "menu display control" and "annotation display control. " Then, the processing returns to step S17 depicted in <FIG> to repeat the processing of step S18.

Further, if the travel amount of the line of sight is found to be low in step S44, then the state estimation block <NUM> determines whether the operator <NUM> is moving or not on basis of the movement of the head 22A in step S47. If the operator <NUM> is found to be moving in step S47, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the close watching state in step S48, thereby notifying the command block <NUM> thereof.

In step S49, the command block <NUM> sets the types of the commands from the operator <NUM> to be permitted other than "mode control" to "menu display control," "annotation display control," and "imaging control. " Then, the processing returns to step S17 depicted in <FIG> to repeat the processing of step S18.

On the other hand, if the operator <NUM> is found to be not moving in step S47, then the state estimation block <NUM> estimates that the state of the operator <NUM> is the observation state in step S50, thereby notifying the command block <NUM> thereof.

In step S51, the command block <NUM> sets the types of the commands from the operator <NUM> to be permitted to "menu display control," "annotation display control," "imaging control," and "camera arm control. " Then the processing returns to step S17 depicted in <FIG> to repeat the processing of step S18.

A described above, on the basis of combinations of two or more types of contactless inputs, the surgical system <NUM> controls the surgical camera <NUM>, the camera arm <NUM>, or the image processing block <NUM>. Therefore, by executing contactless input operations suited for the types of input contents, for example, the operator <NUM> is able to easily and intuitively control the surgical camera <NUM>, the camera arm <NUM>, and the image processing block <NUM>. That is, the surgical system <NUM> can realize NUI (Natural User Interface). As a result, the load of the operator <NUM> is mitigated.

Further, as compared with the case in which the surgical camera <NUM>, the camera arm <NUM>, or the image processing block <NUM> is controlled by the contactless input of one type, the above-mentioned two or more types of contactless inputs enhance the hit ratio of input recognition, which in turn enhancing the safety of surgical procedures.

Since the surgical system <NUM> allows the execution of input operations in a contactless manner or by the contact by the leg 22B, the operator <NUM> himself can execute input operations even if both hands are occupied by the execution of surgical procedure. As a result, as compared with the case in which the operator <NUM> executes input operations, there is no need for interrupting surgical procedure because of input operations, thereby saving the surgical time. In addition, as compared with the case in which a person other than the operator <NUM> executes input operations, the surgical system <NUM> allows the operator <NUM> execute control as intended by the operator <NUM>, thereby mitigating the load of the operator <NUM>.

Further, the surgical system <NUM> can restrict the execution of commands issued by the operator <NUM> in accordance with a state of the operator <NUM> so as to real failsafe, thereby preventing the control operations that are not intended by the operator <NUM> due to the erroneous recognition of a command from the operator <NUM>. Consequently, the safety of surgical procedure is enhanced.

Still further, since the surgical system <NUM> can change the operation mode from the hands-free mode to the manual mode, if a control operation that is not intended by the operator <NUM> is executed due to the erroneous recognition of a command from the operator <NUM>, the unintended control operation can be stopped.

<FIG> is a block diagram illustrating one example of a configuration of a second embodiment of a surgical system to which the present disclosure is applied.

With the configuration depicted in <FIG>, the configurational components same as those previously described with <FIG> are denoted by the same reference symbols. The duplicate description will appropriately skipped.

In configuration, a surgical system <NUM> depicted in <FIG> differs from the surgical system <NUM> depicted in <FIG> in that the surgical system <NUM> has a display <NUM> and a control apparatus <NUM> instead of the display <NUM> and the control apparatus <NUM> and does not have the surgical glasses <NUM> and the marker <NUM>.

With the surgical system <NUM>, the distance between the display <NUM> and the operator <NUM> is shorter than the distance between the display <NUM> and the operator <NUM>, so that the operator <NUM> recognizes a surgical field image displayed on the display <NUM> as a 3D image with the naked eyes without using the surgical glasses <NUM>.

To be more specific, the display <NUM> of the surgical system <NUM> is a 3D display having a comparatively small screen and is arranged at a position comparatively near the operator <NUM> (in the example depicted in <FIG>, a position on the operating table <NUM> and near the operator <NUM>). The display <NUM> displays surgical field images and so on sent from the control apparatus <NUM>. On top of the display <NUM>, the action recognition camera <NUM> is arranged.

Except for a method of recognizing the line of sight and the movement and direction of the head 22A, the control apparatus <NUM> operates in the similar manner to the control apparatus <NUM>, so that the following describes only this recognition method. The control apparatus <NUM> detects the position of the head 22A inside an operator image sent from the action recognition camera <NUM> so as to recognize the movement and direction of the head 22A. Further, the control apparatus <NUM> detects the direction of the line of sight of the operator <NUM> from an operator image so as to recognize the position of the line of sight on the screen of the display <NUM> on the basis of the detected direction.

It should be noted that, with the surgical system <NUM>, the operator <NUM> does not use the surgical glasses <NUM>, so that the detection of line of sight is executed by use of an operator image taken with the action recognition camera <NUM>; however, it is also practicable to execute the detection of line of sight by a line-of-sight detection device by making the operator <NUM> wear the surgical glasses having the line-of-sight detection device.

Further, with the surgical system <NUM>, since the distance between the action recognition camera <NUM> and the operator <NUM> is short, the movement and direction of the head 22A are detected from an operator image; however, it is also practicable to for the operator <NUM> to wear the marker <NUM> so as to detect the movement and direction of the head 22A from a position of the marker <NUM> inside an operator image.

Still further, the display <NUM> may be arranged at a position comparatively far from the operator <NUM>. The display <NUM> is a 3D display with which the operator <NUM> can recognize 3D images through 3D polarized glasses, so that the operator <NUM> may use 3D polarized glasses.

The above-mentioned sequence of processing operations by the control apparatus <NUM> (<NUM>) can be executed by hardware or software. In the execution of the sequence of processing operations by software, the programs constituting that software are installed on a computer.

It should be noted that the computer includes a computer built in dedicated hardware and a general-purse personal computer, for example, in which various programs can be installed for the execution of various functions.

<FIG> is a block diagram illustrating one example of the hardware of a computer for executing the above-mentioned sequence of processing operations by programs.

In a computer <NUM>, a CPU (Central Processing Unit) <NUM>, a ROM (Read Only Memory) <NUM>, and a RAM (Random Access Memory) <NUM> are interconnected by a bus <NUM>.

The bus <NUM> is further connected with an input/output interface <NUM>. The input/output interface <NUM> is connected with an input block <NUM>, an output block <NUM>, a storage block <NUM>, a communication block <NUM>, and a drive <NUM>.

The input block <NUM> includes a keyboard, a mouse, a microphone, and so on. The output block <NUM> includes a display, a speaker, and so on. The storage block <NUM> includes a hard disk drive, a nonvolatile memory, and so on. The communication block <NUM> includes a network interface and so on. The drive <NUM> drives a removable medium <NUM> such as a magnetic disc, an optical disc, a magneto-optical disc, or a semiconductor memory.

With the computer <NUM> configured as described above, the CPU <NUM> loads programs from the storage block <NUM> into the RAM <NUM> via the input/output interface <NUM> and the bus <NUM> and executes the loaded programs so as to execute the above-mentioned sequence of processing operations.

Programs to be executed by the computer <NUM> (the CPU <NUM>) may be provided as recorded to the removable medium <NUM> that is a package medium or the like. In addition, programs may be provided through a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting.

With the computer <NUM>, programs can be installed in the storage block <NUM> via the input/output interface <NUM> by loading the removable medium <NUM> on the drive <NUM>. In addition, programs can be installed in the storage block <NUM> by receiving by the communication block <NUM> the programs via a wired or wireless transmission medium. Otherwise, programs can be installed in the ROM <NUM> or the storage block <NUM> in advance.

It should be noted that programs to be executed by the computer <NUM> may be programs in which processing is executed in a time sequence manner by following the sequence described in the present description or in a parallel manner or on an on-demand basis with required timings.

In the present description, a system denotes an aggregation of two or more configurational components (apparatuses, modules (parts), etc.) regardless whether all the configurational components are within a same housing or not. Therefore, two or more apparatuses accommodated in separate housings and interconnected via a network are a system or one apparatus with two or more modules accommodated in one housing is also a system.

While preferred embodiments of the present disclosure are not limited to the embodiments described above and variations may be made without departing from the scope of the appended claims.

For example, in the first embodiment through the third embodiment, the control apparatus <NUM> (<NUM>) executes control operations on the basis of two or more types of contactless input combinations and the control operations are restricted in accordance with states of the operator <NUM>, both thereby enhancing the safety of surgical procedure.

Further, targets of the restriction by the control apparatus <NUM> (<NUM>) may be anything as far as the targets are surgical apparatuses. For example, the control apparatus <NUM> (<NUM>) can also control such surgical imaging apparatuses as endoscopes and video microscopes.

Moreover, it is also practicable for zoom control to be realized not by the imaging control of the surgical camera <NUM> but by processing a surgical field image in the image processing block <NUM>.

In this case, in accordance with a zoom-in imaging command, the image processing block <NUM> enlarges a surgical field image sent from the surgical camera <NUM> so as to execute electronic zooming in which a zoom-in image taken in a zoom-in manner around a subject corresponding to the position of line of sight is generated from the surgical field image. Likewise, in accordance with a zoom-out imaging command, the image processing block <NUM> reduces a surgical field image sent from the surgical camera <NUM> so as to generate a zoom-out image taken in a zoom-out manner around a subject corresponding to the position of line of sight from the surgical field image. It should be noted that, at this moment, on the basis of the line-of-sight positional information, the image processing block <NUM> may superimpose a marker on the position corresponding to the line of sight inside the zoom-in image or the zoom-out image.

Further, while a surgical field image is displayed on the display <NUM>, annotation display may be always executed. The contactless inputs are not restricted to the voice and line of sight of the operator <NUM>, the movement and direction of the head 22A, and the gesture of the operator <NUM>. For example, the contactless inputs may include the movement and attitude of other than the head 22A of the operator <NUM>.

The means of accepting contactless inputs may be wearable like the surgical glasses <NUM> and the microphone <NUM> or may not be wearable.

Even if the operation mode is the manual mode, the control apparatus <NUM> (<NUM>) may estimate a state of the operator <NUM> and, in accordance with the estimated state, restrict the control of the surgical camera <NUM>, the camera arm <NUM>, and the image processing block <NUM>.

Claim 1:
A surgical control apparatus (<NUM>) comprising:
a first contactless recognition block (<NUM>, <NUM>, <NUM>, <NUM>) configured to recognize a first type of contactless input from a user as first input information;
a second contactless recognition block (<NUM>, <NUM>, <NUM>, <NUM>) configured to recognize a second type of contactless input from the user as second input information, the first type of contactless input being different from the second type of contactless input;
a state estimation block (<NUM>) configured to estimate a state of the user from a plurality of predetermined user states on the basis of a line of sight input and an operator movement input; wherein the state estimation block is configured to estimate the state of the user as: an action-other-than-surgical-procedure state, a downward viewing state, a close watching state in which the user is executing a surgical operation while closely watching a single point inside a surgical field image, or an observation state in which the user temporarily interrupts a surgical procedure to observe a patient; wherein a degree of necessity for changing surgical field images increases in the order from the action-other-than-surgical-procedure state to the downward viewing state to the close watching state to the observation state; and
a command block (<NUM>) configured to set at least one command type from a plurality of command types as a permitted command type and to set other commands types from the plurality of command types as restricted command types, in accordance with the state of the user estimated by the state estimation block, wherein more command types are set as permitted command types in a state higher in the degree of necessity for changing surgical field images, and wherein in a state higher in the necessity for changing surgical field images all of the command types that are permitted in the lower states are permitted, the at least one permitted command type comprising one or more commands for controlling a control operation of a surgical apparatus, each command having an associated recognition condition, wherein the command block is configured to output one of the one or more commands in dependence upon whether the first input information and the second input information satisfy a recognition condition of one of the one or more commands.