Patent Description:
There is known an assistance system where a surgical operator can perform surgical operation on a patient while receiving instructions of an instructor at a remote location. This assistance system is used when, for example, a less experienced operation doctor (the surgical operator) performs surgery while receiving instructions of an experienced instruction doctor (an instructor).

According to the invention disclosed in <CIT>, a monitor displaying a captured image (hereinafter also noted as an endoscope image) by an endoscope inserted into a patient is located at a remote location, and an instruction doctor at the remote location can input instructions while observing the endoscope image displayed on the monitor. On the other hand, on a monitor on a surgery room side, a combined image where instruction contents input at the remote location are superimposed on the endoscope image is displayed, and an operation doctor can perform surgery on the patient while receiving the instructions of the instruction doctor at the remote location.

The above-described assistance system allows an instructor to observe captured images displayed on a monitor and to provide instructions. Thus, even a less experienced surgical operator can perform reliable surgical operation on a patient, and skills of the surgical operator himself/herself can also be improved.

Document <CIT> represents the closest prior art and discloses an instruction side system and an operation-side endoscopic system of a surgical assistance system, wherein the instruction side system sends drawing information to the operation-side system for superimposing on the local endoscopic image. The document does not teach the instruction-side combining of the drawing information with the superimposed image received from the operation side, for verification of the correct displaying of the operation side.

When such an assistance system is employed in a field where safety of a surgical operation subject (a patient) of the above-described endoscopic surgery and the like is required, in order to allow a surgical operator during surgery to confirm a state of a patient in real time, it is important that an endoscope image on which instruction contents of an instructor are reflected is displayed on a monitor on a surgical operator side at as low latency as possible.

In order for the instructor side also to input the instruction contents without a sense of incompatibility, it is preferred that the endoscope image on which the input is reflected is displayed on the monitor of the instructor himself/herself at as low latency as possible.

The present invention is made in consideration of the above-described circumstances, and it is an object of the present invention to provide an assistance system in which an instructor at a remote location can perform appropriate assistance to a surgical operator.

The present invention discloses an instruction-side system information processing device according to claim <NUM>, an operation-side system information processing device according to claim <NUM>, and a surgical assistance system as defined by claim <NUM>.

An information processing device according to the present disclosure is included in a second system of an assistance system including a first system and the second system separated from the first system. The information processing device includes; a data transmission unit that transmits drawing information that is based on an operation input and is necessary for generating annotation image data, to the first system; an image generation unit that generates the annotation image data from the drawing information; and an image combining unit that combines the annotation image data generated by the image generation unit and image data received from the first system.

The annotation image data here is image data that is superimposed on an endoscope image and is image data generated based on the drawing information.

The drawing information means information such as a coordinate, a line type, a color, and a line width specified by the operation input.

In the first system in the above-described assistance system, the annotation image data is generated from the drawing information, and combined image data is generated by combining the generated annotation image data and captured image data, and in the information processing device included in the second system, the image combining unit combines the combined image data received from the first system and the annotation image data generated by the image generation unit.

This generates the annotation image data at each of the first system and the second system.

It is considered that, in the first system, the annotation image data is generated from the drawing information, and combined image data is generated by combining a plurality of pieces of image data including the generated annotation image data, and the image combining unit of the information processing device included in above-described the second system combines the combined image data received from the first system and the annotation image data generated by the image generation unit.

This allows confirmation of whether or not the drawing information transmitted to the first system side is reflected on the combined image data received from the first system, on the second system side.

The image generation unit in the information processing device included in the above-described second system is considered to generate annotation image data that is distinguishable from the annotation image data included in the combined image data.

This allows distinguishing a display based on the annotation image data included in the combined image data from a display based on the annotation image data generated by the image generation unit, and allows confirming whether or not the annotation image data on which the drawing information based on the input at the second system is reflected is generated on the first system side.

An information processing device according to the present disclosure included in a first system of an assistance system including the first system and a second system separated from the first system. The information processing device includes; an image generation unit that receives drawing information based on an operation input from the second system and generates annotation image data from the received drawing information; an image combining unit that generates combined image data by combining image data that is input and the annotation image data generated by the image generation unit; and a data transmission unit that transmits the combined image data generated by the image combining unit, to the second system.

Here, when the combined image data is generated on the first system side, instead of the annotation image data, the drawing information having a communication volume less than the annotation image data is received from the second system.

In the above-described information processing device, a delay time from when a plurality of image data are input until when the combined image data is output by the image combining unit is considered to be less than <NUM> milliseconds.

As a result, when the display based on the input image data is performed on the monitor on the first system side, the surgical operator visually recognizing the monitor does not feel any delay.

The assistance system according to the present disclosure includes the information processing device includes in the above-described first system and the information processing device included in the above-described second system. The assistance system is achieved by each of the information processing devices described above communicating with one another.

With the present invention, the instruction contents, which are based on the input of the instructor at a remote location, displayed on each of the monitors of the surgical operator side and the instructor side with significantly low latency allow the surgical operator and the instructor to share the instruction contents in real time.

The following describes an embodiment with reference to <FIG>.

Each configuration described in the referenced drawings merely indicates one example for achieving the present invention. Accordingly, it is possible to make various modifications corresponding to design and the like as long as they are within the scope not departing from the technical idea of the present invention. In order to avoid duplication for the configuration that has been described once, identical reference numerals are given hereinafter, and re-description will be omitted, in some cases.

<FIG> illustrates an assistance system <NUM>.

In the present embodiment, as one example of the assistance system <NUM>, an assistance system in which a surgical operator <NUM> in a surgery room Rm1 can perform surgery on a patient <NUM> while confirming instructions by an instruction doctor <NUM> in an instruction room Rm2 separated from the surgery room Rm1 will be described. In the surgery assistance system, for example, the surgical operator <NUM> is an operation doctor for the patient <NUM>, and the instructor <NUM> is the instruction doctor instructing the operation doctor.

For example, by inserting an endoscope <NUM> into a body cavity of the patient <NUM>, an endoscope image <NUM> inside the body cavity illustrated in <FIG> is displayed on a monitor <NUM>. The surgical operator <NUM> can confirm the endoscope image <NUM> on the monitor <NUM>.

The endoscope image <NUM> is also displayed on a monitor <NUM> of an instruction terminal <NUM> disposed in the instruction room Rm2. The instructor <NUM> can confirm the endoscope image <NUM> by the monitor <NUM> while staying in the instruction room Rm2.

The instruction terminal <NUM> is, for example, a tablet terminal having a touch panel <NUM>, and the instructor <NUM> can input instruction contents on the endoscope image <NUM> by operating the touch panel <NUM> with his/her finger or a stylus pen.

By the input, a combined image <NUM> in which annotation images <NUM>, <NUM> indicating the instruction contents of the instructor <NUM> as illustrated in <FIG> are superimposed on the endoscope image <NUM> is displayed on the monitor <NUM>. The surgical operator <NUM> can perform a surgical operation on the patient <NUM> while confirming the instructions of the instructor <NUM> displayed on the monitor <NUM>.

Next, the configuration of the assistance system <NUM> will be described with reference to <FIG> is a block diagram illustrating one example of the configuration of the assistance system <NUM>.

The assistance system <NUM> has an operation-side system <NUM> constituted on the surgery room Rm1 side and an instruction-side system <NUM> constituted on the instruction room Rm2 side. The operation-side system <NUM> and the instruction-side system <NUM> are separated from one another and can communicate with one another through a wired or a wireless transmission line.

The operation-side system <NUM> has the endoscope <NUM>, an image generation unit <NUM>, an image combining unit <NUM>, the monitor <NUM>, and a data transmission unit <NUM>.

The endoscope <NUM> includes imaging devices, and captured image signals obtained by the imaging devices are each A/D converted and converted into endoscope image data Id (which means image data for displaying the endoscope image <NUM> as illustrated in <FIG>) representing a luminance value by predetermined gradation in a pixel unit.

The image generation unit <NUM> is constituted of, for example, an image processor or the like, and generates annotation image data Ad1 based on drawing information PI transmitted from the instruction-side system <NUM>.

Here, the drawing information PI includes information such as a coordinate, a line type, a color, a line width that are specified by the operation input.

The annotation image data Ad1 is the image data such as the annotation images <NUM>, <NUM> illustrated in <FIG> so as to be superimposed on the endoscope image <NUM> and generated based on the drawing information PI. This also applies to the annotation image data Ad2, which will be described later.

The image combining unit <NUM> is achieved by, for example, a video mixer device constituted of hardware or a dedicated circuit implemented with a Field Programmable Gate Array (FPGA) and the like, and combines various kinds of pieces of input image data. In the present example, the endoscope image data Id and the annotation image data Ad1 are combined to generate operation-side combined image data Cd1 in the image combining unit <NUM>.

The monitor <NUM> is constituted of, for example, a liquid crystal display device or the like and displays an image on a display panel based on the supplied image data. In the present example, a display based on the supplied endoscope image data Id or operation-side combined image data Cd1 is performed on the monitor <NUM>.

The data transmission unit <NUM> is configured, for example, to be able to communicate with the instruction-side system <NUM> and can transmit various kinds of pieces of data to the instruction-side system <NUM>. In the present example, particularly, the data transmission unit <NUM> transmits the endoscope image data Id or the operation-side combined image data Cd1 obtained from the image combining unit <NUM>.

Subsequently, the instruction-side system <NUM> will be described.

The instruction-side system <NUM> includes an annotation interface <NUM>, a data transmission unit <NUM>, an image generation unit <NUM>, an image combining unit <NUM>, and the monitor <NUM>. The respective components are integrally constituted as, for example, the instruction terminal <NUM>.

The annotation interface <NUM> is constituted of, for example, input devices, such as a touch panel, a touchpad, a computer mouse, and a keyboard, a processor, and the like generating the drawing information PI corresponding to the operation input to those input devices. In the present embodiment, an input to the annotation interface <NUM> is performed by, for example, operating the touch panel <NUM> in <FIG> by the instructor <NUM> with his/her finger, a stylus pen, or the like.

The data transmission unit <NUM> is configured, for example, to be able to communicate with the operation-side system <NUM> and can transmit various kinds of pieces of data to the operation-side system <NUM>. In the present example, particularly, the data transmission unit <NUM> transmits the drawing information PI obtained by the annotation interface <NUM>.

The image generation unit <NUM> is constituted of, for example, an image processor or the like, and generates the annotation image data Ad2 from the drawing information PI.

The image combining unit <NUM> is achieved by, for example, a video mixer device as hardware or a software mixer constituted in the information processing device and combines various kinds of pieces of supplied image data. In the present example, by the image combining unit <NUM>, the annotation image data Ad2 is combined with the endoscope image data Id or the operation-side combined image data Cd1, and thus the instruction-side combined image data Cd2 is generated.

The monitor <NUM> is constituted of, for example, a liquid crystal display device or the like, and displays an image on a display panel based on the supplied image data. In the present example, for example, a display based on the instruction-side combined image data Cd2 is performed.

While, as one example of the above-described respective systems (<NUM>, <NUM>), it is considered that the image generation units <NUM>, <NUM>, the image combining units <NUM>, <NUM>, and the data transmission units <NUM>, <NUM> are constituted of hardware, the constitution of each functional unit is not limited to the above. For example, all or a part of each functional unit may be achieved by one or a plurality of hardware devices.

For example, any of each functional unit may be constituted as a microcomputer including a Central Processing Unit (CPU), a Read Only Memory (ROM), a Random Access Memory (RAM), and the like. In this case, each function can be achieved by the CPU executing processes based on a program as software stored in the ROM or the like.

Details of the assistance system <NUM> according to the present embodiment will be described with reference to <FIG>.

First, in the operation-side system <NUM>, the endoscope image data Id obtained by the endoscope <NUM> is supplied to the image combining unit <NUM>.

When the annotation image data Ad1 is not supplied from the image generation unit <NUM>, the image combining unit <NUM> supplies the endoscope image data Id to the monitor <NUM> and the data transmission unit <NUM>.

By the endoscope image data Id being supplied to the monitor <NUM>, the endoscope image <NUM> is displayed on the monitor <NUM> (see <FIG>). This allows the surgical operator <NUM> to perform the surgical operation while observing a state inside the body cavity of the patient <NUM> on the monitor <NUM>.

The data transmission unit <NUM> transmits the endoscope image data Id supplied from the image combining unit <NUM> to the instruction-side system <NUM>.

In the instruction-side system <NUM>, the image combining unit <NUM> receives the endoscope image data Id from the operation-side system <NUM> (the data transmission unit <NUM>).

When the annotation image data Ad2 is not supplied from the image generation unit <NUM>, the image combining unit <NUM> supplies the endoscope image data Id to the monitor <NUM>.

By the endoscope image data Id being supplied, the endoscope image <NUM> is displayed on the monitor <NUM> (see <FIG>). This allows the instructor <NUM> to share a viewpoint with the surgical operator <NUM> in the surgery room Rm1 while staying in the instruction room Rm2 and to observe the state inside the body cavity of the patient <NUM> and operation status of the surgical operation by the surgical operator <NUM>.

By operating the input devices, such as the touch panel <NUM>, while observing the endoscope image <NUM> displayed on the monitor <NUM>, the instructor <NUM> inputs a pictorial figure, a character, a symbol, and the like indicating the instruction contents for the surgical operator <NUM> on the endoscope image <NUM>.

In the annotation interface <NUM>, the drawing information PI corresponding to the operation input to the touch panel <NUM> is generated. The drawing information PI obtained by the annotation interface <NUM> is supplied to the image generation unit <NUM> and the data transmission unit <NUM>.

The data transmission unit <NUM> transmits the drawing information PI obtained from the annotation interface <NUM> to the operation-side system <NUM>.

In transmission to the operation-side system <NUM>, by transmitting the drawing information PI having a communication volume less than the annotation image data Ad2 generated by the image generation unit <NUM>, a communication delay when data is transmitted from the instruction-side system <NUM> to the operation-side system <NUM> is reduced.

In the operation-side system <NUM>, the image generation unit <NUM> receives the drawing information PI from the instruction-side system <NUM> (the data transmission unit <NUM>). The image generation unit <NUM> generates the annotation image data Ad1 for causing the annotation images <NUM>, <NUM> (see <FIG>) to be displayed from the received drawing information PI.

The annotation image data Ad1 generated by the image generation unit <NUM> is supplied to the image combining unit <NUM>.

The image combining unit <NUM> combines the endoscope image data Id supplied from the endoscope <NUM> and the annotation image data Ad1 supplied from the image generation unit <NUM> to generate the operation-side combined image data Cd1. The operation-side combined image data Cd1 is the image data for causing the combined image <NUM> (see <FIG>) in which the annotation images <NUM>, <NUM> (see <FIG>) are superimposed on the endoscope image <NUM> (see <FIG>) to be displayed.

Here, since the image combining unit <NUM> is constituted of the video mixer device by the hardware, a delay time from when the endoscope image data Id and the annotation image data Ad1 are input until when the operation-side combined image data Cd1 is output is significantly short.

Instead of the video mixer device by the hardware, the image combining unit <NUM> can also be achieved with a processor having a high computation capability. By the processor executing a combining process specified by a program, it is possible to achieve a processing speed comparable to that of the video mixer device by the hardware, for example, a processing speed such that the delay time from input until output is less than <NUM> milliseconds. In such a case, the image combining unit <NUM> may be achieved as a software mixer.

By quickly performing the combining process at the above-described processing speed, the operation-side combined image data Cd1 generated by the combining process is supplied to the monitor <NUM> without delay and displayed on the monitor <NUM> with less time lag.

This allows the surgical operator <NUM> in the surgery room Rm1 to perform the surgical operation on the patient <NUM> without feeling a time lag while confirming the instructions of the instructor <NUM> in the instruction room Rm2 on the monitor <NUM>.

In particular, when the image combining unit <NUM> is the video mixer device by the hardware or a software mixer device using a processor having a significantly high processing speed, and a delay time is less than <NUM> milliseconds as described above, the surgical operator <NUM> is likely to feel almost no time lag with respect to the image.

The data transmission unit <NUM> transmits the operation-side combined image data Cd1 supplied from image combining unit <NUM> to the instruction-side system <NUM>.

On the other hand, in the instruction-side system <NUM>, the image generation unit <NUM> generates the annotation image data Ad2 for causing the annotation images <NUM>, <NUM> (see <FIG>) to be displayed, based on the drawing information PI obtained from the annotation interface <NUM>. The annotation image data Ad2 obtained by the image generation unit <NUM> is supplied to the image combining unit <NUM>.

The image combining unit <NUM> combines the operation-side combined image data Cd1 received from the operation-side system <NUM> (the data transmission unit <NUM>) and the annotation image data Ad2 supplied from the image generation unit <NUM> to generate the instruction-side combined image data Cd2. The generated instruction-side combined image data Cd2 is supplied to the monitor <NUM>.

When having received the endoscope image data Id instead of the operation-side combined image data Cd1 from the operation-side system <NUM>, the image combining unit <NUM> combines the endoscope image data Id and the annotation image data Ad2 to generate the instruction-side combined image data Cd2. The instruction-side combined image data Cd2 generated at this time is supplied to the monitor <NUM>, similarly as described above.

In any cases described above, since the annotation image data Ad2 is combined when the instruction-side combined image data Cd2 is generated, the combined image <NUM> (see <FIG>) in which the annotation images <NUM>, <NUM> (see <FIG>) are superimposed on the endoscope image <NUM> (see <FIG>) is displayed on the monitor <NUM>.

Thus, by generating the instruction-side combined image data Cd2 in the instruction-side system <NUM>, the instruction-side combined image data Cd2 can be supplied to the monitor <NUM> without delay compared with the case where the instruction-side combined image data Cd2 is generated on the operation-side system <NUM> based on the transmitted drawing information PI.

This allows performing an image display that does not cause the instructor <NUM> performing an operation input on the touch panel <NUM> to feel a stress, on the monitor <NUM>.

In particular, by further combining the annotation image data Ad2 on the operation-side combined image data Cd1 combined with the annotation image data Ad1 to generate the instruction-side combined image data Cd2, the instructor <NUM> can confirm on the monitor <NUM> whether or not there is any displacement in the annotation images <NUM>, <NUM> displayed based on each of the operation-side combined image data Cd1 and the annotation image data Ad2.

When the displacement occurs, for example, the instructor <NUM> can confirm the displacements of the annotation images <NUM>, <NUM> as illustrated in <FIG> on the monitor <NUM>.

Here, in considering a point of confirmation of the display contents, the image generation unit <NUM> may generate the annotation image data Ad2 as image data having a display mode different from that of the annotation image data Ad1 generated by the image generation unit <NUM>.

For example, as illustrated in <FIG>, displaying the annotation images <NUM>, <NUM> based on the annotation image data Ad2 with dashed lines allows displaying them to be on the monitor <NUM> in a state of being distinguished from the display (solid line) of the annotation image <NUM> based on the annotation image data Ad1 included in the operation-side combined image data Cd1.

In <FIG>, while the portion where the dashed line runs along the solid line is indicated as such for convenience of explanation, it is actually a portion where they overlap with one another.

As described above, causing both the annotation images <NUM>, <NUM> based on the annotation image data Ad1 and the annotation images <NUM>, <NUM> based on the annotation image data Ad2 to be displayed and allowing them to be confirmed has the following meaning.

In the drawing information PI transmitted from the instruction-side system <NUM>, it is likely to occur that a part of the drawing information PI is lost when received by the operation-side system <NUM> due to, for example, factors such as communication failure, and/or display coordinates of the annotation images <NUM>, <NUM> and the like are displaced due to, for example, unadjusted settings between respective systems (<NUM>, <NUM>) or unadjusted difference in resolution between the monitors <NUM> and <NUM>, and the like.

In this case, an image like <FIG> is displayed on the monitor <NUM> due to displacement of the display coordinates, in some cases.

When a part of the information is lost, in the image generation unit <NUM> of the operation-side system <NUM>, the annotation image data Ad1 for displaying the annotation images <NUM>, <NUM> as illustrated in <FIG> are supposed to be generated based on the received drawing information PI. However, due to the loss of a part of the drawing information PI, the annotation image data Ad1 in which only the annotation image <NUM> as illustrated in <FIG> is displayed (hereinafter also noted as deficient annotation image data Ad1) is generated, in some cases.

At this time, in the image combining unit <NUM>, the deficient annotation image data Ad1 generated by the image generation unit <NUM> and the endoscope image data Id are combined, and the combined image data Cd1 for displaying a combined image <NUM> in which only the annotation image <NUM> is superimposed on the endoscope image <NUM> is generated, as illustrated in <FIG>.

Thus, based on the supplied operation-side combined image data Cd1, the combined image <NUM> lacking the annotation image <NUM> is displayed on the monitor <NUM>. In such a state, the inputs of the instructor <NUM> are not accurately reflected on the monitor <NUM>, and the surgical operator <NUM> performs the surgical operation in a state where the surgical operator <NUM> cannot grasp all of the instructions of the instructor <NUM>.

On the other hand, in the instruction-side system <NUM>, since the operation-side combined image data Cd1 received from the operation-side system <NUM> and the annotation image data Ad2 for displaying the annotation images <NUM>, <NUM> generated by the image generation unit <NUM> are combined, the instruction-side combined image data Cd2 for displaying the combined image <NUM> (see <FIG>) is generated regardless of whether or not the annotation image data Ad1 of the operation-side combined image data Cd1 is lost.

Since this displays the combined image <NUM> based on the instruction-side combined image data Cd2 on the monitor <NUM>, the instructor <NUM> cannot confirm that the annotation image <NUM> is lost in the display of the monitor <NUM> on the surgical operator <NUM> side.

In order for the instructor <NUM> to confirm the display condition of the monitor <NUM> on the surgical operator <NUM> side, it is preferred to cause the images based on the annotation image data Ad1, Ad2 to be displayed in a different mode.

Thus, the image generation unit <NUM> generates the annotation image data Ad2 for causing the annotation images <NUM>, <NUM> to be displayed by, for example, dashed lines, based on the drawing information PI.

The image combining unit <NUM> combines the annotation image data Ad2 supplied from the image generation unit <NUM> and the operation-side combined image data Cd1 received from the operation-side system <NUM> to generate the combined image data Cd2.

By the generated instruction-side combined image data Cd2 being supplied to the monitor <NUM>, a combined image <NUM> as illustrated in <FIG> is displayed on the monitor <NUM>.

This allows the instructor <NUM> to confirm that the portion of the annotation image <NUM> displayed with only the dashed line is not displayed on the monitor <NUM> in the operation-side system <NUM>.

That is, on the monitor <NUM>, the instructor <NUM> can easily compare his/her own instruction contents with the contents on the monitor <NUM> viewed by the surgical operator <NUM>.

As described above, by observing the image based on the instruction-side combined image data Cd2, the instructor <NUM> can see both the images based on the annotation image data Ad1, Ad2 and can confirm whether or not his/her own instructions are correctly transmitted to the surgical operator <NUM>. When being able to recognize that the instructions by the image is not correctly transmitted, the instructor <NUM> can take necessary countermeasures.

In the present embodiment, as an example of a display mode for distinguishing the annotation image data Ad1 from the annotation image data Ad2 in the instruction-side combined image data Cd2, the dashed line and the solid line are used to perform distinction. However, the display mode is not limited to this as long as each of the annotation image data Ad1 and the annotation image data Ad2 is distinguishable, and the distinction can be performed by differentiating the line type such as the line width, the color, the luminance, or the like. The distinction can be performed by highlighting any one of them.

Among the annotation image data Ad2 generated by the image generation unit <NUM>, it is also conceivable to distinguish and display a portion that does not match the annotation image data Ad1.

According to the present embodiment described above, the assistance system <NUM> includes a first system (the operation-side system <NUM>) and a second system (the instruction-side system <NUM>) separated from the operation-side system <NUM>.

Among them, the information processing device included in the instruction-side system <NUM> includes the data transmission unit <NUM>, the image generation unit <NUM>, and the image combining unit <NUM> (see <FIG>).

The data transmission unit <NUM> transmits the drawing information PI based on the operation input, which is necessary for generating the annotation image data Ad1, to the operation-side system <NUM>. The image generation unit <NUM> generates the annotation image data Ad2 from the drawing information PI. Further, the image combining unit <NUM> combines the annotation image data Ad2 generated by the image generation unit <NUM> and the operation-side combined image data Cd1 (or the endoscope image data Id) received from the operation-side system <NUM>.

By generating the annotation image data Ad2 and combining the generated annotation image data Ad2 and the operation-side combined image data Cd1 (or the endoscope image data Id) to generate the instruction-side combined image data Cd2 in the instruction-side system <NUM>, the instruction-side combined image data Cd2 can be obtained more quickly than receiving one generated by the operation-side system <NUM>.

Accordingly, the instructor <NUM> can confirm the pictorial figure and the like input by himself/herself on the monitor <NUM> without being affected by communication delay or the like. Consequently, since the instructor <NUM> can comfortably and accurately perform the input, appropriate instructions for the surgical operator <NUM> can be performed.

According to the present embodiment, the image generation unit <NUM> of the instruction-side system <NUM> generates the annotation image data Ad2 that is distinguishable from the annotation image data Ad1 included in the operation-side combined image data Cd1.

This allows distinguishing the display based on the annotation image data Ad1 from the display based on the annotation image data Ad2 and confirming whether or not the annotation image data Ad1 reflecting the drawing information PI based on the input in the instruction-side system <NUM> is generated in the operation-side system <NUM> (see <FIG>).

Accordingly, the instructor <NUM>, who is looking at the monitor <NUM>, can easily confirm whether or not his/her own instruction contents are appropriately displayed on the monitor <NUM> at which the surgical operator <NUM> is looking.

According to the present embodiment, the information processing device included in the operation-side system <NUM> includes the image generation unit <NUM>, the image combining unit <NUM>, and the data transmission unit <NUM> (see <FIG>).

The image generation unit <NUM> receives the drawing information PI based on the operation input from the instruction-side system <NUM> and generates the annotation image data Ad1 from the received drawing information PI. The image combining unit <NUM> combines the input image data (the endoscope image data Id) and the annotation image data Ad1 generated by the image generation unit <NUM> to generate the operation-side combined image data Cd1. Further, the data transmission unit <NUM> transmits the operation-side combined image data Cd1 (or the endoscope image data Id) generated by the image combining unit <NUM> to the instruction-side system <NUM>.

Here, when the operation-side combined image data Cd1 is generated in the operation-side system <NUM>, instead of the annotation image data Ad2, the drawing information PI having the communication volume less than the annotation image data Ad2 is received from the instruction-side system <NUM>.

This allows reducing the communication delay when the data is transmitted to the operation-side system <NUM> from the instruction-side system <NUM> and generating the annotation image data Ad1 based on the drawing information PI without delay in the operation-side system <NUM>.

Consequently, the instructions by the instructor <NUM> can be displayed on the monitor <NUM> without delay, and the surgical operator <NUM> can immediately confirm the instructions from the instructor <NUM>. That is, the surgical operator <NUM> can perform a safer surgical operation on the patient <NUM>.

In the present embodiment, while the example in which the image data input to the image combining unit <NUM> is the endoscope image data Id has been described, the input image data is not limited to the captured image data like the endoscope image data Id supplied from the endoscope <NUM>, image data recorded in a memory and obtained by reading from the memory, image data received from an external computer device or the like, and the like can be considered in various manners.

In the present embodiment, it is preferred that the delay time from when the endoscope image data Id and the annotation image data Ad1 are input until the operation-side combined image data Cd1 is output by the image generation unit <NUM> is less than <NUM> milliseconds.

Since this allows the surgical operator <NUM> to visually recognize the combined image <NUM> and the like on the monitor <NUM> with low latency, the accurate and safety surgical operation can be performed on the patient <NUM> with the instructions of the instructor <NUM> confirmed.

In the present embodiment, as one example of the assistance system <NUM>, the surgery assistance system in which the surgical operator <NUM> can perform the surgical operation on the patient <NUM> while receiving the instructions of the instructor <NUM> at a remote location has been described. However, the assistance system <NUM> can be widely applied to a situation in which the instructor <NUM> at a remote location gives the instructions to the surgical operator <NUM> while visually recognizing the captured image on the operation-side.

For example, the assistance system <NUM> can be applied to various uses, such as athletes and head coaches in sports instruction, instructors and students in learning assistance, such as education and vocational training, and presenters and listeners in remote conferences.

In the present embodiment, while the example in which the endoscope <NUM> captures the image as illustrated in <FIG> has been described, it is only necessary that an image capturing device is one that can capture an image of the surgical operation on the surgical operator <NUM> side, and it is not limited to the endoscope <NUM>.

In the present embodiment, as one example of the instruction terminal <NUM> in <FIG>, the tablet terminal having the touch panel has been described. However, the instruction terminal <NUM> can be achieved by various devices, for example, a computer mouse for an operation input, a Personal Computer (PC) having a keyboard and the like, a Virtual Reality (VR) having a head-mounted display and a remote controller for an operation input, a master console for a surgical robot, and the like.

Claim 1:
An information processing device included in an instruction-side system (<NUM>) of an assistance system (<NUM>) including an operation-side system (<NUM>) and the instruction-side system (<NUM>) separated from the operation-side system (<NUM>), the information processing device comprising:
an instruction-side data transmission unit (<NUM>) that transmits surgical instructions drawing information (PI) to the operation-side system (<NUM>), wherein the surgical instructions drawing information (PI) is based on an operation input of an instructor (<NUM>) and is necessary for generating operation-side annotation image data (Ad1) in the operation-side system (<NUM>);
an image generation unit (<NUM>) that generates instruction-side annotation image data (Ad2) from the surgical instructions drawing information (PI); and
an instruction-side image combining unit (<NUM>) that combines the instruction-side annotation image data (Ad2) generated by the instruction-side image generation unit (<NUM>) and operation-side combined image data (Cd1) received from the operation-side system (<NUM>) and outputs instruction-side combined image data (Cd2) on a monitor (<NUM>) of the instruction-side system (<NUM>), wherein
in the operation-side system (<NUM>), the annotation image data (Ad1) is generated from the surgical instructions drawing information (PI), and the operation-side combined image data (Cd1) is generated by combining a plurality of pieces of image data (Id) including the generated annotation image data (Ad1).