Patent Description:
Heretofore, in an image pickup environment where a video camera is used, an intercom is known to be used by a monitoring person to vocally give a photographer (i.e., camera operator) image pickup instructions such as an image pickup targeting instruction, zoom-in and zoom-out instructions, and a pan instruction. In a case where such instructions are given by voice, there may conceivably be cases in which the monitoring person has difficulty in correctly conveying the details of an instruction to the camera operator.

For example, PTL <NUM> describes a remote control apparatus having a display unit that displays the same details as those displayed on a viewfinder screen of the video camera when menus are set on the video camera.

[PTL <NUM>]
<CIT> Prior Art Includes: <CIT>; <CIT>; <CIT>; and <CIT>.

An object of the present technology is to enable the display of instruction information regarding image data from external equipment.

Embodiments of the present disclosure are defined by the appended Claims.

According to the present technology, the display of instruction information regarding image data from external equipment is made possible. Incidentally, the advantageous effects stated in this description are only examples and are not limitative of the present technology. There may be additional advantageous effects derived from this description.

Preferred embodiments for implementing the present technology (referred to as the embodiments) are described below. The description will be given under the following headings:.

<FIG> depicts an example of the configuration of an image pickup system <NUM> as one embodiment of the present technology. The image pickup system <NUM> includes a video camera <NUM> and a picked-up image monitoring apparatus <NUM>.

The video camera <NUM> obtains moving image data by imaging a subject. The video camera <NUM> further generates monitoring-use image data on the basis of the moving image data. The video camera <NUM> transmits the monitoring-use image data to the picked-up image monitoring apparatus <NUM> via a network such as any one of wireless networks including Wi-Fi or of wired networks. The monitoring-use image data is made smaller in size and lower in transmission bit rate than the original image data.

The picked-up image monitoring apparatus <NUM> is configured with terminal equipment such as a smartphone, a tablet, or a personal computer, for example. The picked-up image monitoring apparatus <NUM> receives the monitoring-use image data transmitted from the video camera <NUM> via the network. The picked-up image monitoring apparatus <NUM> then causes a display unit to display an image (picked-up image) derived from the monitoring-use image data. Incidentally, the picked-up image monitoring apparatus <NUM> need not necessarily be physically integrated with the display unit.

On the picked-up image monitoring apparatus <NUM>, the user as the monitoring person (i.e., operator) may perform various image pickup instruction operations accompanied by position designation on the image displayed on the display unit. For example, the user performs such image pickup instruction operations using a touch panel provided on the display unit or a pointing device such as a mouse.

<FIG> depicts an example of a correspondence table listing the correspondence between the operations performed by the user on the touch panel on the one hand and instruction types and display details on the other hand. For example, the picked-up image monitoring apparatus <NUM> is provided with multiple default correspondence tables. The picked-up image monitoring apparatus <NUM> is enabled to select one of the multiple default correspondence tables for actual use. The picked-up image monitoring apparatus <NUM> is further enabled either to modify the selected default correspondence table as needed for actual use or to create a new correspondence table.

Although the example in the illustration indicates the correspondence table assumed to be used for operations on the touch panel, a correspondence table may similarly be configured for operations involving the use of a pointing device such as a mouse. In the illustrated example, the items of color, shape, and movement constitute display information representing display details (marks) displayed on the screen to indicate the instruction types. Also in the illustrated example, the item of instruction type constitutes display information representing display details (texts) displayed on the screen to indicate the instruction types.

In a case where, as will be discussed later, the user performs an image pickup instruction operation on the picked-up image monitoring apparatus <NUM>, both the picked-up image monitoring apparatus <NUM> and the video camera <NUM> displays the instruction type on the screen displaying the picked-up image. In this case, three types of display details are conceivable: a mark only, a text only, and both a mark and a text. Both the picked-up image monitoring apparatus <NUM> and the video camera <NUM> are enabled to set one of the types of display details as the type of display.

The picked-up image monitoring apparatus <NUM> transmits beforehand information regarding the correspondence table to be actually used to the video camera <NUM>. This allows the picked-up image monitoring apparatus <NUM> and the video camera <NUM> to share the correspondence table information therebetween.

What follows is an explanation of the illustrated example. Identification information ID = "<NUM>" is associated with the operation "single tap," the instruction type "point," the color "blue," the shape "circle," and the movement "fixed. " Identification information ID = "<NUM>" is associated with the operation "double tap," the instruction type "zoom out," the color "orange," the shape "circle," and the movement "from inside to outside.

Identification information ID = "<NUM>" is associated with the operation "long tap," the instruction type "zoom in," the color "blue," the shape "rectangle," and the movement "from outside to inside. " Identification information ID="<NUM>" is associated with the operation "flick," the instruction type "fast pan," the color "light blue," the shape "arrow," and the movement "fixed. " Identification information ID="<NUM>" is associated with the operation "swipe," the instruction type "slow pan," the color "yellowish green," the shape "arrow," and the movement "fixed.

In a case where the user performs an image pickup instruction operation on the touch panel, the picked-up image monitoring apparatus <NUM> displays, using a mark for example, the instruction type in a manner corresponding to the operation position on the image displayed on the display unit. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the image pickup instruction operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet including position information and display information (ID) as image pickup instruction information. On the basis of the position information and the display information, the video camera <NUM> displays, using a mark for example, the details corresponding to the display information, i.e., the instruction type in the position corresponding to the position information associated with the image displayed on the display unit (finder). This allows the camera operator of the video camera <NUM> to recognize that the image pickup instruction of the displayed instruction type has been issued and to carry out accordingly an image pickup operation in line with the image pickup instruction.

<FIG> depicts an example of the configuration of a data packet. A field (a) holds identification information identifying this data packet including the image pickup instruction information. A field (b) holds ID (see <FIG>) identifying the instruction type. A field (c) holds the x coordinate of the operation position. A field (d) holds the y coordinate of the operation position. A field (e) holds the x coordinate of the ending point of a displayed arrow originating from the operation position as the starting point. A field (f) holds the y coordinate of the ending point. The origin of the coordinates (<NUM>, <NUM>) is at the top left corner, for example.

The display unit of the picked-up image monitoring apparatus <NUM> also displays a cancel button. The user (operator) of the picked-up image monitoring apparatus <NUM> may cancel the most-recently performed image pickup instruction operation by operating the cancel button. The picked-up image monitoring apparatus <NUM> need not be configured to have the cancel button displayed on the image. Alternatively, the picked-up image monitoring apparatus <NUM> may be equipped with a mechanical cancel button.

In a case where the user (operator) of the picked-up image monitoring apparatus <NUM> operates the cancel button, the picked-up image monitoring apparatus <NUM> indicates cancellation of the most-recently issued image pickup instruction by displaying, for example, characters such as "cancel" on the image, not depicted. In this case, the picked-up image monitoring apparatus <NUM> transmits cancellation information to the video camera <NUM>. On the basis of the cancellation information, the video camera <NUM> indicates cancellation of the most-recently issued image pickup instruction by displaying, for example, characters such as "cancel" on the image. This allows the camera operator of the video camera <NUM> to recognize cancellation of the most-recently issued image pickup instruction.

Explained below as an example is a case where the user performs the "single tap" operation on the touch panel. The picked-up image monitoring apparatus <NUM> determines that the user has issued a "point" instruction, i.e., an image pickup targeting instruction. In this case, the picked-up image monitoring apparatus <NUM> displays the image pickup targeting instruction corresponding to the tapped position on the image displayed on the display unit. In this case, for example, a blue, fixed circular mark is displayed for a predetermined time period. Alternatively, a text "point" is displayed for a predetermined time period singly or in combination with the mark.

<FIG> depicts an exemplary display state in a case where the user performs a "single tap" operation, i.e., an image pickup targeting instruction operation, on the image displayed on the display unit of the picked-up image monitoring apparatus <NUM>. In this case, a mark indicative of the image pickup targeting instruction is displayed corresponding to the position where the user has performed the "single tap" operation on the image. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the image pickup targeting instruction operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet holding the display information "ID="<NUM>"" accompanied by position information regarding the tapped position as the image pickup instruction information. On the basis of the position information and the display information, the video camera <NUM> displays on the image a mark indicative of the image pickup targeting instruction corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "single tap" operation.

<FIG> depicts an example of the display state on the display unit of the video camera <NUM> in the above case. This allows the camera operator of the video camera <NUM> to recognize that the image corresponding to the mark display position is the image pickup target. Accordingly, the camera operator performs an image pickup operation such that the image pickup target is at the center of the picked-up image, for example. Note that an example in <FIG> indicates that the instruction type is displayed using the text "point. " Conceivably, the instruction type may also be displayed using both a mark and a text, not illustrated.

Explained below as another example is a case in which the user performs a "double tap" operation on the touch panel. The picked-up image monitoring apparatus <NUM> determines that the user has issued a "zoom out" instruction. In this case, the picked-up image monitoring apparatus <NUM> displays the zoom-out instruction being issued corresponding to the tapped position on the image displayed on the display unit. In the present case, for example, an orange circular mark expanding repeatedly from inside to outside is displayed for a predetermined time period. Alternatively, a text "zoom out" is displayed for a predetermined time period singly or in combination with the mark. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the zoom-out instruction operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet holding the display information "ID="<NUM>"" accompanied by position information regarding the tapped position as the image pickup instruction information. On the basis of the position information and the display information, the video camera <NUM> displays on the image a mark and/or a text indicative of the zoom-out instruction in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "double tap" operation.

<FIG> depicts an example of the display state on the display unit of the video camera <NUM> in the above case. In this example, the display unit displays a mark indicative of the zoom-out instruction, i.e., an orange circular mark expanding repeatedly from inside to outside in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "double tap" operation. This allows the camera operator of the video camera <NUM> to recognize the zoom-out instruction for a zoom-out centering on the image corresponding to the mark display position. Accordingly, the camera operator performs a zoom-out operation centering on the image corresponding to the mark display position. <FIG> depicts an example of a fixed mark indicative of the zoom-out instruction.

Furthermore, explained below as another example is a case in which the user performs a "long tap" operation on the touch panel. The picked-up image monitoring apparatus <NUM> determines that the user has issued a "zoom in" instruction. In this case, the picked-up image monitoring apparatus <NUM> displays the zoom-in instruction being issued corresponding to the tapped position on the image displayed on the display unit. In the present case, for example, a blue rectangular mark contracting repeatedly from outside to inside is displayed for a predetermined time period. Alternatively, a text "zoom in" is displayed for a predetermined time period singly or in combination with the mark. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the zoom-in instruction operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet holding the display information "ID="<NUM>"" accompanied by position information regarding the tapped position as the image pickup instruction information. On the basis of the position information and the display information, the video camera <NUM> displays on the image a mark and/or a text indicative of the zoom-in instruction in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "long tap" operation.

<FIG> depicts an example of the display state on the display unit of the video camera <NUM> in the above case. In this example, the display unit displays a mark indicative of the zoom-in instruction, i.e., a blue rectangular mark contracting repeatedly from outside to inside, in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "long tap" operation. This allows the camera operator of the video camera <NUM> to recognize the zoom-in instruction for a zoom-in centering on the image corresponding to the mark display position. Accordingly, the camera operator performs a zoom-in operation centering on the image corresponding to the mark display position. <FIG> depicts an example of a fixed mark indicative of the zoom-in instruction.

Explained below as a further example is a case in which the user performs a "flick" operation on the touch panel. The picked-up image monitoring apparatus <NUM> determines that the user has issued a "fast pan" instruction. In this case, the picked-up image monitoring apparatus <NUM> displays the fast-pan instruction being issued corresponding to the flicked position on the image displayed on the display unit.

In the present case, for example, a fixed light blue arrow mark is displayed for a predetermined time period. Alternatively, a text "fast pan" is displayed for a predetermined time period singly or in combination with the mark. For example, the fixed arrow mark originates from the starting point of the flick and is given a predetermined length in the flicked direction pointing to the pan direction. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the fast-pan operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet holding the display information "ID = "<NUM>"" accompanied by position information regarding the flicked position as the image pickup instruction information. On the basis of the position information and the display information, the video camera <NUM> displays on the image a mark and/or a text indicative of the fast-pan instruction in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "flick" operation.

<FIG> depicts an example of the display state on the display unit of the video camera <NUM> in the above case. In this example, the display unit displays a mark indicative of the fast-pan instruction as well as the pan direction, i.e., a fixed light blue arrow mark, in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "flick" operation. This allows the camera operator of the video camera <NUM> to recognize the fast-pan instruction for a fast-pan in the arrowed direction. Accordingly, the camera operator performs a fast-pan operation in that direction.

Explained below as another example is a case in which the user performs a "swipe" operation on the touch panel. The picked-up image monitoring apparatus <NUM> determines that the user has issued a "slow pan" instruction. In this case, the picked-up image monitoring apparatus <NUM> displays the slow-pan instruction being issued corresponding to the swiped position on the image displayed on the display unit.

In the present case, for example, a fixed yellowish green arrow mark is displayed for a predetermined time period. Alternatively, a text "slow pan" is displayed for a predetermined time period singly or in combination with the mark. For example, the fixed arrow mark originates from the starting point of the swipe and is given a predetermined length in the swiped direction pointing to the pan direction. This allows the user (operator) of the picked-up image monitoring apparatus <NUM> to verify the slow-pan operation that the user has carried out.

Also in this case, the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> a data packet holding the display information "ID = "<NUM>" accompanied by position information regarding the swiped position. On the basis of the position information and the display information, the video camera <NUM> displays on the image a mark and/or a text indicative of the slow-pan instruction in a manner corresponding to the position where the user of the picked-up image monitoring apparatus <NUM> has performed the "swipe" operation. This allows the camera operator of the video camera <NUM> to recognize the slow-pan instruction for a slow-pan in the arrowed direction. Accordingly, the camera operator performs a slow-pan operation in that direction.

<FIG> depicts an example of the configuration of the video camera <NUM>. The video camera <NUM> includes a control unit <NUM>, an operation inputting unit <NUM>, an image pickup unit <NUM>, a picked-up image signal processing unit <NUM>, an encoding unit <NUM>, a recording/reproducing unit <NUM>, a recording medium <NUM>, a communication unit <NUM>, a finder display processing unit <NUM>, and a finder <NUM>.

The control unit <NUM> controls the operations of the components in the video camera <NUM>. The operation inputting unit <NUM> connected with the control unit <NUM> provides a user interface that accepts various operations performed by the user.

The image pickup unit <NUM> has an image pickup lens and an image pickup element (imager), not depicted. The image pickup unit <NUM> images a subject and outputs a picked-up image signal. The image pickup element may be a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, for example. The picked-up image signal processing unit <NUM> generates picked-up image data by performing such processes as sample hold, gain control, conversion from analog signal to digital signal, white balance adjustment, and gamma correction on the picked-up image signal (analog signal) output from the image pickup unit <NUM>.

The encoding unit <NUM> generates encoded image data by performing data compression processing using the MPEG method, for example, on the picked-up image data generated by the picked-up image signal processing unit <NUM>. On the basis of the picked-up image data, the encoding unit <NUM> further generates monitoring-use image data. The monitoring-use image data is made smaller in size and lower in transmission bit rate than the picked-up image data.

The recording/reproducing unit <NUM> records the encoded image data obtained by the encoding unit <NUM> to the recording medium <NUM>. As needed, the recording/reproducing unit <NUM> reproduces the encoded image data from the recording medium <NUM>. The recording medium <NUM> is configured with a memory card, for example. Here, a series of image data from the start to the end of recording is recorded as data of one clip. On the recording medium, the image data is filed in clips and managed by a file system.

The communication unit <NUM> communicates with the picked-up image monitoring apparatus <NUM> via any one of wireless networks including Wi-Fi or of wired networks. At the time of image pickup, the communication unit <NUM> sends to the picked-up image monitoring apparatus <NUM> the monitoring-use image data obtained by the encoding unit <NUM>.

Also at the time of image pickup, when the user (operator) of the picked-up image monitoring apparatus <NUM> performs an image pickup instruction operation, the communication unit <NUM> receives a data packet (see <FIG>) including position information and display information (ID) as image pickup instruction information from the picked-up image monitoring apparatus <NUM> via the network. The data packet is sent to the control unit <NUM>. This enables the control unit <NUM> to recognize the position information and the display information (ID) as the image pickup instruction information.

Also, the communication unit <NUM> receives beforehand the information regarding the correspondence table (see <FIG>) from the picked-up image monitoring apparatus <NUM> via the network, the correspondence table denoting the correspondence between the operations performed by the user on the touch panel on the one hand and instruction types and display details on the other hand, the table being used by the picked-up image monitoring apparatus <NUM> in operation. The correspondence table information is sent to the control unit <NUM> and held in a memory. This enables the control unit <NUM> to reference the correspondence table in preparing display data for displaying the image pickup instruction details (instruction type) corresponding to the display information (ID).

Also at the time of image pickup, when the user (operator) of the picked-up image monitoring apparatus <NUM> performs an operation to cancel the most-recently issued image pickup instruction, the communication unit <NUM> receives information for cancelling the image pickup instruction from the picked-up image monitoring apparatus <NUM> via the network. Furthermore, the communication unit <NUM> communicates with an editor terminal apparatus, not depicted, and transmits the clip data reproduced from the recording medium <NUM> to the editor terminal apparatus.

The finder display processing unit <NUM> generates finder-use image data on the basis of the picked-up image data generated by the picked-up image signal processing unit <NUM>. In this case, the finder display processing unit <NUM> under control of the control unit <NUM> synthesizes and sends display data to the finder <NUM> serving as the display unit, the display data representing the image pickup instruction details (instruction type) corresponding to the display information (ID) supplied from the control unit <NUM>, the display data being synthesized in such a manner that the image pickup instruction details are displayed in the position based on the position information.

In turn, the finder <NUM> displays, for example, a mark indicative of the instruction type in the position corresponding to the position information associated with the image. This allows the camera operator of the video camera <NUM> to recognize that an image pickup instruction of the instruction type has been issued and to perform an image pickup operation in line with that image pickup instruction.

Further, upon receipt of the information for cancelling the image pickup instruction from the picked-up image monitoring apparatus <NUM>, the finder display processing unit <NUM> inserts display data for displaying the cancellation into the finder-use image data. In this case, an overlay display of the image pickup instruction cancellation appears on the image displayed on the finder <NUM>. This prompts the camera operator of the video camera <NUM> to immediately recognize the image pickup instruction being cancelled and to avoid carrying out the image pickup operation in line with the image pickup instruction.

<FIG> depicts an example of the configuration of the picked-up image monitoring apparatus <NUM>. The picked-up image monitoring apparatus <NUM> includes a control unit <NUM>, an operation inputting unit <NUM>, a communication unit <NUM>, a decoding unit <NUM>, a display processing unit <NUM>, and a display panel <NUM>.

The control unit <NUM> controls the operations of the components in the picked-up image monitoring apparatus <NUM>. The operation inputting unit <NUM> connected with the control unit <NUM> provides a user interface that accepts various operations performed by the user. The operation inputting unit <NUM> is configured, for example, with mechanical operation buttons or a touch panel mounted on a screen of the display panel <NUM>.

The user may carry out image pickup instruction operations using the touch panel mounted on the screen of the display panel <NUM>. The control unit <NUM> recognizes the instruction type of the operation performed by the user on the touch panel and the display details to which the operation corresponds on the basis of the correspondence table (see <FIG>) held in a memory of the control unit <NUM>. The user may operate the operation inputting unit <NUM> to set beforehand the correspondence table to be actually used by the picked-up image monitoring apparatus <NUM>.

The communication unit <NUM> communicates with the video camera <NUM> via any one of wireless networks including Wi-Fi or of wired networks. At the time of image pickup with the video camera <NUM>, the communication unit <NUM> receives the monitoring-use image data from the video camera <NUM> via the network.

Also at the time of image pickup with the video camera <NUM>, when the user (camera operator) performs an image pickup instruction operation, the communication unit <NUM> transmits a data packet (see <FIG>) including position information and display information (ID) as image pickup instruction information to the video camera <NUM> via the network. The data packet is supplied from the control unit <NUM> to the communication unit <NUM>.

Also at the time of image pickup with the video camera <NUM>, when the user (camera operator) performs an operation to cancel the most-recently issued image pickup instruction, the communication unit <NUM> transmits information for cancelling the image pickup instruction to the video camera <NUM> via the network.

Further, before image pickup with the video camera <NUM>, the communication unit <NUM> transmits the information regarding the correspondence table to be actually used by the picked-up image monitoring apparatus <NUM> to the video camera <NUM> via the network so that the two apparatuses may share the information therebetween. The correspondence table information is supplied from the control unit <NUM> to the communication unit <NUM>.

The decoding unit <NUM> decodes the monitoring-use image data (encoded image data) received by the communication unit <NUM>. The display processing unit <NUM> generates image data for display panel use on the basis of the monitoring-use image data obtained by the decoding unit <NUM>.

In the above case, when the user (camera operator) performs an image pickup instruction operation, the display processing unit <NUM> under control of the control unit <NUM> synthesizes and sends display data to the display panel <NUM> serving as the display unit, the display data representing the image pickup instruction details (instruction type) corresponding to the display information (ID) supplied from the control unit <NUM>, the display data being synthesized in such a manner that the image pickup instruction details are displayed in the position based on the position information.

As discussed above, the video camera <NUM> in the image pickup system <NUM> in <FIG> transmits the monitoring-use image data corresponding to the picked-up image data to the picked-up image monitoring apparatus <NUM>. The video camera <NUM> receives display information accompanied by position information from the picked-up image monitoring apparatus <NUM>. The video camera <NUM> then provides finder display data in such a manner that the image data corresponding to moving image data is overlaid with the display data indicative of the image pickup instruction details (instruction type) corresponding to the display information, the image pickup instruction details being displayed in the position based on the position information. This makes it possible to display the image pickup instruction details set by the picked-up image monitoring apparatus <NUM> in a manner associated with the position designated by the picked-up image monitoring apparatus <NUM> on the picked-up image.

Further, the picked-up image monitoring apparatus <NUM> in the image pickup system <NUM> in <FIG> receives from the video camera <NUM> the monitoring-use image data corresponding to the picked-up image data. In turn, the picked-up image monitoring apparatus <NUM> generates display information accompanied by position information based on the user's operations including designation of the position on the display image derived from the monitoring-use image data. The picked-up image monitoring apparatus <NUM> transmits the display information thus generated to the video camera <NUM>. This enables the video camera <NUM> to display the image pickup instruction details set in association with the position designated on the picked-up image.

Furthermore, the image pickup system <NUM> in <FIG> permits accurate issuing of the image pickup targeting instruction and other image pickup instructions from a remote location connected via the network. For example, it is possible to issue precise image pickup instructions from a TV station to a remotely-located camera operator. Because the image pickup target is designated without recourse to voice information, the system can be utilized extensively in situations where the use of voice is ruled out, such as on golf courses, on tennis courts, or in concert halls.

Moreover, the utilization of instruction marks or texts makes it possible to issue instructions intuitively without using language. Having the instructions displayed on the finder contributes to training novice camera operators. Also, because the image pickup system <NUM> alone completes the cycle of issuing image pickup instructions and carrying out image pickup operations, those in the field are little affected by potential troubles of audio equipment.

The technology according to the present disclosure may be applied to diverse products. For example, the technology may be applied to a surgery room system.

<FIG> is a view schematically depicting a general configuration of a surgery room system <NUM> to which the technology according to an embodiment of the present disclosure can be applied. Referring to <FIG>, the surgery room system <NUM> is configured such that a group of apparatus installed in a surgery room are connected for cooperation with each other through an audiovisual (AV) controller <NUM> and a surgery room controlling apparatus <NUM>.

In the surgery room, various apparatus may be installed. In <FIG>, as an example, various apparatus group <NUM> for endoscopic surgery, a ceiling camera <NUM>, a surgery field camera <NUM>, a plurality of display apparatus 5103A to 5103D, a recorder <NUM>, a patient bed <NUM> and an illumination <NUM> are depicted. The ceiling camera <NUM> is provided on the ceiling of a surgery room and images the hands of a surgeon. The surgery field camera <NUM> is provided on the ceiling of the surgery room and images a state of the entire surgery room.

Among the apparatus mentioned, the apparatus group <NUM> belongs to an endoscopic surgery system <NUM> hereinafter described and include an endoscope, a display apparatus which displays an image picked up by the endoscope and so forth. Various apparatus belonging to the endoscopic surgery system <NUM> are referred to also as medical equipment. Meanwhile, the display apparatus 5103A to 5103D, the recorder <NUM>, the patient bed <NUM> and the illumination <NUM> are apparatus which are equipped, for example, in the surgery room separately from the endoscopic surgery system <NUM>. The apparatus which do not belong to the endoscopic surgery system <NUM> are referred to also as non-medical equipment. The audiovisual controller <NUM> and/or the surgery room controlling apparatus <NUM> cooperatively control operation of the medical equipment and the non-medical equipment with each other.

The audiovisual controller <NUM> integrally controls processes of the medical equipment and the non-medical equipment relating to image display. Specifically, each of the apparatus group <NUM>, the ceiling camera <NUM> and the surgery field camera <NUM> from among the apparatus provided in the surgery room system <NUM> may be an apparatus having a function of sending information to be displayed during surgery (such information is hereinafter referred to as display information, and the apparatus mentioned is hereinafter referred to as apparatus of a sending source). Meanwhile, each of the display apparatus 5103A to 5103D may be an apparatus to which display information is outputted (the apparatus is hereinafter referred to also as apparatus of an output destination). Further, the recorder <NUM> may be an apparatus which serves as both of an apparatus of a sending source and an apparatus of an output destination. The audiovisual controller <NUM> has a function of controlling operation of an apparatus of a sending source and an apparatus of an output destination to acquire display information from the apparatus of a sending source and transmit the display information to the apparatus of an output destination so as to be displayed or recorded. It is to be noted that the display information includes various images picked up during surgery, various kinds of information relating to the surgery (for example, physical information of a patient, inspection results in the past or information regarding a surgical procedure) and so forth.

Specifically, to the audiovisual controller <NUM>, information relating to an image of a surgical region in a body lumen of a patient imaged by the endoscope may be transmitted as the display information from the apparatus group <NUM>. Further, from the ceiling camera <NUM>, information relating to an image of the hands of the surgeon picked up by the ceiling camera <NUM> may be transmitted as display information. Further, from the surgery field camera <NUM>, information relating to an image picked up by the surgery field camera <NUM> and illustrating a state of the entire surgery room may be transmitted as display information. It is to be noted that, if a different apparatus having an image pickup function exists in the surgery room system <NUM>, then the audiovisual controller <NUM> may acquire information relating to an image picked up by the different apparatus as display information also from the different apparatus.

Alternatively, for example, in the recorder <NUM>, information relating to such images as mentioned above picked up in the past is recorded by the audiovisual controller <NUM>. The audiovisual controller <NUM> can acquire, as display information, information relating to the images picked up in the past from the recorder <NUM>. It is to be noted that also various pieces of information relating to surgery may be recorded in advance in the recorder <NUM>.

The audiovisual controller <NUM> controls at least one of the display apparatus 5103A to 5103D, which are apparatus of an output destination, to display acquired display information (namely, images picked up during surgery or various pieces of information relating to the surgery). In the example depicted, the display apparatus 5103A is a display apparatus installed so as to be suspended from the ceiling of the surgery room; the display apparatus 5103B is a display apparatus installed on a wall face of the surgery room; the display apparatus 5103C is a display apparatus installed on a desk in the surgery room; and the display apparatus 5103D is a mobile apparatus (for example, a tablet personal computer (PC)) having a display function.

Further, though not depicted in <FIG>, the surgery room system <NUM> may include an apparatus outside the surgery room. The apparatus outside the surgery room may be, for example, a server connected to a network constructed inside and outside the hospital, a PC used by medical staff, a projector installed in a meeting room of the hospital or the like. Where such an external apparatus is located outside the hospital, also it is possible for the audiovisual controller <NUM> to cause display information to be displayed on a display apparatus of a different hospital through a teleconferencing system or the like to perform telemedicine.

The surgery room controlling apparatus <NUM> integrally controls processes other than processes relating to image display on the non-medical equipment. For example, the surgery room controlling apparatus <NUM> controls driving of the patient bed <NUM>, the ceiling camera <NUM>, the surgery field camera <NUM> and the illumination <NUM>.

In the surgery room system <NUM>, a centralized operation panel <NUM> is provided such that it is possible to issue an instruction regarding image display to the audiovisual controller <NUM> or issue an instruction regarding operation of the non-medical equipment to the surgery room controlling apparatus <NUM> through the centralized operation panel <NUM>. The centralized operation panel <NUM> is configured by providing a touch panel on a display face of a display apparatus.

<FIG> is a view depicting an example of display of an operation screen image on the centralized operation panel <NUM>. In <FIG>, as an example, an operation screen image is depicted which corresponds to a case in which two display apparatus are provided as apparatus of an output destination in the surgery room system <NUM>. Referring to <FIG>, the operation screen image <NUM> includes a sending source selection region <NUM>, a preview region <NUM> and a control region <NUM>.

In the sending source selection region <NUM>, the sending source apparatus provided in the surgery room system <NUM> and thumbnail screen images representative of display information the sending source apparatus have are displayed in an associated manner with each other. A user can select display information to be displayed on the display apparatus from any of the sending source apparatus displayed in the sending source selection region <NUM>.

In the preview region <NUM>, a preview of screen images displayed on two display apparatus (Monitor <NUM> and Monitor <NUM>) which are apparatus of an output destination is displayed. In the example depicted, four images are displayed by picture in picture (PinP) display in regard to one display apparatus. The four images correspond to display information sent from the sending source apparatus selected in the sending source selection region <NUM>. One of the four images is displayed in a comparatively large size as a main image while the remaining three images are displayed in a comparatively small size as sub images. The user can exchange between the main image and the sub images by suitably selecting one of the images from among the four images displayed in the region. Further, a status displaying region <NUM> is provided below the region in which the four images are displayed, and a status relating to surgery (for example, elapsed time of the surgery, physical information of the patient and so forth) may be displayed suitably in the status displaying region <NUM>.

A sending source operation region <NUM> and an output destination operation region <NUM> are provided in the control region <NUM>. In the sending source operation region <NUM>, a graphical user interface (GUI) part for performing an operation for an apparatus of a sending source is displayed. In the output destination operation region <NUM>, a GUI part for performing an operation for an apparatus of an output destination is displayed. In the example depicted, GUI parts for performing various operations for a camera (panning, tilting and zooming) in an apparatus of a sending source having an image pickup function are provided in the sending source operation region <NUM>. The user can control operation of the camera of an apparatus of a sending source by suitably selecting any of the GUI parts. It is to be noted that, though not depicted, where the apparatus of a sending source selected in the sending source selection region <NUM> is a recorder (namely, where an image recorded in the recorder in the past is displayed in the preview region <NUM>), GUI parts for performing such operations as reproduction of the image, stopping of reproduction, rewinding, fast-feeding and so forth may be provided in the sending source operation region <NUM>.

Further, in the output destination operation region <NUM>, GUI parts for performing various operations for display on a display apparatus which is an apparatus of an output destination (swap, flip, color adjustment, contrast adjustment and switching between two dimensional (2D) display and three dimensional (3D) display) are provided. The user can operate the display of the display apparatus by suitably selecting any of the GUI parts.

It is to be noted that the operation screen image to be displayed on the centralized operation panel <NUM> is not limited to the depicted example, and the user may be able to perform operation inputting to each apparatus which can be controlled by the audiovisual controller <NUM> and the surgery room controlling apparatus <NUM> provided in the surgery room system <NUM> through the centralized operation panel <NUM>.

<FIG> is a view illustrating an example of a state of surgery to which the surgery room system described above is applied. The ceiling camera <NUM> and the surgery field camera <NUM> are provided on the ceiling of the surgery room such that it can image the hands of a surgeon (medical doctor) <NUM> who performs treatment for an affected area of a patient <NUM> on the patient bed <NUM> and the entire surgery room. The ceiling camera <NUM> and the surgery field camera <NUM> may include a magnification adjustment function, a focal distance adjustment function, an imaging direction adjustment function and so forth. The illumination <NUM> is provided on the ceiling of the surgery room and irradiates at least upon the hands of the surgeon <NUM>. The illumination <NUM> may be configured such that the irradiation light amount, the wavelength (color) of the irradiation light, the irradiation direction of the light and so forth can be adjusted suitably.

The endoscopic surgery system <NUM>, the patient bed <NUM>, the ceiling camera <NUM>, the surgery field camera <NUM> and the illumination <NUM> are connected for cooperation with each other through the audiovisual controller <NUM> and the surgery room controlling apparatus <NUM> (not depicted in <FIG>) as depicted in <FIG>. The centralized operation panel <NUM> is provided in the surgery room, and the user can suitably operate the apparatus existing in the surgery room through the centralized operation panel <NUM> as described hereinabove.

In the following, a configuration of the endoscopic surgery system <NUM> is described in detail. As depicted, the endoscopic surgery system <NUM> includes an endoscope <NUM>, other surgical tools <NUM>, a supporting arm apparatus <NUM> which supports the endoscope <NUM> thereon, and a cart <NUM> on which various apparatus for endoscopic surgery are mounted.

In endoscopic surgery, in place of incision of the abdominal wall to perform laparotomy, a plurality of tubular aperture devices called trocars 5139a to 5139d are used to puncture the abdominal wall. Then, a lens barrel <NUM> of the endoscope <NUM> and the other surgical tools <NUM> are inserted into body lumens of the patient <NUM> through the trocars 5139a to 5139d. In the example depicted, as the other surgical tools <NUM>, a pneumoperitoneum tube <NUM>, an energy treatment tool <NUM> and forceps <NUM> are inserted into body lumens of the patient <NUM>. Further, the energy treatment tool <NUM> is a treatment tool for performing incision and peeling of a tissue, sealing of a blood vessel or the like by high frequency current or ultrasonic vibration. However, the surgical tools <NUM> depicted are mere examples at all, and as the surgical tools <NUM>, various surgical tools which are generally used in endoscopic surgery such as, for example, a pair of tweezers or a retractor may be used.

An image of a surgical region in a body lumen of the patient <NUM> picked up by the endoscope <NUM> is displayed on a display apparatus <NUM>. The surgeon <NUM> would use the energy treatment tool <NUM> or the forceps <NUM> while watching the image of the surgical region displayed on the display apparatus <NUM> on the real time basis to perform such treatment as, for example, resection of an affected area. It is to be noted that, though not depicted, the pneumoperitoneum tube <NUM>, the energy treatment tool <NUM>, and the forceps <NUM> are supported by the surgeon <NUM>, an assistant or the like during surgery.

The supporting arm apparatus <NUM> includes an arm unit <NUM> extending from a base unit <NUM>. In the example depicted, the arm unit <NUM> includes joint portions 5147a, 5147b and 5147c and links 5149a and 5149b and is driven under the control of an arm controlling apparatus <NUM>. The endoscope <NUM> is supported by the arm unit <NUM> such that the position and the posture of the endoscope <NUM> are controlled. Consequently, stable fixation in position of the endoscope <NUM> can be implemented.

The endoscope <NUM> includes the lens barrel <NUM> which has a region of a predetermined length from a distal end thereof to be inserted into a body lumen of the patient <NUM>, and a camera head <NUM> connected to a proximal end of the lens barrel <NUM>. In the example depicted, the endoscope <NUM> is depicted which is configured as a hard mirror having the lens barrel <NUM> of the hard type. However, the endoscope <NUM> may otherwise be configured as a soft mirror having the lens barrel <NUM> of the soft type.

The lens barrel <NUM> has, at a distal end thereof, an opening in which an objective lens is fitted. A light source apparatus <NUM> is connected to the endoscope <NUM> such that light generated by the light source apparatus <NUM> is introduced to a distal end of the lens barrel <NUM> by a light guide extending in the inside of the lens barrel <NUM> and is applied toward an observation target in a body lumen of the patient <NUM> through the objective lens. It is to be noted that the endoscope <NUM> may be a direct view mirror or may be a perspective view mirror or a side view mirror.

An optical system and an image pickup element are provided in the inside of the camera head <NUM> such that reflected light (observation light) from an observation target is condensed on the image pickup element by the optical system. The observation light is photoelectrically converted by the image pickup element to generate an electric signal corresponding to the observation light, namely, an image signal corresponding to an observation image. The image signal is transmitted as RAW data to a CCU <NUM>. It is to be noted that the camera head <NUM> has a function incorporated therein for suitably driving the optical system of the camera head <NUM> to adjust the magnification and the focal distance.

It is to be noted that, in order to establish compatibility with, for example, a stereoscopic vision (3D display), a plurality of image pickup elements may be provided on the camera head <NUM>. In this case, a plurality of relay optical systems are provided in the inside of the lens barrel <NUM> in order to guide observation light to the plurality of respective image pickup elements.

The CCU <NUM> includes a central processing unit (CPU), a graphics processing unit (GPU) or the like and integrally controls operation of the endoscope <NUM> and the display apparatus <NUM>. Specifically, the CCU <NUM> performs, for an image signal received from the camera head <NUM>, various image processes for displaying an image based on the image signal such as, for example, a development process (demosaic process). The CCU <NUM> provides the image signal for which the image processes have been performed to the display apparatus <NUM>. Further, the audiovisual controller <NUM> depicted in FIG. B1 is connected to the CCU <NUM>. The CCU <NUM> provides the image signal for which the image processes have been performed also to the audiovisual controller <NUM>. Further, the CCU <NUM> transmits a control signal to the camera head <NUM> to control driving of the camera head <NUM>. The control signal may include information relating to an image pickup condition such as a magnification or a focal distance. The information relating to an image pickup condition may be inputted through the inputting apparatus <NUM> or may be inputted through the centralized operation panel <NUM> described hereinabove.

The display apparatus <NUM> displays an image based on an image signal for which the image processes have been performed by the CCU <NUM> under the control of the CCU <NUM>. If the endoscope <NUM> is ready for imaging of a high resolution such as <NUM> (horizontal pixel number <NUM> × vertical pixel number <NUM>), <NUM> (horizontal pixel number <NUM> × vertical pixel number <NUM>) or the like and/or ready for 3D display, then a display apparatus by which corresponding display of the high resolution and/or 3D display are possible may be used as the display apparatus <NUM>. Where the apparatus is ready for imaging of a high resolution such as <NUM> or <NUM>, if the display apparatus used as the display apparatus <NUM> has a size of equal to or not less than <NUM> inches, then a more immersive experience can be obtained. Further, a plurality of display apparatus <NUM> having different resolutions and/or different sizes may be provided in accordance with purposes.

The light source apparatus <NUM> includes a light source such as, for example, a light emitting diode (LED) and supplies irradiation light for imaging of a surgical region to the endoscope <NUM>.

The arm controlling apparatus <NUM> includes a processor such as, for example, a CPU and operates in accordance with a predetermined program to control driving of the arm unit <NUM> of the supporting arm apparatus <NUM> in accordance with a predetermined controlling method.

An inputting apparatus <NUM> is an input interface for the endoscopic surgery system <NUM>. A user can perform inputting of various kinds of information or instruction inputting to the endoscopic surgery system <NUM> through the inputting apparatus <NUM>. For example, the user would input various kinds of information relating to surgery such as physical information of a patient, information regarding a surgical procedure of the surgery and so forth through the inputting apparatus <NUM>. Further, the user would input, for example, an instruction to drive the arm unit <NUM>, an instruction to change an image pickup condition (type of irradiation light, magnification, focal distance or the like) by the endoscope <NUM>, an instruction to drive the energy treatment tool <NUM> or a like through the inputting apparatus <NUM>.

The type of the inputting apparatus <NUM> is not limited and may be that of any one of various known inputting apparatus. As the inputting apparatus <NUM>, for example, a mouse, a keyboard, a touch panel, a switch, a foot switch <NUM> and/or a lever or the like may be applied. Where a touch panel is used as the inputting apparatus <NUM>, it may be provided on the display face of the display apparatus <NUM>.

The inputting apparatus <NUM> is otherwise a device to be mounted on a user such as, for example, a glasses type wearable device or a head mounted display (HMD), and various kinds of inputting are performed in response to a gesture or a line of sight of the user detected by any of the devices mentioned. Further, the inputting apparatus <NUM> includes a camera which can detect a motion of a user, and various kinds of inputting are performed in response to a gesture or a line of sight of a user detected from a video picked up by the camera. Further, the inputting apparatus <NUM> includes a microphone which can collect the voice of a user, and various kinds of inputting are performed by voice through the microphone. By configuring the inputting apparatus <NUM> such that various kinds of information can be inputted in a contactless fashion in this manner, especially a user who belongs to a clean area (for example, the surgeon <NUM>) can operate an apparatus belonging to an unclean area in a contactless fashion. Further, since the user can operate an apparatus without releasing a possessed surgical tool from its hand, the convenience to the user is improved.

A treatment tool controlling apparatus <NUM> controls driving of the energy treatment tool <NUM> for cautery or incision of a tissue, sealing of a blood vessel or the like. A pneumoperitoneum apparatus <NUM> feeds gas into a body lumen of the patient <NUM> through the pneumoperitoneum tube <NUM> to inflate the body lumen in order to secure the field of view of the endoscope <NUM> and secure the working space for the surgeon. A recorder <NUM> is an apparatus capable of recording various kinds of information relating to surgery. A printer <NUM> is an apparatus capable of printing various kinds of information relating to surgery in various forms such as a text, an image or a graph.

In the following, especially a characteristic configuration of the endoscopic surgery system <NUM> is described in more detail.

The supporting arm apparatus <NUM> includes the base unit <NUM> serving as a base, and the arm unit <NUM> extending from the base unit <NUM>. In the example depicted, the arm unit <NUM> includes the plurality of joint portions 5147a, 5147b and 5147c and the plurality of links 5149a and 5149b connected to each other by the joint portion 5147b. In <FIG>, for simplified illustration, the configuration of the arm unit <NUM> is depicted in a simplified form. Actually, the shape, number and arrangement of the joint portions 5147a to 5147c and the links 5149a and 5149b and the direction and so forth of axes of rotation of the joint portions 5147a to 5147c can be set suitably such that the arm unit <NUM> has a desired degree of freedom. For example, the arm unit <NUM> may preferably be included such that it has a degree of freedom equal to or not less than <NUM> degrees of freedom. This makes it possible to move the endoscope <NUM> freely within the movable range of the arm unit <NUM>. Consequently, it becomes possible to insert the lens barrel <NUM> of the endoscope <NUM> from a desired direction into a body lumen of the patient <NUM>.

An actuator is provided in the joint portions 5147a to 5147c, and the joint portions 5147a to 5147c include such that they are rotatable around predetermined axes of rotation thereof by driving of the actuator. The driving of the actuator is controlled by the arm controlling apparatus <NUM> to control the rotational angle of each of the joint portions 5147a to 5147c thereby to control driving of the arm unit <NUM>. Consequently, control of the position and the posture of the endoscope <NUM> can be implemented. Thereupon, the arm controlling apparatus <NUM> can control driving of the arm unit <NUM> by various known controlling methods such as force control or position control.

For example, if the surgeon <NUM> suitably performs operation inputting through the inputting apparatus <NUM> (including the foot switch <NUM>), then driving of the arm unit <NUM> may be controlled suitably by the arm controlling apparatus <NUM> in response to the operation input to control the position and the posture of the endoscope <NUM>. After the endoscope <NUM> at the distal end of the arm unit <NUM> is moved from an arbitrary position to a different arbitrary position by the control just described, the endoscope <NUM> can be supported fixedly at the position after the movement. It is to be noted that the arm unit <NUM> may be operated in a master-slave fashion. In this case, the arm unit <NUM> may be remotely controlled by the user through the inputting apparatus <NUM> which is placed at a place remote from the surgery room.

Further, where force control is applied, the arm controlling apparatus <NUM> may perform power-assisted control to drive the actuators of the joint portions 5147a to 5147c such that the arm unit <NUM> may receive external force by the user and move smoothly following the external force. This makes it possible to move the arm unit <NUM> with comparatively weak force when the user directly touches with and moves the arm unit <NUM>. Accordingly, it becomes possible for the user to move the endoscope <NUM> more intuitively by a simpler and easier operation, and the convenience to the user can be improved.

Here, generally in endoscopic surgery, the endoscope <NUM> is supported by a medical doctor called scopist. In contrast, where the supporting arm apparatus <NUM> is used, the position of the endoscope <NUM> can be fixed with a higher degree of certainty without hands, and therefore, an image of a surgical region can be obtained stably and surgery can be performed smoothly.

It is to be noted that the arm controlling apparatus <NUM> may not necessarily be provided on the cart <NUM>. Further, the arm controlling apparatus <NUM> may not necessarily be a single apparatus. For example, the arm controlling apparatus <NUM> may be provided in each of the joint portions 5147a to 5147c of the arm unit <NUM> of the supporting arm apparatus <NUM> such that the plurality of arm controlling apparatus <NUM> cooperate with each other to implement driving control of the arm unit <NUM>.

The light source apparatus <NUM> supplies irradiation light upon imaging of a surgical region to the endoscope <NUM>. The light source apparatus <NUM> includes a white light source which includes, for example, an LED, a laser light source or a combination of them. In this case, where a white light source includes a combination of red, green, and blue (RGB) laser light sources, since the output intensity and the output timing can be controlled with a high degree of accuracy for each color (each wavelength), adjustment of the white balance of a picked up image can be performed by the light source apparatus <NUM>. Further, in this case, if laser beams from the RGB laser light sources are applied time-divisionally on an observation target and driving of the image pickup elements of the camera head <NUM> is controlled in synchronism with the irradiation timings, then images individually corresponding to the R, G and B colors can be picked up time-divisionally. According to the method just described, a color image can be obtained even if a color filter is not provided for the image pickup element.

Further, driving of the light source apparatus <NUM> may be controlled such that the intensity of light to be outputted is changed for each predetermined time. By controlling driving of the image pickup element of the camera head <NUM> in synchronism with the timing of the change of the intensity of light to acquire images time-divisionally and synthesizing the images, an image of a high dynamic range free from underexposed blocked up shadows and overexposed highlights can be created.

Further, the light source apparatus <NUM> may be configured to supply light of a predetermined wavelength band ready for special light observation. In special light observation, for example, by utilizing the wavelength dependency of absorption of light of a body tissue, narrow band light observation (narrow band imaging) of imaging a predetermined tissue such as a blood vessel of a superficial portion of the mucous membrane or the like in a high contrast is performed by applying light of a narrower band in comparison with irradiation light upon ordinary observation (namely, white light). Alternatively, in special light observation, fluorescent observation for obtaining an image from fluorescent light generated by irradiation of excitation light may also be performed. In fluorescent observation, it is possible to perform observation of fluorescent light from a body tissue by irradiating excitation light on the body tissue (autofluorescence observation) or to obtain a fluorescent light image by locally injecting a reagent such as indocyanine green (ICG) into a body tissue and irradiating excitation light corresponding to a fluorescent light wavelength of the reagent upon the body tissue. The light source apparatus <NUM> can be configured to supply such narrow-band light and/or excitation light suitable for special light observation as described above.

Functions of the camera head <NUM> of the endoscope <NUM> and the CCU <NUM> are described in more detail with reference to <FIG> is a block diagram depicting an example of a functional configuration of the camera head <NUM> and the CCU <NUM> depicted in <FIG>.

Referring to <FIG>, the camera head <NUM> has, as functions thereof, a lens unit <NUM>, an image pickup unit <NUM>, a driving unit <NUM>, a communication unit <NUM> and a camera head controlling unit <NUM>. Further, the CCU <NUM> has, as functions thereof, a communication unit <NUM>, an image processing unit <NUM> and a control unit <NUM>. The camera head <NUM> and the CCU <NUM> are connected to be bidirectionally communicable to each other by a transmission cable <NUM>.

First, a functional configuration of the camera head <NUM> is described. The lens unit <NUM> is an optical system provided at a connecting location of the camera head <NUM> to the lens barrel <NUM>. Observation light taken in from a distal end of the lens barrel <NUM> is introduced into the camera head <NUM> and enters the lens unit <NUM>. The lens unit <NUM> includes a combination of a plurality of lenses including a zoom lens and a focusing lens. The lens unit <NUM> has optical properties adjusted such that the observation light is condensed on a light receiving face of the image pickup element of the image pickup unit <NUM>. Further, the zoom lens and the focusing lens include such that the positions thereof on their optical axis are movable for adjustment of the magnification and the focal point of a picked up image.

The image pickup unit <NUM> includes an image pickup element and disposed at a succeeding stage to the lens unit <NUM>. Observation light having passed through the lens unit <NUM> is condensed on the light receiving face of the image pickup element, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the image pickup unit <NUM> is provided to the communication unit <NUM>.

As the image pickup element which is included by the image pickup unit <NUM>, an image sensor, for example, of the complementary metal oxide semiconductor (CMOS) type is used which has a Bayer array and is capable of picking up an image in color. It is to be noted that, as the image pickup element, an image pickup element may be used which is ready, for example, for imaging of an image of a high resolution equal to or not less than <NUM>. If an image of a surgical region is obtained in a high resolution, then the surgeon <NUM> can comprehend a state of the surgical region in enhanced details and can proceed with the surgery more smoothly.

Further, the image pickup element which is included by the image pickup unit <NUM> is configured such that it has a pair of image pickup elements for acquiring image signals for the right eye and the left eye compatible with 3D display. Where 3D display is applied, the surgeon <NUM> can comprehend the depth of a living body tissue in the surgical region with a higher degree of accuracy. It is to be noted that, if the image pickup unit <NUM> is configured as that of the multi-plate type, then a plurality of systems of lens units <NUM> are provided corresponding to the individual image pickup elements of the image pickup unit <NUM>.

The image pickup unit <NUM> may not necessarily be provided on the camera head <NUM>. For example, the image pickup unit <NUM> may be provided just behind the objective lens in the inside of the lens barrel <NUM>.

The driving unit <NUM> includes an actuator and moves the zoom lens and the focusing lens of the lens unit <NUM> by a predetermined distance along the optical axis under the control of the camera head controlling unit <NUM>. Consequently, the magnification and the focal point of a picked up image by the image pickup unit <NUM> can be adjusted suitably.

The communication unit <NUM> includes a communication apparatus for transmitting and receiving various kinds of information to and from the CCU <NUM>. The communication unit <NUM> transmits an image signal acquired from the image pickup unit <NUM> as RAW data to the CCU <NUM> through the transmission cable <NUM>. Thereupon, in order to display a picked up image of a surgical region in low latency, preferably the image signal is transmitted by optical communication. This is because, since, upon surgery, the surgeon <NUM> performs surgery while observing the state of an affected area through a picked up image, in order to achieve surgery with a higher degree of safety and certainty, it is demanded for a moving image of the surgical region to be displayed on the real time basis as far as possible. Where optical communication is applied, a photoelectric conversion module for converting an electric signal into an optical signal is provided in the communication unit <NUM>. After the image signal is converted into an optical signal by the photoelectric conversion module, it is transmitted to the CCU <NUM> through the transmission cable <NUM>.

Further, the communication unit <NUM> receives a control signal for controlling driving of the camera head <NUM> from the CCU <NUM>. The control signal includes information relating to image pickup conditions such as, for example, information that a frame rate of a picked up image is designated, information that an exposure value upon image picking up is designated and/or information that a magnification and a focal point of a picked up image are designated. The communication unit <NUM> provides the received control signal to the camera head controlling unit <NUM>. It is to be noted that also the control signal from the CCU <NUM> may be transmitted by optical communication. In this case, a photoelectric conversion module for converting an optical signal into an electric signal is provided in the communication unit <NUM>. After the control signal is converted into an electric signal by the photoelectric conversion module, it is provided to the camera head controlling unit <NUM>.

It is to be noted that the image pickup conditions such as the frame rate, exposure value, magnification or focal point are set automatically by the control unit <NUM> of the CCU <NUM> on the basis of an acquired image signal. In other words, an auto exposure (AE) function, an auto focus (AF) function and an auto white balance (AWB) function are incorporated in the endoscope <NUM>.

The camera head controlling unit <NUM> controls driving of the camera head <NUM> on the basis of a control signal from the CCU <NUM> received through the communication unit <NUM>. For example, the camera head controlling unit <NUM> controls driving of the image pickup element of the image pickup unit <NUM> on the basis of information that a frame rate of a picked up image is designated and/or information that an exposure value upon image picking up is designated. Further, for example, the camera head controlling unit <NUM> controls the driving unit <NUM> to suitably move the zoom lens and the focus lens of the lens unit <NUM> on the basis of information that a magnification and a focal point of a picked up image are designated. The camera head controlling unit <NUM> may include a function for storing information for identifying of the lens barrel <NUM> and/or the camera head <NUM>.

It is to be noted that, by disposing the components such as the lens unit <NUM> and the image pickup unit <NUM> in a sealed structure having high airtightness and high waterproof, the camera head <NUM> can be provided with resistance to an autoclave sterilization process.

Now, a functional configuration of the CCU <NUM> is described. The communication unit <NUM> includes a communication apparatus for transmitting and receiving various kinds of information to and from the camera head <NUM>. The communication unit <NUM> receives an image signal transmitted thereto from the camera head <NUM> through the transmission cable <NUM>. Thereupon, the image signal may be transmitted preferably by optical communication as described above. In this case, for the compatibility with optical communication, the communication unit <NUM> includes a photoelectric conversion module for converting an optical signal into an electric signal. The communication unit <NUM> provides the image signal after conversion into an electric signal to the image processing unit <NUM>.

Further, the communication unit <NUM> transmits, to the camera head <NUM>, a control signal for controlling driving of the camera head <NUM>. Also the control signal may be transmitted by optical communication.

The image processing unit <NUM> performs various image processes for an image signal in the form of RAW data transmitted thereto from the camera head <NUM>. The image processes include various known signal processes such as, for example, a development process, an image quality improving process (a bandwidth enhancement process, a super-resolution process, a noise reduction (NR) process and/or an image stabilization process) and/or an enlargement process (electronic zooming process). Further, the image processing unit <NUM> performs a detection process for an image signal for performing AE, AF and AWB.

The image processing unit <NUM> includes a processor such as a CPU or a GPU, and when the processor operates in accordance with a predetermined program, the image processes and the detection process described above can be performed. It is to be noted that, where the image processing unit <NUM> includes a plurality of GPUs, the image processing unit <NUM> suitably divides information relating to an image signal such that image processes are performed in parallel by the plurality of GPUs.

The control unit <NUM> performs various kinds of control relating to image picking up of a surgical region by the endoscope <NUM> and display of the picked up image. For example, the control unit <NUM> generates a control signal for controlling driving of the camera head <NUM>. Thereupon, if image pickup conditions are inputted by the user, then the control unit <NUM> generates a control signal on the basis of the input by the user. Alternatively, where the endoscope <NUM> has an AE function, an AF function and an AWB function incorporated therein, the control unit <NUM> suitably calculates an optimum exposure value, focal distance and white balance in response to a result of a detection process by the image processing unit <NUM> and generates a control signal.

Further, the control unit <NUM> controls the display apparatus <NUM> to display an image of a surgical region on the basis of an image signal for which the image processes have been performed by the image processing unit <NUM>. Thereupon, the control unit <NUM> recognizes various objects in the surgical region image using various image recognition technologies. For example, the control unit <NUM> can recognize a surgical tool such as forceps, a particular living body region, bleeding, mist when the energy treatment tool <NUM> is used and so forth by detecting the shape, color and so forth of edges of the objects included in the surgical region image. The control unit <NUM> causes, when it controls the display apparatus <NUM> to display a surgical region image, various kinds of surgery supporting information to be displayed in an overlapping manner with an image of the surgical region using a result of the recognition. Where surgery supporting information is displayed in an overlapping manner and presented to the surgeon <NUM>, the surgeon <NUM> can proceed with the surgery more safety and certainty.

The transmission cable <NUM> which connects the camera head <NUM> and the CCU <NUM> to each other is an electric signal cable ready for communication of an electric signal, an optical fiber ready for optical communication or a composite cable thereof.

Here, while, in the example depicted in the figure, communication is performed by wired communication using the transmission cable <NUM>, the communication between the camera head <NUM> and the CCU <NUM> may be performed otherwise by wireless communication. Where the communication between the camera head <NUM> and the CCU <NUM> is performed by wireless communication, there is no necessity to lay the transmission cable <NUM> in the surgery room. Therefore, such a situation that movement of medical staff in the surgery room is disturbed by the transmission cable <NUM> can be eliminated.

An example of the surgery room system <NUM> to which the technology according to an embodiment of the present disclosure can be applied has been described above. It is to be noted here that, although a case in which the medical system to which the surgery room system <NUM> is applied is the endoscopic surgery system <NUM> has been described as an example, the configuration of the surgery room system <NUM> is not limited to that of the example described above. For example, the surgery room system <NUM> may be applied to a soft endoscopic system for inspection or a microscopic surgery system in place of the endoscopic surgery system <NUM>.

The technology according to the present disclosure may be applied, in the above-described configurations, to the interface block that gives the image pickup instructions regarding each of the cameras making up the surgery room system <NUM>. Specifically, the interface including the video camera <NUM> and the picked-up image monitoring apparatus <NUM> explained above using <FIG> may be employed in the interface block constituted by the camera head <NUM> and the display unit <NUM> in <FIG> and <FIG> and by PCs (including tablets) through which medical staff such as doctors monitoring the surgery behind the surgeon <NUM> or from outside the surgery room issue instructions and advice to the surgeon as well as to the operator of the centralized operation panel <NUM>. Utilization of the technology according to the present disclosure makes it possible to associate a particular position on the picked-up image with the display of the image pickup instructions such as the image pickup targeting instruction, zoom-in and zoom-out instructions, and pan instruction addressed to the camera operator of the camera head <NUM> or the instructions and advice addressed to the surgeon <NUM>.

The above-described embodiment has indicated the example in which the picked-up image monitoring apparatus <NUM> transmits to the video camera <NUM> the data packet including the position information and the display information (ID) as the image pickup instruction information. Alternatively, the image pickup instruction information may conceivably be conveyed by voice from the picked-up image monitoring apparatus <NUM> to the video camera <NUM>.

The above-described embodiment has also indicated the example in which the instruction type (mark, text) is displayed on the image position designated by the position information obtained in accordance with the user's operations on the touch panel of the picked-up image monitoring apparatus <NUM>. In this case, if the position of the subject recognized by subject recognition (e.g., facial recognition or human body recognition) fails to match the position designated by the position information, position adjustment (including position information adjustment) may conceivably be performed in such a manner that the instruction type (mark, text) is displayed corresponding to the position of the recognized subject. This makes it possible, for example, to issue the image pickup targeting instruction to the camera operator more appropriately.

Whereas the above-described embodiment has further indicated the image pickup system including one video camera <NUM> and one picked-up image monitoring apparatus <NUM>, there may conceivably be an image pickup system that includes multiple video cameras <NUM> and one picked-up image monitoring system <NUM>, or an image pickup system that includes multiple units of either or both of the video camera <NUM> and the picked-up image monitoring apparatus <NUM>.

Claim 1:
An image processing apparatus (<NUM>) comprising:
a reception unit (<NUM>) configured to receive image data from external equipment (<NUM>);
a processing unit (<NUM>) configured to generate instruction information regarding the image data;
a transmission unit (<NUM>) configured to transmit the instruction information to the external equipment;
the processing unit (<NUM>) further generates a display-use image based on and reflecting the instruction information; and wherein the processing unit (<NUM>) generates the display-use image having either a displayed mark or a displayed text corresponding to details of the instruction information; and characterized in that a display unit (<NUM>) is configured to display the display-use image; and
wherein either the display unit (<NUM>) is configured to display a cancel button or the image processing apparatus (<NUM>) further comprises a mechanical cancel button; and
the transmission unit (<NUM>) is configured to transmit cancellation information to the external equipment (<NUM>) indicating cancellation of a most-recently transmitted instruction information in accordance with operation of the displayed cancel button or of the mechanical cancel button.