Patent Description:
Some imaging devices are provided with automatic focus control (autofocus control) and manual focus control. In such an imaging device, it is required to perform appropriate focus control on a subject intended by a user by linking autofocus control with manual focus control.

For example, Patent Document <NUM> proposes a method of determining a main subject with high accuracy in order to keep focusing on a subject intended by a user.

However, in the method described in Patent Document <NUM>, the main subject is automatically determined, and thus the focus control intended by the user is not always performed.

Accordingly, the present disclosure proposes a technology for executing focus control more in accordance with a user's intention.

The present disclosure is defined by the appended set of claims.

Hereinafter, embodiments will be described in the following order with reference to the accompanying drawings.

An external appearance of an imaging device <NUM> according to the present embodiment is illustrated in <FIG> and <FIG>.

Note that, in the following examples, closeness and distances of a subject and a target of focus control may be described, where "close" indicates being closer to the imaging device <NUM>, and "far" indicates being farther from the imaging device <NUM>. That is, the "close subject" indicates a subject closer to the imaging device <NUM>.

Furthermore, the imaging device <NUM> is an example including an interchangeable lens but is not limited thereto, and can be widely applied to various imaging devices incorporated in, for example, a still camera, a video camera, and other devices.

The imaging device <NUM> includes a camera housing <NUM> in which necessary units are disposed inside and outside, and a lens barrel <NUM> attached to a front surface portion 2a of the camera housing <NUM>.

A rear monitor <NUM> is arranged on a rear surface portion 2b of the camera housing <NUM>. A through image, a recorded image, and the like are displayed on the rear monitor <NUM>.

The rear monitor <NUM> is, for example, a display device such as a liquid crystal display (LCD) or an organic electro-luminescence (EL) display.

An electric view finder (EVF) <NUM> is disposed on an upper surface portion 2c of the camera housing <NUM>. The EVF <NUM> includes an EVF monitor 5a and a frame-shaped enclosing part 5b projecting backward so as to surround upper and left and right sides of the EVF monitor 5a.

The EVF monitor 5a is formed using an LCD, an organic EL display, or the like. Note that an optical view finder (OVF) may be provided instead of the EVF monitor 5a.

Various operation elements <NUM> are provided on the rear surface portion 2b and the upper surface portion 2c. Examples of operation elements <NUM> include a reproduction menu activation button, an enter button, a cross key, a cancel button, a zoom key, a slide key, a shutter button <NUM> (release button), and the like.

The various operation elements <NUM> include elements in various modes such as a button, a dial, and a pressable and rotatable composite operation element. With the operation element <NUM> of various modes, for example, a menu operation, a reproduction operation, a mode selection-switching operation, a focus operation, a zoom operation, and selection and setting of parameters such as a shutter speed and an F-number can be performed.

Various lenses are disposed inside the lens barrel <NUM>, and a ring-shaped focus ring <NUM> and a ring-shaped zoom ring <NUM> are included.

The focus ring <NUM> is rotatable in a circumferential direction, and a focus position can be moved in an optical axis direction by various lenses moving in the optical axis direction according to a rotation direction.

The "focus position" is a position in focus from the imaging device <NUM> in the optical axis direction. This is, for example, the position of the subject with respect to the imaging device <NUM> in a case where there is a subject in focus. The focus position is changed by focus control.

By rotating the focus ring <NUM>, the focus position on the imaging device <NUM> can be made closer or farther. Furthermore, by rotating the focus ring <NUM>, manual focus control for manually adjusting an in-focus state can be achieved.

The zoom ring <NUM> is rotatable in the circumferential direction, and manual zooming control can be performed by the various lenses moving in the optical axis direction according to the rotation direction.

<FIG> is a block diagram of the imaging device <NUM>.

Inside and outside the camera housing <NUM> and the lens barrel <NUM> of the imaging device <NUM>, a lens system <NUM>, an imaging element unit <NUM>, a signal processing unit <NUM>, a recording control unit <NUM>, a display unit <NUM>, an output unit <NUM>, an operation unit <NUM>, a camera control unit <NUM>, a memory unit <NUM>, a driver unit <NUM>, a sensor unit <NUM>, and the like are provided. Furthermore, a power supply unit and the like are appropriately provided.

The lens system <NUM> includes various lenses such as an incident end lens, a zoom lens, a focus lens, and a condenser lens, a diaphragm mechanism that performs exposure control by adjusting, for example, an aperture amount of a lens or an iris (diaphragm) such that sensing is performed in a state where signal charges are not saturated and are within a dynamic range, and a shutter unit such as a focal plane shutter.

Note that a part of optical system components such as the lens system <NUM> may be provided in the camera housing <NUM>.

The imaging element unit <NUM> is of a charge coupled device (CCD) type or a complementary metal-oxide semiconductor (CMOS) type, for example, and includes a sensing element in which a plurality of pixels is two-dimensionally arranged, and thereby performs exposure control of light from a subject incident through the lens system <NUM>.

The imaging element unit <NUM> includes a processing unit that performs, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, or analog/digital (A/D) conversion processing on an electric signal photoelectrically converted by a pixel. Therefore, the imaging element unit <NUM> outputs a captured image signal as digital data to the signal processing unit <NUM> and the camera control unit <NUM>.

The signal processing unit <NUM> includes, for example, a microprocessor specialized in digital signal processing such as a digital signal processor (DSP), a microcomputer, or the like.

The signal processing unit <NUM> includes units for performing various types of signal processing on the digital signal (captured image signal) transmitted from the imaging element unit <NUM>.

Specifically, processing such as correction processing between R, G, and B color channels, white balance correction, aberration correction, and shading correction is performed.

Furthermore, the signal processing unit <NUM> performs YC generation processing of generating (separating) a luminance (Y) signal and a color (C) signal from R, G, and B image data, processing of adjusting luminance and color, and processing such as knee correction and gamma correction.

Moreover, the signal processing unit <NUM> performs conversion into a final output format by performing resolution conversion processing, codec processing for performing encoding for recording or communication, and the like. Image data converted into the final output format is stored in the memory unit <NUM>. Furthermore, by outputting the image data to the display unit <NUM>, an image is displayed on the rear monitor <NUM> or the EVF monitor 5a. Moreover, by being output from the external output terminal, it is displayed on a device such as a monitor provided outside the imaging device <NUM>.

The recording control unit <NUM> includes, for example, a nonvolatile memory, and functions as a storage unit that stores image files (content file) of still image data, moving image data, and the like, attribute information of the image files, thumbnail images, and the like.

The image files are stored in a format such as joint photographic experts group (JPEG), tagged image file format (TIFF), or graphics interchange format (GIF), for example.

Various actual forms of the recording control unit <NUM> can be considered. For example, the recording control unit <NUM> may be configured as a flash memory built in the imaging device <NUM>, or may be configured by a memory card (for example, a portable flash memory) that can be attached to and detached from the imaging device <NUM> and an access unit that accesses the memory card for storage and reading. Furthermore, it may be implemented as a hard disk drive (HDD) or the like as a form built in the imaging device <NUM>.

The display unit <NUM> executes processing for performing various displays for an imaging person. The display unit <NUM> is, for example, the rear monitor <NUM> or the EVF monitor 5a. The display unit <NUM> performs processing of displaying the image data input from the signal processing unit <NUM> and converted into an appropriate resolution. Thus, what is called a through image, which is a captured image during standby for release, is displayed.

Moreover, the display unit <NUM> achieves display of various operation menus, icons, messages, and the like as a graphical user interface (GUI) on the screen on the basis of an instruction from the camera control unit <NUM>.

Furthermore, the display unit <NUM> can display a reproduced image of the image data read from a recording medium in the recording control unit <NUM>.

Note that, in the present example, while both the EVF monitor 5a and the rear monitor <NUM> are provided, the embodiment of the present technology is not limited to such a configuration, and only one of the EVF monitor 5a and the rear monitor <NUM> may be provided, or either one or both of the EVF monitor 5a and the rear monitor <NUM> may be detachable.

The output unit <NUM> performs data communication and network communication with an external device in a wired or wireless manner. For example, captured image data (a still image file or a moving image file) is transmitted to an external display device, recording device, reproduction device, or the like.

In addition, the output unit <NUM> may function as a network communication unit. For example, communication may be performed by various networks such as the Internet, a home network, and a local area network (LAN), and various data may be transmitted and received to and from a server, a terminal, or the like on the network.

The operation unit <NUM> includes not only the above-described various operation elements <NUM> but also the rear monitor <NUM> employing a touch panel system and the like, and outputs operation information corresponding to various operations such as a tap operation and a swipe operation of the imaging person to the camera control unit <NUM>.

Note that the operation unit <NUM> may function as a reception unit of an external operation device such as a remote controller separate from the imaging device <NUM>. Examples of the external operation device include a smartphone, a tablet, a Bluetooth (registered trademark) remote controller, a wired remote controller, a wireless operation device for focus operation, and the like.

The focus ring <NUM> that detects an operation for manual focus control and the zoom ring <NUM> that detects an operation for zooming control are one aspect of the operation unit <NUM>.

The camera control unit <NUM> includes a microcomputer (arithmetic processing device) including a central processing unit (CPU), and performs overall control of the imaging device <NUM>. For example, control of the shutter speed according to an operation of the imaging person, an instruction on various types of signal processing in the signal processing unit <NUM>, imaging operation and recording operation according to an operation of the user, and reproduction operation of a recorded image file are performed.

The camera control unit <NUM> switches various image capturing modes and the like. Examples of the various image capturing modes include a still image capturing mode, a moving image capturing mode, a continuous image capturing mode for continuously acquiring still images, and the like.

The camera control unit <NUM> performs user interface control for enabling the user to operate these functions. The user interface (UI) control performs processing of detecting an operation with respect to each operation element <NUM> provided in the imaging device <NUM>, display processing with respect to the rear monitor <NUM>, operation detection processing, and the like.

Furthermore, the camera control unit <NUM> instructs the driver unit <NUM> to control various lenses included in the lens system <NUM>.

For example, processing of specifying an aperture value in order to secure a light amount necessary for auto focus (AF) control, an operation instruction of an aperture mechanism according to the aperture value, and the like are performed.

The memory unit <NUM> stores information and the like used for processing executed by the camera control unit <NUM>. As the illustrated memory unit <NUM>, for example, a read only memory (ROM), a random access memory (RAM), a flash memory, and the like are comprehensively illustrated.

The memory unit <NUM> may be a memory area built in a microcomputer chip as the camera control unit <NUM> or may be configured by a separate memory chip.

Programs and the like used by the camera control unit <NUM> are stored in the ROM, the flash memory, and the like of the memory unit <NUM>. The ROM, the flash memory, and the like store an operating system (OS) for the CPU to control each unit, content files such as image files, and application programs, firmware, and the like for various operations.

The camera control unit <NUM> executes the program to control the entire imaging device <NUM> including the lens barrel <NUM>.

The RAM of the memory unit <NUM> is used as a work area of the camera control unit <NUM> by temporarily storing data, programs, and the like used in various data processing executed by the CPU of the camera control unit <NUM>.

The driver unit <NUM> is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a diaphragm mechanism driver for a motor that drives a diaphragm mechanism, and the like.

Each driver supplies a drive current to a corresponding drive motor according to an instruction from the camera control unit <NUM>.

The sensor unit <NUM> comprehensively indicates various sensors mounted on the imaging device <NUM>. As the sensor unit <NUM>, for example, a position information sensor, an illuminance sensor, an acceleration sensor, and the like are mounted.

A sensor provided in the focus ring <NUM> or the zoom ring <NUM> to detect a rotation direction or an operation amount of the focus ring <NUM> or the zoom ring <NUM> is one aspect of the sensor unit <NUM>.

The camera control unit <NUM> executes various functions by executing a program stored in the memory unit <NUM>.

Each function of the camera control unit <NUM> will be described with reference to <FIG>. Note that a part of each function may be included in the signal processing unit <NUM>. In addition, a part of each function may be achieved by cooperation of the camera control unit <NUM> and the signal processing unit <NUM>.

The camera control unit <NUM> has functions as a user operation detection unit <NUM>, a focus position movement detection unit <NUM>, a depth information detection unit <NUM>, an in-screen target position setting unit <NUM>, a subject recognition unit <NUM>, an area setting unit <NUM>, a mode switching unit <NUM>, a focus control unit <NUM>, and a display control unit <NUM>.

The user operation detection unit <NUM> performs processing of detecting a user operation on the operation unit <NUM>. Specifically, a process of detecting an operation of pressing the shutter button <NUM>, a process of rotating the focus ring <NUM>, a process of detecting an operation of rotating the zoom ring <NUM>, and the like are performed.

The user operation detection unit <NUM> also detects an operation amount and an operation mode. For example, an operation mode of half pressing the shutter button <NUM> or an operation mode of full pressing the shutter button <NUM>, or the like is distinguished and detected.

Furthermore, when detecting an operation of rotating the focus ring <NUM> or the zoom ring <NUM>, the user operation detection unit <NUM> detects the rotation direction and the rotation amount.

The user operation detection unit <NUM> detects the start and end of a user operation for rotating the focus ring <NUM>. Various types of detection of the start of the user operation can be considered, for example, such as determining the start of the user operation by detecting that the user touches the focus ring <NUM> using a sensor or the like provided for the focus ring <NUM>.

Various types of detection of the end of the user operation can be considered, for example, such as determining the end of the user operation in a case where the state in which the operation on the focus ring <NUM> is detected changes to a state in which the operation is not detected. Specifically, the end of the user operation may be detected according to an output of a sensor that detects contact of a finger of a person with the focus ring <NUM>, or the end of the user operation may be detected in a case where the operation is not performed for a certain period of time.

The user operation detection unit <NUM> may detect an operation by the external operation device.

For example, in a case where the operation unit <NUM> receives an operation signal from the external operation device such as a smartphone, a tablet, a Bluetooth remote controller, a wired remote controller, or a wireless operation device for focus operation, the user operation detection unit <NUM> may recognize these operations as focus operation.

In the following description, a focus operation on the focus ring <NUM> will be described as an example, but the present technology can also be applied to a focus operation by these remote operations.

The focus position movement detection unit <NUM> detects (calculates) a moving direction (moving direction of the focus position) and a moving distance (moving amount of the focus position) of the focus position on the basis of an operation direction and an operation amount of the focus ring <NUM> detected by the user operation detection unit <NUM>.

Various modes of detecting the moving direction of the focus position can be considered. As an example, a minute operation amount (or a difference in rotation position) of the focus ring <NUM> detected for each frame (<NUM>/<NUM> sec, <NUM>/<NUM> sec, or the like) is acquired, and an operation direction of the user with respect to the focus ring <NUM>, that is, the rotation direction of the focus ring <NUM> is determined on the basis of a plurality of minute operation amounts.

Furthermore, the moving distance of the focus position can be obtained by integrating minute operation amounts.

In addition, the operation amount may be acquired not every frame but every predetermined time.

For example, the depth information detection unit <NUM> detects a piece of depth information for each pixel area including a plurality of pixels. The depth information is depth information of the subject, and may be detected on the basis of, for example, a phase difference signal output from an image plane phase difference pixel included in the imaging element unit <NUM>, or may be detected by providing a depth sensor separately from the imaging element unit <NUM> and receiving reflected light of reference light such as near infrared rays or acquiring distortion or intensity of distribution of the reflected light.

Note that the depth information may be detected for each pixel.

In a case where a focus control target area FA to be described later is set, the depth information may be detected only for the focus control target area FA.

The in-screen target position setting unit <NUM> sets an in-screen target position PS on the basis of the depth information. The in-screen target position PS is an area set on the captured image, and is an area including a group of pixels having similar depth information. That is, the in-screen target position PS set by the in-screen target position setting unit <NUM> is an area that can be set for each subject, and is an area in which a subject that is a possible target for the focus control is captured.

<FIG> illustrates an example of the in-screen target position PS. This is an example in which an in-screen target position PS1 set as a depth value DP1 and an in-screen target position PS2 set as a depth value DP2 are set on the captured image.

The subject recognition unit <NUM> performs processing of recognizing a subject in the captured image by performing image processing on the captured image. The subject recognition unit <NUM> may be capable of executing processing of determining the type of the subject. Thus, it is possible to perform appropriate focus control reflecting the user's desire to perform the focus control for a person and the user's desire to perform the focus control for a cat.

Furthermore, the subject recognition unit <NUM> sets the position of the recognized subject on the captured image as an in-screen subject position PO. The in-screen subject position PO is set to indicate the display position of the subject recognized by the image recognition, unlike the in-screen target position PS set on the basis of the depth information.

The area setting unit <NUM> sets the focus control target area FA. The focus control target area FA is an area where autofocus control is performed. For example, in a case where a central portion of the captured image is selected as the focus control target area FA, the focus control target is selected from among subjects captured in the central portion. The focus control target area FA can be set in the entire captured image or in a part of the captured image.

The setting of the focus control target area FA may be performed according to an operation of the user, or may be automatically performed by automatically determining a situation by image recognition.

In addition, a plurality of focus control target areas FA may be set. Then, the plurality of focus control target areas FA may be given priority.

The mode switching unit <NUM> switches various modes. For example, the mode switching unit <NUM> switches between an AF mode for performing the autofocus control and a manual focus (MF) mode for performing the manual focus control. The switching processing is executed, for example, when the user operation detection unit <NUM> detects that the user has operated the operation element <NUM> for switching between the AF mode and the MF mode. Alternatively, the switching processing may be executed by detecting a menu operation of the user.

The mode switching unit <NUM> switches a plurality of types of AF modes. Examples of the AF mode include a tracking focus mode in which the focus control is performed according to movement of a subject set as the focus control target, a single focus mode in which the focus control is performed only once in a case where the shutter button <NUM> is half-pressed, and the like. These switching is performed according to a menu operation of the user.

In particular, in the present embodiment, the mode switching unit <NUM> temporarily switches to the MF mode by detecting a specific user operation during execution of the AF mode. Furthermore, in the temporary MF mode, the end of a specific user operation is detected and the mode is switched again to the AF mode. That is, by the user performing a specific operation in the AF mode, the mode transitions to the MF mode, and by ending the specific operation, the mode transitions to the AF mode again.

The specific user operation is, for example, a rotation operation of the focus ring <NUM>. During the AF mode, the focus ring <NUM> is rotated in the circumferential direction to temporarily transition to the MF mode.

The focus control unit <NUM> performs the autofocus control corresponding to each mode. For example, in the single focus mode, lens drive control is performed so as to focus on the subject in the focus control target area. Furthermore, in the tracking focus mode, lens drive control for keeping focusing on the focus control target is performed according to movement of the focus control target (particularly, movement in the optical axis direction).

Furthermore, when the AF mode is switched to the temporary MF mode by the above-described specific user operation, the focus control unit <NUM> performs processing of estimating the focus control target (subject) changed according to a manual operation mode of the focus ring <NUM> by the user. Then, when the specific user operation ends and the AF mode is set, control to focus on the estimated new focus control target is performed. For example, in a case where tracking focus control has been performed before the temporary MF mode, the tracking focus control is performed for a new focus control target when the mode returns from the temporary MF mode to the AF mode again.

Note that the processing of estimating the changed new focus control target may be executed after returning from the temporary MF mode to the AF mode.

In order to perform such focus control, the focus control unit <NUM> sets a target focus position. In the setting of the target focus position, in the tracking focus mode, the in-focus state of the focus control target is maintained by executing processing of finely adjusting the target focus position according to the movement of the focus control target. The fine adjustment of the target focus position is performed, for example, on the basis of the depth information.

Furthermore, when shifting to the temporary MF mode, the focus control unit <NUM> performs the focus control on the basis of the operation direction (the rotation direction of the focus ring <NUM>) and the operation amount in a specific operation of the user detected by the user operation detection unit <NUM>. Specifically, when returning to the AF mode again, the focus control unit <NUM> calculates the moving direction and the moving distance of the focus position on the basis of the operation direction and the operation amount in the specific operation, and estimates a new focus control target. Then, the estimated focus control target is selected to reset the target focus position. A specific example of processing of estimating and selecting a new focus control target will be described later.

The display control unit <NUM> performs display processing of displaying a menu screen to the user using the display unit <NUM>, display processing of displaying a captured image, display processing of displaying a warning or the like, and the like.

The display control unit <NUM> executes processing of displaying the through image. When displaying the through image, the display control unit <NUM> superimposes and displays a frame image on the subject recognized by the subject recognition unit <NUM>. The frame image may be changed according to the focus state of the subject.

For example, the frame image to be superimposed on a subject that is selected as the focus control target is a first frame image FP1 that is a solid-line frame. Furthermore, the frame image to be superimposed on a subject that is not selected as the focus control target is a second frame image FP2 that is a broken-line frame.

Specifically, an example of the first frame image FP1 and the second frame image FP2 is illustrated in <FIG>.

The image illustrated in <FIG> illustrates a state in which two persons are recognized as subjects by the subject recognition unit <NUM>. One of the two persons is regarded as a close subject, and the other is regarded as a far subject.

Furthermore, the focus control unit <NUM> is in a state of selecting the close subject as the focus control target.

The focus control unit <NUM> performs the focus control on the focus control target to focus on a close subject. Furthermore, a far subject is out of focus.

The display control unit <NUM> superimposes and displays the first frame image FP1 on the subject as the focus control target selected by the focus control unit <NUM>.

Furthermore, the display control unit <NUM> superimposes and displays the second frame image FP2 on other subjects.

Note that various frame images can be considered as frame images to be superimposed and displayed on the image on the display unit <NUM> by the display control unit <NUM>. For example, instead of a rectangular frame, a circular or elliptical frame may be superimposed and displayed. Alternatively, a rectangular frame may be superimposed on each of the four corners of the subject being recognized to indicate an area where the subject is displayed. In addition, the frames of the first frame image FP1 and the second frame image FP2 may be hatched.

Furthermore, while <FIG> illustrates an example in which the display modes of the subject as the focus control target and other subjects are changed depending on the line type, they may be distinguished by a line color or a line thickness, or may be distinguished by combining them.

In addition, the subject as the focus control target and other subjects may be displayed separately without using a frame. For example, depth map display for displaying the subject by changing the display color according to the depth amount may be performed, or peaking display for displaying the subject by changing the display color according to the contrast amount in the screen may be performed.

As described above, in the imaging device <NUM>, a new focus control target is selected when transitioning to the AF mode again on the basis of a user operation when shifting from the AF mode to the temporary MF mode.

Here, some examples will be described for selection examples of the new focus control target.

A first selection example is an example in which, in a case where the user performs an operation of rotating the focus ring <NUM> when transitioning to the temporary MF mode, the new focus control target is selected according to the rotation position of the focus ring <NUM> at the end time point of the user operation.

<FIG> illustrates an example of the through image displayed on the rear monitor <NUM> as the display unit <NUM>. On the display unit <NUM>, the focus control target area FA is displayed by a thick frame line. Furthermore, in the display unit <NUM>, respective in-screen target positions PS3, PS4, and PS5 for three subjects recognized on the basis of the depth information by the in-screen target position setting unit <NUM> within the frame of the focus control target area FA are indicated by frame images.

The subject located at the in-screen target position PS3 is a subject A of a smile mark, the subject located at the in-screen target position PS4 is a subject B of a soccer ball, and the subject located at the in-screen target position PS5 is a subject C of a plant.

Here, the subjects A, B, and C are located at different positions in the depth direction. That is, distances from the imaging device <NUM> to the respective subjects are different as illustrated in <FIG>.

Then, as illustrated in <FIG>, a depth value of the in-screen target position PS3 is set as a depth value DP3, a depth value of the in-screen target position PS4 is set as a depth value DP4, and a depth value of the in-screen target position PS5 is set as a depth value DP5.

Furthermore, the magnitude relationship among the depth value DP3, the depth value DP4, and the depth value DP5 is set as depth value DP3 < depth value DP5 < depth value DP4. That is, the subject A is the closest to the imaging device <NUM>, the subject C is the next closest to the imaging device <NUM>, and the subject B is the farthest from the imaging device <NUM> (see <FIG>).

Note that, as illustrated in <FIG>, the depth information is calculated for each small area <NUM> including a plurality of pixels.

In the case of the example illustrated in <FIG>, the subject A is selected as the focus control target, and the first frame image FP1 as a solid-line frame image is superimposed and displayed at the position. Thus, the area of the in-screen target position PS3 is indicated as the focus control target in the screen.

The subject B is not selected as the focus control target, and on its position, the second frame image FP2 as a broken-line frame image is superimposed. Thus, an area of the in-screen target position PS4 is indicated as a subject that is not the focus control target in the screen.

The subject C is not selected as the focus control target, and on its position, the second frame image FP2 as a broken-line frame image is superimposed. Thus, an area of the in-screen target position PS5 is indicated as a subject that is not the focus control target in the screen.

In this manner, it is assumed that, at a time point before the user performs an operation on the focus ring <NUM>, the subject A is selected as the focus control target (see <FIG> and <FIG>). That is, the focus position before the user operation is the position of the subject A.

Here, by the user operating the focus ring <NUM>, the imaging device <NUM> transitions from the tracking focus mode to the temporary MF mode.

Then, it is assumed that, by the user operating the focus ring <NUM>, the focus position moves to a position between the subject C and the subject B and close to the subject C (see <FIG>).

In a case where the user finishes the operation of the focus ring <NUM> in this state, the imaging device <NUM> estimates the user's intention and selects a subject as the focus control target.

In the first selection example, the subject C, which is the subject closest to the focus position after the user operation, is selected as the focus control target. Thus, on the display unit <NUM>, the in-screen target position PS5 for the subject C is indicated by the first frame image FP1 that is a solid-line frame image, and the in-screen target positions PS3 and PS4 for the subjects A and B are indicated by the second frame images FP2 which are broken-line frame images (see <FIG>).

Note that, in the present example, an example has been described in which the in-screen target position PS is set for each subject detected on the basis of the depth information by the in-screen target position setting unit <NUM>, and the focus control target is selected from the in-screen target positions PS, but other examples are also conceivable.

For example, in a case where the in-screen target position PS set for each subject detected by the in-screen target position setting unit <NUM> on the basis of the depth information matches the in-screen subject position PO set for each subject recognized by the subject recognition unit <NUM> performing image processing, it may be indicated on the display unit <NUM> as a possible focus control target. Alternatively, possible focus control targets may be indicated on the display unit <NUM> only on the basis of the in-screen subject position PO without using the in-screen target position PS.

Then, according to the present example, the subject closest to the focus position after the user operation is selected as the focus control target from among subjects detected as subjects in both the detection processing by the depth information and the detection processing by the image recognition processing, thereby preventing the focus control on the inappropriate subject from being performed.

In a second selection example, a new focus control target is selected in consideration of the moving direction of the focus position, that is, the rotation direction of the focus ring <NUM>.

<FIG> is a diagram illustrating a positional relationship between the imaging device <NUM> and the subjects A, B, and C. Furthermore, <FIG> illustrates the focus position before an operation of the user and the focus position after the operation.

As illustrated in the drawing, similarly to the first selection example, the second selection example illustrates a state in which, by the user operates the focus ring <NUM>, the imaging device <NUM> transitions from the tracking focus mode to the temporary MF mode, and then by the user operating the focus ring <NUM>, the focus position is moved to a position between the subject C and the subject B and close to the subject C. That is, the focus position at the end time point of the user operation (end time focus position) is set as the position between the subject C and the subject B.

Here, in the second selection example, the subject closest to the focus position after the user operation is the subject C, but instead of simply selecting the subject C as the focus control target, the focus control target is selected in consideration of the rotation direction of the focus ring <NUM>. This is synonymous with selecting the focus control target in consideration of the moving direction of the focus position.

Specifically, it is determined whether or not the deviation between the focus position and the position of the subject C after the user operation is equal to or more than a predetermined threshold. Then, in a case where it is determined that the deviation is equal to or more than the predetermined threshold, the imaging device <NUM> estimates that the focus control target intended by the user is not the subject C but the subject B ahead in the moving direction of the focus position.

Therefore, as illustrated in <FIG>, the imaging device <NUM> selects the subject B as the focus control target. Thus, the subject B is indicated by the first frame image FP1 on the display unit <NUM>.

Note that, similarly to the first selection example, the focus control target may be selected by further using the in-screen subject position PO for each subject recognized by the subject recognition unit <NUM> by image processing.

In a third selection example, a new focus control target is selected in consideration of the moving direction of the focus position and the focus control target area FA.

<FIG> illustrates a relationship between the focus control target area FA set in the tracking focus mode and each subject. That is, the subject A and the subject B are located in the focus control target area FA, and the subject C is located outside the focus control target area FA.

Furthermore, the subject A is selected as the focus control target.

<FIG> illustrates a state in which the user performs a rotation operation of the focus ring <NUM> in this state to change the focus position and the focus position. As illustrated, the subject C and the subject B are located ahead in the moving direction of the focus position as viewed from the end time focus position. Then, the subject C is closer to the focus position than the subject B.

In the state illustrated in <FIG>, the subject C is located ahead in the moving direction, but the subject C is a subject located outside the focus control target area FA as illustrated in <FIG>.

Here, in the first selection example and the second selection example described above, the subject as the focus control target is selected without considering the focus control target area FA. Therefore, depending on conditions, a subject outside the focus control target area FA may be selected as the focus control target.

Specifically, as illustrated in <FIG> and <FIG>, in a case where the user performs the rotation operation of the focus ring <NUM>, the subject C located outside the focus control target area FA is selected as the focus control target in the first selection example and the second selection example.

On the other hand, in the third selection example, by selecting the focus control target in consideration of the focus control target area FA, the following occurs.

That is, since the subject C is located ahead in the moving direction of the focus position but is located outside the focus control target area FA, the subject B is selected as the focus control target.

Note that although <FIG> illustrates an example in which the in-screen target position PS5 is set for the subject C outside the focus control target area FA, the subject for which the in-screen target position PS is set may be limited to the subject in the focus control target area FA.

In another selection example, in a case where a specific operation of the user is detected in the MF mode, the mode transitions to the MF mode (the above-described temporary MF mode) as a preparation stage before shifting to the AF mode, and thereafter transitions to the AF mode in response to detection of the end of the specific operation. Then, in the AF mode, the focus control for a new focus control target estimated in the MF mode as the preparation stage is executed.

This will be specifically described with reference to <FIG>.

In a state before detection of the specific operation by the user, the focus position is a position between the subject A and the subject C.

In this state, it is assumed that the user performs a specific operation (rotation operation of the focus ring <NUM>) to move the focus position to a position between the subject C and the subject B.

Then, in a case where the user ends the specific operation, the mode shifts to the AF mode. The focus control target at this time corresponds to the specific operation of the user.

For example, as in the first selection example, the subject C closest to the end time focus position may be selected as the focus control target.

Alternatively, as in the second selection example, instead of selecting the subject C closest to the end time focus position as the focus control target, the subject B may be selected as the focus control target in consideration of the rotation direction of the focus ring <NUM>.

Alternatively, as in the third selection example, in consideration of the end time focus position, the rotation direction of the focus ring <NUM>, and the focus control target area FA, the subject inside the focus control target area FA and located ahead in the moving direction of the focus position may be selected as the focus control target.

After selecting a new focus control target, the tracking focus control is performed in accordance with the movement of the selected subject.

Note that, although the example of transitioning to the tracking focus mode after the end of the specific operation by the user has been described here, a subject as the focus control target may be automatically selected and focused, and thereafter the transition to the MF mode may be performed again. In this case, a state of what is called "focus lock" is set. That is, as long as the subject as the focus control target does not move, the state of focusing on the subject is maintained.

Here, an example of a display mode in the display unit <NUM> will be described.

<FIG> illustrates a scene in which tracking focus is performed as the AF mode. As illustrated, on the display unit <NUM>, with respect to a through image in which the subject H1 on the near side and the subject H2 on the deep side are captured, an in-screen target position PS6 corresponding to a subject H1 is indicated by the first frame image FP1, and an in-screen target position PS7 corresponding to a subject H2 is indicated by the second frame image FP2.

In this state, it is assumed that the user performs a rotation operation of the focus ring <NUM> to move the focus position to a far side (subject side), and ends the operation of the focus ring <NUM>.

At this time, the imaging device <NUM> transitions to the temporary MF mode during the operation of the focus ring <NUM>, and thereafter returns to the AF mode (in this case, the tracking focus mode) again.

After returning to the tracking focus mode, the imaging device <NUM> selects the subject H2 as a new focus control target according to the relationship between the focus position and the subject position at the end time point of the rotation operation of the focus ring <NUM>, the rotation direction of the focus ring <NUM>, and the like. Thus, as illustrated in <FIG>, in the through image displayed on the display unit <NUM>, the in-screen target position PS7 of the subject H2 selected as the new focus control target is indicated by the solid-line first frame image FP1, and the in-screen target position PS6 of the subject H1 not to be the focus control target is indicated by the broken-line second frame image FP2.

Thereafter, in a case where the subject H2 moves toward the imaging device <NUM>, the focus control in the tracking focus mode is executed, to thereby change the focus position according to the change in the position of the subject H2. That is, even if the subject moves, the imaging device <NUM> continues to focus on the tracking target subject (see <FIG>).

By interposing a user operation on the focus ring <NUM> during the tracking focus mode, it is possible to keep focusing on the subject while changing the tracking target subject to the subject intended by the user.

Note that another example of the display mode of the in-focus state on the display unit <NUM> will be described with reference to <FIG>.

<FIG> is a diagram illustrating a through image in the tracking focus mode. Furthermore, the focus control target is the subject H1, and a third frame image FP3 including four frame images is superimposed and displayed on the pupil area, thereby indicating that focusing is performed on the pupil of the subject H1.

At this time, no frame image is superimposed on the subject H2. However, the imaging device <NUM> has already recognized the subject H1 and the subject H2 on the basis of the depth information and a result of the image processing.

When the user performs a rotation operation of the focus ring <NUM>, the imaging device <NUM> transitions from the tracking focus mode to the temporary MF mode. <FIG> illustrates a state in which the focus position is moved to the far side by the operation of the user, that is, a state in which the focus position is located between the subject H1 and the subject H2.

As illustrated, although the focus control target is the subject H1, the focus position is shifted from the pupil position, and the in-focus state with respect to the pupil could not be maintained, causing a state in which the first frame image FP1 is superimposed on the subject H1. That is, the user can grasp that the in-focus state with respect to the pupil could not be maintained due to a change in the display mode of the frame image.

When the user finishes the rotation operation of the focus ring <NUM> in the state illustrated in <FIG>, the imaging device <NUM> transitions from the temporary MF mode to the tracking focus mode again. At this time, a subject to be a new focus control target (tracking target) is selected. Here, the subject H2 is selected as the new focus control target, and this state is illustrated in <FIG>.

As illustrated, the first frame image FP1 is superimposed on the subject H2 that is the focus control target. This state indicates that the subject H2 is the focus control target, and also indicates that the pupil position is not recognized and the focus control on the pupil is not performed.

Next, in a case where the subject H2 moves so as to approach the imaging device <NUM>, a through image as illustrated in <FIG> is displayed on the display unit <NUM>.

In this state, when the subject H2 approaches the imaging device <NUM>, the pupil position of the subject H2 can be recognized, and the pupil of the subject H2 is appropriately focused.

Therefore, the third frame image FP3 including four frame images is superimposed and displayed on the pupil portion of the subject H2.

Note that no frame image is superimposed and displayed for the subject H1, but the imaging device <NUM> has already recognized the subject H1 on the basis of the depth information and the result of the image processing.

A flow of processing executed by the camera control unit <NUM> of the imaging device <NUM> to implement the above-described various functions will be described with an example.

In step S101 in <FIG>, the camera control unit <NUM> determines whether or not a specific operation has been detected. The specific operation is, for example, the above-described rotation operation of the focus ring <NUM>, and is an operation performed in a case where it is desired to manually change the focus control target in the AF mode, in a case where it is desired to change the focus control target in the MF mode, in a case where it is desired to smoothly shift to the AF mode while changing the focus control target in the MF mode, or the like.

In a case where it is determined that the specific operation has not been detected, the camera control unit <NUM> proceeds to step S102 and executes control according to the current mode. For example, when the current mode is the tracking focus mode, the focus position is moved according to the movement of the subject of the focus control target, and the focus is kept on the subject. Furthermore, in a pupil AF mode in which the pupil is automatically focused, control to focus on the pupil is performed by detecting the pupil of the subject.

On the other hand, in a case where it is determined that the specific operation has been detected, the camera control unit <NUM> proceeds to step S103 and changes the mode to the temporary MF mode. Moreover, in step S104, the camera control unit <NUM> acquires the moving direction of the focus position (the rotation direction of the focus ring <NUM>) and the moving distance of the focus position (the operation amount of the focus ring <NUM>).

In step S105, the camera control unit <NUM> determines whether or not the end of the specific operation has been detected. The camera control unit <NUM> repeats the process of step S104 until the end of the specific operation is detected. Note that, although not illustrated, driving of the lens system <NUM> in accordance with the specific operation of the user is executed in the imaging device <NUM> until the end of the specific operation of the user is detected, thereby achieving manual adjustment of the focus position by the user.

On the other hand, in a case where it is determined that the end of the specific operation has been detected, the camera control unit <NUM> proceeds to step S106 and performs processing of transitioning to the AF mode before transitioning to the MF mode. The case where the end of the specific operation has been detected is, for example, a case where a state in which the user's finger touches the focus ring <NUM> is changed from a detected state to a non-detected state, or the like.

In step S107, the camera control unit <NUM> acquires information in the focus control target area FA.

Two examples of the information acquisition processing in the focus control target area FA will be described.

A first example is illustrated in <FIG>. In step S151, the camera control unit <NUM> acquires the depth information in the focus control target area FA.

Next, in step S152, the camera control unit <NUM> specifies a subject to be a possible focus control target on the basis of the acquired depth information.

A second example is illustrated in <FIG>. In step S151, the camera control unit <NUM> acquires the depth information. Subsequently, in step S153, the camera control unit <NUM> acquires information of the subject recognized by the image processing.

Next, in step S154, the camera control unit <NUM> selects the subject recognized in step S153 as the possible focus control target among the subjects recognized on the basis of the depth information. That is, the example illustrated in <FIG> is an example in which it is selected as the possible focus control target in a case where the in-screen target position PS set for each subject detected by the in-screen target position setting unit <NUM> on the basis of the depth information and the in-screen subject position PO set for each subject recognized by the subject recognition unit <NUM> performing the image processing match.

After selecting or specifying the possible focus control target, the camera control unit <NUM> sets a new focus control target in step S108 on the basis of the moving direction and the moving distance of the focus position acquired in step S104 in the MF mode and the depth information in the focus control target area FA acquired in step S107.

Note that, as a result based on these pieces of information, it may be determined not to change the focus control target. For example, in a case where an appropriate subject is not detected, or the like, the focus control target is not changed.

After executing the process of step S108, the camera control unit <NUM> returns to step S101, determines again whether or not the specific operation has been detected, and in a case where the specific operation has not been detected, control according to the current mode is executed in step S102.

That is, in a case where a new focus control target is selected in step S108, the tracking focus control is executed by setting an object target distance according to the subject selected as the new focus control target in step S102 until the next specific operation is detected.

A first example of the processing of selecting a new focus control target illustrated in step S108 of <FIG> will be described with reference to <FIG>.

The first example corresponds to the first selection example of the focus control target described above.

In step S201, the camera control unit <NUM> specifies the subject closest to the focus position after the user operation, that is, the end time focus position from among the subjects extracted as possible subjects.

Subsequently, in step S202, the camera control unit <NUM> selects the specified subject as the focus control target.

Thus, the selection mode described in the first selection example is achieved.

A second example of the processing of selecting a new focus control target will be described with reference to <FIG>.

The second example corresponds to the second selection example of the focus control target described above.

In step S211, the camera control unit <NUM> specifies the closest subject among the subjects located ahead of the end time focus position in the moving direction of the focus position.

Thus, the selection mode described in the second selection example is achieved.

A third example of processing of selecting a new focus control target will be described with reference to <FIG> described above.

The third example corresponds to the third selection example of the focus control target described above.

The camera control unit <NUM> performs respective processes of step S101 to step S106 in <FIG> to thereby perform the process of detecting the specific operation of the user and transitioning to the temporary MF mode, and thereafter performs the process of detecting the end of the specific operation and transitioning to the original mode again. Since each process is similar to each process denoted by the same reference numeral described in the first example, a detailed description thereof will be omitted.

Next, instead of step S107, the camera control unit <NUM> acquires information regarding the entire area regardless of the focus control target area FA. Specifically, specification of the subject based on the depth information and specification of the subject by image processing are performed for the entire area.

Next, in step S108, the camera control unit <NUM> performs processing of selecting a new focus control target from among the subjects specified in the focus control target area FA.

However, in a case where there is no possible subject in the focus control target area FA, a new focus control target is selected from among subjects outside the focus control target area FA.

This can be achieved by acquiring information for the entire area in step S107.

For example, in a case where the imaging device <NUM> cannot be moved in accordance with the movement of the subject, the subject may be temporarily located outside the focus control target area FA. Even in such a case, it is possible to select an appropriate subject as the focus control target by acquiring information on the outside of the focus control target area FA.

Note that information on the focus control target area FA may be acquired in step S107 in <FIG>. Also in this case, the subject C (see <FIG>) located outside the focus control target area FA is not focused, and the subject B in the focus control target area FA is appropriately focused. Furthermore, by limiting the information to be acquired to the focus control target area FA, it is possible to reduce the burden of processing executed by the imaging device <NUM>.

A fourth example of the processing of selecting a new focus control target executed in step S108 in <FIG> will be described with reference to <FIG>.

The fourth example is an example in which the first example and the second example are selectively used according to conditions.

In order to select a new focus control target, the camera control unit <NUM> specifies the subject closest to the focus position after the user operation in step S201. The specified subject is a subject that can be selected as the focus control target depending on conditions.

Subsequently, in step S222, the camera control unit <NUM> determines whether or not a distance (deviation) between the focus position after the user operation and the position of the specified subject is equal to or more than a predetermined threshold.

In a case where it is determined that the distance is less than the predetermined threshold, that is, in a case where it is determined that the focus position after the user operation and the position of the subject are close, the camera control unit <NUM> selects the specified subject as a new focus control target in step S202.

On the other hand, in a case where it is determined that the distance between the focus position after the user operation and the position of the specified subject is equal to or more than the predetermined threshold, the camera control unit <NUM> specifies the closest subject among the subjects located ahead of the end time focus position in the moving direction of the focus position in step S211.

Then, in step S202, the camera control unit <NUM> selects the specified subject as a new focus control target.

By performing such processing, a subject intended by the user can be selected as the focus control target.

In each of the above-described examples, it is conceivable to adjust, to the operation mode of the user, the time and speed (focus moving speed) from selection of the subject as the focus control target until actual focusing.

For example, a case is considered in which the user gradually moves the focus position from the subject H1 toward the subject H2 by slowly moving the focus ring <NUM>, and thereafter ends the user operation before the focus position moves to the position of the subject H2. At this time, the imaging device <NUM> can immediately focus on the subject H2 after estimating that the subject suitable as the focus control target is the subject H2.

However, lens driving for focusing on the subject H2 may be performed slowly in consideration of the operation mode in which the user slowly moves the focus ring <NUM> to move the focus position.

Thus, the user can specify the speed (time) to focus in the focus control only by changing the operation mode, and thus, it is possible to achieve preferred autofocus control by a simple operation. Thus, the convenience of the user is improved.

<FIG> illustrates an example of processing executed by the camera control unit <NUM> in order to adjust, to the operation mode of the user, the time and speed (focus moving speed) until focusing. Note that processes similar to those in <FIG> are denoted by the same reference numerals, and description thereof is omitted as appropriate.

The camera control unit <NUM> determines whether or not the specific operation has been detected in step S101, and in a case where it is determined that the specific operation has been detected, the camera control unit <NUM> performs processing of transitioning to the temporary MF mode in step S103, and acquires information on the specific operation of the user in step S111.

In the information acquired in step S111, not only the moving direction and the moving distance of the focus position but also the moving speed of the focus position are acquired in response to that the user has performed an operation of moving the focus position in the optical axis direction as the specific operation.

In step S105, the camera control unit <NUM> determines whether or not the end of the specific operation has been detected. The camera control unit <NUM> repeats the process of step S111 until the end of the specific operation is detected.

On the other hand, in a case where it is determined that the end of the specific operation has been detected, the camera control unit <NUM> proceeds to step S106 and performs processing of transitioning to the mode before transitioning to the MF mode.

The camera control unit <NUM> acquires information in the focus control target area FA in step S107, and selects a new focus control target in step S108.

Then, the camera control unit <NUM> returns to step S101, determines again whether or not the specific operation has been detected, and in a case where the specific operation has not been detected, control according to the current mode is executed in step S112. At this time, in a case where the moving speed of the focus position has been acquired in step S111, the camera control unit <NUM> performs focus control according to the mode while taking the moving speed into consideration in step S112.

Thus, control for focusing on the subject as the focus control target newly selected in step S108 at a speed corresponding to the operation mode of the user is performed.

Note that the focus moving speed can be set in advance by a menu operation, in addition to the changing according to the specific operation of the user as described above. For example, it may be configured such that a setting to increase the focus speed, a setting to decrease the focus speed, and a setting that it is variable according to the specific operation of the user can be selected.

The technology according to the present disclosure can be applied to a medical imaging system. The medical imaging system is a medical system using an imaging technology, and is, for example, an endoscope system or a microscope system.

An example of the endoscope system will be described with reference to <FIG> and <FIG>. <FIG> is a diagram illustrating an example of a schematic configuration of an endoscope system <NUM> to which the technology according to the present disclosure can be applied. <FIG> is a diagram illustrating an example of a configuration of an endoscope <NUM> and a camera control unit (CCU) <NUM>. <FIG> illustrates a state in which an operator (for example, a surgeon) <NUM> who is a surgery participant is performing surgery on a patient <NUM> on a patient bed <NUM> using an endoscope system <NUM>. As illustrated in <FIG>, the endoscope system <NUM> includes an endoscope <NUM> that is a medical imaging apparatus, the CCU <NUM>, a light source device <NUM>, a recording device <NUM>, an output device <NUM>, and a support apparatus <NUM> that supports the endoscope <NUM>.

In endoscopic surgery, an insertion aid called a trocar <NUM> punctures the patient <NUM>. Then, a scope <NUM> and a surgical instrument <NUM> connected to the endoscope <NUM> are inserted into the body of the patient <NUM> via the trocar <NUM>. The surgical instrument <NUM> is, for example, an energy device such as an electric scalpel, forceps, or the like.

A surgical image that is a medical image showing the inside of the body of the patient <NUM> captured by the endoscope <NUM> is displayed on a display device <NUM>. The operator <NUM> performs treatment on an operation target using the surgical instrument <NUM> while viewing the surgical image displayed on the display device <NUM>. Note that the medical image is not limited to the surgical image, and may be a diagnostic image captured during diagnosis.

The endoscope <NUM> is a camera that images the inside of the body of the patient <NUM>, and is, for example, as illustrated in <FIG>, a camera head including a condensing optical system <NUM> that condenses incident light, a zoom optical system <NUM> that enables optical zooming by changing the focal length of the camera, a focus optical system <NUM> that enables focus adjustment by changing the focal length of the camera, and a light receiving element <NUM>. The endoscope <NUM> condenses light on the light receiving element <NUM> via the connected scope <NUM> to generate a pixel signal, and outputs the pixel signal to the CCU <NUM> through the transmission system. Note that the scope <NUM> is an insertion unit that has an objective lens at a distal end and guides light from the connected light source device <NUM> into the body of the patient <NUM>. The scope <NUM> is, for example, a rigid scope for a rigid endoscope and a flexible scope for a flexible endoscope. Furthermore, the pixel signal is only required to be a signal based on a signal output from a pixel, and is, for example, a RAW signal or an image signal. In addition, a memory may be mounted in the transmission system that connects the endoscope <NUM> and the CCU <NUM>, and parameters related to the endoscope <NUM> and the CCU <NUM> may be stored in the memory. The memory may be disposed, for example, on a connection portion of the transmission system or on a cable. For example, parameters at the time of shipment of the endoscope <NUM> and parameters that have changed at the time of energization may be stored in the memory of the transmission system, and the operation of the endoscope may be changed on the basis of the parameters read from the memory. In addition, the endoscope and the transmission system may be in a set and referred to as an endoscope. The light receiving element <NUM> is a sensor that converts received light into a pixel signal, and is, for example, a complementary metal oxide semiconductor (CMOS) type imaging element. The light receiving element <NUM> is preferably an imaging element capable of color image capturing having a Bayer array. Furthermore, the light receiving element <NUM> is preferably an imaging element having the number of pixels corresponding to a resolution of, for example, <NUM> (the number of horizontal pixels of <NUM> × the number of vertical pixels of <NUM>), <NUM> (the number of horizontal pixels of <NUM> × the number of vertical pixels of <NUM>), or a square <NUM> (the number of horizontal pixels of <NUM> or more × the number of vertical pixels of <NUM> or more). The light receiving element <NUM> may be one sensor chip or a plurality of sensor chips. For example, a prism that separates incident light for each predetermined wavelength band may be provided, and each wavelength band may be imaged by different light receiving elements. In addition, a plurality of light receiving elements may be provided for stereoscopic viewing. In addition, the light receiving element <NUM> may be a sensor including an arithmetic processing circuit for image processing in a chip structure, or may be a sensor for time of flight (ToF). Note that the transmission system is, for example, an optical fiber cable or wireless transmission. The wireless transmission is only required to be capable of transmitting the pixel signal generated by the endoscope <NUM>. For example, the endoscope <NUM> and the CCU <NUM> may be wirelessly connected or the endoscope <NUM> and the CCU <NUM> may be connected via a base station in an operating room. At this time, the endoscope <NUM> may simultaneously transmit not only the pixel signal but also information (for example, the processing priority of the pixel signal, the synchronization signal, and the like) related to the pixel signal. Note that, in the endoscope, the scope and the camera head may be integrated, or a light receiving element may be provided at the distal end portion of the scope.

The CCU <NUM> is a control device that integrally controls the connected endoscope <NUM> and light source device <NUM>, and is, for example, an information processing device including an FPGA <NUM>, a CPU <NUM>, a RAM <NUM>, a ROM <NUM>, a GPU <NUM>, and an I/F <NUM> as illustrated in <FIG>. In addition, the CCU <NUM> may integrally control the connected display device <NUM>, recording device <NUM>, and output device <NUM>. For example, the CCU <NUM> controls the irradiation timing, the irradiation intensity, and the type of an irradiation light source of the light source device <NUM>. Furthermore, the CCU <NUM> performs image processing such as development processing (for example, demosaic processing) or correction processing on the pixel signal output from the endoscope <NUM>, and outputs a pixel signal after processing (for example, an image) to an external device such as the display device <NUM>. Furthermore, the CCU <NUM> transmits a control signal to the endoscope <NUM> to control driving of the endoscope <NUM>. The control signal is, for example, information regarding imaging conditions such as a magnification and a focal length of the camera. Note that the CCU <NUM> may have a function for image down-conversion, and may be capable of simultaneously outputting a high resolution (for example, <NUM>) image to the display device <NUM> and a low resolution (for example, HD) image to the recording device <NUM>.

In addition, the CCU <NUM> may be connected to an external device via an IP converter that converts a signal into a predetermined communication protocol (for example, Internet Protocol (IP)). The connection between the IP converter and the external device may be configured by a wired network, or a part or all of the networks may be constructed by a wireless network. For example, the IP converter on the CCU <NUM> side may have a wireless communication function and transmit a received video to an IP switcher or an output-side IP converter via a wireless communication network such as a 5th generation mobile communication system (<NUM>) or a 6th generation mobile communication system (<NUM>).

The light source device <NUM> is a device capable of emitting light in a predetermined wavelength band, and includes, for example, a plurality of light sources and a light source optical system that guides light of the plurality of light sources. The light source is, for example, a xenon lamp, an LED light source, or an LD light source. The light source device <NUM> has, for example, LED light sources corresponding to three respective primary colors R, G, and B, and emits white light by controlling output intensity and output timing of each light source. In addition, the light source device <NUM> may have a light source capable of emitting special light used for special light observation, separately from a light source that emits normal light used for normal light observation. The special light is light in a predetermined wavelength band different from that of normal light which is light for normal light observation, and examples thereof include near-infrared light (light having a wavelength of <NUM> or more), infrared light, blue light, and ultraviolet light. The normal light is, for example, white light or green light. In narrow band light observation which is a type of special light observation, blue light and green light are alternately emitted, whereby an image of a predetermined tissue such as a blood vessel in a mucosal surface layer can be captured with high contrast by utilizing wavelength dependency of light absorption in a body tissue. Furthermore, in fluorescence observation which is a type of special light observation, excitation light that excites a reagent injected into a body tissue is emitted, and fluorescence emitted from the body tissue or the reagent is received to obtain a fluorescence image, so that it is possible to make it easy for the operator to visually recognize the body tissue or the like that is difficult for the operator to visually recognize with the normal light. For example, in infrared light observation using infrared light, by emitting near infrared light as excitation light for exciting a reagent such as indocyanine green (ICG) injected into a body tissue, a deep structure of the body tissue can be easily visually recognized. In addition, in the fluorescence observation, a reagent (for example, PDD or <NUM>-ALA) that is excited by special light in a blue wavelength band and emits fluorescence in a red wavelength band may be used. Note that, in the light source device <NUM>, the type of irradiation light is set by control of the CCU <NUM>. The CCU <NUM> may have a mode in which the normal light observation and the special light observation are alternately performed by controlling the light source device <NUM> and the endoscope <NUM>. At this time, information based on the pixel signal obtained by the special light observation is preferably superimposed on the pixel signal obtained by the normal light observation.

The recording device <NUM> is a device that records the pixels acquired from the CCU <NUM>, and is, for example, a recorder. The recording device <NUM> records the image acquired from the CCU <NUM> in an HDD, an SDD, or an optical disk. The recording device <NUM> may be connected to a network in the hospital and accessible from a device outside the operating room. In addition, the storage device <NUM> may have a function for image down-conversion or a function for image up-conversion.

The display device <NUM> is a device capable of displaying an image, and is, for example, a display monitor. The display device <NUM> displays a display image based on the pixel signal subjected to the image processing by the CCU <NUM> under the control of the CCU <NUM>. Note that the display device <NUM> may also function as an input device that enables line-of-sight recognition, voice recognition, and instruction input by a gesture by including a camera and a microphone.

The output device <NUM> is a device that outputs information acquired from the CCU <NUM>, and is, for example, a printer. The output device <NUM> prints, for example, a print image on paper based on the pixel signal acquired from the CCU <NUM>.

The support apparatus <NUM> is an articulated arm including a base unit <NUM> having an arm control device <NUM>, an arm unit <NUM> extending from the base unit <NUM>, and a holding unit <NUM> attached to a distal end of the arm unit <NUM>. The arm control device <NUM> includes a processor such as a CPU, and operates according to a predetermined program to control driving of the arm unit <NUM>. The support apparatus <NUM> controls the position and posture of the endoscope <NUM> held by the holding unit <NUM>, for example, by controlling parameters such as the length of each link <NUM> constituting the arm unit <NUM> and the rotation angle and torque of each joint <NUM> by the arm control device <NUM>. Thus, the endoscope <NUM> can be changed to a desired position or posture, the scope <NUM> can be inserted into the patient <NUM>, and the observation area in the body can be changed. The support apparatus <NUM> functions as an endoscope support arm that supports the endoscope <NUM> during surgery. Thus, the support apparatus <NUM> can serve as a substitute for a scopist who is an assistant holding the endoscope <NUM>. In addition, the support apparatus <NUM> may be a device that supports a microscope device <NUM> to be described later, and can also be referred to as a medical support arm. Note that the control of the support apparatus <NUM> may be an autonomous control method by the arm control device <NUM>, or may be a control method controlled by the arm control device <NUM> on the basis of a user input. For example, the control method may be a master slave method in which the support apparatus <NUM> as the slave device is controlled on the basis of the movement of the master device at hand of the user. In addition, the support apparatus <NUM> may be remotely controllable from outside the operating room.

An example of the endoscope system <NUM> to which the technology according to the present disclosure can be applied has been described above. For example, the technology according to the present disclosure may be applied to a microscope system.

<FIG> is a diagram illustrating an example of a schematic configuration of a microscopic surgery system to which the technology according to the present disclosure can be applied. Note that, in the following description, components similar to those of the endoscope system <NUM> are denoted by the same reference numerals, and redundant description thereof will be omitted.

<FIG> schematically illustrates a state in which the operator <NUM> is performing surgery on the patient <NUM> on the patient bed <NUM> using a microscopic surgery system <NUM>. Note that, in <FIG>, for the sake of simplicity, a cart <NUM> is not illustrated in the configuration of the microscopic surgery system <NUM>, and the microscope device <NUM> in place of the endoscope <NUM> is illustrated in a simplified manner. However, the microscope device <NUM> in the present description may refer to a microscope unit <NUM> provided at the distal end of the link <NUM>, or may refer to the entire configuration including the microscope unit <NUM> and the support apparatus <NUM>.

As illustrated in <FIG>, at the time of surgery, an image of a surgical site captured by the microscope device <NUM> is enlarged and displayed on the display device <NUM> installed in the operating room using the microscopic surgery system <NUM>. The display device <NUM> is installed at a position facing the operator <NUM>, and the operator <NUM> performs various treatments on the surgical site such as resection of an affected part, for example, while observing the state of the surgical site by a video projected on the display device <NUM>.

Examples of the endoscope system <NUM> and the microscopic surgery system <NUM> to which the technology according to the present disclosure can be applied have each been described above. Note that the system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the support apparatus <NUM> can support another observation device or another surgical instrument at the distal end thereof instead of the endoscope <NUM> or the microscope unit <NUM>. As the other observation device, for example, forceps, tweezers, a pneumoperitoneum tube for pneumoperitoneum, an energy treatment tool for incising a tissue or sealing a blood vessel by cauterization, or the like can be applied. By supporting these observation devices and surgical instruments by the support apparatus, the position can be more stably fixed and the burden on the medical staff can be reduced as compared with a case where the medical staff manually supports them. The technology according to the present disclosure may be applied to a support apparatus that supports such a configuration other than the microscope unit.

The technology according to the present disclosure can be suitably applied to the endoscope system <NUM>, the microscope device <NUM>, and the like among the configurations described above. Specifically, the subject intended by the operator is estimated according to the moving direction and the moving distance of the focus position, and is selected as the focus control target. Thus, the focus position can be quickly and accurately adjusted as compared with a case where the focus position can be manually adjusted to the focus control target, and therefore, the operation time can be shortened and the efficiency can be improved, and the burden on the patient can be reduced.

As described above, the imaging device <NUM> includes the user operation detection unit <NUM> that detects an operation of a user (for example, a rotation operation on the focus ring <NUM>) for focus control, the focus position movement detection unit <NUM> that detects movement (change) of a focus position (a position in the optical axis direction of a subject in focus) based on the operation of the user detected by the user operation detection unit <NUM>, the depth information detection unit <NUM> that detects depth information in the focus control target area FA, and the focus control unit <NUM> that sets a target focus position on the basis of the movement of the focus position and the depth information.

That is, a change in the focus position due to a user operation is detected, and setting of the target focus position reflecting the user operation and focus control are performed after the user operation.

For example, in a case where the user operation is performed in the AF mode, a transition to the MF mode is temporarily made to detect the movement of the focus position by the user. This detection processing is processing for estimating the focus control target (subject) intended by the user, and is also processing for collecting information for selecting a subject as the focus control target in the subsequent AF control.

The camera control unit <NUM> of the imaging device <NUM> can perform optimum focus control using the MF mode and the AF mode by detecting manual operation of the focus ring <NUM> by the user, estimating the user's intention by detecting an operation direction (moving direction of the focus position) and the operation amount (moving amount of the focus position), and performing focus control based on the estimation after shifting to the AF mode.

Furthermore, in a case where only the AF mode is used, there are many scenes where a subject different from the subject intended by the user is in focus. By using the imaging device <NUM> having the present configuration, it is possible to specify the subject intended by the user via the temporary MF mode during execution of the AF mode, and thus, it is possible to achieve desired AF control. In particular, the AF mode and the MF mode can be seamlessly switched, and then such highly convenient control can be achieved by using information acquired in a certain mode also in another mode to which it is shifting.

Note that, in each of the above-described examples, the description has been given focusing on the operation of rotating the focus ring <NUM> as the focus operation, but as described above, even in a case where the focus operation is performed by the remote operation by the external operation device, the same processing can be performed, and as a result, a similar effect can be obtained.

In this case, usability in remote operation can be improved. In particular, it is suitable in a case where the subject intended by the user is brought into focus by remote operation with respect to the fixedly arranged imaging device <NUM>, and since it is not necessary to touch the imaging device <NUM>, occurrence of blurring at the time of the focus operation can be eliminated.

In other words, the configuration of the imaging device <NUM> as described above includes the user operation detection unit <NUM> that detects the user operation for the focus control, the focus position movement detection unit <NUM> that detects the movement of the focus position by the user operation detected by the user operation detection unit <NUM>, the depth information detection unit <NUM> that detects the depth information in the focus control target area FA of the focus control, and the focus control unit <NUM> that sets the target focus position on the basis of the depth information in a case where the user operation detection unit <NUM> detects the end of the movement of the focus position.

As described in the section (<FIG>, <FIG>, <FIG>, and <FIG>) of the functional configuration of the imaging device <NUM>, and the like, the focus position movement detection unit <NUM> in the imaging device <NUM> may detect the moving direction of the focus position.

The user's intention can be more accurately estimated by detecting the moving direction of the focus position. Therefore, user-friendly autofocus control can be achieved.

As described in the section (<FIG>) of the functional configuration of the imaging device <NUM>, and the like, the focus position movement detection unit <NUM> of the imaging device <NUM> may detect the moving direction every predetermined time (for example, one frame).

By detecting the moving direction every predetermined time, the operation direction of the user is appropriately detected.

Therefore, it is possible to appropriately estimate the focus control target intended by the user.

As described in the section (<FIG>, <FIG>, and <FIG>) of the functional configuration of the imaging device <NUM>, and the like, the depth information detection unit <NUM> of the imaging device <NUM> may detect the depth information of the entirety of the focus control target area FA, and the focus control unit <NUM> may set the target focus position on the basis of the depth information of the entirety of the focus control target area FA, the moving direction of the focus position, and an end time focus position (focus position at the movement end time point) at the time of detecting that an operation of the user on the movement of the focus position has ended.

By setting the target focus position on the basis of the end time focus position, the user's intention is reflected in the focus control.

Furthermore, even in a case where the mode is changed so as to return to the tracking focus mode again after the transition from the tracking focus mode to the temporary MF mode, the information of the user operation acquired in the MF mode is used in the tracking focus mode, whereby the tracking focus control in accordance with the user's intention can be performed.

As described in the section of the functional configuration of the imaging device <NUM> (<FIG>), and the like, the imaging device <NUM> may include the in-screen target position setting unit <NUM> that sets the in-screen target position PS on the basis of the depth information, and the focus control unit <NUM> may set the focus position corresponding to the in-screen target position PS as the target focus position.

The in-screen target position PS is an area set on the captured image, and is an area including a group of pixels having similar depth information. For example, depth information for each pixel is similar in a pixel area in which a certain subject is captured, and thus the pixel area is set as the in-screen target position PS.

Thus, it is possible to perform appropriate focus control on the subject recognized on the basis of the depth information.

In other words, such a configuration of the imaging device <NUM> includes an area setting unit (in-screen target position setting unit <NUM>) that sets an area where a focus possible object is located in an image as a focus possible area (in-screen target position) on the basis of the depth information, and the focus control unit <NUM> sets the target focus position so as to focus on the focus possible object in the focus possible area set by the area setting unit.

As described in the first selection example (<FIG>, and <FIG>) and the like, in a case where the in-screen target position setting unit <NUM> sets a plurality of in-screen target positions PS, the focus control unit <NUM> of the imaging device <NUM> may set, as the target focus position, a focus position closest to the end time focus position among focus positions corresponding to the in-screen target positions PS.

Thus, in a case where the user moves the focus position to the vicinity of the target subject, the focus control for the target subject is performed.

That is, the user can perform the autofocus control on an intended subject without moving the focus ring <NUM> so that the target subject is accurately focused, and convenience can be improved.

In other words, in such a configuration of the imaging device <NUM>, in a case where the area setting unit (in-screen target position setting unit <NUM>) sets a plurality of focus possible areas (in-screen target positions), the focus control unit <NUM> sets the target focus position so as to focus on the focus possible object at the position closest to the end time focus position in the optical axis direction.

As described in the first selection example and the second example (<FIG>) of the information acquisition processing in the focus control target area FA, and the like, the imaging device <NUM> may include the subject recognition unit <NUM> that recognizes the subject, and in a case of determining that the in-screen target position PS matches an in-screen subject position that is a position of the subject in a screen recognized by the subject recognition unit <NUM>, the focus control unit <NUM> may set a focus position corresponding to the in-screen subject position as the target focus position.

Thus, for example, a subject as a focus control target is selected using image recognition processing or the like.

Therefore, a possibility that an appropriate subject is selected as the focus control target can be increased, and appropriate focus control can be executed.

In particular, in a case where a category such as a person, a cat, or a car is specified as a subject to be detected, an appropriate subject belonging to the specified subject category is a possible focus control target, and thus, it is possible to reduce the possibility of focusing on an unintended subject.

In other words, such a configuration of the imaging device <NUM> includes the subject recognition unit <NUM> that recognizes the subject on the basis of the captured image, and the focus control unit <NUM> sets the target focus position so as to focus on the focus possible object in the focus possible area (in-screen target position) that matches the position of the subject in the image.

As described in the first selection example and the like, in a case where the subject recognition unit <NUM> recognizes a plurality of subjects, the focus control unit <NUM> of the imaging device <NUM> may set, as the target focus position, a focus position closest to the end time focus position among focus positions corresponding to the in-screen subject positions.

Thus, an appropriate subject can be selected as the focus control target, and a subject for which the user's intention has been determined can be selected as the focus control target.

In other words, in such a configuration of the imaging device <NUM>, in a case where the subject recognition unit <NUM> recognizes a plurality of subjects, the focus control unit <NUM> sets the target focus position so as to focus on the subject at the position closest to the end time focus position in the optical axis direction.

As described in the section of the functional configuration of the imaging device <NUM> (<FIG>), <FIG>, and the like, the focus control unit <NUM> of the imaging device <NUM> may perform tracking focus control for the subject focused according to the target focus position set after an end of the movement of the focus position.

Thus, the user does not need to perform an operation of continuing focusing on the subject that has been manually focused. This can be achieved by selecting the focus control target in the tracking focus mode in consideration of content of the operation of the user on the focus ring <NUM>.

By performing such control, convenience can be improved.

As described in the section (<FIG>, <FIG>, <FIG>, and <FIG>) of the functional configuration of the imaging device <NUM> and the like, the imaging device <NUM> may include the area setting unit <NUM> that sets the focus control target area FA according to an operation of a user.

Thus, it is possible to select the focus control target by further reflecting the user's intention from among the subjects.

Therefore, focus control appropriate for the user can be executed.

As described in the section (<FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>) of the functional configuration of the imaging device <NUM>, and the like, the imaging device <NUM> may include the display control unit <NUM> that performs processing of superimposing and displaying the first frame image FP1 on a subject focused by the focus control unit <NUM> among the subjects recognized by the subject recognition unit <NUM>, and processing of superimposing and displaying the second frame image FP2 on other subjects.

Thus, the user can confirm the focus control target by confirming the display unit <NUM>.

As described in the section of focus moving speed control (<FIG>), the focus control unit <NUM> of the imaging device <NUM> may perform the focus control according to a set focus moving speed.

Thus, the speed (time) until focusing in the focus control can be specified only by changing the operation mode.

Therefore, preferred autofocus control can be achieved by a simple operation, and the convenience of the user is improved.

As described in the section (<FIG>) of the configuration of the imaging device <NUM>, and the like, the user operation detection unit <NUM> of the imaging device <NUM> may detect an operation of the user on the focus ring <NUM>, and the focus position movement detection unit <NUM> may detect the moving direction of the focus position on the basis of the operation direction of the focus ring <NUM>.

Thus, the moving direction of the focus position is appropriately detected.

Therefore, appropriate focus control can be performed on the basis of the operation of the user on the focus ring <NUM>.

As described in the section of the functional configuration of the imaging device <NUM> (<FIG>), and the like, the focus control unit <NUM> of the imaging device <NUM> is capable of switching between a first mode (AF mode) in which autofocus control on a subject is performed and a second mode (MF mode) in which manual focus control is performed, and may switch to the second mode in a case where the user operation detection unit <NUM> detects an operation on the focus ring <NUM> in the first mode.

Thus, the user only needs to operate the focus ring <NUM> when switching to the MF mode.

That is, since mode switching can be performed by a simple operation, convenience can be improved.

As described in the section of the functional configuration of the imaging device <NUM> (<FIG>), and the like, the user operation detection unit <NUM> of the imaging device <NUM> may determine that an end of the movement of the focus position is detected in a case where a state in which the operation on the focus ring <NUM> is detected as the user operation changes to a state in which the operation is not detected.

Thus, for example, it is possible to detect the end of the user operation on the focus ring <NUM> using a sensor or the like.

Therefore, the user operation end time point can be appropriately determined, and the subsequent focus control can be appropriately performed.

As described in the fourth example (<FIG>) of the processing of selecting the focus control target, and the like, the focus control unit <NUM> of the imaging device <NUM> is capable of executing a first control (selection mode illustrated in the first selection example) of setting a focus position closest to the end time focus position among focus positions corresponding to the in-screen target position PS as the target focus position, and a second control (selection mode illustrated in the second selection example) of setting the target focus position according to the focus position and the moving direction at the end time point of the movement, and may switch from the first control to the second control in a case where a deviation between the closest focus position and the end time focus position is equal to or more than a predetermined threshold.

Thus, the subject of the focus control target is selected according to the operation mode of the user and the position of the subject.

Therefore, the focus control can be performed on the subject of the focus control target in which the user's intention is appropriately reflected.

In other words, such a configuration of the imaging device <NUM> includes an area setting unit (in-screen target position setting unit <NUM>) that sets an area in which a focus possible object is located in an image as a focus possible area (in-screen target position) on the basis of depth information, the focus position movement detection unit <NUM> detects a moving direction of the focus position, the focus control unit <NUM> is capable of executing first control of setting the target focus position so as to focus on the focus possible object at a position closest to the focus position (end time focus position) at the end time point of movement in the optical axis direction and second control of setting the target focus position according to the focus position at the end time point of movement and the moving direction, and switches from the first control to the second control in a case where the deviation between the target focus position and the focus position at the end time point of movement in the first control is equal to or more than a predetermined threshold.

Furthermore, the program to be executed by the imaging device <NUM> is a program to be executed by a CPU, a DSP, or the like, or a device including the CPU, the DSP, or the like, for example, to execute each processing illustrated in each of <FIG>.

That is, this program is a program for causing the imaging device <NUM> or the like to execute processing of detecting an operation of a user for focus control, processing of detecting movement of a focus position based on the operation of the user detected, processing of detecting depth information in the focus control target area FA, and processing of setting a target focus position on the basis of the movement of the focus position and the depth information.

With such a program, the above-described imaging device <NUM> can be achieved.

A program for achieving such an imaging device <NUM> can be recorded in advance in an HDD as a recording medium built in a device such as the imaging device <NUM>, a ROM in a microcomputer having a CPU, or the like.

Alternatively, the program can also be temporarily (or permanently) stored (recorded) in a removable recording medium such as a flexible disc, a compact disc read only memory (CD-ROM), a magnet optical (MO) disc, a digital versatile disc (DVD), a Blu-ray disc (registered trademark), a magnetic disc, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as what is called package software.

Furthermore, such a program can be installed from the removable recording medium into a personal computer or the like, or can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

Furthermore, such a program is suitable for providing the imaging device <NUM> of the embodiment in a wide range. For example, by downloading the program to a mobile terminal device such as a smartphone or a tablet having a camera function, a mobile phone, a personal computer, a game device, a video device, a personal digital assistant (PDA), or the like, these devices can be caused to function as the imaging device <NUM> of the present disclosure.

Claim 1:
An imaging device (<NUM>), comprising:
a user operation detection unit (<NUM>) that detects an operation of a user for focus control;
a focus position movement detection unit (<NUM>) that detects movement of a focus position based on the operation of the user detected by the user operation detection unit;
a depth information detection unit (<NUM>) that detects depth information in a focus control target area; and
a focus control unit (<NUM>) that sets a target focus position on a basis of the movement of the focus position and the depth information,
wherein the focus position movement detection unit (<NUM>) detects a moving direction of the focus position;
wherein the depth information detection unit (<NUM>) detects depth information of an entirety of the focus control target area;
wherein the focus control unit (<NUM>) sets the target focus position on a basis of the depth information of the entirety of the focus control target area, the moving direction of the focus position, and an end time focus position at a time of detecting that an operation of the user with respect to the movement of the focus position has ended;
wherein the imaging device (<NUM>) further comprises an in-screen target position setting unit (<NUM>) that sets an in-screen target position on a basis of the depth information;
wherein the focus control unit (<NUM>) sets a focus position corresponding to the in-screen target position as the target focus position; and
characterised in that the focus control unit (<NUM>) is capable of executing a first control of setting a focus position closest to the end time focus position among focus positions corresponding to the in-screen target position as the target focus position, and a second control of setting the target focus position according to the focus position and the moving direction at an end time point of the movement, and
switches from the first control to the second control in a case where a deviation between the closest focus position and the end time focus position is equal to or more than a predetermined threshold.