Source: https://patents.google.com/patent/US9392159B2/en
Timestamp: 2020-03-29 15:41:28
Document Index: 136769373

Matched Legal Cases: ['Application No. 2011', 'Application No. 2010', 'Application No. 2010', 'Application No. 201280042875', 'Application No. 11150773', 'Application No. 2010', 'Application No. 11', 'Application No. 11150773', 'Application No. 201280042875', 'Application No. 2012', 'Application No. 2010', 'Application No. 201110008088', 'Application No. 11']

US9392159B2 - Focus estimating device, imaging device, and storage medium storing image processing program - Google Patents
Focus estimating device, imaging device, and storage medium storing image processing program Download PDF
US9392159B2
US9392159B2 US14/186,575 US201414186575A US9392159B2 US 9392159 B2 US9392159 B2 US 9392159B2 US 201414186575 A US201414186575 A US 201414186575A US 9392159 B2 US9392159 B2 US 9392159B2
estimating device
US14/186,575
US20140232927A1 (en
2011-09-02 Priority to JP2011191555 priority Critical
2011-09-02 Priority to JP2011-191555 priority
2012-08-31 Priority to PCT/JP2012/005539 priority patent/WO2013031238A1/en
2014-02-21 Application filed by Nikon Corp filed Critical Nikon Corp
2014-05-06 Assigned to NIKON CORPORATION reassignment NIKON CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KAWAI, ATSUSHI
2014-05-19 Assigned to NIKON CORPORATION reassignment NIKON CORPORATION CORRECTIVE ASSIGNMENT TO CORRECT THE ADDRESS PREVIOUSLY RECORDED ON REEL 032831 FRAME 0164. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: KAWAI, ATSUSHI
2014-06-23 Assigned to NIKON CORPORATION reassignment NIKON CORPORATION CORRECTIVE ASSIGNMENT TO CORRECT THE DOCKET NUMBER PREVIOUSLY RECORDED ON REEL 032927 FRAME 0945. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Assignors: KAWAI, ATSUSHI
2014-08-21 Publication of US20140232927A1 publication Critical patent/US20140232927A1/en
2016-07-12 Publication of US9392159B2 publication Critical patent/US9392159B2/en
238000003384 imaging method Methods 0 title claims description 29
A focus estimating device has an area setting unit which sets an area in which focusing of an image is determined and divides the area into a plurality of blocks, a block selecting unit which selects a block to be used for focus determination, a block focus determination unit which performs focus determination for each block from an edge in each color component of the block selected by the block selecting unit, and a focus determination unit which determines focusing of the area based on the focus determination.
This application is a continuation application of international Application No. PCT/JP2012/005539, filed on Aug. 31, 2012, designating the U.S., in which the International Application claims a priority date of Sep. 2, 2011, based on prior filed Japanese Patent Application No. 2011-191555, the entire contents of which are incorporated herein by reference.
The present application relates to a focus estimating device, an imaging device, and a storage medium storing an image processing program.
An autofocus device which detects a focal position deviation of a lens by using chromatic aberration of light and modifies blur of each color light by performing focusing based on a detection result has been known (see Japanese Unexamined Patent Application Publication No. H06-138362 as an example).
However, the prior-art autofocus device cannot detect in what direction in an optical axis direction a lens position is deviated with respect to a focusing position, and determination accuracy of a focus state is low.
A focus estimating device which is an aspect of the present invention has an area setting unit which sets an area in which focusing of an image is determined and divides the area into a plurality of blocks, a block selecting unit which selects a block to be used for focus determination, a block focus determination unit which performs focus determination for each block from an edge in each color component of the block selected by the block selecting unit, and a focus determination unit which determines focusing of the area based on the focus determination.
An imaging device which is an aspect of the present invention has an imaging unit and the focus estimating device of the above described one aspect.
A non-transitory computer readable storage medium which is an aspect of the present invention stores an image processing computer program causing a computer to execute the processing of setting an area in which focusing of an image is determined, dividing the area into a plurality of blocks, selecting a block to be used for focus determination, performing focus determination for each block from an edge in each color component of the selected block, and determining focusing of the area based on the focus determination.
A focus estimating device which is another aspect of the present invention has an area setting unit which sets an area in which focusing of an image is determined and divides the area into a plurality of blocks, a block selecting unit which selects the block including an edge, a block focus determination unit which performs focus determination of each block from the edge in each color component of the block selected by the block selecting unit, and a focus determination unit which determines focusing of the area based on the focus determination.
An imaging device which is another aspect of the present invention has an imaging unit and the focus estimating device of the above described another aspect.
A non-transitory computer readable storage medium which is another aspect of the present invention stores an image processing computer program causing a computer to execute the processing of setting an area in which focusing of an image is determined, dividing the area into a plurality of blocks, selecting a block including an edge, performing focus determination for each block from the edge in each color component of the selected block, and determining focusing of the area based on the focus determination.
FIG. 2 is a flowchart illustrating an example of an AF operation of the electronic camera in the first embodiment.
FIG. 3 is a diagram illustrating a setting example of a focus estimating area.
FIG. 4 is a diagram illustrating a determination example of a block type in a scene corresponding to FIG. 3.
FIG. 5 is a diagram illustrating a case in which a gradient of an edge is inclined in opposite directions in two color components.
FIG. 6A is a diagram illustrating an example of a mountain-shaped structure, and FIG. 6B is a diagram illustrating an example of a valley-shaped structure.
FIG. 7A is a diagram illustrating a state before color correction, and FIG. 7B is a diagram illustrating a state after the color correction.
FIGS. 8A to 8C are diagrams illustrating an outline of focus determination.
FIG. 9 is a diagram illustrating a selection example of a block to be estimated in a scene corresponding to FIG. 4.
FIG. 10 is a diagram illustrating an example of focus estimation of the focus estimating area in a scene corresponding to FIG. 9.
FIG. 11 is a diagram illustrating a configuration example of a focus estimating device of a second embodiment.
FIG. 12 is a flowchart illustrating an operation example of the focus estimating device of the second embodiment.
DETAILED DESCRIPTION OF THE EMBODIMENTS Explanation of First Embodiment
FIG. 1 is a diagram illustrating a configuration example of an electronic camera of a first embodiment which is an example of a focus estimating device and an imaging device.
An electronic camera 11 includes a photographing optical system 12, an imaging unit 13, a camera microcomputer 14, a first memory 15, a second memory 16, a recording I/F 17, a display unit 18, and an operation unit 19. The imaging unit 13, the first memory 15, the second memory 16, the recording I/F 17, the display unit 18, and the operation unit 19 are connected to the camera microcomputer 14, respectively.
The photographing optical system 12 has a plurality of lenses including a focusing lens for AF (AutoFocus) and a zoom lens. In FIG. 1, the photographing optical system 12 is expressed by a single lens for simplification. The photographing optical system 12 may be replaceable with respect to a main body of the electronic camera 11.
The imaging unit 13 is a module for imaging (photographing) a subject image formed by the photographing optical system 12. For example, the imaging unit 13 includes an image sensor performing photoelectric conversion, an analog frontend circuit performing analog signal processing, and a digital frontend circuit performing A/D conversion and digital signal processing. The imaging unit 13 outputs an image signal acquired by imaging to the camera microcomputer 14.
Here, in each of pixels of the image sensor of the imaging unit 13, red (R), green (G), and blue (B) color filters are arranged in accordance with known Bayer array. Thus, each pixel of the image sensor outputs an image signal corresponding to the respective color by color separation using the color filters. As a result, the imaging unit 13 can acquire color images at the time of imaging.
Moreover, in a photographing mode of the electronic camera 11, the imaging unit 13 photographs a still image for recording involving recording in a non-volatile recording medium in accordance with a photographing instruction by a user. Moreover, the imaging unit 13 photographs an image for observation (a through image) at predetermined intervals even during photographing standby for a still image.
The camera microcomputer 14 is a processor for integrally controlling an operation of the electronic camera 11. For example, in the photographing mode, the camera microcomputer 14 causes the imaging unit 13 to photograph a still image for recording in accordance with the photographing instruction by the user. Moreover, the camera microcomputer 14 applies image processing such as color interpolation processing, white balance correction, gradation conversion, color conversion, contour enhancement processing and the like to data of the image acquired from the imaging unit 13.
Moreover, the camera microcomputer 14 functions, by executing a program, as an area setting unit 21, an edge detecting unit 22, a block selecting unit 23, a focus determination unit 24, and a focus estimating unit 25 (operations of the area setting unit 21, the edge detecting unit 22, the block selecting unit 23, the focus determination unit 24, and the focus estimating unit 25 will be described later).
The first memory 15 is a memory for temporarily storing data of the image and includes an SDRAM which is a volatile recording medium, for example. Moreover, the second memory 16 is a memory for storing a program executed by the camera microcomputer 14 and various types of data and includes a non-volatile memory such as a flash memory, for example.
The recording I/F 17 has a connector for connecting a non-volatile recording medium 26. The recording I/F 17 executes writing/reading of data of the image with respect to the recording medium 26 connected to the connector. The above described recording medium 26 is a hard disk and a memory card incorporating a semiconductor memory, for example. FIG. 1 illustrates a memory card as an example of the recording medium 26.
The display unit 18 is a display device (such as a liquid crystal monitor, an organic EL monitor and the like) displaying various images. For example, the display unit 18 performs moving image display (viewfinder display) of a through image or reproduction display of a still image by control of the camera microcomputer 14.
The operation unit 19 includes a plurality of switches (a power button, a release button, a zoom button, a cross key, a determination button, a reproduction button, a deletion button and the like, for example) for receiving various operations of the user.
An example of an AF operation of the electronic camera in the first embodiment will be described.
In the present embodiment, when an input of a through image from the imaging unit 13 is started, the camera microcomputer 14 executes AF processing and continuously performs focusing during display of the through image. As a result, the electronic camera 11 performs AF control during display of the through image all the time.
If a frame acquired from the imaging unit 13 is a color image expressed by an RGB colorimetric system, a focusing position on an optical axis of each color component is different due to a difference in axial chromatic aberration in each color component of the RGB. Paying attention to a point that a size relationship of a blur width w is reversed before and after the focusing position on the optical axis in each color component in an edge area, it is known that the current focus state can be determined based on a difference in the blur width w (Δw) of the edge between two colors. Thus, by determining in what direction in an optical axis direction the current position of the AF lens is shifted from the focusing position, the electronic camera 11 can execute AF control.
Here, when an optical system or an image sensor of a general electronic camera is to be used, the above described blur width of each color component has a range of several to several tens of pixels, while a difference in the blur width between two colors is only approximately a tenth or a hundredth thereof. That is, since the difference in the blur width is small with respect to a measurement error of the blur width that can be measured only by the unit of a pixel, it is difficult to directly acquire the difference Δw of the blur width with accuracy from the measurement result of the blur width. Thus, the electronic camera of the present embodiment determines the focus state by the following method.
FIG. 2 is a flowchart illustrating an example of the AF operation of the electronic camera in the first embodiment. The processing in FIG. 2 is started by the camera microcomputer 14 in accordance with start of the photographing mode.
Here, in the explanation in FIG. 2, on an assumption that there is axial chromatic aberration between R and G, an example of acquiring a difference ΔwR−G of the blur width in color difference R−G will be mainly described. It is assumed that ΔwR−G calculated from the color difference R−G is a positive value on a front focus side and a negative value on a rear focus side. If a positive/negative relationship is reversed between the front focus and the rear focus, it is only necessary to use ΔwG−R calculated from the color difference G−R. It is needless to say that the determination can be made in the same method also between R and B or G and B.
Step #101: the camera microcomputer 14 starts photographing of the through image by driving the imaging unit 13. Subsequently, the through image is sequentially generated at predetermined intervals.
Moreover, the camera microcomputer 14 in the photographing mode causes the display unit 18 to perform moving image display of the through image. Therefore, the user can perform framing for determining a photographing composition by referring to the through image on the display unit 18.
Step #102: the camera microcomputer 14 acquires a color image (through image) to be processed from the imaging unit 13. Then, the camera microcomputer 14 executes pre-processing for the color image to be processed such as smoothing processing for noise reduction and resolution correction for correcting direction dependence of the edge. The camera microcomputer 14 may partially apply the pre-processing at #102 only to a focus estimating area, to be described later, in the color image to be processed.
In the present specification, RGB pixel values in a coordinate (x, y) of a color image are expressed as R(x, y), G(x, y), and B(x, y), respectively.
Step #103: The area setting unit 21 sets an area (focus estimating area) for which focus estimation is made in a part of the color image to be processed. The focus estimating area is divided into areas in a lattice state by rectangular blocks. As an example, an image size of each block is the number of pixels of BLOCK_SIZE×BLOCK_SIZE (8×8 pixels, for example). FIG. 3 illustrates a setting example of the focus estimating area.
Step #104: The edge detecting unit 22 detects an edge (a gradient component of an image) of each color component of a pixel for each block. Then, the edge detecting unit 22 sorts the blocks by type in accordance with a gradient amount of the pixel in the block.
Here, assuming that a coordinate of the block is (u, v), a set BL of the pixels included in the block (u, v) of interest can be expressed by an equation (1):
x1=u×BLOCK_SIZE,x2=(u+1)×BLOCK_SIZE−1
y1=v×BLOCK_SIZE,y2=(v+1)×BLOCK_SIZE−1
BL={(x,y)|x1≦x≦x2∩y1≦y≦y2} (1)
Moreover, a horizontal gradient (dxR(x, y)) and a vertical gradient (dyR(x, y)) of an R component in each pixel and a horizontal gradient (dxG(x, y)) and a vertical gradient (dyG(x, y)) of a G component in each pixel can be acquired by an equation (2):
dxR(x,y)=(R(x+1,y)−R(x−1,y))/2
dyR(x,y)=(R(x,y+1)−R(x,y−1))/2
dxG(x,y)=(G(x+1,y)−G(x−1,y))/2
dyG(x,y)=(G(x,y+1)−G(x,y−1))/2 (2)
Moreover, average values (RBlock(u, v), GBlock(u, v)) of the R and G pixels in the block and its horizontal gradient sum (dxRBlock(u, v), dxGBlock(u, v)) as well as vertical gradient sum (dyRBlock(u, v), dyGBlock(u, v)) can be acquired by an equation (3):
R Block ⁢ ⁡ ( u , v ) = [ ∑ ( x , y ) ∈ BL ⁢ R ⁡ ( x , y ) ] / ( BLOCK_SIZE ) 2 ⁢ ⁢ G Block ⁡ ( u , v ) = [ ∑ ( x , y ) ∈ BL ⁢ G ⁡ ( x , y ) ] / ( BLOCK_SIZE ) 2 ⁢ ⁢ d ⁢ ⁢ x ⁢ ⁢ R Block ⁡ ( u , v ) = ∑ ( x , y ) ∈ BL ⁢ d ⁢ ⁢ xR ⁡ ( x , y ) ⁢ ⁢ d ⁢ ⁢ y ⁢ ⁢ R Block ⁡ ( u , v ) = ∑ ( x , y ) ∈ BL ⁢ d ⁢ ⁢ yR ⁡ ( x , y ) ⁢ ⁢ d ⁢ ⁢ x ⁢ ⁢ G Block ⁡ ( u , v ) = ∑ ( x , y ) ∈ BL ⁢ d ⁢ ⁢ xG ⁡ ( x , y ) ⁢ ⁢ d ⁢ ⁢ y ⁢ ⁢ G Block ⁡ ( u , v ) = ∑ ( x , y ) ∈ BL ⁢ d ⁢ ⁢ yG ⁡ ( x , y ) ( 3 )
Moreover, in order to determine front or rear focus by axial chromatic aberration, edge gradients need to be inclined in the same direction between two colors to be compared (see FIGS. 8A to 8C). This condition is satisfied if a relationship in an equation (4) is true in the horizontal direction and the vertical direction, respectively, for the gradient sums of each block.
dxR(x,y)×dxG(x,y)≧0
dyR(x,y)×dyG(x,y)≧0 (4)
As an example, at #104, it is only necessary for the edge detecting unit 22 to execute processing of (A1)) to (A5) below for each block and to sort the blocks in the focus estimating area by type.
(A1) The edge detecting unit 22 extracts a set of pixels (HGPR−G) having gradients in the same direction in two colors in the horizontal direction at a threshold value TH_GRADPIXEL or more in the block of interest by an equation (5). Then, the edge detecting unit 22 acquires the number of elements of HGPR−G (nHGradPixelR−G(u, v)) by an equation (6). In the present specification, a value of TH_GRADPIXEL is set so as to become larger than amplitude of a noise that can be included in the image in order to prevent misdetection of a noise.
H ⁢ ⁢ G ⁢ ⁢ P R · G = { ( x , y ) ❘ ( x , y ) ∈ B ⁢ ⁢ L ⋂ dxR ⁡ ( x , y ) × dxG ⁡ ( x , y ) ≥ 0 ⋂ dxR ⁡ ( x , y ) ≥ TH_GRADPIXEL ⋂ dxG ⁡ ( x , y ) ≥ TH_GRADPIXEL } ( 5 ) ⁢ nHGradPixel R · G ⁡ ( u , v ) =  H ⁢ ⁢ G ⁢ ⁢ P R · G  ( 6 )
(A2) The edge detecting unit 22 extracts a set of pixels (VGPR−G) having gradients in the same direction in two colors in the vertical direction at the threshold value TH_GRADPIXEL or more in the block of interest by an equation (7). Then, the edge extracting unit acquires the number of elements of VGPR−G (nVGradPixeIR−G(u, v)) by an equation (8).
V ⁢ ⁢ G ⁢ ⁢ P R · G = { ( x , y ) ❘ ( x , y ) ∈ BL ⋂ dyR ⁡ ( x , y ) × dyG ⁡ ( x , y ) ≥ 0 ⋂ dyR ⁡ ( x , y ) ≥ TH_GRADPIXEL ⋂ dyG ⁡ ( x , y ) ≥ TH_GRADPIXEL } ( 7 ) ⁢ nVGradPixel R · G ⁡ ( u , v ) ⁢  V ⁢ ⁢ G ⁢ ⁢ P R · G  ( 8 )
(A3) The edge detecting unit 22 extracts a set of pixels (FPR−G) having no gradient at the threshold value TH_FLATPIXEL or more in the block of interest by an equation (9). Then, the edge extracting unit acquires the number of elements of FPR−G (nFlatPixeIR−G(u, v)) by an equation (10). In the present specification, a value of TH_FLATPIXEL is set as appropriate within a range in which contrast can be considered to be substantially flat even if there is some amplitude of a pixel value.
FP R - G = { ( x , y ) | ( x , y ) ∈ BL ⋂ dxR ⁡ ( x , y ) < TH_FLATPIXEL ⋂ dxG ⁢ ( x , y ) < TH_FLATPIXEL ⋂ dyR ⁡ ( x , y ) < TH_FLATPIXEL ⋂ dyG ⁡ ( x , y ) < TH_FLATPIXEL } ( 9 ) nFlatPixel R - G ⁡ ( u , v ) =  FP R - G  ( 10 )
(A4) The edge detecting unit 22 determines a block type of the block (u, v) of interest in accordance with the following conditions (i) to (iv):
(i) If the block (u, v) of interest satisfies an equation (11), the edge detecting unit 22 makes the block (u, v) of interest “horizontal edge block HEBLR−G”. In the present specification, “NPixel” refers to the total number of pixels of the block of interest (NPixel=BLOCK_SIZE2). Moreover, “RATIO_EDGEBLOCK” indicates a coefficient specifying a ratio of a content of the pixel with a gradient to all the pixels in the block.
HEBL R - G = { ( u , v ) | nHGradPixel R - G ⁡ ( u , v ) ≥ N Pixel × RATIO_EDGEBLOCK ⋂ nVGradPixel R - G ⁡ ( u , v ) < N Pixel × RATIO_EDGEBLOCK } ( 11 )
That is, in the block (u, v) of interest, if a ratio of nHGradPixeIR−G(u, v) to all the pixels in the block is at RATIO_EDGEBLOCK or more and if a ratio of nVGradPixeIR−G(u, v) to all the pixels in the block is less than RATIO_EDGEBLOCK, the block type is determined to be the “horizontal edge block HEBLR−G”.
(ii) If the block (u, v) of interest satisfies an equation (12), the edge detecting unit 22 makes the block (u, v) of interest “vertical edge block VEBLR−G”.
VEBL R - G = { ( u , v ) | nHGradPixel R - G ⁡ ( u , v ) < N Pixel × RATIO_EDGEBLOCK ⋂ nVGradPixel R - G ⁡ ( u , v ) ≥ N Pixel × RATIO_EDGEBLOCK } ( 12 )
That is, in the block (u, v) of interest, if a ratio of nHGradPixeIR−G(u, v) to all the pixels in the block is less than RATIO_EDGEBLOCK and if a ratio of nVGradPixeIR−G(u, v) to all the pixels in the block is at RATIO_EDGEBLOCK or more, the block type is determined to be the “vertical edge block VEBLR−G”.
(iii) If the block (u, v) of interest satisfies an equation (13), the edge detecting unit 22 makes the block (u, v) of interest “flat block FBLR−G”. “RATIO_FLATBLOCK” indicates a coefficient specifying a ratio of a content of the pixel without a gradient to all the pixels in the block.
FBL R - G = { ( u , v ) | ( u , v ) ∉ HEBL R - G ⋂ ( u , v ) ∉ VEBL R - G ⋂ nFlatPixel R - G ⁡ ( u , v ) ≥ N Pixel × RATIO_FLATBLOCK } ( 13 )
That is, if the block of interest does not belong to HEBLR−G or VEBLR−G, and if the ratio of nFlatPixeIR−G(u, v) to all the pixels in the block is at RATIO_FLATBLOCK or more, the block type is determined to be a “flat block FBLR−G”.
(iv) If the block (u, v) of interest does not fall under any one of the above described (i) to (iii), the edge detecting unit 22 excludes the block from a target for subsequent processing.
(A5) The edge detecting unit 22 changes the block of interest and executes processing of the above described (A1) to (A4). Then, the edge detecting unit 22 repeats the above described operation until the block types of all the blocks are determined.
FIG. 4 illustrates a determination example of the block type in a scene corresponding to FIG. 3. In this figure, “H” indicates a horizontal edge block, “V” indicates a vertical edge block, and “F” indicates a flat block.
Step #105: The block selecting unit 23 selects a block to be estimated to be applied to focus estimation in the edge blocks (horizontal edge block, vertical edge block) selected at #104. At this time, the block selecting unit 23 groups a plurality of the blocks to be estimated if an edge continues in the same direction between the adjacent blocks to be estimated. Hereinafter, the block to be estimated is also referred to as a continuous edge block in some cases.
As an example, a case of selection of the block to be estimated from the horizontal edge block will be described. The block selecting unit 23 groups the edge blocks which satisfy all of the following (condition 1) to (condition 5) and continue from the horizontal direction (u1, v) to (u2, v) in the horizontal edge blocks into one block to be estimated (continuous edge block).
(Condition 1) The block selecting unit 23 selects the edge blocks having edge gradients inclined in the same direction for two color components having axial chromatic aberration as blocks to be estimated. That is because, for example, as illustrated in FIG. 5, if the edge gradients in two color components are inclined in opposite directions, detection of the difference in the blur width by the axial chromatic aberration becomes difficult. Specifically, it is only necessary for the horizontal edge block in the (condition 1) to satisfy the following equation (14):
(u1,v)εHEBLR−G∩ . . . ∩(u2,v)εHEBLR−G (14)
(Condition 2) The block selecting unit 23 selects the block to be estimated by excluding the block including an uneven structure (thin-line/mountain-shaped/valley-shaped structure) at the edge part. That is because, for example, color correction which will be described later cannot be made correctly for the mountain-shaped structure as in FIG. 6A or the valley-shaped structure as in FIG. 6B.
If the structure as above is included, a sum of the gradient absolute values and an absolute value of a gradient sum do not match each other. Thus, it is only necessary for the horizontal edge block in (condition 2) to satisfy the following equations (15) and (16):
∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢  d ⁢ ⁢ xR Block ⁡ ( i , v )  -  ∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢ d ⁢ ⁢ xR Block ⁡ ( i , v )  = 0 ( 15 ) ∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢  d ⁢ ⁢ xG Block ⁡ ( i , v )  -  ∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢ d ⁢ ⁢ xG Block ⁡ ( i , v )  = 0 ( 16 )
(Condition 3) The block selecting unit 23 selects the horizontal edge block in which at least one of the blocks adjacent in a gradient direction (horizontal direction) of an edge is a flat block as the block to be estimated. That is because, if the above described condition is satisfied, color correction which will be described later can be made.
For example, in the horizontal edge blocks, it is only necessary that the two blocks (u1−1, v), (u2+1, v) adjacent in the horizontal direction of the continuous edge blocks are flat blocks. Thus, it is only necessary for the horizontal edge block in (condition 3) to satisfy the following equation (17):
(u1−1,v)εFBLR−G∩(u2+1,v)εFBLR−G (17)
(Condition 4) The block selecting unit 23 selects the block having a difference in contrast generated at the edge to be a threshold value TH_CONTRAST or more for both the two color components as the block to be estimated. That is because, if the contrast difference is small, detection of the blur width of the edge by the axial chromatic aberration becomes relatively difficult.
For example, it is only necessary for the horizontal edge block in (condition 4) to satisfy the following equation (18):
|R Block(u1−1,v)−R Block(u2+1,v)|≧TH_CONTRAST
|G Block(u1−1,v)−G Block(u2+1,v)|≧TH_CONTRAST (18)
(Condition 5) The block selecting unit 23 selects the block having an edge length (length of the continuous edge blocks) less than a threshold value TH_LENGTH as the block to be estimated. The edge length is in inverse proportion to sharpness of the edge in general. Thus, by excluding the edge with too much blur or the edge which is not sharp, determination accuracy can be improved.
For example, it is only necessary for the horizontal edge block in (condition 5) to satisfy the following equation (19):
u2−u1≦TH_LENGTH (19)
Summarizing the above described (condition 1) to (condition 5), the condition which should be satisfied by the set HEBSR−G of the continuous edge blocks in the horizontal direction can be expressed by an equation (20). The continuous edge block at this time continues in the horizontal direction from (u1, v) to (u2, v). Moreover, the two blocks (u1−1, v) (u2+1, v) located in the horizontal direction of the continuous edge blocks are both flat blocks.
HEBS R - G = { ( u ⁢ ⁢ 1 , u ⁢ ⁢ 2 , v ) | ( u ⁢ ⁢ 1 , v ) ∈ HEBL R - G ⋂ ⁢ … ⁢ ⋂ ( u ⁢ ⁢ 2 , v ) ∈ HEBL R - G ⋂ ∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢ ⁢  dxR Block ⁡ ( i , v )  -  ∑ i = u ⁢ ⁢ 1 u2 ⁢ ⁢ dxR Block ⁡ ( i , v )  = 0 ⋂ ∑ i = u ⁢ ⁢ 1 u ⁢ ⁢ 2 ⁢ ⁢  dxG Block ⁡ ( i , v )  -  ∑ i = u ⁢ ⁢ 1 u2 ⁢ ⁢ dxG Block ⁡ ( i , v )  = 0 ⋂ ( u ⁢ ⁢ 1 - 1 , v ) ∈ FBL R - G ⋂ ( u ⁢ ⁢ 2 + 1 , v ) ∈ FBL R - G ⋂  R Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) - R Block ⁡ ( u ⁢ ⁢ 2 + 1 , v )  ≥ TH_CONTRAST ⋂  G Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) - G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v )  ≥ TH_CONTRAST ⋂ u ⁢ ⁢ 2 - 1 ≤ TH_EB ⁢ _LENGTH } ( 20 )
Moreover, all of the above described (condition 1) to (condition 5) can be also applied to selection of the set VEBSR−G of the continuous edge blocks in the vertical direction. The condition which should be satisfied by the set VEBSR−G of the continuous edge blocks in the vertical direction can be expressed by an equation (21). The continuous edge blocks at this time continues in the vertical direction from (u, v1) to (u, v2). Moreover, the two blocks (u, v1−1) (u, v2+1) located in the vertical direction of the continuous edge blocks are both flat blocks.
VEBS R - G = { ( u , v ⁢ ⁢ 1 , v ⁢ ⁢ 2 ) | ( u , v ⁢ ⁢ 1 ) ∈ VEBL R - G ⋂ ⁢ … ⁢ ⋂ ( u , v ⁢ ⁢ 2 ) ∈ VEBL R - G ⋂ ∑ j = v ⁢ ⁢ 1 v ⁢ ⁢ 2 ⁢ ⁢  dyR Block ⁡ ( u , j )  -  ∑ j = v ⁢ ⁢ 1 v2 ⁢ ⁢ dyR Block ⁡ ( u , j )  = 0 ⋂ ∑ j = v ⁢ ⁢ 1 v ⁢ ⁢ 2 ⁢ ⁢  dyG Block ⁡ ( u , j )  -  ∑ j = v ⁢ ⁢ 1 v ⁢ ⁢ 2 ⁢ ⁢ dyG Block ⁡ ( u , j )  = 0 ⋂ ( u , v ⁢ ⁢ 1 - 1 ) ∈ FBL R - G ⋂ ( u , v ⁢ ⁢ 2 + 1 ) ∈ FBL R - G ⋂  R Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) - R Block ⁡ ( u , v ⁢ ⁢ 2 + 1 )  ≥ TH_CONTRAST ⋂  G Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) - G Block ⁡ ( u , v ⁢ ⁢ 2 + 1 )  ≥ TH_CONTRAST ⋂ v ⁢ ⁢ 2 - v ⁢ ⁢ 1 ≤ TH_EB ⁢ _LENGTH } ( 21 )
Moreover, regarding the continuous edge blocks defined at #105, a set of pixels (x, y) in the continuous edge blocks is assumed to be P. At this time, P is expressed by an equation (22):
For the continuous edge blocks (u1,u2,v) ε HEBSR−G in the horizontal direction
x1=u1×BLOCK_SIZEx2=(u2+1)×BLOCK_SIZE−1
y1=v×BLOCK_SIZEy2=(v+1)×BLOCK_SIZE−1
For the continuous edge blocks (u, v1, v2) ε VEBSR−G in the vertical direction
x1=u×BLOCK_SIZEx2=(u+1)×BLOCK_SIZE−1
y1=v1×BLOCK_SIZEy2=(v2+1)×BLOCK_SIZE−1
P={(x,y)|x1≦x≦x2∩y1≦y≦y2} (22)
A selection example of the block to be estimated in a scene corresponding to FIG. 4 is illustrated in FIG. 9. In the example in FIG. 9, an edge block selected as a block to be estimated is indicated by a bold line. In FIG. 9, a horizontal edge block (H) sandwiched in the horizontal direction by two flat blocks (F) and a vertical edge block (V) sandwiched in the vertical direction by two flat blocks (F) are the blocks to be estimated.
Step #106: The focus determination unit 24 applies color correction to each of the continuous edge blocks (blocks to be estimated) acquired at #105.
As an example, the focus determination unit 24 acquires FLR and FLG as well as FHR and FHG by the following equations (23) and (24) by using an average pixel value of two flat blocks adjacent in the gradient direction of the edge with respect to the continuous edge blocks. Here, “FL” indicates a flat block on a low contrast side and “FH” indicates a flat block on a high contrast side.
For the continuous edge blocks HEBSR−G in the horizontal direction
G Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) < G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) → { FL R = R Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) , FH R = R Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) FL G = G Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) , FH G = G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) ⁢ ⁢ G Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) > G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) → { FL R = R Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) , FH R = R Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) FL G = G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) , FH G = G Block ⁡ ( u ⁢ ⁢ 1 - 1 , v ) ( 23 )
For the continuous edge blocks VEBSR−G in the vertical direction
G Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) < G Block ⁡ ( u , v ⁢ ⁢ 2 + 1 ) → { FL R = R Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) , FH R = R Block ⁡ ( u , v ⁢ ⁢ 2 + 1 ) FL G = G Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) , FH G = G Block ⁡ ( u , v ⁢ ⁢ 2 + 1 ) ⁢ ⁢ G Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) > G Block ⁡ ( u , v ⁢ ⁢ 2 + 1 ) → { FL R = R Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) , FH R = R Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) FL G = G Block ⁡ ( u ⁢ ⁢ 2 + 1 , v ) , FH G = G Block ⁡ ( u , v ⁢ ⁢ 1 - 1 ) ( 24 )
Subsequently, the focus determination unit 24 determines primary coefficients (aR0, aR1, aG0, aG1) by an equation (25) so that FLR, FLG, and FHR, FHG all become gray between CONT_L and CONT_H.
a R ⁢ ⁢ 0 = CONT_L × FH R - CONT_H × FL R FH R - FL R ⁢ ⁢ a R ⁢ ⁢ 1 = CONT_H - CONT_L FH R - FL R ⁢ ⁢ a G ⁢ ⁢ 0 = CONT_L × FH G - CONT_H × FL G FH G - FL G ⁢ ⁢ a G ⁢ ⁢ 1 = CONT_H - CONT_L FH G - FL G ( 25 )
Then, the focus determination unit 24 normalizes pixel values R(x, y) and G(x, y) of a pixel (pixel of (x, y) ε P) in the continuous edge block by the following equations (26) and (27).
R′(x,y)=a R0 +a R1 ×R(x,y) (26)
G′(x,y)=a G0 +a G1 ×G(x,y) (27)
FIGS. 7A and 7B are diagrams illustrating an outline of color correction at #106. For example, an edge R(x) of an R component indicated by a solid line and an edge G(x) of a G component indicated by a broken line in FIG. 7A are subjected to color correction so that a contrast difference on the both sides of an edge area becomes equal as illustrated in FIG. 7B.
Step #107: The focus determination unit 24 acquires the difference Δw of the blur width of the edge between two colors having axial chromatic aberration in each of the continuous edge blocks (blocks to be estimated), respectively, subjected to color correction at #106. As an example, the focus determination unit 24 may acquire Δw by the following method:
The focus determination unit 24 sorts pixel sets ER, EG of an edge part having a gradient at a threshold value TH_EDGE or more from the pixel (pixel of (x, y) ε P) in the continuous edge block subjected to color correction by an equation (28).
E R = { ( x , y ) ❘ ( x , y ) ∈ P ⋂ dxR ⁡ ( x , y ) 2 + dyR ⁡ ( x , y ) 2 >= TH_EDGE } ⁢ ⁢ E G = { ( x , y ) ❘ ( x , y ) ∈ P ⋂ dxG ⁡ ( x , y ) 2 + dyG ⁡ ( x , y ) 2 >= TH_EDGE } ( 28 )
Then, the focus determination unit 24 acquires an average value EmeanR of an R pixel and an average value EmeanG of a G pixel by an equation (29):
Emean R = ∑ ( x , y ) ∈ E R ⁢ ⁢ R ⁡ ( x , y )  E R  ⁢ ⁢ Emean G = ∑ ( x , y ) ∈ E G ⁢ ⁢ G ⁡ ( x , y )  E G  ( 29 )
Here, the above described set ER can be sorted to a set ELR of pixel values each having R(x, y) smaller than EmeanR and a set EHR of pixel values each having R(x, y) larger than EmeanR. Moreover, the above described set EG can be sorted to a set ELG of pixel values each having G(x, y) smaller than EmeanG and a set EHG of pixel values each having G(x, y) larger than EmeanG (equation (30)).
EL R={(x,y)|R(x,y)<=EmeanR∩(x,y)εE R}
EH R={(x,y)|R(x,y)>=EmeanR∩(x,y)εE R}
EL G={(x,y)|G(x,y)<=EmeanR∩(x,y)εE G}
EH G={(x,y)|G(x,y)>=EmeanG∩(x,y)εE G} (30)
Moreover, a sum ELR−G of sets ELR and ELG and a sum EHR−G of sets EHR and EHG can be defined by an equation (31), respectively:
EL R−G =EL R ∪EL G
EH R−G =EH R ∪EH G (31)
The focus determination unit 24 acquires a sum of color differences ΔALR−G and ΔAHR−G respectively, for the pixels of the sets ELR−G and EHR−G by an equation (32). Then, the focus determination unit 24 acquires an index value IR−G by an equation (33):
Δ ⁢ ⁢ AL R - G = ∑ ( x , y ) ∈ EL R - G ⁢ ( R ⁡ ( x , y ) - G ⁡ ( x , y ) ) ⁢ ⁢ Δ ⁢ ⁢ AH R - G = ∑ ( x , y ) ∈ EH R - G ⁢ ( R ⁡ ( x , y ) - G ⁡ ( x , y ) ) ( 32 ) I R - G = 4 × ( Δ ⁢ ⁢ AL R - G - Δ ⁢ ⁢ AH R - G ) CONT_H - CONT_L ( 33 )
The above described index value IR−G forms a relationship of IR−G=ΔwR−G×NW for the difference ΔwR−G in the blur width of the edge in two colors in the continuous edge blocks and the number of edges NW by the unit of pixel. Thus, the focus determination unit 24 can acquire ΔwR−G by roughly calculating NW.
Step #108: The focus determination unit 24 performs focus determination for each block to be estimated by using the difference Δw in the blur width of the edge between two colors.
In the present embodiment, ΔwR−G calculated from the color difference R−G is a positive value on the front focus side and a negative value on the rear focus side. Thus, the focus determination at #108 is performed as follows, for example:
ΔwR−G > ΔwAFR−G + TH _ FOCUS1 REAR FOCUS ΔwAFR−G + TH _ FOCUS2 ≦ ΔwR−G ≦ ΔwAFR−G + FOCUS TH _ FOCUS1 ΔwR−G < ΔwAFR−G + TH _ FOCUS2 FRONT FOCUS
Here, “ΔwAFR−G” is a reference value indicating a blur width at focusing. A value of ΔwAFR−G is adjusted as appropriate by considering characteristics of axial chromatic aberration of the optical system, a color of the subject, a sectional shape of the edge, an interpolation method of the color image, AF means of the camera and the like. Moreover, “TH_FOCUS1, TH_FOCUS2” are parameters indicating blur allowable values on the rear focus side and the front focus side, respectively.
FIGS. 8A to 8C are diagrams illustrating an outline of the above described focus determination. For example, the edge R(x) of the R component and the edge G(x) of the G component after normalization by color correction will be considered.
Since R blurs more than G in rear focus, a relationship of wR>wG is formed between an edge length wR of the R component and an edge length wG of the G component as illustrated in FIG. 8A. Thus, if Δw (=wR−wG) is larger than the threshold value TH_FOCUS1, the focus determination unit 24 determines it to be rear focus.
On the other hand, since G blurs more than R in front focus, a relationship of wR<wG is formed between an edge length wR of the R component and an edge length wG of the G component as illustrated in FIG. 8C. Thus, if Δw (=wR−wG) is smaller than the threshold value TH_FOCUS2, the focus determination unit 24 determines it to be front focus.
At focusing, a relationship of wR=wG is formed between an edge length wR of the R component and an edge length wG of the G component in principle as illustrated in FIG. 8B. Thus, if Δw (=wR−wG) is at the threshold value TH_FOCUS2 or more and at the threshold value TH_FOCUS 1 or less, the focus determination unit 24 determines that rear focus or front focus is not clear since the focus determination unit 24 is in the focus state.
Step #109: The camera microcomputer 14 determines whether or not focus determination will be further made by a different parameter.
If the above described requirement is satisfied (YES side), the camera microcomputer 14 performs focus determination with the processing similar to #104 to #108 for the other color components (B−G and R−B, for example). Alternatively, the camera microcomputer 14 may perform focus determination with the processing similar to #104 to #108 by changing the block size.
On the other hand, if the above described requirement is not satisfied (NO side), the camera microcomputer 14 proceeds to processing at #108.
Step #110: The focus estimating unit 25 integrates the focus determination results at each of the blocks to be estimated and estimates the focus state of the subject image in the focus estimating area.
The focus estimating unit 25 gives serial numbers with a suffix i to all the determination results (NResult pieces) for each setting of the color difference (R−G, B−G and the like), the gradient direction (vertical/horizontal and the like), and the block size for each of the blocks to be estimated (that is, the maximum value of i is the total number of the determination results). For total determination, the following value is substituted for the i-th determination result Ai:
A i = { 1 ( REAR ⁢ ⁢ FOCUS ⁢ ⁢ DETERMINATION ) 0 ( FOCUS ⁢ ⁢ DETERMINATION ) ⁢ ⁢ ( i = 1 , 2 ⁢ ⁢ … ⁢ ⁢ N Result ) - 1 ( FRONT ⁢ ⁢ FOCUS ⁢ ⁢ DETERMINATION )
The above described determination result Ai is an example of a focus determination value and indicates a determination result of focusing at the block to be estimated.
Moreover, the focus estimating unit 25 at #110 may estimate the focus state of the subject image in the focus estimating area by applying weight-addition to at least either one of the focus determination result by a combination of the different color components and the focus determination result with different block sizes. Examples of (B1) to (B7) as an example of the weight-addition will be described below. The weight-addition illustrated below may be performed in combination as appropriate.
(B1): For example, the focus estimating unit 25 may multiply the result of focus determination between the color components with large axial chromatic aberration by a large weighting factor.
(B2): The larger the block size (that is, an area of an individual block) is, the relatively higher reliability of focus determination becomes. Therefore, the focus estimating unit 25 may increase the value of the weighting factor in proportion to the block size.
(B3): The larger the contrast difference of the edge included in the block is, the relatively higher reliability of focus determination becomes. Therefore, the focus estimating unit 25 may increase the value of the weighting factor in the block to be estimated as the contrast difference of the edge increases.
(B4): The smaller the blur width of the edge included in the block is, the edge is in a state close to the focusing, and reliability of focus determination becomes relatively high. Moreover, if a subject contour has smooth gradation, the blur width of the edge looks large, and reliability of focus determination becomes relatively low. Therefore, the focus estimating unit 25 may increase the value of the weighting factor in the block to be estimated as the apparent blur width of the edge decreases.
(B5): In accordance with a zoom position of the photographing optical system 12, a behavior of axial chromatic aberration changes, respectively. Therefore, the focus estimating unit 25 may determine the value of the weighting factor in the block to be estimated in accordance with the zoom position of the photographing optical system when a color image is acquired. At this time, a factor table indicating correspondence between the zoom position of the photographing optical system 12 and the weighting factor may be prepared in advance so that the focus estimating unit 25 reads the weighting factor from the factor table in accordance with the zoom position of the photographing optical system 12 and determines the weighting factor. If the electronic camera is a camera with interchangeable lenses, it is needless to say that the above described factor table is prepared for each of the interchangeable lenses.
(B6): When the difference in the blur width of the edge is to be acquired, if the edge was extracted from a part close to an achromatic color in at least one of the flat blocks adjacent on the both sides in the gradient direction, reliability of focus determination becomes relatively high. Therefore, the focus estimating unit 25 may increase the value of the weighting factor in the block to be estimated if a color difference in the two color components in the flat blocks adjacent in the gradient direction of the edge is lower than the threshold value.
(B7): For a moving subject, reliability of focus determination becomes relatively low. For example, when a motion vector of the subject is calculated by using a through image, the larger the motion vector of the subject in the block to be estimated is, the focus estimating unit 25 may decrease the value of the weighting factor in the block to be estimated.
When all the focus estimation results are handled equally, the weighting factor ωi may be set to “1”.
Then, the focus estimating unit 25 may acquire a comprehensive determination result of the block to be estimated by an equation (34) by using the total number AOnFocus of the blocks to be estimated which was determined to be focused and the weighted sum ATotalResult of the determination result. “TH_N_ONFOCUS” is a threshold value indicating a focus state, and “TH_REARFOCUS, TH_FRONTFOCUS” are threshold values indicating a rear focus state and a front focus state, respectively.
A OnFocus = { A i | A i = 0 } ⁢ ⁢ ( i = 1 , 2 ⁢ ⁢ … ⁢ ⁢ N Result ) ⁢ ⁢ A TotalResult = ∑ i = 0 N Result ⁢ ⁢ ω ⁢ ⁢ iA i ⁢ ⁢ For ⁢ ⁢  A OnFocus  ≥ TH_N ⁢ _ONFOCUS , ⁢ FOCUS ⁢ ⁢ For ⁢ ⁢  A OnFocus  < TH_N ⁢ _ONFOCUS ⋂ A TotalResult ≥ TH_REARFOCUS , ⁢ REAR ⁢ ⁢ FOCUS ⁢ ⁢ For ⁢ ⁢  A OnFocus  < TH_N ⁢ _ONFOCUS ⋂ A TotalResult ≤ TH_FRONTFOCUS , ⁢ FRONT ⁢ ⁢ FOCUS ( 34 )
Subsequently, the focus estimating unit 25 determines the focus state of the whole focus estimating area by using the determination result of the above described block to be estimated. As an example, the focus estimating unit 25 may employ the determination result of the block to be estimated in the largest number in the focus estimation of the focus estimating area.
FIG. 10 is a diagram illustrating an example of focus estimation in the focus estimating area in a scene corresponding to FIG. 9. In FIG. 10, the blocks to be estimated which were determined to be front focus, rear focus, and focused are noted as “FR”, “RE”, and “FO”, respectively. In the example in FIG. 10, a ratio of rear focus: focused: front focus of the determination result of the block to be estimated (continuous edge block) in the focus estimating area is 1:1:3, and the entire focus estimating area is determined to be front focus.
Step #111: The camera microcomputer 14 moves the AF lens in a direction such that Δw gets closer to zero based on the determination result of the whole focus estimating area acquired at #110. Explanation of the processing in FIG. 2 ends here.
A working effect of the first embodiment will be described below.
The electronic camera 11 of the first embodiment sorts the block type based on the gradient amount of the pixel in the block in the focus estimating area of the color image (#104) and selects the block to be estimated to be applied to focus estimation (#105). The electronic camera 11 performs focus determination for a block to be estimated (#107, #108) by using the difference in the blur width (Δw) of the edge in the block to be estimated and determines the focus state of the focus estimating area by integrating the focus results (#110).
As a result, since the focus estimation for the entire focus estimating area can be made by accumulating the focus determination by the unit of individual blocks to be estimated, focus estimation with higher accuracy can be realized even if a focus state of the color image including edges in various colors/contrasts is estimated.
Moreover, in the electronic camera 11 in the first embodiment, when the edge continues in the same direction between the adjacent blocks to be estimated, a plurality of blocks to be estimated is grouped (#104). As a result, since the block to be estimated with a small area and the block to be estimated with a large area can be extracted from the focus estimating area at the same time in one session of processing, efficiency of calculation is improved.
At #104 in the above described embodiment, the edge detecting unit 22 may sort the block type by considering presence of the mountain/valley-shaped structure on a scale of the block size or less. In the variation 1, based on the idea similar to (Condition 2) at #105, it is only necessary for the edge detecting unit 22 to apply an equation (35) instead of the equation (11) in the determination of (i) at #104. Similarly, in the determination of (ii) at #104, it is only necessary for the edge detecting unit 22 to apply an equation (36) instead of the equation (12):
HEBL R - G = { ( u , v ) | nHGradPixel R - G ⁡ ( u , v ) ≥ N Pixel × RATIO_EDGEBLOCK ⋂ nVGradPixel R - G ⁡ ( u , v ) < N Pixel × RATIO_EDGEBLOCK ⋂ ∑ ( x , y ) ∈ BL ⁢ ⁢  dxR ⁡ ( x , y )  -  dxR Block ⁡ ( u , v )  = 0 } ( 35 ) VEBL R - G = { ( u , v ) | nHGradPixel R - G ⁡ ( u , v ) < N Pixel × RATIO_EDGEBLOCK ⋂ nVGradPixel R - G ⁡ ( u , v ) ≥ N Pixel × RATIO_EDGEBLOCK ⋂ ∑ ( x , y ) ∈ BL ⁢ ⁢  dyR ⁡ ( x , y )  -  dyR Block ⁡ ( u , v )  = 0 } ( 36 )
According to the above variation 1, since an area giving a bad influence on the focus determination by presence of an image structure having a block size or less can be eliminated in advance, determination accuracy can be improved.
At #110 of the above described embodiment, the focus estimating unit 25 may change the order of determination as appropriate.
As an example, the focus estimating unit 25 performs focus determination after applying weighted-averaging to the difference Δw of the blur width calculated by the unit of blocks to be estimated. Then, the focus estimating unit 25 sets a weight αi of the determination result by using the difference in the blur width calculated for M pieces of the blocks to be estimated in the focus estimating area as Δwi. In the variation 2, Δwi indicating the difference in blur width becomes a focus determination value.
Moreover, the weight of the determination result may be given by a length of a continuous edge block or a contrast difference between the adjacent flat blocks, or any one of the above described (B1) to (B7) described at #110 may be applied. When all the determination results are to be handled equally, αi=1 may be set.
An average (Δwmean) of the difference in the blur width in the focus estimating area can be acquired by an equation (37):
Δ ⁢ ⁢ w Mean = ∑ i = 1 i <= M Result ⁢ ⁢ α i ⁢ Δ ⁢ ⁢ w i ∑ i = 1 i <= M Result ⁢ ⁢ α i ( 37 )
The focus estimating unit 25 may use the result of focus determination by comparison between the average Δwmean of the difference in the blur width and the threshold value as the determination result of the focus estimating area.
FIG. 11 is a diagram illustrating a configuration example of a focus estimating device of a second embodiment. The focus estimating device of the second embodiment is a personal computer in which an image estimating program is installed. In the example of the second embodiment, a color image photographed by the electronic camera is read by the focus estimating device, and the image is subjected to focus estimation in a post-processing process by the image estimating program.
A computer 31 illustrated in FIG. 11 has a data reading unit 32, a storage device 33, a CPU 34, a memory 35, an input/output I/F 36, and a bus 37. The data reading unit 32, the storage device 33, the CPU 34, the memory 35, and the input/output I/F 36 are connected to each other via the bus 37. Moreover, to the computer 31, an input device 38 (keyboard, pointing device and the like) and a monitor 39 are connected via the input/output I/F 36, respectively. The input/output I/F 36 receives various inputs from the input device 38 and also outputs data for display to the monitor 39.
The data reading unit 32 is used when image data or a program is read from the outside. For example, the data reading unit 32 is a reading device (reading devices of an optical disk, a magnetic disk, a magneto-optical disk and the like) for acquiring data from a detachable recording medium, a communication device (a USB interface, an LAN module, a wireless LAN module and the like) for conducting communication with an external device in compliance with a known communication standards. The data reading unit 32 of the second embodiment functions as an acquiring unit for acquiring a color image.
The storage device 33 includes a hard disk, a recording medium such as a non-volatile semiconductor memory, for example. This storage device 33 records an image processing program. The storage device 33 can also store image data read from the data reading unit 32 and the like.
The CPU 34 is a processor for integrally controlling each part of the computer 31. This CPU 34 operates as the area setting unit 21, the edge detecting unit 22, the block selecting unit 23, the focus determination unit 24, and the focus estimating unit 25 in the first embodiment by executing the image estimating program.
The memory 35 temporarily stores various calculation results by the image processing program. This memory 35 is a volatile SDRAM, for example.
FIG. 12 is a flowchart illustrating an operation example of the focus estimating device of the second embodiment. Since processing at #201 to #209 in FIG. 12 corresponds to processing at #102 to #110 in FIG. 2, duplicated explanation will be omitted. At #201, it is assumed that the CPU 34 acquires a color image photographed by the electronic camera from the data reading unit 32. Then, at #210, the CPU 34 executes quality judgment of the image based on the focus estimation and displays the result on the monitor 39. The CPU 34 may automatically select a favorable image from among a plurality of color images in accordance with the result of the quality judgment of the image. In such second embodiment, too, the effect substantially similar to the first embodiment can be acquired.
Supplementary Matters of Embodiments
(Supplement 1) In the above described embodiments, only the edge block in which both of the blocks adjacent in the gradient direction of the edge are flat blocks is handled as a block to be estimated. However, as described also at #105, as long as one of the blocks adjacent in the gradient direction of the edge is a flat block, the edge block may be used as a block to be estimated.
(Supplement 2) An example of weighting to the focus determination value in the above described embodiments is not limited to the examples in the above described (B1) to (B7). For example, if (condition 2) is not applied to #105 in the above embodiment (that is, a block including an uneven structure in the edge portion is not excluded from the block to be estimated), the focus estimating unit 25 may decrease the value of the weighting factor in the block to be estimated including the uneven structure in the edge portion.
In the above embodiments, the example in which each function of the image processing device is realized by a program in a software manner is described. However, in the present invention, each of the functions of the area setting unit 21, the edge detecting unit 22, the block selecting unit 23, the focus determination unit 24, and the focus estimating unit 25 may be realized in a hardware manner by using ASIC.
In the above described second embodiment, as an example of the focus estimating device which reads an image photographed by the electronic camera, a personal computer is described, but the focus estimating device in the second embodiment may be other electronic devices such as a photo viewer and a printer, for example.
1. A focus estimating device comprising:
an area setting unit setting an area in which focusing of an image is determined and dividing the area into a plurality of blocks;
a block selecting unit selecting a block to be used for focus determination:
a block focus determination unit performing focus determination for each block from an edge of each color component within the block selected by the block selecting unit; and
a focus determination unit determining focusing of the area based on the focus determination.
2. The focus estimating device according to claim 1, wherein
the block selecting unit selects a block which is adjacent to the block including a flat part and includes an edge.
3. The focus estimating device according to claim 2, wherein
when pixel values of adjacent pixels in each color in the block are differentiated and a ratio of the pixel having the difference at a first threshold value or less is acquired, the block including the flat part is a block having the ratio at a second threshold value or more.
4. The focus estimating device according to claim 1, wherein
the block selecting unit groups the blocks having a continuous edge.
5. The focus estimating device according to claim 1, wherein
the block selecting unit selects a block in which edge inclinations in two color components are in the same direction.
6. The focus estimating device according to claim 1, wherein
the block selecting unit excludes the block including an uneven structure in the edge from the selection.
7. The focus estimating device according to claim 6, wherein
the block including the uneven structure is a block including any one of a thin-line structure, a mountain-shaped structure and a valley-shaped structure.
8. The focus estimating device according to claim 1, wherein
the block selecting unit selects a block in which a difference in contrast generated at the edge is at a threshold value or more.
9. The focus estimating device according to claim 1, wherein
the block selecting unit selects a block in which a length of the edge is less than a threshold value.
10. The focus estimating device according to claim 1, wherein
the focus determination unit determines focusing of the area by applying weight-addition to at least either one of the focus determination result by a combination of the different color components and the focus determination result at different block sizes.
11. The focus estimating device according to claim 10, wherein
in the focus determination unit, the larger a difference in contrast at an edge is, the larger a value of weighting in the block is set.
12. The focus estimating device according to claim 10, wherein
in the focus determination unit, the smaller a blur width at an edge is, the larger a value of weighting in the block is set.
13. The focus estimating device according to claim 10, wherein
the focus determination unit determines a value of weighting in the block in accordance with a zoom position of a photographing optical system when the image is acquired.
14. The focus estimating device according to claim 10, wherein
the focus determination unit increases a value of weighting in the block when a color difference in two color components which acquire a difference in a blur width in at least one of the blocks adjacent to a gradient direction of the edge is lower than a threshold value.
15. The focus estimating device according to claim 10, wherein
in the focus determination unit, the larger the motion vector of the subject in the block is, the smaller a value of weighting in the block is set.
the focus estimating device according to claim 1.
17. The focus estimating device according to claim 1, wherein the block focus determination unit obtains a blur width of a first edge of a first color and a blur width of a second edge of a second color and performs the focus determination based on the blur width of the first edge and the blur width of the second edge.
18. A focus estimating device comprising:
a block selecting unit selecting the block including an edge;
19. The focus estimating device according to claim 18, wherein
20. The focus estimating device according to claim 18, wherein
21. The focus estimating device according to claim 18, wherein
22. The focus estimating device according to claim 21, wherein
23. The focus estimating device according to claim 18, wherein
24. The focus estimating device according to claim 18, wherein
25. The focus estimating device according to claim 18, wherein
26. The focus estimating device according to claim 25, wherein
27. The focus estimating device according to claim 25, wherein
28. The focus estimating device according to claim 25, wherein
29. The focus estimating device according to claim 25, wherein
30. The focus estimating device according to claim 25, wherein
the focus estimating device according to claim 18.
32. The focus estimating device of claim 18, wherein the block focus determination unit obtains a blur width of a first edge of a first color and a blur width of a second edge of a second color and performs the focus determination based on the blur width of the first edge and the blur width of the second edge.
33. A non-transitory computer readable storage medium stores an image processing computer program causing a computer to execute:
setting an area in which focusing of an image is determined and dividing the area into a plurality of blocks;
selecting a block to be used for focus determination;
performing focus determination for each block from an edge of each color component within the selected block; and
determining focusing of the area based on the focus determination.
34. A non-transitory computer readable storage medium stores an image processing computer program causing a computer to execute:
selecting the block including an edge;
performing focus determination for each block from the edge of each color component within the selected block; and
35. A focus estimating device comprising:
a computer processor that operates as:
36. A focus estimating device comprising:
an area setting unit, executed by the processor, setting an area in which focusing of an image is determined and dividing the area into a plurality of blocks;
a block selecting unit, executed by the processor, selecting a block to be used for focus determination:
a block focus determination unit, executed by the processor, performing focus determination for each block from an edge of each color component within the block selected by the block selecting unit; and
a focus determination unit, executed by the processor, determining focusing of the area based on the focus determination.
37. A focus estimating device comprising:
38. A focus estimating device comprising:
a block selecting unit, executed by the processor, selecting the block including an edge;
US14/186,575 2011-09-02 2014-02-21 Focus estimating device, imaging device, and storage medium storing image processing program Active US9392159B2 (en)
JP2011191555 2011-09-02
JP2011-191555 2011-09-02
PCT/JP2012/005539 WO2013031238A1 (en) 2011-09-02 2012-08-31 Focus evaluation device, imaging device, and program
US15/183,091 US9648227B2 (en) 2011-09-02 2016-06-15 Focus estimating device, imaging device, and storage medium storing image processing program
PCT/JP2012/005539 Continuation WO2013031238A1 (en) 2011-09-02 2012-08-31 Focus evaluation device, imaging device, and program
US15/183,091 Continuation US9648227B2 (en) 2011-09-02 2016-06-15 Focus estimating device, imaging device, and storage medium storing image processing program
US20140232927A1 US20140232927A1 (en) 2014-08-21
US9392159B2 true US9392159B2 (en) 2016-07-12
ID=47755765
US14/186,575 Active US9392159B2 (en) 2011-09-02 2014-02-21 Focus estimating device, imaging device, and storage medium storing image processing program
US15/183,091 Active US9648227B2 (en) 2011-09-02 2016-06-15 Focus estimating device, imaging device, and storage medium storing image processing program
US (2) US9392159B2 (en)
JP (1) JP6074954B2 (en)
CN (1) CN103765276B (en)
WO (1) WO2013031238A1 (en)
CN103765276B (en) 2011-09-02 2017-01-18 株式会社尼康 Focus evaluation device, imaging device, and program
JP6565326B2 (en) * 2014-08-11 2019-08-28 セイコーエプソン株式会社 Imaging display device and control method thereof
JP6467823B2 (en) * 2014-08-29 2019-02-13 株式会社ニコン Imaging device
JP6467824B2 (en) * 2014-08-29 2019-02-13 株式会社ニコン Imaging device
CN105527778A (en) * 2016-01-16 2016-04-27 上海大学 Automatic focusing method for electric adjustable liquid lens
US9762790B2 (en) * 2016-02-09 2017-09-12 Panasonic Intellectual Property Management Co., Ltd. Image pickup apparatus using edge detection and distance for focus assist
JP2017225071A (en) * 2016-06-17 2017-12-21 キヤノン株式会社 Information processing device, information processing method and program
EP0335656A1 (en) 1988-03-29 1989-10-04 Sony Corporation Automatic focus control apparatus
JPH03136012A (en) 1989-10-23 1991-06-10 Olympus Optical Co Ltd Focusing detecting device
JPH03204277A (en) 1990-01-05 1991-09-05 Canon Inc Focus detection method
US5192998A (en) 1990-01-05 1993-03-09 Canon Kabushiki Kaisha In-focus detecting device
JPH06138362A (en) 1991-02-06 1994-05-20 Sony Corp Autofocusing device
US5376991A (en) 1990-02-27 1994-12-27 Canon Kabushiki Kaisha Control system
JPH09139940A (en) 1995-04-18 1997-05-27 Sun Microsyst Inc Decoder for video delivery system expandable between terminals
JP2000165892A (en) 1998-11-30 2000-06-16 Matsushita Electric Ind Co Ltd Pixel interpolation method and image processor
JP2000299874A (en) 1999-04-12 2000-10-24 Sony Corp Signal processor, signal processing method, image pickup device and image pickup method
JP2002214513A (en) 2001-01-16 2002-07-31 Minolta Co Ltd Optical system controller, optical system control method, and recording medium
JP2003015026A (en) 2001-07-02 2003-01-15 Konica Corp Image processing method and image processing device
US20030020819A1 (en) 2001-07-27 2003-01-30 Hidetoshi Fukuda Image pickup apparatus
US20030099044A1 (en) 2001-11-29 2003-05-29 Minolta Co. Ltd. Autofocusing apparatus
CN1487728A (en) 2002-08-27 2004-04-07 奥林巴斯株式会社 camera
JP2004157456A (en) 2002-11-08 2004-06-03 Olympus Corp Camera and range-finding method of camera
JP2004219461A (en) 2003-01-09 2004-08-05 Minolta Co Ltd Imaging apparatus and auto-focusing device
JP2005156597A (en) 2003-11-20 2005-06-16 Canon Inc Automatic focusing apparatus and method, program, and storage medium
US20060078217A1 (en) 2004-05-20 2006-04-13 Seiko Epson Corporation Out-of-focus detection method and imaging device control method
US20070116375A1 (en) 2004-04-12 2007-05-24 Nikon Corporation Image processing device having color shift correcting function, image processing program and electronic camera
JP2008042937A (en) 2007-09-14 2008-02-21 Olympus Corp Imaging apparatus
US20090102963A1 (en) 2007-10-22 2009-04-23 Yunn-En Yeo Auto-focus image system
JP2009103800A (en) 2007-10-22 2009-05-14 Sony Corp Automatic focusing device and method, and program
US20090208114A1 (en) 2008-02-20 2009-08-20 Seiko Epson Corporation Image Processing Apparatus
US20100128144A1 (en) 2008-11-26 2010-05-27 Hiok Nam Tay Auto-focus image system
US20100202683A1 (en) * 2009-02-06 2010-08-12 Robinson Ian S Optimized imaging system for collection of high resolution imagery
US20100315521A1 (en) 2009-06-15 2010-12-16 Keiji Kunishige Photographing device, photographing method, and playback method
JP2010286752A (en) 2009-06-15 2010-12-24 Olympus Imaging Corp Imaging device and imaging method
JP2011107682A (en) 2009-07-17 2011-06-02 Nikon Corp Focusing device and camera
US20110150349A1 (en) * 2009-12-17 2011-06-23 Sanyo Electric Co., Ltd. Image processing apparatus and image sensing apparatus
JP2011154194A (en) 2010-01-27 2011-08-11 Nikon Corp Imaging apparatus
US20110199534A1 (en) 2010-01-12 2011-08-18 Nikon Corporation Image-capturing device
JP4497567B2 (en) * 1998-09-14 2010-07-07 キヤノン株式会社 Focus detection device, distance measuring device and optical instrument
CN1184502C (en) * 2002-06-13 2005-01-12 北京中星微电子有限公司 Process for automatic focusing
JP5141129B2 (en) 2007-08-07 2013-02-13 富士通株式会社 Receipt format control method, receipt format control program, receipt format control device, and receipt format control system
JP5144202B2 (en) * 2007-10-05 2013-02-13 オリンパス株式会社 Image processing apparatus and program
JP5515698B2 (en) * 2009-12-04 2014-06-11 パナソニック株式会社 Refrigeration cycle equipment
JP5223912B2 (en) * 2010-06-17 2013-06-26 株式会社ニコン Imaging apparatus and focus determination program
2012-08-31 CN CN201280042875.7A patent/CN103765276B/en active IP Right Grant
2012-08-31 JP JP2012191152A patent/JP6074954B2/en active Active
2012-08-31 WO PCT/JP2012/005539 patent/WO2013031238A1/en active Application Filing
2014-02-21 US US14/186,575 patent/US9392159B2/en active Active
2016-06-15 US US15/183,091 patent/US9648227B2/en active Active
JP2003046876A (en) 2001-07-27 2003-02-14 Olympus Optical Co Ltd Imaging equipment
JP2009199232A (en) 2008-02-20 2009-09-03 Seiko Epson Corp Image processor
Apr. 17, 2012 Office Action issued in Japanese Patent Application No. 2010-274724 (with translation).
Apr. 17, 2012 Office Action issued in Japanese Patent Application No. 2010-274725 (with translation).
Apr. 7, 2014 Office Action issued in U.S. Appl. No. 12/987,437.
Dec. 14, 2015 Office Action issued in Chinese Patent Application No. 201280042875.7.
Dec. 19, 2014 Office Action issued in European Application No. 11150773.7.
Dec. 20, 2011 Office Action issued in Japanese Patent Application No. 2010-004218 (with translation).
Dec. 4, 2012 Search Report issued in International Patent Application No. PCT/JP2012/005539 (with translation).
Jul. 31, 2012 Office Action issued in European Patent Application No. 11 150 773.7.
Jun. 27, 2011 Search Report issued in European Patent Application No. 11150773.7.
Jun. 3, 2015 Office Action issued in Chinese Patent Application No. 201280042875.7.
Jun. 6, 2013 Office Action issued in U.S. Appl. No. 12/987,437.
Mar. 15, 2016 Office Action issued in Japanese Patent Application No. 2012-191152.
Mar. 4, 2014 International Preliminary Report on Patentability issued in International Patent Application No. PCT/JP2012/005539 (with translation).
Nov. 20, 2012 Office Action issued in Japanese Patent Application No. 2010-274725 (with translation).
Oct. 10, 2014 Office Action issued in Chinese Application No. 201110008088.8.
Sep. 20, 2013 Search Report issued in European Patent Application No. 11 150 773.7-1562.
U.S. Appl. No. 12/987,437, filed Jan. 10, 2011.
JP6074954B2 (en) 2017-02-08
WO2013031238A1 (en) 2013-03-07
CN103765276A (en) 2014-04-30
CN103765276B (en) 2017-01-18
US20160301856A1 (en) 2016-10-13
JP2013065005A (en) 2013-04-11
US9648227B2 (en) 2017-05-09
US20140232927A1 (en) 2014-08-21
US10015472B2 (en) 2018-07-03 Image processing using distance information
US8849062B2 (en) 2014-09-30 Eye defect detection in international standards organization images
US8285067B2 (en) 2012-10-09 Method of notifying users regarding motion artifacts based on image analysis
US8730341B2 (en) 2014-05-20 Image processing apparatus, image pickup apparatus, control method for image processing apparatus, and storage medium storing control program therefor
US8648927B2 (en) 2014-02-11 Imaging device, imaging method and program storage medium
KR101246738B1 (en) 2013-03-25 Image processing apparatus and image processing method, and data processing apparatus and data processing method
US7586520B2 (en) 2009-09-08 Image correction apparatus, image correction program storage medium, image correction method, and image correction system
US8305487B2 (en) 2012-11-06 Method and apparatus for controlling multiple exposures
KR101324268B1 (en) 2013-11-01 Image identifying method and imaging apparatus
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KAWAI, ATSUSHI;REEL/FRAME:032831/0164
Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ADDRESS PREVIOUSLY RECORDED ON REEL 032831 FRAME 0164. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:KAWAI, ATSUSHI;REEL/FRAME:032927/0945
Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE DOCKET NUMBER PREVIOUSLY RECORDED ON REEL 032927 FRAME 0945. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:KAWAI, ATSUSHI;REEL/FRAME:033216/0718