Image pickup apparatus, image pickup system, method of controlling image pickup apparatus, and non-transitory computer-readable storage medium

An image pickup apparatus (100) that generates a three-dimensional image, includes an image pickup element (114) including a micro lens (2) and a pixel unit cell (1) including a plurality of pixels (1a, 1b) arranged to receive light from the microlens (2), wherein the image pickup element is configured to output separated first image and second image signals for respective pixels of the pixel unit cell, a parallax detector (123) configured to calculate a parallax comprising an image shift amount between the first image signal and the second image signal, and an image correcting portion (120) configured to perform a correction to change the parallax, wherein the amount of change of the parallax depends on a change of the aperture stop (111).

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus that generates a three-dimensional image.

2. Description of the Related Art

Recently, an image pickup element that includes a pixel unit cell configured by arranging a plurality of pixels for one microlens (a composite pixel structure) and that outputs a first image signal and a second image signal separated for each pixel of the pixel unit cell is proposed.

Japanese Patent No. 4027113 discloses a configuration in which a part of an image pickup element has a composite pixel structure and this composite pixel is used as a pixel for obtaining information (ranging). However, considering the performance of obtaining the information (ranging performance), it is effective that an entire region of the image pickup element is the composite pixel structure. Therefore, it is proposed that the composite pixel structure is adopted for an entire surface of the image pickup element and that a pixel unit in the composite pixel structure is used for both shooting an image and obtaining the information (for the ranging).

Furthermore, recently, two images (a left-sided image and a right-sided image) having a parallax can be obtained using the pixel having the composite pixel structure. Therefore, achieving a monocular three-dimensional image pickup apparatus in which it is not necessary to use binocular image pickup apparatus (a plurality of image pickup apparatuses) is considered.

When the binocular image pickup apparatus (the plurality of image pickup apparatuses) are used, a three-dimensional effect is determined by a congestion angle of two image pickup apparatuses (cameras), and therefore a stable three-dimensional effect can be obtained even when an aperture stop of an image pickup optical system is changed.

However, when the three-dimensional image is obtained by using the monocular image pickup apparatus, an image shift amount generated on an imaging plane changes in accordance with the aperture stop (a diameter of the aperture stop) of the image pickup optical system. In other words, even when an identical object is shot, the image shift amount is changed by changing the aperture stop and therefore the three-dimensional effect is changed.

BRIEF SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus, an image pickup system, and a method of controlling the image pickup apparatus that reduce a dependency of a three-dimensional effect on an aperture stop when a three-dimensional image is generated using an image pickup element that includes a pixel unit cell configured by arranging a plurality of pixels for one microlens. In addition, the present invention provides a non-transitory computer-readable storage medium that stores a computer-executable program.

An image pickup apparatus as one aspect of the present invention generates a three-dimensional image, and includes an image pickup element comprising a microlens and a pixel unit cell comprising a plurality of pixels configured to receive light from the microlens, the image pickup element being configured to output a separated first image signal and second image signal for respective pixels of the pixel unit cell, a parallax detector configured to calculate a parallax comprising an image shift amount between the first image signal and the second image signal, and an image correcting portion configured to perform a correction to change the parallax, wherein the amount of change of the parallax depends on a change in the aperture stop.

An image pickup system as another aspect of the present invention includes the image pickup apparatus and an image pickup optical system configured to form an optical image on the image pickup element.

A method of controlling an image pickup apparatus as another aspect of the present invention is a method of controlling an image pickup apparatus that generates a three-dimensional image, and includes the steps of outputting a separate first image signal and second image signal for respective pixels of a pixel unit cell of an image pickup apparatus, calculating a parallax comprising an image shift amount between the first image signal and the second image signal, and performing a correction to change the parallax, wherein the amount of change of the parallax depends on a change of the aperture stop.

A non-transitory computer-readable storage medium as another aspect of the present invention is a non-transitory computer-readable storage medium that stores a program that upon execution causes a computer to perform the method of controlling the image pickup apparatus.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments of the present invention will be described below with reference to the accompanied drawings.

First of all, referring toFIG. 1, an image pickup apparatus in Embodiment 1 of the present invention will be described.FIG. 1is a block diagram of illustrating an overall configuration of an image pickup apparatus100(an image pickup system). The image pickup apparatus100is an image pickup apparatus that generates a three-dimensional image. Reference numeral110denotes a lens (an image pickup lens) that forms an optical image (an object image) on an image pickup element114. Reference numeral111denotes an aperture stop that limits light entering the image pickup element114via the lens110. Reference numeral112denotes a shutter (SH) that blocks incident light on the image pickup element114via the lens110. Reference numeral113denotes a lens controller that performs a focus control of the lens110, an opening diameter control of the aperture stop111, and the like. A lens unit101(an image pickup optical system) is configured by the lens110, the aperture stop111, the shutter112, and the lens controller113. The image pickup apparatus100is configured by the lens unit101and an image pickup apparatus body.

Reference numeral114denotes an image pickup element such as a CMOS sensor or a CCD sensor that converts the optical image obtained via the lens unit101into an electric signal (an analog signal). The image pickup element114includes a pixel unit cell configured by arranging a plurality of pixels for one microlens, and outputs separated image signals for each pixel of the pixel unit cell (a first image signal and a second image signal). In such a configuration, a different image signal for each pixel unit cell can be outputted. Details of the structure of the image pickup element114will be described below with reference toFIGS. 2 and 3.

Reference numeral116denotes an output signal processing circuit. The output signal processing circuit116performs a processing of an OB (optical black) clamp to adjust a level of an optical black to a standard level for the analog signal outputted from the image pickup element114or the like. The output signal processing circuit116includes an analog front end (AFE) that converts this analog signal into a digital signal, a digital front end (DFE) that performs various kinds of correction processing or a digital processing such as reordering by receiving a digital output of each pixel, and the like.

Reference numeral117denotes a timing generating circuit (TG) that supplies a control signal to the image pickup element114and the output signal processing circuit116. Reference numeral118denotes a memory that temporarily stores an output of the output signal processing circuit116in order to perform a ranging calculation (focus detecting calculation) and an image processing at subsequent stages. Reference numeral119denotes a ranging processor that performs a ranging calculation using left-sided and right-sided image signals (image outputs) of the image pickup element114stored in the memory118, i.e. signals digitally processed by the output signal processing circuit116. The ranging processor119performs a correlation calculation in an output direction where the images are separated with respect to the image pickup element114so as to obtain a defocus amount of the lens110, and transfers information related to the defocus amount to a system control circuit150described below.

Reference numeral123denotes a parallax detector that performs a correlation calculation for the right-sided and left-sided image signals of the image pickup element114stored in the memory118so as to calculate an image shift amount at corresponding points (matching points) of the right-sided and left-sided images. The parallax detector123makes a well-known parallax map based on the image shift amount between the right-sided and left-sided images at each point (at each corresponding point) so as to obtain the image shift amount of the right-sided and left-sided images. In the present embodiment, the “image shift amount of the right-sided and left-sided images” or the “image shift amount of the corresponding points of the right-sided and left-sided images” is defined as a “parallax”. Therefore, the parallax detector123calculates an image shift amount of a first image signal and a second image signal as a parallax. In other words, the parallax detector123is configured to calculate the parallax including the image shift amount between the first image signal and the second image signal. In the present embodiment, the parallax detector123is configured differently (separately) from the ranging processor119, but the present embodiment is not limited to this. For example, the parallax detector123may also be provided inside the ranging processor119so as to make the parallax map in the middle of the ranging calculation.

Reference numeral120denotes an image correcting portion (a three-dimensional effect correcting portion). The image correcting portion120corrects an output of each pixel of an A image (the left-sided image) and a B image (the right-sided image) based on information of the parallax map made by the parallax detector123and lens information (a diameter of the aperture stop, a focal length, or the like) obtained at the time of shooting the image so as to correct the three-dimensional effect. In other words, the image correcting portion120is configured to perform a correction to change the parallax whose amount of change depends on a change of the aperture stop111. Preferably, the image correcting portion120performs a correction so as to reduce an amount of change of the parallax depending on a change of the aperture stop111(the diameter of the aperture stop).

Reference numeral124denotes a related-information providing portion that generates each of related information of the right-sided and left-sided images in a saving format. The related-information providing portion124specifically generates related information containing information to correct the three-dimensional effect obtained by the image correcting portion120or the parallax detector123. Reference numeral122denotes a memory control circuit. The memory control circuit122performs a control to transfer image data generated by the image processing circuit121and the related information generated by the related-information providing portion124to a memory130described below, or the like.

Reference numeral128denotes an image display including a TFT-type LCD. Reference numeral130denotes a memory that stores shot still image data or moving image data and the related information in connection with each other. Reference numeral150denotes a system control circuit that controls an entire of an image processing apparatus. The system control circuit150contains a well-known signal processing unit (a CPU) and the like. In a common image pickup apparatus, a photometric portion, a power supply controller, a switch operating portion, and the like are also included, but they are not main elements in the present embodiment and therefore descriptions of the elements are omitted.

Next, referring toFIG. 2, an image pickup element114of the present embodiment will be described.FIG. 2is a conceptual diagram (a top view) of the pixel unit cell in the image pickup element114(an image pickup element having a composite pixel structure). Reference numeral1denotes a pixel unit cell of the image pickup element114. Each of reference numerals1aand1bdenotes a pixel that includes a structure of a well-known image pickup pixel containing a photoelectric conversion element. Each of signals (the first image signal and the second image signal) from the pixels1aand1bcan be outputted independently of each other. A well-known color filter having an identical color is arranged on each of upper surfaces of the pixels1aand1b.

Reference numeral2denotes a microlens. The plurality of pixels1aand1bthat are arranged under the one microlens2are treated as a pixel unit cell that obtains incident light via the identical microlens2. The pixels1aand1bcan obtain a plurality of different images that are obtained by pupil division according to their arrangements. Thus, the image pickup element114includes the micro lens2and the pixel unit cell1including the plurality of pixels1aand1barranged to receive the light from the microlens2, and the image pickup element114is configured to output separated first image and second image signals for respective pixels of the pixel unit cell1. In the present embodiment, as the composite pixel structure, two separated pixels are provided inside the pixel unit cell1, but the present embodiment is not limited to this. As the composite pixel structure, for example, a composite pixel configured by dividing the pixel unit cell into four, five, or nine pixels may also be adopted. In this case, an output of each of the divided pixels of the pixel unit cell1is added so as to obtain at least two image outputs (image signals) separated for the left eye and the right eye, and thus the similar effect can be obtained.

As a method of separating the image signal into the left-sided and right-sided images, for example, there is a method of adding two pixels at the upper left and the lower left and also adding two pixels at the upper right and the lower right when the pixel unit cell is configured by four pixels having two pixels vertically and horizontally. Alternatively, only a pair of two pixels separated to right and left (for example, two pixels at the upper right and the upper left) may be used to obtain outputs for the right and the left. When the pixel unit cell is configured by five pixels having two pixels vertically and horizontally and one pixel at the center, except the one pixel at the center, the outputs of the pixels separated to right and left can be obtained by adding the outputs similarly to the case of the configuration of the four pixels. Also in the case of the divided pixels of nine pixels where three pixels are arranged vertically and horizontally, the three vertical pixels at the left and the three vertical pixels at the right side are added and on the other hand the three vertical pixels at the center are excluded, and thus the outputs separated to the right and left images can be obtained similarly to the above case.

Next, referring toFIG. 3, a pixel arrangement of the image pickup element114will be described.FIG. 3is a diagram of illustrating one example of the pixel arrangement of the image pickup element114. Reference numeral201denotes an optical black (OB) pixel region that is a standard in the image processing. Reference numeral202denotes an effective pixel region in which the plurality of pixel unit cells1described with reference toFIG. 2are arranged so as to obtain the image signals. Reference numeral203denotes a recording pixel region that is basically used as a recording pixel. The recording pixel region203is a basic recording pixel region in which a center of the effective pixel region202and a center of the recording pixel region203coincide with each other. The recording pixel region203is set so as to be narrower than the effective pixel region202. A region (recording pixel margin regions M1and M2) between an edge portion of the effective pixel region202and an edge portion of the recording pixel region203may be used when controlling the correction of the three-dimensional effect described below.

When image data are stored, it is preferred that data of the recording pixel region203are only stored so as to store image data for which the adjustment of the three-dimensional effect has been performed. However, generally, it is preferred that the recording pixel region203is stored in a multi-picture format (MPO) for which a post-processing is unnecessary, JPEG, a moving picture file such as an AVI file, or the like. When the recording pixel region203is stored, in addition to storing only the recording pixel region203for all the files, for example the effective pixel region202or all pixel regions may be stored when the file is stored in a RAW format. In this case, as related information, information related to the correction of the three-dimensional effect such as aperture stop information or focal length information of an image pickup optical system in adjusting the three-dimensional effect are given to be stored. Giving the related information will be described below with reference toFIG. 11and the like.

Next, referring toFIG. 4, the shooting operation of the image pickup apparatus100in the present embodiment will be described.FIG. 4is a flowchart of illustrating the shooting operation of the image pickup apparatus100. This flow starts when a shooting start switch SW such as a live-view start switch or a shooting preparation switch (not shown) is turned on. Each step ofFIG. 4is performed based on an instruction of the system control circuit150.

First of all, in Step S301, a preliminary shooting that is required for a ranging operation and a photometric operation is performed before an actual shooting. The preliminary shooting is performed by the ranging processor119calculating a current focus position (defocus amount) based on the output signals (A image and B image) divided by the pixel unit cell1described with reference toFIG. 2. Subsequently, in Step S302, the system control circuit150obtains an in-focus position based on the defocus amount calculated in Step S301. The lens controller113performs a focus operation in which the lens110is driven to an in-focus target. Then, in Step S303, the photometric operation in which exposure information of an object are obtained is performed based on the information obtained in Step S301. Subsequently, in Step S304, a shooting condition such as an appropriate shutter speed at the time of the actual shooting is determined based on the exposure information of the object obtained in Step S303. The shooting condition (information related to the lens unit101) determined in Step S304, specifically an aperture stop, a focal length, lens unique information, or the like is also stored in a memory (not shown).

Next, in Step S305, it is confirmed whether the shooting switch SW2such as a still image recording switch or a moving image recording switch is turned on. When the shooting switch SW2is turned on, the flow proceeds to Step S307. On the other hand, the shooting switch SW2is not turned on, the flow proceeds to Step S306. In Step S306, it is confirmed whether the shooting start switch SW that is turned on at the time of starting the shooting is turned off. When the shooting start switch SW is turned off, the shooting operation is finished. On the other hand, when the shooting start switch SW remains in the on-state, the flow proceeds to Step S301and the preliminary shooting is performed again.

In Step S307, in accordance with the shooting condition determined in Step S304, the shooting operation such as a shutter control, an accumulation or readout operation of the image pickup element114is performed. In some cases such as the moving image shooting, a time to mechanically block the light for each frame cannot be ensured. Therefore, the image pickup element114may be driven (a slit rolling shutter drive may be performed) so that the operation is performed while the shutter112is always open. The read output signal for each of the left-sided and right-sided images is temporarily stored in the memory118.

Subsequently, in Step S308, the parallax detector123detects an image shift amount (a parallax) of corresponding points of the left-sided image (the A image) and the right-sided image (the B image) so as to make a parallax map. The detection of the parallax (the detection of the image shift amount) is performed based on an image that is read from the image pickup element114in Step S307and that is temporarily stored in the memory118via the output signal processing circuit116. The detection of the image shift amount in Step S308will be described below with reference toFIGS. 5A and 5B.

Subsequently, in Step S351, the ranging processor119performs a correlation calculation of the image shot in Step S307and the like so as to obtain the defocus amount, and this defocus amount is stored in a memory (not shown) as ranging data for the shooting of the next frame. In the present embodiment, since the ranging processor119and the parallax detector123are separately configured, Steps S308and S351are described as separated operations. However, since the detection result of the parallax detector123can also be used for the ranging operation, the parallax detection may be performed as a part of information of the ranging operation, and the detected parallax may also be used for the ranging information.

Next, in Step S309, the image correcting portion120obtains an appropriate correction amount for each of the pixel unit based on the parallax map made in Step S308and the shooting condition (information related to the lens unit101) stored in Step S304. Then, the image correcting portion120performs the processing of correcting the three-dimensional effect so as to compensate the shortfall of the parallax for the shot A image and B image based on this correction amount. Details of the processing of correcting the three-dimensional effect will be described below with reference toFIGS. 6A and 6B through 10A to 10H.

Subsequently, in Step S310, the image processing circuit121performs an image processing such as a color conversion for each of the A image and the B image obtained by the correction processing of the three-dimensional effect in Step S309. The image processing circuit121performs a development to JPEG or the like, a cutout of an image size, and a compression processing by this image processing, and stores the A image and the B image in a multi-picture format. In other words, according to this image processing, images obtained by correcting the three-dimensional effect of the A image and the B image (after adjusting the three-dimensional effect) are generated.

Subsequently, in Step S311, the related-information providing portion124reads image information containing information related to the correction processing of the three-dimensional effect and shooting related information from the correction parameter setting portion115or the like. Then, the related-information providing portion124generates related information and makes data formed by giving the related information to the image generated in Step S310.

Subsequently, in Step S312, the memory control circuit122transfers, to the memory130, the data obtained by giving the related information generated in Step S311to the image-processed data of Step S310, and then records the data in the memory130. When the data are recorded in a RAW image, the image recording in Step S310is performed in a state where the correction of the three-dimensional effect is not performed (in an unprocessed state), and parallax map information or image pickup lens related information are stored as the related information in Step S311. According to this configuration, they can also be treated as useful information when the correction of the three-dimensional effect is performed by the display unit after the image is stored. The present embodiment is not limited to the RAW image, and the related information may also be given when an image in the JPEG format is recorded.

Next, in Step S313, the system control circuit150determines whether the shooting start switch SW is turned off. When the shooting start switch SW is not turned off, the shooting is continued and the flow returns to Step S302. Then, the system control circuit150obtains an in-focus position based on the defocus amount calculated by the ranging processor119in last Step S351. Then, the lens controller113performs a focus operation in which the lens110is driven toward the in-focus target. On the other hand, in Step S313, when the shooting start switch SW is turned off, the shooting operation of this flow is finished, and the mode of the image pickup apparatus returns to be in a standby mode.

Next, referring toFIGS. 5A and 5B, the detection of the image shift amount (making the parallax map) in Step S308ofFIG. 4will be described.FIGS. 5A and 5Bare diagrams of describing the detection of the image shift amount (making the parallax map). The parallax map is data to obtain a parallax of each of the corresponding points based on a displacement (the image shift amount) of the corresponding points on the image pickup element114that is generated in accordance with a difference of light beams caused by the divided pupils between the A image (the left-sided image) and the B image (the right-sided image). Using the data, the defocus amount of the image pickup optical system, a distance of the object, and the like can be estimated. As a method of making the parallax map, a “stereo matching method” that calculates area correlation value of the left-sided and right-sided images (the A and B images) for each of partial images (an image of each block that is obtained by dividing an entire image into predetermined regions as blocks) or the like is generally used.

There are various kinds of methods as the stereo matching method, and in the present embodiment, a case where a simple template matching method using a difference and a sum is used to match a feature point will be described.FIGS. 5A and 5Billustrate the A image (the left-sided image) and the B image (the right-sided image), respectively.FIG. 5Aillustrates each pixel point in the image, andFIG. 5Bomits unnecessary pixel points for convenience of description. As illustrated inFIGS. 5A and 5B, a partial image of the A image (the left-sided image) which corresponds to a left-sided point of view position of the shooting image is referred to as a standard image401and a partial image of the B image (the right-sided image) which corresponds to a right-sided point of view position of the shooting image is referred to as a reference image402so as to perform the template matching. In the present embodiment, since the inside of the identical pixel unit cell1is divided into the left-sided and right-sided images (the A and B images), basically, there is only a one-dimensional shift in an x direction. Therefore, in the present embodiment, a one-dimensional correlation will be described, but the embodiment can also be applied to a two-dimensional shift.

First of all, a specific point in the standard image401is selected as an attention pixel (x, y)404. Then, a region with a predetermined size around the attention pixel404is cut as a template403. Next, considering roughly a moving amount of a point of view, a corresponding point searching region407in the reference image402is arbitrarily set. Then, a reference pixel405that exists inside the corresponding point searching region407is sequentially selected, and also the similarity of a window region406around the reference405having the same size as that of the template403with reference to the template403inside the standard image401is calculated. The calculation of the similarity of the attention pixel404is sequentially performed while the window region406moves in the corresponding point searching region407.

The similarity in the vicinity of a candidate of the corresponding point can be calculated by the sum of squares of a difference of pixel values as represented by the following Expression (1).

In Expression (1), symbol (x,y) is a position of the attention pixel404for the left-sided image (the A image), and symbol I(x, y) is an image signal of the attention pixel404for the left-sided image (the A image). Symbol (xb+i,y) is a position of the reference pixel405for the right-sided image (the B image), and symbol I0(xb+i,y) is an image signal of the reference pixel405for the right-sided image (the B image). Symbol K is the corresponding point searching region407. Symbol JSDD is called a “residual sum of squares”, which indicates zero when the pixel values precisely match with each other. Accordingly, the reference pixel405at which the similarity JSDD corresponding to the attention pixel404is minimized is determined as a corresponding point (xmin,y). Then, a shift between the attention pixel (x,y)404used for the calculation of the similarity and the determined corresponding point (xmin,y) in a horizontal direction, i.e. the shift X=x−xmin, is obtained.

Using the template matching process as described above, corresponding information (matching information) of the B image (the right-sided image) with reference to the A image (the left-sided image) are obtained, and the difference X (=x−xmin) that is a shift (a displacement) between a coordinate of the attention pixel and a coordinate of the corresponding point in the horizontal direction is obtained as an image shift amount of the left-sided and right-sided images. This calculation is performed for each pixel unit cell and the image shift amount is arranged on a position of each pixel cell so as to make the parallax map. The present embodiment is not limited to the calculation of the image shift amount using the method described above, and another method may also be used if it is a method capable of calculating the image shift amount for each pixel in images including two points of view for which the matching process is to be performed.

Next, referring toFIGS. 6A and 6B through 10A to 10H, the calculation of a parameter related to the correction of the three-dimensional effect in the present embodiment will be described. The correction of the three-dimensional effect is performed to obtain an image which is multiplied by a coefficient depending on an aperture stop so that the image shift amount (the parallax) described with reference toFIGS. 5A and 5Bis changed to be an image shift amount (a parallax) corresponding to a diameter of the aperture stop that is a reference.

FIGS. 6A and 6Bare diagrams of describing the shooting of the three-dimensional image (depending on the aperture stop) of the image pickup element114, which illustrate a relationship between a focus position determined by the object and the image pickup optical system and an imaging state on the imaging plane. InFIGS. 6A and 6B, at a focal length p, states of focusing on a main object indicated by a circle are illustrated when the diameter of the aperture stop is d1inFIG. 6Aand is d2inFIG. 6B. The light beam from the object for the pixels1aand1bis divided into a light beam passing through a divided pupil corresponding to an a-pixel in the image pickup optical system so as to enter the a-pixel and a light beam passing through a divided pupil corresponding to a b-pixel in the image pickup optical system so as to enter the b-pixel. Since these two light beams enter the pixels from an identical point on the object, they pass through an identical microlens2so as to reach one point on the image pickup element114. Accordingly, the A image (the left-side image) and the B image (the right-sided image) substantially coincide with each other on the image pickup element114. InFIGS. 6A and 6B, both the A image (the left-sided image) and the B image (the right-sided image) are formed on a pixel unit cell P7.

On the other hand, with respect to an indent light beam from an object at a far distance indicated by a triangle, as indicated by a solid line, an imaging point exists at a side of the image pickup optical system. Therefore, a state where the imaging plane exists behind the focus position, a so-called rear focus state, is obtained. In the rear focus state, positions that the light beams of the a-pixel and the b-pixel reaches are shifted from each other, and therefore the shift (the parallax) of the corresponding points is generated for the A image (the left-sided image) and the B image (the right-sided image). InFIG. 6A, the A image (the left-sided image) is formed on P9, and the B image (the right-sided image) is formed on P5. InFIG. 6B, the A image (the left-sided image) is formed on P8, and the B image (the right-sided image) is formed on P6.

FIG. 7is a diagram of a relationship between the defocus amount and the image shift amount (the parallax) of the left-sided and right-sided images, which illustrates the relationship in the state ofFIG. 6A. InFIG. 7, a lateral axis indicates the defocus amount, and a vertical axis indicates the image shift amount. As illustrated inFIG. 7, the defocus amount and the image shift amount of the left-sided and the right-sided images are proportional to each other. Since the defocus amount is proportional to an inverse of a distance, the distance can be calculated by obtaining the defocus amount. When the defocus amount is defined as D and the image shift amount of the left-sided and the right-sided images is defined as X, the following Expression (2) is satisfied.
D=K×X+H(2)

In Expression (2), H is a hyperfocal offset, and K is a coefficient (an inclination coefficient). The coefficient K is calculated as represented by the following Expression (3), using a distance B (B1inFIG. 6Aand B2inFIG. 6B) between centroids of the divided pupils of the left-sided and the right-sided images depending on the aperture stop111of the lens unit101and the focal length p between the lens110and the image pickup element114.
K=B/p(3)

FIG. 8is a diagram of a relationship between the defocus amount D and the image shift amount (the parallax) of the left-sided and the right-sided images in each of the states ofFIGS. 6A and 6Bbased on the relation ofFIG. 7. In the present embodiment, the diameter d1of the aperture stop inFIG. 6Ais set as a reference condition that is a target of correcting the three-dimensional effect, and the diameter d1of the aperture stop is a maximum diameter of the aperture stop (an open state of the aperture stop) that is settable by the image pickup optical system (the lens unit101) that is used at the time of shooting an image. The reason to set the aperture stop to the open state is that it is a condition that obtains a maximum image shift amount (a maximum parallax) settable by the mounted image pickup optical system (the lens unit101). In the present embodiment, the image correcting portion120performs the correction so that the parallax is a parallax obtained in the open state (full-open state) of the aperture stop111or the parallax comes close to the parallax obtained in the open state.

As illustrated inFIG. 8, the state of the diameter d1of the aperture stop inFIG. 6Ahas an inclination coefficient K1and the diameter d2of the aperture stop inFIG. 6Bhas an inclination coefficient K2. Thus, at the in-focus point indicated by the circle, independently of the diameter d1or d2of the aperture stop, the image shift amount (the parallax) of the left-sided and the right-sided images is not generated. On the other hand, in the rear focus state (indicated by the triangle) or a front focus state (indicated by a square), the image shift amount (the parallax) of the left-sided and the right-sided images is different depending on the diameter d1or d2of the aperture stop.

Specifically, the relation of image shift amounts X1and X2(parallaxes) of the left-sided and right-sided images in the rear focus state (indicated by the triangle) ofFIGS. 6A and 6Bis X1>X2. In this case, a coefficient Z of correcting the three-dimensional effect to adjust the inclination coefficient K2to the inclination coefficient K1is set to be the following Expression (4).
Z=X1/X2=B1/B2=K1/K2  (4)

In other words, the coefficient Z of correcting the three-dimensional effect that is a correction magnification (a parallax magnification) of the image shift amount (the parallax) depending on the aperture stop111based on the inclination coefficient K1in the diameter d1as a reference and the inclination coefficient K2in the diameter d2of the aperture stop when actually shooting an image. Using the coefficient Z of correcting the three-dimensional effect (the parallax magnification), the image shift amount (the parallax) of the left-sided and right-sided images can be obtained independently of the diameter of the aperture stop.

FIG. 9is a diagram of a relation between the diameter of the aperture stop (an F-number) and the inclination coefficient K, which illustrates a case where the focal length p is 50 mm. When an image shot by a diameter of the aperture stop of F5.6 is corrected to an image shift amount (a parallax) shot on condition that corresponds to F1.8, the correction may be performed using the coefficient Z of correcting the three-dimensional effect (Z=1.80/0.7=2.58). The inclination coefficient K can be obtained by previously storing data such as the diameter of the aperture stop, the focal length, and a light intensity distribution of the image pickup element and calculating it. Actually, however, since a vignetting caused by an aberration or a structure of the lens or the like is different in accordance with a type of the lens to be used, it is preferred that the data are previously stored for each lens (for each aperture stop). The present embodiment describes the case of the focal length p, but an actual lens has a structure that combines a plurality of lenses. Therefore, calculating the focal length p as an exit pupil distance, a higher accuracy result can be obtained.

The relation of the image shift amount X (the parallax) of the left-sided and right-side images measured inFIGS. 5A and 5B, the coefficient Z of correcting the three-dimensional effect described with reference toFIG. 6A, 6BtoFIG. 9, and an image shift amount X′ of the correction target (a target parallax) is represented as the following Expression (5).
X′=X×Z(5)

In other words, the three-dimensional effect is corrected by correcting the image shift amount (the parallax) of each pixel to be magnified by Z times in a horizontal direction so that the image shift amount X (the parallax) of the left-sided and right-sided images are the image shift amount X′ (the parallax) of the correction target. As described above, in the present embodiment, the parameter of correcting the three-dimensional effect (the coefficient Z of correcting the three-dimensional effect) is obtained based on the aperture stop information of the lens unit101(the image pickup optical system).

FIGS. 10A to 10Hare schematic diagrams of correcting the three-dimensional effect in the present embodiment.FIG. 10Ais a top view of illustrating the relation of positions of objects for the image pickup apparatus. With reference to a main object (indicated by a circle), an object at a far distance (indicated by a triangle) exists, and the image pickup optical system is controlled so as to focus on the main object.FIG. 10Billustrates a case where the objects are viewed from the image pickup apparatus, and the image is recorded as illustrated in the drawing when the 2D shooting (the two-dimensional shooting) is performed.

FIGS. 10C to 10Eare diagrams of schematically illustrating the A image (the left-sided image) and the B image (the right-sided image) of the image pickup element on condition that each of the objects is on an identical line.FIG. 10Cillustrates a state before correcting the three-dimensional effect for the diameter d1of the aperture stop that is a reference (a large diameter of the aperture stop),FIG. 10Dillustrates a state before correcting the three-dimensional effect for the diameter d2of the aperture stop (a small diameter of the aperture stop), andFIG. 10Eillustrates a state after correcting the three-dimensional effect for the diameter d2of the aperture stop (the small diameter of the aperture stop).FIG. 10Cdoes not indicate an image shift of the main object (indicated by circles) and indicates an image shift amount X1of the left-sided and right-sided images of the object at a far distance (indicated by triangles). On the other hand,FIG. 10Ddoes not indicate the image shift of the main object (indicated by circles) and indicates an image shift amount X2of the left-sided and right-sided images of the object at the far distance (indicated by triangles) which is smaller than the image shift amount X1. In this case, there is a possibility that the object at the far distance (indicated by triangles) inFIG. 10Cis a blurred image compared to the object at the far distance (indicated by triangles) inFIG. 10D.FIG. 10Eillustrates a state which is obtained by performing the correction of the three-dimensional effect forFIG. 10D.FIG. 10Eis a state where the image shift amount X of the left-sided and right-sided images is multiplied by the coefficient Z of correcting the three-dimensional effect depending on the diameter of the aperture stop for the corresponding object of each of the A image (the left-sided image) and the B image (the right-sided image), using the method of illustrating inFIGS. 5A and 5B to 9.

FIGS. 10F to 10Hare schematic diagrams (top views) of illustrating the three-dimensional effect that is visible by an observer when the shot image for which the correction of the three-dimensional effect has been performed is displayed as illustrated inFIGS. 10C to 10E. InFIG. 10F, there is no image shift of the main object, and therefore the main object appears to exist on a surface of the display unit that is a reference plane. On the other hand, with respect to the object at the far distance, an intersection point (a cross point) existing on an extended line of the points of view of the left and right eyes that are caused by the image shift amount (the parallax) of the left-sided and right-sided images, depending on the diameter d1of the aperture stop, is generated at the rear side compared to the surface of the display unit, and therefore a deep three-dimensional effect is obtained. InFIG. 10G, with respect to the object at the far distance, the cross point depending on the diameter d2of the aperture stop is generated at the rear side compared to the surface of the display unit, but the cross point exists at a position near the surface of the display unit compared toFIG. 10F. Therefore, the three-dimensional effect ofFIG. 10Gis lower than that ofFIG. 10F.

On the other hand, inFIG. 10H, the correction of the three-dimensional effect is performed for the image on the surface of the display unit. Therefore, the cross point of the object at the far distance is substantially the same as that at the time of shooting an image on condition of the diameter d1of the aperture stop that is a reference. Accordingly, even when the shooting is performed in the diameter d2of the aperture stop, a three-dimensional effect similar to that in the larger diameter d1of the aperture stop can be obtained. Since a depth of field in the small diameter d2of the aperture stop is deeper than that in the reference diameter d1of the aperture stop, a more focused depth effect (a more focused three-dimensional effect) can be obtained. The embodiment describes the object at the far distance, and additionally, the similar correction is also performed for an object at a near distance to be able to obtain an embossed effect (three-dimensional effect) similar to the three-dimensional effect generated in a desired diameter of the aperture stop.

FIGS. 11A and 11Bare schematic diagrams of three-dimensional image data containing the image data and the related information described with reference to Steps S311and S312inFIG. 4. InFIG. 11A, reference numeral601denotes three-dimensional image data that are obtained as a pair of three-dimensional images. Left-sided image data602L and right-sided image data602R in which the image shift amount has been corrected are stored in the three-dimensional image data. Reference numeral603denotes related information of the three-dimensional image data601. The related information603includes information to correct the three-dimensional effect determined based on the image shift amount described above (for example, the diameter d of the aperture stop, or the coefficient Z of correcting the three-dimensional effect). The related information603may also include other information related to shooting an image such as a shutter speed, a frame rate, or shooting lens information (a focal length, an exit pupil distance, an F-number, or an open F-number). In the embodiment, the related information603are indicated as one region including the left-sided image data602L and the right-sided image data602R, and alternatively the related information603may be stored separately in different regions of a left-sided image dedicated region and a right-sided image dedicated region.

FIG. 11Billustrates a case where an image file in a plurality of formats are related to each other. InFIG. 11B, reference numeral611denotes data (three-dimensional image data) of images (left-sided and right-sided images) that are shot as a pair of three-dimensional images. Reference numerals612and614are image data in which data of the shot images (the left-sided and right-sided images) are stored in formats different from each other. For example, reference numeral612denotes image data stored in a JPEG format as a first format, and reference numeral614denotes image data stored in a RAW format as a second format, and thus the image data612and614are image data stored in formats different from each other. Each of the image data612and614includes the left-sided image data612L and614L and the right-sided image data612R and614R. An image size of the image data stored in each format does not need to be the same, and for example the image data612may be image data of a recording pixel region where the image shift amount has been corrected, and the image data614may be image data of an effective pixel region where the image shift amount has not been corrected.

Reference numeral613denotes related information of the three-dimensional image data611that are stored in the first format, and reference numeral615denotes related information of the three-dimensional image data611that are stored in the second format. The related information613and615include information for correcting the three-dimensional effect determined based on the image shift amount information described above (for example, the coefficient Z of correcting the three-dimensional effect). Similarly to the related information603inFIG. 11A, other information related to shooting the image such as the shutter speed, the frame rate, or the shooting lens information (the diameter of the aperture stop, i.e. the F-number, the focal length, the exit pupil distance, or the open F-number) may be included. When information in different formats of an identical image are stored, the related information for each format do not need to be identical. For example, also with respect to the information to correct the three dimensional effect, the coefficient Z of correcting the three-dimensional effect may be only stored for the first format, and both the diameter of the aperture stop (the F-number) and the coefficient Z of correcting the three-dimensional effect may be stored for the second format.

InFIGS. 11A and 11B, the image shift amount (the parallax) of the left-sided and right-sided images is corrected in the image data (the left-sided and right-sided images) shot as the pair of three-dimensional images, but the present embodiment is not limited to this. For example, at the time of recording the image, the information related to the correction of the three-dimensional effect such as the shooting lens information (the focal length, the exit pupil distance, the F-number, the open F-number, or the like) or the coefficient Z of correcting the three-dimensional effect are given to the three-dimensional data, and thus a desired image shift amount (parallax) of the information at the time of shooting the image can also be corrected at the side of the display unit. Therefore, the image shift amount of the left-sided and right-sided images does not need to be corrected for the image data (the left-sided and right-sided images) that are to be stored. In other words, there may be a case where the corrected image is not made in the operation of correcting the three-dimensional effect in Step S310ofFIG. 4.

In the present embodiment, the diameter d1of the aperture stop that is a reference of the three-dimensional effect (the correction target) is described as the open aperture stop in the image pickup optical system that is used to shoot an image, but the present embodiment is not limited to this. The image pickup apparatus100, for example, may have a mode in which a desired F-number can be set as a reference. The diameter d1of the aperture stop may also be corrected by using theoretical values of the focal length and the diameter of the aperture stop, or a predetermined diameter of an aperture stop in another lens as a reference, regardless of the lens that is used to shoot the image. According to this method, an image shift amount (a parallax) more than that in the open aperture stop of the image pickup optical system can also be generated.

As described above, according to the present embodiment, in an image pickup apparatus using the divided pixels so as to perform the ranging operation and obtain the three-dimensional image, an image shift by an image shift amount corresponding to an object at a predetermined distance can be performed. Therefore, a pseudo image shift (an artificial image shift) is given to the image object (the in-focus position), and thus the three-dimensional effect can be given to the main object.

Next, an image pickup apparatus in Embodiment 2 of the present invention will be described. In Embodiment 1, the single-shot operation is mainly described. In this regard, when a continuous shooting operation (a continuous shooting, a moving image, or the like) is performed, considering occurrence of a feeling of strangeness for the change of the three-dimensional effect from a first frame, it is preferred that the diameter of the aperture stop for the first frame is set to the reference diameter d1of the aperture stop (the target of correcting the three-dimensional effect) so as to correct the three-dimensional effect for a second frame and its subsequent frames.

Referring toFIG. 12, a sequence of setting the diameter of the aperture stop at the time of shooting the first frame (at the time of starting the shooting) as a reference of correcting the three-dimensional effect will be described.FIG. 12is a flowchart of illustrating the shooting operation of an image pickup apparatus in the present embodiment, which illustrates a simple shooting sequence at the time of the continuous shooting operation. The flow ofFIG. 12starts when the shooting start switch SW is turned on in a state where the continuous shooting operation such as the continuous shooting or the moving image is set by a continuous shooting mode setting switch (not shown) of the image pickup apparatus. Each step ofFIG. 12is performed based on an instruction of the system control circuit150.

First of all, in Step S901, a first frame is shot. The shooting operation is the same as the shooting sequence described with reference toFIG. 4. However, since it is not necessary to correct the three-dimensional effect in Step S901, Step S309ofFIG. 4is not performed in the present embodiment. Subsequently, in Step S902, the diameter of the aperture stop used in Step S901is stored as the reference diameter d1of the aperture stop that is a correction target. Then, in Step S903, the system control circuit150determines whether the shooting start switch SW maintains the on-state or is turned off. When the shooting start switch SW is set to on, the flow proceeds to Step S904. On the other hand, when the shooting start switch SW is set to off, this sequence is finished and the image pickup apparatus returns to a standby state.

In Step S904, the system control circuit150, similarly to the case of the first frame, performs the shooting sequence described with reference toFIG. 4. In this case, as the reference diameter d1of correcting the three-dimensional effect, the diameter of the apertures stop stored in Step S902(the diameter of the aperture stop for the first frame) is used. When the shooting operation is finished, the flow returns to Step S903. The first frame may also be used as a preliminary shooting before an actual shooting.

According to the present embodiment, the diameter of the aperture stop initially set at the time of the continuous shooting operation is set as a reference of the three-dimensional effect, and the correction of the three-dimensional effect is performed by using the diameter of the aperture stop. In other words, the image correcting portion120performs the correction so that the parallax becomes the parallax obtained when starting the shooting or the parallax comes close to the parallax obtained when starting the shooting. Therefore, the feeling of strangeness for the change of the three-dimensional effect can be reduced.

As the sequences illustrated inFIGS. 4 and 12, the diameter d1of the aperture stop as a reference (a correction target) may also be set so as not to be changed in the continuous shooting (a continuous shooting, a moving image, or the like). In this case, in the continuous shooting, the image correcting portion120performs the correction using a parameter set at the initial time of the shooting until the continuous shooting is finished. According to this configuration, a stable three-dimensional effect can be obtained even when the diameter of the aperture stop is changed while the continuous shooting is performed.

Next, an image pickup apparatus in Embodiment 3 of the present invention will be described. In Embodiment 1, the coefficient (the correction coefficient) that is previously stored or that is calculated is used. However, in view of the efficiency of the memory, it is not preferable that the plurality of data including data related to the interchangeable lens are previously stored. In addition, when a lens which is different in a type from the lens stored in the memory is mounted on the image pickup apparatus body, there is a possibility that an unexpected problem occurs.

The present embodiment obtains the image shift amounts (the parallaxes) X1and X2of the left-sided and right-sided images for an object at a predetermined distance so as to calculate the coefficient Z of correcting the three-dimensional effect during the operation of the preliminary shooting before the actual shooting, and thus a value of correcting the three-dimensional effect in which it is not necessary to previously store individual lens data is calculated. Specifically, the image pickup apparatus100of the present embodiment has an operation mode of obtaining the parameter (the coefficient Z of correcting the three-dimensional effect or the like) used for the correction by the image correcting portion120before the actual shooting.

FIG. 13is a flowchart of illustrating the shooting operation of the image pickup apparatus in the present embodiment, which illustrates a method of calculating the correction coefficient. This flow may be an operation of obtaining the correction value performed by a switch (not shown) before the actual shooting, or may be operated as a part of the operation of the preliminary shooting of Step S301inFIG. 4. Each step ofFIG. 13is performed based on an instruction of the system control circuit150.

First of all, in Step S1101, the aperture stop111in the lens unit101is set to the reference diameter d1(F-number) of the aperture stop (setting a reference aperture stop). In order to obtain a strong three-dimensional effect, it is preferred that the aperture stop is set to the open aperture stop of the mounted lens, but the present embodiment is not limited to this. For example, the reference diameter1of the aperture stop may also be set to a predetermined diameter of the aperture stop desired by a user. Subsequently, in Step S1102, a preliminary shooting operation is performed for a ranging and a photometry.

Next, in Step S1103, the system control circuit150obtains an in-focus position based on the defocus amount calculated in Step S1102. Then, the lens controller113performs a focus operation that drives the lens110toward an in-focus target. Subsequently, in Step S1104, a photometric operation is performed to obtain exposure information of an object based on the information obtained in Step S1102. Then, in Step S1105, according to a shooting condition set in Steps S1101to S1104, a shutter control and an accumulation and a readout operation of the image pickup element114(a first shooting) are performed.

Next, in Step S1106, the parallax detector123makes the parallax map between the left-sided image (the A image) and the right-sided image (the B image), and extracts (determines) a feature point in this time. The parallax map is read from the image pickup element114in Step S1105, which is made based on an image temporarily stored in the memory118via the output signal processing circuit116. As a method of determining the feature point, for example, there is a method of recognizing a region where a predetermined image shift amount is generated between the left-sided and right-sided images as a feature point based on a shape, a color, an output value, or the like. Alternatively, a distance from a main object (a region where there is no image shift amount of the left-sided and right-sided images) or the like may be used to set the feature point. However, since the region is set to detect the image shift amount, the region which is not at the in-focus position is set as the feature point.

Next, in Step S1107, the parallax detector123detects a shift (a displacement) between the left-sided and right-sided images at the feature point extracted in Step S1106as an image shift amount X1(a first parallax). As a method of detecting the parallax, for example, the method described with reference toFIGS. 5A and 5Bis used.

Subsequently, in Step S1108, the diameter d1of the aperture stop set in Step S1101is changed to the diameter d2of the aperture stop (the F-number) different from the diameter d1of the aperture stop. The diameter d2of the aperture stop set in Step S1108is a diameter of the aperture stop (an F-number) which is used at the time of shooting an image. Then, in Step S1109, according to the shooting condition set in Steps S1101to S1104, a shutter control and an accumulation and a readout operation of the image pickup element114(a second shooting) are performed. In this case, since the diameter d1of the aperture stop is changed to the diameter d2of the aperture stop, the settings of the shutter control and the accumulation time are changed to be an appropriate exposure.

Next, in Step S1110, the parallax detector123makes the parallax map between the left-sided image (the A image) and the right-sided image (the B image), and extracts (determines) a feature point in this time. The parallax map is read from the image pickup element114in Step S1109, which is made based on an image temporarily stored in the memory118via the output signal processing circuit116. Subsequently, in Step S1111, the parallax detector123, similarly to the case of Step S1107, detects a shift between the left-sided and right-sided images at the feature point extracted in Step S1110as an image shift amount X2(a second parallax).

Next, in Step S1112, the system control circuit150compares the image shift amount X1(the first parallax) detected in Step S1107with the image shift amount X2(the second parallax) detected in Step S1111so as to calculate the coefficient Z of correcting the three-dimensional effect. Then, in Step S1113, the coefficient Z of correcting the three-dimensional effect calculated in Step S1112is stored in the correction parameter setting portion115or the like, and then this flow is finished. The flow ofFIG. 13only compares the two diameters of the aperture stop so as to obtain the coefficient of correcting the three-dimensional effect, but the present embodiment is not limited to this. The present embodiment may also change the diameter of the aperture stop to repeat the flow ofFIG. 13more than once so as to obtain correction coefficients of a plurality of diameters of the aperture stop.

According to the present embodiment, the data is obtained while the diameter of the aperture stop in the mounted lens changes so as to calculate the coefficient of correcting the three-dimensional effect, and thus a storage region of the data can be reduced and an efficient operation of correcting the three-dimensional effect can be performed.

According to each of the embodiments described above, when a three-dimensional image is generated by an image pickup element having a pixel unit cell configured by arranging a plurality of pixels for one microlens, an image pickup apparatus, an image pickup system, and a method of controlling the image pickup apparatus that reduce dependency of an aperture stop for a three-dimensional effect can be provided. Furthermore, a program that describes a procedure of the method of controlling the image pickup apparatus and that is executable by a computer can be provided. In addition, a non-transitory computer-readable storage medium that stores the program that causes the computer to execute the method of controlling the image pickup apparatus can be provide.

This application claims the benefit of Japanese Patent Application No. 2012-176657, filed on Aug. 9, 2012, which is hereby incorporated by reference herein in its entirety.