Processor

A processor includes: an imaging control unit which causes an imaging unit for capturing an image with changing a focused object distance, to change the focused object distance with a first change width in the case of a first focused object distance, and to change the focused object distance with a second change width that is smaller than the first change width in the case of a second focused object distance that is shorter than the first focused object distance; and an image processing unit which extracts focused object image data from captured image data for each of the focused object distances, and generates image data by shifting the extracted object image data by parallax amounts corresponding to the respective focused object distances, and by synthesizing the extracted object image data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2010-147766, filed on Jun. 29, 2010, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein relate to a processor which generates a parallax image for a three-dimensional image.

BACKGROUND

A method for generating a three-dimensional image using parallax images, which includes of an image for a left eye (hereafter “left eye image”) and an image for a right eye (hereafter “right eye image”), which have parallax from each other, is known. Japanese Patent Application Laid-Open Nos. H5-7373, H9-116932, 2002-125246 and 2007-129436 disclose technologies related to parallax images.

As one parallax image generation method, a method for independently capturing a left eye image and a right eye image of a same object using two systems of image capturing units having parallax, such as two digital still cameras or double lenses of a digital still camera, which are set for generating parallax, has been proposed. Another method that has been proposed is generating a parallax image using image processing from an image captured by a single lens digital still camera. According to this method, a left eye image is captured by a single lens image capturing unit first, then the object image extracted from the left eye image is shifted by the amount of parallax, thereby a right eye image is generated.

A method for using a single lens image capturing unit may be implemented using a simple and compact configuration, but the following procedures are required. For example, in order to determine parallax for each of a plurality of objects when capturing the plurality of objects of which distances from the image pickup unit are different, the distance information of each object is required. Therefore instead of disposing a distance measuring unit, a plurality of images are captured while changing the focused object distance. Then by extracting the focused object image data from each captured image data, the extracted object image data and the focused object distance upon capturing the image are corresponded. Then by shifting the extracted object image data by the amounts of parallax according to the respective focused object distances and synthesizing this data, the right eye image data is generated.

If such a method is used, some degree of image capturing procedures is inevitably required. Hence an efficient image capturing operation is demanded in order to prevent a drop in throughput. Also when the extracted object image data is shifted and synthesized, image processing, which reproduces natural brightness and a perspective of an object, is demanded.

SUMMARY

According to an aspect of an embodiment, a processor includes: an imaging control unit which causes an imaging unit for capturing an image with changing a focused object distance, to change the focused object distance with a first change width in the case of a first focused object distance, and to change the focused object distance with a second change width that is smaller than the first change width in the case of a second focused object distance that is shorter than the first focused object distance; and an image processing unit which extracts focused object image data from captured image data for each of the focused object distances, and generates image data by shifting the extracted object image data by parallax amounts corresponding to the respective focused object distances, and by synthesizing the extracted object image data.

DESCRIPTION OF EMBODIMENTS

Embodiments will now be described with reference to the drawings. The technical scope, however, is not limited to these embodiments, but extends to matters stated in Claims and equivalents thereof.

FIG. 1is a diagram illustrating a configuration of an imaging device to which the processor of this embodiment is applied. This imaging device2is a single lens digital still camera, for example. The imaging device2captures an image of an object Ob by an imaging control unit41of the processor4causing an imaging unit7to change a focused object distance. Then an image processing unit42performs image processing on a captured image data for each focused object distance, and generates parallax image data (left eye image data and right eye image data).

The imaging unit7has a single lens unit6and a picture element8. The light from the object Ob forms an object image on the picture element8via a lens unit6. The lens unit6has a zoom lens in which a plurality of single focus lenses are combined, so that the focused object distance is changed. The picture element8includes a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, for example, and converts the received light into analog electric signals.

An analog/digital converter10converts an analog electric signal, which is output from the picture element8, into a digital signal. An image memory12stores the digital signal, which is output from the analog/digital converter10, as captured image data. The captured image data is read by the processor4and processed by the image processing unit42, whereby parallax data is generated. The generated parallax image data is stored in the image memory12, and is transferred to a display device16and storage device14. The display device16includes an LCD (Liquid Crystal Display) having a polarizing filter corresponding to parallax of the left and right eyes, and displays a three-dimensional image based on the parallax image data. The storage device14writes or reads the parallax data to/from a storage media, such as an IC memory, SD card, memory stick, hard disk and floppy disk®.

The processor4includes a microcomputer or ASIC (Application Specific Integrated Circuit), for example. The processor4systematically controls the operation of the imaging device2by synchronously controlling the imaging unit7, AD converter10, image memory12, storage device14and display device16. The imaging control unit41outputs a control signal, which indicates the change width of the focused object distance, to the lens unit6of the imaging unit7. Thereby the focused object distance for the imaging unit7, to capture an image, is controlled. In the processing unit4, the image processing unit42performs such processing as white balance, gamma correction, image filtering such as a noise filter, and image compression/decompression, for the captured image data. Also as described later, the first image processing unit42aextracts the focused object image data from the captured image data, and the second image processing unit42bgenerates parallax image data from the focused object image data.

The present embodiment relates to the control of the image capturing operation of the imaging unit7by the imaging control unit41, and image processing by the first image processing unit42aand the second image processing unit42b.

FIG. 2is a diagram illustrating a basic operation of the imaging control unit41and the image processing unit42.FIG. 2illustrates an example of capturing an image with the front face of the lens unit6as the left eye direction, and generating a parallax image data from the captured image data. Here the parallax image data includes of left eye image data and right eye image data, which has parallax in the virtual right eye.

First asFIG. 2Aillustrates, images are captured under control of the imaging control unit7at a plurality of focused object distances D1to D3, while changing the focused object distances of the lens unit6. The first image processing unit42aof the image processing unit42detects an object portion which is focused (hereafter focused object portion) Ob1to Ob3from the captured image data P1to P3obtained for each focused object distance D1to D3respectively. At this time, the image processing unit42ameasures the high frequency component amount in the entire captured image data in each of the captured image data P1to P3, and if the high frequency component amount is high, exceeding a given threshold, for example, the signal processing unit42adetermines that the focused object portions Ob1to Ob3exist. Hence the focused object portions Ob1to Ob3are detected by detecting the high frequency components. For example, in the case of the captured image data P1, when the focused object distance D1is the longest, the focused object portion Ob1, which corresponds to a cloud or mountain at a long distance, is detected. In the case of the captured image data P2when the focused object distance D2is intermediate, the focused object portion Ob2corresponding to a house in an intermediate distance is detected. In the case of the captured data P3when the focused object distance D3is shortest, the focused object portion Ob3corresponding to a bus at a short distance is detected.

Then asFIG. 2Billustrates, the first image processing unit42adeletes the images other than the detected focused object portions Ob1to Ob3, whereby the object image data OD1to OD3corresponding to each focused object portion Ob1to Ob3(hereafter called “focused object image data”) are extracted. Here the focused object image data OD1to OD3and the focused object distance D1to D3, as the distance information, are corresponded.

Then asFIG. 2Cillustrates, the second image processing unit42bshifts the focused object image data OD1to OD3to the left by the parallax amounts (arrows Ar21to Ar23), to generate the object image data (hereafter called “right eye object image data”) rOD1to rOD2. Here the parallax amount for shifting the focused object image data OD1to OD3is determined based on the distance d between the lens unit6and lens unit6r, which is the virtual lens unit6rcorresponding to the right eye, and the focused object distance D1to D3, as illustrated inFIG. 2A.

The second image processing unit42bcombines the right eye object image data rOD1to rOD3, and generates the right eye image data PRm. On the other hand, the second image processing unit42bgenerates the left eye image data either by using any of the captured image data P1to P3as the left eye image data, or by synthesizing the focused object image data OD1to OD3. In this way, the parallax image data, which includes of the right eye image data PRm and the left eye image data, is generated. Although a case of generating the right eye image data using the left eye image data as a reference is illustrated, left and right may be reversed in this processing.

FIG. 3is a flow chart illustrating the operation procedure of the imaging device2. Steps S1to S4relate to the imaging operation by the imaging unit7which is performed under control of the imaging control unit41. In step S1, the lens unit6of the imaging unit7responds to the control signal from the imaging control unit41, and sets the focused object distance for starting image capturing. In step S2, the imaging unit7captures an image, and loads the captured image data into the processor4. In step S3, the imaging control unit41determines whether image capturing is completed for a given number of times, set in advance. If not completed (NO in step S3), the imaging control unit41outputs a control signal to indicate the next focused object distance to the imaging unit7in step S4, and responding to this, the imaging unit7sets the focused object distance indicated by the control signal. And processing returns to step S2. This procedure is repeated until image capturing is completed for a number of times being set.

When image capturing is completed (YES in step S3), the first image processing unit42adetects a focused object portion in each captured image data in step S5, and deletes the other images to extract the focused object image data.

Steps S6to S12relate to image processing by the second image processing unit42b. The detail of each processing will be described later. In step S6, the second image processing unit42bperforms processing to transform the focused object image data. The transforming processing includes enlarging/reducing the size of the object image. Then in step S7, the second image processing unit42bperforms brightness correction processing for the focused object image data. Then in step S8, the second image processing unit42bperforms color saturation correction processing for the focused object image data. Then in step S9, the second image processing unit42bperforms gradation processing for the focused object image data. Then in step S10, the second image processing unit42bsynthesizes the focused object image data, and generates the left eye image data. One of the captured image data may be used as the left eye image data in step S10. Then in step S11, the second image processing unit42bshifts the focused object image data by the parallax amounts and synthesizes the data, so as to generate temporary right eye image data. Then in step S12, the second image processing unit42bperforms interpolation processing for a portion which is behind the object for the temporary right eye image data. By this procedure, the parallax image data is generated.

Now the imaging operation in steps S1to S4and the image processing in step S6and later will be described separately.

According to this embodiment, when the imaging unit7captures an image while changing the focused object distance, the imaging control unit41outputs a control signal to indicate the change width of the focused object distance to the lens unit6of the imaging unit7, and changes the focused object distance with this change width. The imaging control unit41controls the change width by the following two control modes.

In the first control mode, the imaging control unit41causes the imaging unit7to change the focused object distance with a large change width if the focused object distance is long, that is, if an image of an object in a long distance is captured, and with a small change width if the focused object distance is short, that is, if an image of an object in a short distance is captured. In the second control mode, the imaging control unit41causes the imaging unit7to change the focused image distances with such a change width that the focal distance ranges corresponding to a depth of field of each focused object distance overlap as little as possible, and preferably become continuous. The first and second control modes will be described in concrete terms with reference toFIG. 4toFIG. 7.

FIG. 4is a diagram illustrating the first control mode. In the first control mode, an image is captured n times (n>1) in a distance range L0 where the lens unit6focuses.FIG. 4AtoFIG. 4Cillustrate the lens unit6, picture element8and objects Ob1to Ob3at respective focused object distances when three types of focus object distances D1to D3are used.FIG. 4Ais a case of the focused object distance D=L0, which is the longest,FIG. 4Bis a case of the focused object distance D=L/2, which is the second longest, andFIG. 4Cis a case of the focused object distance D=L/n, which is the shortest.

The imaging control unit41outputs a control signal to set the focused object distance D of the lens unit6to, for example, the longest focused object distance L0 in step S1inFIG. 3, for example. Then steps S2to S4are executed, and the imaging control unit41changes the focused object distance in the imaging unit7. At this time, the imaging control unit41changes a focused object distance to have a larger change width as the focused object distance is longer, and to have a smaller change width as the focused object distance is shorter. Specifically, the focused object distance is controlled to be decreased gradually, such as L0 (FIG.4A)→L0/2 (FIG.4B)→L0/3→L0/4 . . . →L0/(n−2)→L0/(n−1)→L0/n (FIG. 4C) (n is a number of times of capturing an image, and is a 1 or greater value). Or the focused object distance D may be gradually increased from L0/n to L0. At this time, the focused object distance D, the longest distance L0 and the number of times of image capturing n have the following relationship.
D=L0/n(n>1)

Instead of n, 2 to the mth power (m is an integer) may be used.

FIG. 5is a diagram illustrating the relationship between the change width of the focused object distance and parallax.FIG. 5illustrates objects Ob11, Ob12, Ob13and Ob14, which are located at equal interval in the front face direction of the lens unit6. The objects Ob11and Ob12are positioned in a relatively long distance, and the objects Ob13and Ob14are position in a relatively short distance. Here the parallaxes of the objects Ob11and Ob12in the right eye virtual lens unit6rare θ1and θ2respectively, and the difference of parallaxes is Δθ1. On the other hand, the parallaxes of the objects Ob13and Ob14are θ3and θ4, and the difference thereof is Δθ2.

Comparing the differences of the parallaxes Δθ1and Δθ2, the difference Δθ2is greater than the difference Δθ1. In other words, the influence of the distance difference between objects in the short distance on the difference of parallaxes is greater than the influence of the distance difference between objects in the long distance on the difference of parallaxes. This means that when observing with the naked eye, the distance difference of the objects in the long distance is less recognized as parallax, but the distance difference of the objects in the short distance is easily recognized as parallax.

Here it is assumed that the images of the objects Ob11and Ob12in the long distance are captured with the same depth of field of the focused object distance, and a same focused object distance is corresponded to the respective distance information. In this case, even if the focused object image data of the objects Ob11and Ob12are shifted respectively with the same parallax amount corresponding to the focused object distance, no odd sensation is generated since the difference Δθ1between parallaxes θ1and θ2is small. But on the other hand, it is assumed that the images of the objects Ob13and Ob14in the short distance are captured with a same depth of field of the focused object distance, and the same focused object distance is corresponded to the respective distance information. In this case, if the focused object image data is shifted respectively with a same parallax amount corresponding to the focused object distance, a highly odd visual sensation is generated, since the difference Δθ2between parallaxes θ3and θ4is large.

In the first control mode in this embodiment, images of the objects Ob13and Ob14in the short distance are captured with different focused object distances by changing the focused object distance with a smaller change width as the focused object distance is shorter. Therefore the respective object image data of the objects Ob13and Ob14are shifted by the parallax amounts according to the respective focused object distances. As a result, a natural right eye image, with less of an odd sensation, is generated.

FIG. 6is a diagram illustrating the second control mode.FIG. 6AtoFIG. 6Cillustrate the lens unit6, picture element8and objects Ob1to Ob3in respective focused object distances when the imaging unit7captures images using focused object distances D1to D3.FIG. 6AtoFIG. 6Calso illustrate the focal distance ranges M1to M3corresponding to the depth of field of each focused object distances D1to D3.

In step S1inFIG. 3, for example, the imaging control unit41sets the focused object distance of the lens unit6to the longest focused object distance D1illustrated inFIG. 6A. Then steps S2to S4are executed. At this time, the imaging control unit41causes the imaging unit7to change the focused object distances D1to D3with such a change width D1-D2and D2-D3that the focal distance ranges M1to M3overlap as little as possible, and preferably the focus distance ranges M1to M3become continuous, as illustrated inFIG. 6AtoFIG. 6C. Since the depth of field of each focused object distance is deeper (longer distance) as the focused object distance is longer, and shallower (shorter distance) as the focused object distance is shorter, the focal distance ranges M1to M3corresponding to the focused object distances D1to D3(here D1>D2>D3) have the relationship of M1>M2>M3. In order to minimize overlapping of the focal distance ranges M1to M3, and preferably the focus distance ranges M1to M3become continuous, the imaging control unit41causes the imaging unit7to change the focused object distance with a gradually decreasing change width D1-D2and D2-D3. The focused object distances may be changed in the reverse sequence, as D3, D2and D1, but in this case, the change width is controlled to be gradually increased.

If a pitch width for changing the focused object distance is fixed, and controlling with a change width with which the focal distance ranges M1to M3become continuous is difficult, and an overlap is generated between the focal distance ranges M1and M2, or between M2and M3, then the imaging control unit41may make the focal distance ranges continuous by adjusting the focal distance ranges. For example, the imaging control unit41changes the focal distance range by controlling the effective aperture of the lens in the lens unit6. In this case, insufficient or excessive quantity of light due to the change of the effective aperture is corrected by adjusting the exposure time, or by adjusting the analog gain or digital gain of the electric signal corresponding to the captured image data.

FIG. 7illustrates a case of changing the focused object distance with an equal change width, to be compared with the second control mode.FIG. 7illustrates the respective focal distance ranges M1to M3of the focused object distances D1to D3when the imaging unit7changes the focused object distances D1to D3(D1>D2>D3) with an equal change width ΔD.

When the focal distance range is deep corresponding to the focused object distance, and the change width ΔD is small compared with the focal distance range, the range where the focal distance ranges M1, M2and M3overlap increases as illustrated inFIG. 7. Particularly in the distance range M4, the focal distance ranges M1and M3overlap before and after changing the two change widths. In this case, the focal distance range M2overlaps in the entire region thereof with one or both of the focal distance ranges M1and M3. Therefore the focused object image data obtained in the focal distance range M2is also redundantly obtained in the focal distance ranges M1and M3. In this case, the image capturing in the focused object distance D2involves unnecessary steps. If redundant captured image data increases, unnecessary processing also increases.

However according to the second control mode of this embodiment, the focused object distances D1to D3are changed with a change width with which the focal distance ranges M1to M3overlap as little as possible, and preferably become continuous. The focal distance ranges may overlap somewhat between adjacent focused object distances. Even if the focal distance ranges M1and M2or M2and M3have some overlapped portion respectively, the generation of unnecessary image capturing steps or unnecessary image processing in the focused object distance D2, as illustrated inFIG. 7, is avoided if at least the focal distance ranges M1and M3, having two change widths, are apart without an overlap. As a result, throughput improves accordingly.

Even in the case when a same object is included in two captured images, since two focal distance ranges partially overlap, and overlapped focused object image data is extracted in step S5inFIG. 3, the first image processing unit42aeliminates an overlap by keeping the focused object image data in the short focal distance range, and deleting the focused object image data in the long focal distance range (or vice versa), for example.

In the second control mode as well, the change width decreases as the focused object distance decreases, so as illustrated inFIG. 5, images of objects in short distance of which parallax is large are captured with different focused object distances respectively. Therefore the focused object image data are shifted with different parallax amounts, and more natural right eye image data is generated.

Now the image processing in step S6or later inFIG. 3will be described. Each image processing to be described below may be in practice alone or in combination of two or more image processings.

FIG. 8is a diagram illustrating the transforming processing of focused object image data. First the first image processing unit42adetects a focused object portion from captured image data P81to P83in the focused object distances D1to D3(D1>D2>D3) of the lens unit6, as illustrated inFIG. 8A. For example, focused object portions Ob811and Ob812, corresponding to an object in a long distance are detected in captured image data P81, focused object portions Ob821and Ob822corresponding to an object in an intermediate distance are detected in captured image data P82, and focused object portions Ob831and Ob832corresponding to an object in a short distance are detected in the captured image data P83respectively.

Then asFIG. 8Billustrates, the first image processing unit42aextracts the focused object image data OD811and OD812corresponding to the focused object portions Ob811and Ob812from the captured image data P81, the focused object image data OD821and OD822corresponding to the focused object portions Ob821and Ob822from the captured image data P82, and the focused object image data OD831and OD832corresponding to the objects Ob831and Ob832from the captured image data P83.

Then asFIG. 8Cillustrates, the second image processing unit42btransforms the focused object image data OD811, OD812, OD821, OD822, OD831and OD832according to the corresponding focused object distances D1to D3. Then the second image processing unit42bshifts the focused object image data OD811, OD812, OD821, OD822, OD831and OD832by the parallax amounts in the virtual lens unit6raccording to the focused object distances D1to D3, so as to generate right eye focused object data rOD811, rOD812, rOD821, rOD822, rOD831and rOD832.

In the transforming processing, transformation is performed so that the image at the left is small and the image at the right is large to a degree according to the focused object distance. In other words, transformation is performed so that the size in a direction (y axis) crossing the parallax direction (x axis) becomes smaller in an image more to the left and larger in an image more to the right. For example, when the horizontal direction of each frame is the X axis and the vertical direction thereof is the Y axis, and the center coordinates of the image are (0, 0), the coordinate positions (x, y) before transformation are converted into coordinate positions (X, Y) after transformation using the following expression.
X=α·x
Y=Y·f(x)
f(x)=β·x+γ

Where α and β are coefficients to indicate the degree of transformation, and α increases according to the focused object distance, β decreases according to the focused object distance, and γ decreases according to the focused object distance. Here α, β and γ are all positive values. According to this processing, the focused object image data is transformed with the horizontal center of the frame as the boundary, so that the image at the left is reduced in both the x and y axis directions, and the image at the right is reduced in the x axis direction, and expanded in the y axis direction.

Therefore the focused object image data OD811and OD812at the longest focused object distance D1are hardly transformed, but the focused image data OD821and OD822at the intermediate focused object distance D2are transformed so that the focused object image data OD821at the left becomes somewhat smaller, and the focused object image data OD822at the right becomes somewhat larger. The focused object image data OD831and OD832at the shortest focused object distance D3are transformed so that the focused object image data OD831at the left becomes still smaller, and the focused object image data OD832at the right becomes still larger. The procedure to perform this transformation processing corresponds to step S6inFIG. 3.

This transformation processing based on that even if objects are located in a same focused object distance, the objects appear smaller as they become closer to the left, and appear larger as they become closer to the right according to the distance from the right eye. This will be described with reference toFIG. 9.

FIG. 9is a plain view illustrating positions of the objects of the example inFIG. 8. Here the same reference symbols as the focused object portions inFIG. 8are denoted for each object. InFIG. 9, the objects Ob811and Ob812are illustrated in the focused object distance D1of the lens unit6, the objects Ob821and OB822are illustrated in the focused object distance D2thereof, and the objects Ob831and Ob832are illustrated in the focused object distance D3thereof. When the horizontal direction is a direction coupling the lens unit6of the left eye and the virtual lens unit6rof the right eye, the distance difference of Ob811and Ob812in the horizontal direction, the distance difference of Ob821and Ob822in the horizontal direction, and the distance difference of Ob831and Ob832in the horizontal direction are all the same.

In terms of the distance from the virtual lens unit6rof the right eye to each object, the parallax amount of an object at the left is greater than the parallax amount of an object at the right if the focused object distance is the same, so the distance to the object at the left is longer than the distance to the object at the right. Therefore in the longest focused object distance D1, the distance d11to the object Ob811is longer than the distance d12to the object Ob812by the distance difference Δd31. In the intermediate focused object distance D2, the distance d21to the object Ob821is longer than the distance d22to the object Ob822by the distance difference Δd32(>Δd31). In the shortest focused object distance D3, the distance d31to the object Ob831is longer than the distance d32to the object Ob832by the distance Δd33(>Δd32).

In this way, even if objects are at the same focused object distance, the distance to the object at the left appears longer and the distance to the object at the right appears shorter from the view of the virtual lens unit6rof the right eye. Hence the object at the left appears larger and the object at the right appears smaller. The distance difference of the left and right objects from the virtual lens unit6rof the right eye decreases as the focused object distance increases, and increases as the focused object distance decreases. Therefore the apparent difference of the sizes of the objects at the left and right decreases as the distance increases, and the apparent difference of the sizes of the objects at the left and right increases as the distance decreases.

As a result, the right eye image data expressing a natural perspective is generated by synthesizing the right eye focused object data rOD811, rOD812, rOD821, rOD822, rOD831and rOD832after the transformation processing inFIG. 8is performed. Then based on this data, a more natural three-dimensional image is generated.

InFIG. 8andFIG. 9, the case of the left and right object image data corresponding to different objects was described as examples. However the description inFIG. 8andFIG. 9are also applied to the object image data of a single object. In this case, each portion of the single object image data corresponds to the object image data. And the above mentioned transformation processing is executed for each object image data according to the coordinates in one frame. Thereby a three-dimensional image which has a perspective transformed to be smaller at the left and larger at the right is generated even for a single object.

FIG. 10is a diagram illustrating a first brightness correction processing of the focused object image data. Here the brightness corresponds to the Y value when RGB (Red, Green, Blue) pixel values of the captured image data is converted into YCbCr, for example.

The first image processing unit42adetects the focused object portions Ob91to Ob95in the captured image data P91to P95in the focused object distances D91to D95of the lens unit6, as illustrated inFIG. 10A. Here an object “1” corresponding to the focused object portion Ob91, object “2” corresponding to the focused object portion Ob92, object “3” corresponding to the focused object portion Ob93, object “4” corresponding to the focused object portion Ob94, and object “5” corresponding to the focused object portion Ob95have a longer distance from the lens unit6in this sequence. Since the focused object distances D91to D95increases in this sequence, the focused object portion Ob91is detected in the captured image data P91in the focused object distance D1, the focused object portion Ob92is detected in the captured image data P92in the focused object distance D2, the focused object portion Ob93is detected in the captured image data P93in the focused object distance D3, the focused object portion Ob94is detected in the captured image data P94in the focused object distance D4, and the focused object portion Ob95is detected in the captured image data P95in the focused object distance D5respectively. Then asFIG. 10Billustrates, the first image processing unit42aextracts the focused object image data OD91to OD95corresponding to the focused object portions Ob91to Ob95from the captured image data P91to P95respectively.

If a fairly uniform quantity of light is supplied to each object when the image is captured, the focused object portions Ob91to Ob95in the captured image data P91to P95have similar brightness because of automatic exposure control. Then in the right eye image data synthesized in this state, both the image in the long distance and the image in the short distance have the same brightness. In a naturally captured image however, the brightness of the image is lower, that is darker, as the distance from the lens unit6increases, and brightness thereof is higher, that is brighter, as the distance decreases.

Hence the second image processing unit42bcorrects the brightness of the focused object image data OD91to OD95to be brightness according to the focused object distance D91to D95. Specifically, the second image processing unit42bselects any of the captured image data P91to P95, such as the captured image data P93of the intermediate focused object distance D93, as a reference captured image data Pref as illustrated inFIG. 10E. Here in the reference captured image data Pref (captured image data P93), the focused object portion Ob93is focused, the rest is not focused. So the second image processing unit42bobtains the respective coordinates of the focused object image data OD91to OD95, and detects the brightnesses C1to C5from the corresponding coordinate regions r1to r5in the reference captured image data Pref (captured image data P3). At this time, representative values, such as the average value, central value or intermediate value, of the brightness in the coordinate regions are detected.

Then asFIG. 10Cillustrates, the second image processing unit42bcorrects the brightness of the focused object image data OD91to OD95to be the detected brightnesses C1to C5. This brightness correction processing procedure corresponds to step S7inFIG. 3.

After this brightness correction processing, the second image processing unit42bshifts the focused object image data OD91to OD95according to the parallax amounts in the virtual lens unit6r(arrows Ar1to Ar5), as illustrated inFIG. 10D, and generates right eye object image data rOD91to rOD95. Then the second image processing unit42bsynthesizes the right eye object image data rOD91to rOD95to generate right eye image data PRm10. On the other hand, the second image processing unit42buses the captured image data P93selected as the reference captured image data Pref as left eye image data. Or the second image processing unit42bsynthesizes the focused object image data having the corrected brightness to generate the left eye captured image data.

By this processing, the right eye image data, that reproduces natural brightness corresponding to the perspective of an object, is generated, even if each focused object portion is corrected to a similar brightness by an automatic exposure adjustment function when the captured image data P91to P95are processed. As a result, a more natural three-dimensional image is generated based on this data.

FIG. 11is a diagram illustrating a second brightness correction processing. The differences betweenFIG. 11andFIG. 10are thatFIG. 11has no drawing corresponding toFIG. 10E, andFIG. 11Cis different fromFIG. 10C. In the case of the second brightness correction processing, the second image processing unit42bcorrects brightness of the object image data OD91to OD95according to the focused object distance P91to P95, instead of using the reference captured image data. For example, the focused object distances D91to D95increases in this sequence, so the brightness of the corresponding focused object image data OD91to OD95decreases in this sequence. Therefore asFIG. 11Cillustrates, the second image processing unit42bcorrects the brightness of the focused object image data OD91, OD92, OD93, OD94and OD95to be −20%, −10%, ±0%, +10% and +20% respectively.

According to the second brightness correction processing, even if each focused object portion is corrected to similar brightness by an automatic exposure adjustment function when the captured image data P91to P95are processed, the increased or decreased brightness is returned to the original brightness according to the focused object distance. Therefore the left eye image data and the right eye image data, which reproduce natural brightness corresponding to the perspective of the object, are generated. Based on this data, a more natural three-dimensional image is generated.

FIG. 12is a flow chart illustrating the procedure of the second brightness correction processing. In step S52, the first image processing unit42aextracts the focused object image data OD91from the captured image data P91in the focused object distance D91. Then in step S54, the second image processing unit42bincreases the brightness of the focused object image data OD91by 20%. Then in step S56, the first image processing unit42aextracts the focused object image data OD92from the captured image data P92in the focused object distance D92. Then in step S58, the second image processing unit42bincreases the brightness of the focused object image data OD92by 10%. Then in step S60, the first image processing unit42aextracts the focused object image data OD93from the captured image data P93in the focused object distance D93. In this case, the brightness of the focused object image data OD93is unchanged. Then in step S62, the first image processing unit42aextracts the focused object image data OD94from the captured image data P94in the focused object distance D94. Then in step S64, the second image processing unit42bdecreases the brightness of the focused object image data OD94by 10%. Then in step S66, the first image processing unit42aextracts the focused object image data OD95from the captured image data P95in the focused object distance D95. Then in step S68, the second image processing unit42bdecreases the brightness of the focused object image data OD95by 20%. Then in step S70, the second image processing unit42bshifts the focused object image data OD91to OD95and generates the right eye object image data rOD91to rOD95, and synthesizes this data.

According to a variant form of the second brightness correction processing, map data, where the focused object distance and brightness are corresponded, is provided in the image processing unit42in advance, and based on this map data, the second image processing unit42bcorrects the brightness of each focused object image data.FIG. 13illustrates an example of map data, in which brightness changes linearly or non-linearly according to the focused object distance. From this map data, the second image processing unit42breads brightness corresponding to the focused object distance of each focused object image data, and corrects the brightness to be this brightness. According to this method, a more accurate brightness correction is implemented considering the optical characteristics of the lens unit6, the quantity of electric charges generated by the picture element8according to the quantity of light, and the output characteristics of the display device16or the like.

The second image processing unit42bcorrects the color saturation of the focused object image data. Here color saturation is a distance from the achromatic color axis in the CbCr (color difference) space when the RGB pixel values of the captured image data are converted into YCbCr, and corresponds to a square root of (Cb2+Cr2).

For the color saturation correction processing, processing similar to the processing described inFIG. 10toFIG. 13is applied. For example, as illustrated inFIG. 10, the second image processing unit42bselects the reference captured image data Pref from the captured image data P91to P95, and corrects the respective color saturation of the focused object image data OD91to OD95to be representative color saturations of the coordinate regions to which the focused object image data OD91to OD95correspond to in the reference captured image data Pref. Or the second image processing unit42badjusts the color saturation of each focused object image data OD91to OD95according to the focused object distances D91to D95, in a similar manner to as illustrated inFIG. 11andFIG. 12. In this case, the correction is performed so that the color saturation is decreased as the distance of the object increases, and the color saturation is increased as the distance of the object decreases. The image processing unit42has the map data illustrated inFIG. 13in which the focused object distance and color saturation are corresponded, and the second image processing unit42brefers to this and corrects the respective color saturation of the focused object image data OD91to OD95, so that the color saturation corresponds to the focused object distance. This color saturation correction procedure corresponds to step S8inFIG. 3.

By this processing, the right eye image data, that reproduces natural color saturation corresponding to the perspective of the object, is generated. And based on this data, a more natural three-dimensional image is generated.

FIG. 14is a diagram illustrating the gradation processing. Here the captured image data P1to P3, illustrated inFIG. 2, will be described as an example. The first image processing unit42adetects the focused object portions Ob1to Ob3from the captured images P1to P3(FIG. 14A), and extracts the focused object image data OD1to OD3(FIG. 14B). Then the second image processing unit42bexecutes the gradation processing on the focused object image data OD1to OD3respectively, with a degree according to the corresponding focused object distances D1to D3. In gradation processing, filter processing for removing high frequency components of the object image using a moving average filter, low pass filter based on a convolution operation, a Gaussian filter or the like, is performed. The degree of gradation processing corresponds to the quantity of the remaining high frequency components. For example, if filtering to decrease the quantity of high frequency components is performed, a strong degree of gradation processing, with which the contours of the obtained images is unclear, is executed. If filtering to increase the quantity of high frequency components is performed, a weak degree of gradation processing, with which the contours of the obtained image is somewhat clear, is executed.

AsFIG. 14Cillustrates, the second image processing unit42bperforms a strong degree of gradation processing on the focused object image data OD1corresponding to the longest focused object distance D1, for example, and shifts the focused object image data OD1by the parallax amount (arrow Ar21) to generate the right eye object image data rOD1′. The second image processing unit42bperforms gradation processing on the focused object image data OD2corresponding to the intermediate focused object distance D2, with a degree weaker than the gradation processing on the focused object image data OD1, shifts the focused object image data OD2by the parallax amount (arrow AR22) to generate the right eye object image data rOD2′. Then the second image processing unit42bshifts the focused object image data OD3corresponding to the shortest object distance D3by the parallax amount (arrow Ar23) with hardly performing or without performing gradation processing, to generate the right eye object image data rOD3′. Then the second image processing unit42bsynthesizes the right eye object image data rOD1′, rOD2′ and rOD3′, to generate the right eye image data PRm'. The second image processing unit42balso generates the left eye image data by synthesizing the gradated focused object image data. The procedure of performing the gradation processing on the focused object image data OD1to OD3corresponds to step S9inFIG. 3.

In this way, the left eye image data and right eye image data, with which contours become less clear as the object image is more distant, are generated. As a result, the left eye image data and right eye image data, which reproduce a natural perspective, are generated, and a more natural three-dimensional image is generated based on this data.

In the above mentioned first and second brightness correction processings, color saturation correction processing and gradation processing, it is also possible that an arbitrary focused object distance is specified by user operation, and the focused object image data corresponding to the specified focused object distance is maintained as specified or corrected to a given brightness, color saturation or quantity of gradation, and the rest of the focused object image is corrected to a lower brightness, color saturation or quantity of gradation. In this case, an image with a visual effect to enhance a specific object, such as a spot lighted portion, according to user taste, is generated.

FIG. 15is a diagram illustrating an interpolation processing.FIG. 15Ais a diagram illustrating a shape and a position of an object. In the upper part ofFIG. 15A, four sided views of objects141,142and143are illustrated, of which direction facing the lens unit6is a front. As illustrated here, a cylinder142protruding toward the lens unit6exists at the right of the object141. The object141is located in the focused object distance D141, and the end portion143of the cylinder142is located in the focused object distance D143(<D141).

AsFIG. 15Billustrates, the first image processing unit42adetects the focused object portion Ob141corresponding to the object141from the captured image data P141in the focused object distance D141, and detects the focused object portion Ob143corresponding to the end portion143of the cylinder142from the captured image data P143in the focused object distance D143. Here the side face of the cylinder142, of which image is not captured from the angle of the lens unit6of the left eye, is not detected as a focused object portion.

Then asFIG. 15Cillustrates, the first image processing unit42aextracts the focused object image data OD141corresponding to the focused object portion Ob141from the captured image data P141, and extracts the focused object image data OD143corresponding to the focused object portion Ob143from the captured image data P143. Then asFIG. 15Dillustrates, the second image processing unit42bshifts the focused object image data OD141and OD143by the parallax amounts according to the corresponding focused object distances D141and D143(arrows Ar141, Ar143) respectively, so as to generate the right eye focused object data rOD141and rOD143.

Here if the right eye object image data rOD141and rOD143are directly synthesized, a right eye image, as illustrated inFIG. 15E, is generated. In other words, the image of the object141is generated near the end portion143of the cylinder142. However, asFIG. 15Aillustrates, the side face of the cylinder142is observed from the virtual lens unit6rof the right eye, due to parallax (L141), and the object141which is located behind the side face of the cylinder142is not observed. Therefore the right eye image illustrated inFIG. 15Eis unnatural, since it is different from an actually observed state.

Hence according to this embodiment, when the focused object image data OD141and OD143are shifted by the parallax amounts, the second image processing unit42binterpolates the intermediate regions M141and M143respectively between the focused object image data OD141and OD143before shifting and the right eye object image data rOD141and rOD143after shifting with a given single color, such as gray, respectively, as illustrated inFIG. 15D. The color to be used for the interpolation may be set to any color, but such an achromatic color as gray is preferably used to generate a natural three-dimensional image, minimizing the influence on the image.

Then the second image processing unit42bsynthesizes the right eye object image data rOD141and the interpolated intermediate region M141, and the right eye object image data rOD143and the interpolated intermediate region M143. At this time, the second image processing unit42brenders the right eye object image data on the image memory12in the sequence of longer focused object distance, that is, in the sequence of the right eye object image data of which distance is longer. A region where the image data overlaps is overwritten with the right eye object image data of which focused object distance is shorter. Thereby the right eye object image data rOD141, of which focused object distance is long, and the interpolated intermediate region M141are overwritten with the image data of the right eye object image data rOD143, of which focused object distance is short, and with the interpolated intermediate region M143. As a result, the right eye image data PRm15illustrated inFIG. 15Fis generated. Here the image142bof the side face of the cylinder142is displayed with the interpolated intermediate region M143, and the object141, which is behind the side face of the cylinder142and therefore may not be seen, is not displayed. In other words, an image, where an object in long distance located behind an object in a short distance is not seen, is generated. As a result, a natural right eye image data, close to the actually observed state, is generated. Based on this right eye image data, a more natural three-dimensional image is generated.

FIG. 16is a flow chart illustrating an image processing procedure to generate a parallax image including the interpolation processing. In step S82, the first image processing unit42aextracts the focused object image data OD141and OD143from the captured image data P141and P143in the focused object distances D141and D143. Then in step S84, the second image processing unit42bsynthesizes the extracted focused object image data OD141and OD143to generate the left eye image data. Then in step S86, the second image processing unit42bshifts the extracted focused object image data OD141and OD143by the parallax amounts, to generate the right eye object image data rOD141and rOD143. Then in step S88, the second image processing unit42binterpolates the intermediate region between the focused object image data OD141and OD143and the right eye object image data rOD141and rOD143using a single color. Then in step S90, the second image processing unit42bsynthesizes the right eye object image data rOD141and rOD143and the respective interpolated intermediate regions to generate the right eye image data.

FIG. 17is a diagram illustrating a variant form of interpolation processing. Here the right eye object image data rOD143and interpolation target intermediate region M143, described inFIG. 15D, are illustrated. In a first variant form, the second image processing unit42binterpolates the intermediate region M143not with such a given color as gray, but with a color of pixel G143(pixel value Ga) located near the intermediate region M143, that is, inside the contour line, in the right eye object image data rOD143.

In a second variant form, the second image processing unit42binterpolates the intermediate region M143with a color of a pixel G145(pixel value Gb) located near the intermediate region M143in the background region, that is, outside the contour line of the intermediate region, or outside the contour line of the focused object image data OD143before shifting.

In a third variant form, the second image processing unit42binterpolates the intermediate region M143with an interpolated color of pixel G143and pixel G145, according to the respective distances from pixel G143and pixel G145. Specifically, when Gx is the pixel value of the interpolation target pixel Mx in the intermediate region M143, and Lx is a distance of the pixel Mx from the pixel G145, the pixel value Gx is determined based on the following expression.
Gx=(Ga−Gb)/Lx+Gb

(where Lx is a distance between pixel G143and G145or less).

Thereby the intermediate region M143is interpolated using a gradation color which becomes closer to the color of the end portion143according to the distance to the lens unit6, based on the color in the background region of the cylinder142as a reference. In the above expression, Ga and Gb may be switched, so as to use the color in the end portion143as a reference. According to the third variant form, an even more natural three-dimensional image is generated, than the case of interpolating using a single color.

As described above, according to the present embodiment, throughput is improved and a natural three-dimensional image is generated when an image is captured using a single lens image capturing unit.