3-dimensional image acquisition apparatus and 3D image acquisition method for simultaneously obtaining color image and depth image

A 3-dimensional (3D) image acquisition apparatus capable of simultaneously obtaining a color image and a depth image in a single shooting operation is provided. The apparatus includes a light source for radiating illumination light having a predetermined wavelength onto an object; a lens unit having at least four object lenses; an image sensor including at least four sensing regions for individually receiving light focused by the object lenses and for generating images; and at least three optical shutters individually facing at least three of the at least four object lenses and for modulating incident light with predetermined gain waveforms.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No. 10-2012-0093888, filed on Aug. 27, 2012, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

Apparatuses and methods consistent with exemplary embodiments relate to 3-dimensional (3D) image acquisition apparatuses and 3D image acquisition methods for simultaneously obtaining a color image and a depth image, and more particularly, to compact 3D image acquisition apparatuses capable of simultaneously obtaining a color image and a depth image in a single shooting operation so as to have a compact configuration and to increase the speed of obtaining a 3D image, and 3D image acquisition methods using the same.

2. Description of the Related Art

Currently, since 3D display apparatuses capable of displaying stereoscopic images are increasingly being developed and are in more demand, people are interested in 3D content. As such, research is being conducted on 3D image acquisition apparatuses, e.g., 3D cameras, for allowing general users to directly generate 3D content. When an object is photographed, a 3D camera may obtain an image including typical 2-dimensional (2D) color information (hereinafter referred to as a color image) and an image including depth information (hereinafter referred to as a depth image) together.

Depth information regarding distances from surfaces of an object and a 3D camera may be obtained by using a stereo vision method using two cameras or an optical triangulation method using patterned light and a camera. In the optical triangulation method, although a color image and a depth image is simultaneously obtained in a single shooting operation and thus the speed of obtaining a 3D image is high, the accuracy of depth information may be greatly reduced if a distance to an object is large, and precise depth information may not be easily obtained due to high dependency on a surface state of the object. Also, since a sufficient distance has to be ensured between a light source and a light detector, a compact configuration may not be easily achieved.

In order to solve the above problems, a time-of-flight (TOF) technology has been introduced. According to TOF technology, a TOF until light illuminates an object and light reflected from the object reaches a light receiving part and is measured. In order to extract depth information, the TOF technology includes a series of operations such as projecting light having a certain wavelength (e.g., near infrared (NIR) light having a wavelength of 850 nm) onto an object by using an illumination optical system including a light emitting diode (LED) or a laser diode (LD), receiving the light by a light receiving part light having the same wavelength as that of the projected light, and then modulating the received light by using a modulator. Various TOF technologies have been suggested according to this series of light processing operations.

However, a TOF technology requires at least three infrared (IR) images to obtain one depth image. For example, one depth image may be obtained by photographing an object by using at least three beams of IR light having the same wavelength and different phases. Accordingly, since at least three shooting operations are required to obtain one depth image, a total exposure time is increased. As such, the TOF technology is not useful to photograph a moving object.

SUMMARY

One or more exemplary embodiments provide a TOF-type 3D image acquisition apparatuses capable of simultaneously obtaining a color image and a depth image in a single shooting operation so as to have a compact configuration and to increase the speed of obtaining a 3D image and 3D image acquisition methods using the same.

According to an aspect of an exemplary embodiment, a 3-dimensional (3D) image acquisition apparatus includes a light source for radiating illumination light having a predetermined wavelength onto an object; a lens unit having at least one first object lens and at least three second object lenses; an image sensor including at least one first sensing region for receiving light focused by the at least one first object lens and generating an image and at least three second sensing regions for individually receiving light focused by the at least three second object lenses and generating images; and at least three optical shutters individually facing the at least three second object lenses and for modulating incident light.

The at least one first object lens and the at least three second object lenses may be individual lenses separated from each other, and the lens unit may include a housing for fixing the separated individual lenses.

The at least one first object lens and the at least three second object lenses may be formed as a lens array sheet including the at least one first object lens and the at least three second lenses formed on one substrate.

The at least one first sensing region and the at least three second sensing regions may be formed by logically dividing an effective sensing region of one image sensor into at least four regions.

The at least one first sensing region and the at least three second sensing regions and the at least one first object lens and the at least three second object lenses may be aligned in the form of matrix arrays corresponding to each other.

The 3D image acquisition apparatus may further include a barrier disposed between the lens unit and the image sensor in order to prevent beams of light focused by different object lenses from overlapping with each other at boundaries of adjacent sensing regions.

The barrier may include two plates that are orthogonal to each other.

The 3D image acquisition apparatus may further include an infrared (IR) cut-off filter disposed in the at least one first sensing region.

The 3D image acquisition apparatus may further include band pass filters disposed on optical axes of the at least three second object lenses, and for transmitting only light within the predetermined wavelength region.

The optical shutters may be disposed between the lens unit and the object and have a size equal to or greater than a size of an effective aperture of the at least three second object lenses.

The optical shutters may be disposed between the lens unit and the image sensor.

The image sensor may be a color image sensor including color filters disposed at every pixel.

Each of the color filters may be configured to simultaneously transmit light of one of red, green, blue, cyan, magenta, and yellow and light within the predetermined wavelength region.

The light source may be an IR light source for generating IR light.

The color filters may be disposed in only the at least one first sensing region.

The 3D image acquisition apparatus may further include an image signal processing unit for generating a 3D image by using at least four images generated in the at least one first sensing region and the at least three second sensing regions; and a control unit for controlling operations of the light source and the optical shutters.

The control unit may control the at least three optical shutters to modulate incident light with gain waveforms having a same frequency as a frequency of the illumination light and having different phases from one another.

The control unit may control the at least three optical shutters in a first mode to modulate incident light with gain waveforms having a same frequency as a frequency of the illumination light and having different phases from one another, and then, in a second mode, to modulate incident light with gain waveforms having different wavelengths from the wavelength of the illumination light and having different phases from one another.

The control unit may control the at least three optical shutters to transmit incident light without modulating it in a third mode.

According to an aspect of another exemplary embodiment, a 3-dimensional (3D) image acquisition method includes projecting illumination light within a predetermined wavelength region onto an object; focusing light reflected from the object onto each of at least one first sensing region of an image sensor using at least one first object lens and at least three second sensing regions of the image sensor using at least three second object lenses; modulating light focused by the at least three second sensing regions, by using at least three optical shutters; and generating a depth image by using at least three images generated in the at least three second sensing regions, and generating a color image in the at least one first sensing regions, wherein the at least three optical shutters modulate incident light with gain waveforms having a same frequency as a frequency of the illumination light and having different phases from one another.

The 3D image acquisition method may further include disposing an infrared (IR) cut-off filter in the at least one first sensing region, so as to transmit only visible light.

The 3D image acquisition method may further include disposing band pass filters in the at least three second sensing regions, so as to transmit only light having a same wavelength region as that of the illumination light.

DETAILED DESCRIPTION

FIG. 1is a conceptual view of a 3D image acquisition apparatus100according to an exemplary embodiment. Referring toFIG. 1, the 3D image acquisition apparatus100may include a light source102for generating illumination light having a predetermined wavelength, an optical system110for generating a color image and a depth image by respectively using visible light and the illumination light reflected from an external object (not shown), an image signal processing unit105for generating a 3D image by using the color image and the depth image, and a control unit103for controlling operations of the light source102, the optical system110, and the image signal processing unit105. Also, the 3D image acquisition apparatus100may further include a display panel104for displaying an image. The optical system110is disposed in a case101, and a transparent window106may be disposed in a region of the case101facing the optical system110, such that the visible light and the illumination light reflected from the object are incident on the optical system110. Also, a transparent window106may prevent components of the optical system110from being exposed to the external environment.

The light source102may be, for example, an LED or an LD for emitting illumination light having a near infrared (NIR) wavelength of about 850 nm and invisible to human eyes for safety purposes. However, the current embodiment is not limited thereto, and illumination light having a different wavelength and another type of light source may be appropriately used. Also, the light source102may emit illumination light having a specially defined waveform, e.g., a sine wave, a ramp wave, or a square wave, according to a control signal received from the control unit103.

The optical system110may include a lens unit111having at least four object lenses, an image sensor113for receiving light focused by each of the object lenses so as to generate an image, and at least three optical shutters112facing at least three of the at least four object lenses and for modulating the illumination light with predetermined gain waveforms. Also, the optical system110may further include band pass filters114for transmitting only light in the same wavelength region as that of the illumination light generated by the light source102. Although the band pass filters114are shown as disposed in front of and are adjacent to the optical shutters112inFIG. 1, the band pass filters114may alternately be disposed between the optical shutters112and the image sensor113. The band pass filters114may be disposed in only a region where the optical shutters112are disposed, and are not disposed in a region where the optical shutters112are not disposed. For example, the band pass filters114may be individually disposed on optical axes of the at least three object lenses facing the at least three optical shutters112.

FIG. 2is a perspective view of the optical system110of the 3D image acquisition apparatus100illustrated inFIG. 1, according to an exemplary embodiment. Referring toFIG. 2, the lens unit111may include four individual object lenses, e.g., first through fourth object lenses111athrough111d. For example, the first through fourth object lenses111athrough111dmay be fixed in a lens housing115. Also, the first through fourth object lenses111athrough111dmay be aligned in the form of, for example, a 2×2 matrix. First through third optical shutters112athrough112cmay be respectively disposed in front of three object lenses, e.g., the first through third object lens111athrough111c, other than one object lens, e.g., the fourth object lens111d. The first through third optical shutters112athrough112cmodulate the amplitude of the illumination light reflected from the object with predetermined gain waveforms. For example, the first through third optical shutters112athrough112cmay be GaAs-based Fabry-Perot semiconductor modulators capable of operating at an ultrahigh speed of several ten to several hundred MHz. The gain waveforms of the first through third optical shutters112athrough112cmay have the same frequency as that of the illumination light and may have different phases from one another. The gain waveforms may be controlled by the control unit103. For example, the phases of the gain waveforms of the first through third optical shutters112athrough112cmay be 0°, 60° and 120°, respectively. Even though three optical shutters112athrough112care shown inFIG. 2, it is possible to use only two optical shutters, because one optical shutter whose gain waveform has the phase of 0° may be omitted. Therefore, it should be noted that, in the following description, three optical shutters112athrough112care illustrated as an example of the present embodiment, but two or more optical shutters may be used according to the design the 3D image acquisition apparatus100.

The first through fourth object lenses111athrough111dmay individually focus light onto the image sensor113. From among four beams of light focused by the first through fourth object lenses111athrough111d, the light focused by the fourth object lens111dis visible light, and light focused by the first through third object lens111athrough111cis NIR light modulated by the first through third optical shutters112athrough112c. In order to form four images by using the four lights, the image sensor113may be logically divided into four sensing regions, e.g., first through fourth sensing regions113athrough113d.That is, according to the current embodiment, instead of using four individual image sensors, an effective sensing region of a single image sensor113may be divided into the first through fourth sensing regions113athrough113d. For example, the control unit103and the image signal processing unit105may extract an NIR image formed by the first object lens111a,from coordinates corresponding to the first sensing region113aof the image sensor113. The first through fourth sensing regions113athrough113dmay be aligned in the form of a 2×2 matrix so as to respectively correspond to the first through fourth object lenses111athrough111d. The image sensor113may be a charge-coupled device (CCD) or a semiconductor imaging device such as a complementary metal oxide semiconductor (CMOS) device.

In a typical 3D image acquisition apparatus, visible light and NIR light are split into separate paths by using a beam splitter, and an image sensor is disposed on each path to obtain a visible light image and an NIR image. However, as described above, the 3D image acquisition apparatus100according to the current embodiment does not need an optical configuration for splitting visible light and NIR light, and may use only one image sensor113. Accordingly, the size and weight of the 3D image acquisition apparatus100may be reduced as compared to the typical 3D image acquisition apparatus.

FIG. 3is a perspective view of the optical system110of the 3D image acquisition apparatus100illustrated inFIG. 1, according to another exemplary embodiment. Referring toFIG. 3, the lens unit111may be a lens array sheet in which the first through fourth object lenses111athrough111dare formed on one substrate. If the lens unit111is a lens array sheet, the lens unit111may not require the lens housing115illustrated inFIG. 2. The configurations of the optical shutters112and the image sensor113may be the same as those described above in relation toFIG. 2. Meanwhile, as illustrated inFIG. 3, in the fourth sensing region113dfor obtaining a color image, an IR cut-off filter117for blocking light in an IR band and transmitting only visible light so as to prevent the color image from being distorted due to illumination light of an IR band may be further disposed. That is, the IR cut-off filter117may be disposed in the fourth sensing region113dcorresponding to the fourth object lens111dwhere the first through third optical shutters112athrough112care not disposed, from among the first through fourth object lenses111athrough111d. Although the IR cut-off filter117is illustrated as disposed between the fourth sensing region113dand the fourth object lens111dinFIG. 3, alternately, the IR cut-off filter117may be disposed in front of the fourth object lens111d. The IR cut-off filter117may also be included in the optical system110illustrated inFIG. 2. Also, although not shown inFIG. 3, the band pass filters114illustrated inFIG. 1, which may be used for transmitting only IR light having the same wavelength region as that of the illumination light generated by the light source102, may be further disposed individually between the first through third object lenses111athrough111cand the first through third sensing regions113athrough113c.

InFIGS. 2 and 3, the optical shutters112may be disposed in front of the lens unit111, i.e., between the lens unit111and an object. In this case, the optical shutters112may have a size similar to that of a valid aperture of the first through fourth object lenses111athrough111d. For example, the optical shutters112may have the same size as that of an effective aperture of the first through fourth object lenses111athrough111d, or may have a size slightly greater than that of the effective aperture in consideration of an alignment error caused when the optical shutters112are aligned. Accordingly, the size of the optical shutters112may be much less than that of the image sensor113. If the size of the optical shutters112is reduced, since an RC time constant of the optical shutters112is also reduced, high frequency operation can be easily achieved. That is, a frequency band of gain waveforms of the optical shutters112may be increased and thus the accuracy of measuring depth information may be improved.

However, as illustrated inFIG. 4, alternately, the optical shutters112may be disposed between the lens unit111and the image sensor113. In this case, the optical shutters112are disposed directly in front of the image sensor113, and the size of the optical shutters112may be the same as that of the image sensor113.

Meanwhile, beams of light focused by the first through fourth object lenses111athrough111dare respectively incident on the first through fourth sensing regions113athrough113d. However, the beams of light focused by the first through fourth object lenses111athrough111dmay overlap with each other at boundaries of the first through fourth sensing regions113athrough113d. As such, the color image may be distorted and the accuracy of measuring the depth information may be reduced. In order to prevent the above problem, as illustrated inFIG. 5, a barrier116for preventing light of neighboring regions from overlapping may be disposed between the lens unit111and the image sensor113. Referring toFIG. 5, the barrier116may include, for example, two long plates that are orthogonal to each other to have a ‘+’-shaped cross-section.

In order to obtain both a color image and a depth image, the image sensor113may be a color image sensor. The color image sensor may include color filters for transmitting only light of certain colors and disposed at every pixel. For example, the color image sensor may include a red color filter for transmitting only red light R, a green color filter for transmitting only green light G, and a blue color filter for transmitting only blue light B. Alternatively, the color image sensor may include a combination of color filters for transmitting light of color regions of cyan C, magenta M, yellow Y, and black K. However, as illustrated inFIGS. 2 and 3, the image sensor113obtains a color image in only the fourth sensing region113dand obtains IR images in the first through third sensing regions113athrough113c. Accordingly, at least the color filters disposed in the first through third sensing regions113athrough113cneed to transmit IR images. As such, the image sensor113may use color filters having transmission characteristics shown in the graph ofFIG. 6. For example, the red color filter transmits red light and IR light. The other color filters may also transmit light of corresponding color regions and IR light having a wavelength equal to or less than 800 nm. Therefore, both color and IR images may be captured by using one image sensor113.

Alternatively, as illustrated inFIG. 7, the image sensor113, in which the color filters are disposed in only the fourth sensing region113dand are not disposed in the first through third sensing regions113athrough113c, may be used. That is, the color filters may be disposed in a region where the optical shutters112are not disposed and may be removed from or not included in regions where the optical shutters112are disposed. Also, if only depth information is required, since a color image is not necessary, the image sensor113may be a black-and-white image sensor having no color filter.

Operation of the 3D image acquisition apparatus100will now be described.

Initially, the light source102projects illumination light having a predetermined frequency onto an object by the control of the control unit103. After that, the illumination light is reflected on the object and then is focused onto the image sensor113by the lens unit111. In this case, a phase of the illumination light is delayed according to a distance (i.e., a depth) between the object and the 3D image acquisition apparatus100. Accordingly, if a phase delay value of the illumination light is accurately measured, the distance between the object and the 3D image acquisition apparatus100may be obtained.

Meanwhile, visible light generated by an external light source such as the sun or a lamp and reflected by the object is also focused onto the image sensor113by the lens unit111. Referring toFIGS. 2 and 3, the optical shutter112is not disposed and the IR cut-off filter117is disposed in the fourth sensing region113d. Accordingly, a color image may be generated in the fourth sensing region113dby using the visible light focused by the lens unit111. On the other hand, the band pass filters114for transmitting only IR light having the same wavelength region as that of the illumination light, and the optical shutters112are disposed in the first through third sensing regions113athrough113c. Accordingly, IR images may be generated in the first through third sensing regions113athrough113cby using the IR light focused by the lens unit111.

In order to calculate a phase delay value of the illumination light, the optical shutters112amplitude-modulate the IR light with predetermined gain waveforms by the control of the control unit103. For example, all of the gain waveforms of the first through third optical shutters112athrough112cmay be controlled to have the same frequency as that of the illumination light emitted from the light source102and to have different phases from one another. As such, the IR images generated in the first through third sensing regions113athrough113cmay have different brightness levels. The IR images and the color image are transmitted to the image signal processing unit105. The image signal processing unit105may generate a depth image by extracting depth information by using the IR images, and may generate a 3D image by combining the depth image and the color image.

A process of extracting depth information performed by the image signal processing unit105may be mathematically modeled as described below.

Initially, reflected light that is reflected off an object and returned to the 3D image acquisition apparatus100may be represented as follows.
PLD=acos(ωt+φobj)+b[Equation 1]

In Equation 1, the unknowns are a reflectivity a of the object, an intensity b of an external light component, and a phase delay Φobj. A frequency ω of illumination light is determined by the control unit103and is a known value. Since there are three unknowns, at least three equations are necessary to calculate a phase delay value.

Accordingly, the first through third optical shutters112athrough112cmay modulate the reflected light represented by Equation 1 with gain waveforms having different phases, as follows.
Tφ1=ccos(ωt+φ1)+d   [Equations 2]
Tφ2=ccos(ωt+φ2)+d
Tφ3=ccos(ωt+φ3)+d

In Equations 2, c is the intensity of a gain waveform, d is a direct current (DC) component of a gain waveform, and Φ1, Φ2, and Φ3are phases of gain waveforms.63The reflected light after being modulated by the first through third optical shutters112athrough112creaches the image sensor113in the form of multiples of a signal represented by Equation 1 and signals represented by Equations 2. After that, the image sensor113may generate IR images by being exposed to the three different modulated beams of light in the first through third sensing regions113athrough113cfor a predetermined exposure time T. Brightness values of the IR images may be represented as follows.

In Equation 3, i is an identifier for identifying the three IR images generated after being modulated by the first through third optical shutters112athrough112c.

In Equation 3, for example, if Φ1=0, Φ2=β, and Φ3=2β, the phase delay value is calculated as follows.

The distance between the 3D image acquisition apparatus100and the object may be calculated by using the phase delay value calculated in Equation 4, as follows.

In Equation 5, f is a frequency of the illumination light, and c is the speed of light. In this manner, if the calculation is performed on every pixel of the image sensor113, a depth image representing distance information may be obtained.

The 3D image acquisition apparatus100may use at least the first through fourth object lenses111athrough111dand at least the first through third optical shutters112athrough112cas described above and thus may simultaneously generate a color image and a depth image in a single shooting operation. Accordingly, the 3D image acquisition apparatus100may have a compact configuration due to using a TOF method, and may obtain a 3D image at a high speed due to using optical triangulation. For example, the 3D image acquisition apparatus100may have a simple configuration and may obtain an image of a moving object by using a TOF method.

Meanwhile, the above method of extracting distance information by using Equations 1 through 5 is merely an example of various TOF methods, and the 3D image acquisition apparatus100may also extract the distance information by using a different method. For example, by modulating illumination light with a plurality of gain waveforms having the same wavelength (or frequency) as that of the illumination light and having different phases in a first shooting operation, and then by modulating the illumination light with a plurality of gain waveforms having different wavelengths from that of the illumination light and having different phases in a second shooting operation, more accurate distance information may be extracted.

Also, the 3D image acquisition apparatus100may operate in a night shot mode in which the optical shutters112transmit IR light without modulating it. In this case, instead of generating a 3D image, the 3D image acquisition apparatus100may capture, by using IR light, an image of an object that is not easily viewed at night time when visible light is insufficient.

So far, the 3D image acquisition apparatus100generating both a color image and a depth image is described. However, a depth image acquisition apparatus generating only a depth image may be provided. For example, the depth image acquisition apparatus may include a light source102, at least three object lenses111athrough111c, at least two optical shutters112aand112b, and an image sensor113comprising at least three sensing region113athrough113c.