Patent Description:
Three-dimensional (3D) content is applied not only to games and culture but also to many fields such as education, manufacturing, and autonomous driving, and a depth map is required to acquire 3D content. The depth map is information indicating a distance in space and represents perspective information of one point with respect to another point of a two-dimensional (2D) image.

One method of acquiring a depth map includes projecting an infrared (IR) structured light onto an object, interpreting light reflected from the object, and extracting a depth map. The IR structured light scheme has a problem in that it is difficult to obtain a desired level of depth resolution for a moving object.

Meanwhile, a time-of-flight (ToF) scheme has attracted attention as a technology that replaces the IR structured light scheme.

According to the ToF scheme, the distance to an object is calculated by measuring a flight time, that is, a time for light to be shot, reflected, and return. The greatest advantage of the ToF scheme is that it quickly provides distance information regarding 3D space in real time. Also, a user can obtain accurate distance information without separate algorithm application or hardware correction. Also, a user can acquire an accurate depth map even if he or she measures a very close subject or a moving subject.

However, the current ToF scheme has a problem in that information obtainable per frame, i.e., resolution, is low.

One way to increase resolution is to increase the number of pixels of an image sensor. However, in this case, there is a problem in that the volume and production cost of a camera module increase significantly.

Accordingly, there is a need for a depth map acquisition method capable of increasing resolution without significantly increasing the volume and production cost of a camera module.

<CIT> discloses an apparatus and a method for generating a depth image using a time of flight (TOF) method. <CIT> discloses a refraction plate tilted by means of a piezoelectric element, so that the incident position can be shifted to a solid state imaging device, when such position shifted images are composited an image of the high 2nd resolution is obtained. <CIT> discloses a wide-angle photographing apparatus using an auto-focus module, wherein the auto-focus module has a wide-angle lens on a photographing module with auto-focusing capabilities and is capable of executing a wide-angle photographing operation.

An object of the present invention is to provide a camera module for extracting a depth map using a time-of-flight (ToF) scheme.

With the camera module according to an embodiment of the present invention, it is possible to acquire a depth map with high resolution without significantly increasing the number of pixels of an image sensor.

Also, according to an embodiment, it is possible to obtain a subpixel shift effect using a simple structure, and also it is possible to protect an image sensor from moisture, foreign matter, and the like.

Although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. For example, a first element may be called a second element, and a second element may also be called a first element without departing from the scope of the present invention.

As used herein, the singular forms "a," "an," and "one" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and/or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or combinations thereof but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or combinations thereof.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals are given to the same elements throughout the drawings and redundant descriptions thereof will be omitted.

<FIG> is a block diagram of a ToF camera module according to an embodiment of the present invention.

Referring to <FIG>, the ToF camera module <NUM> includes a lighting unit <NUM>, a lens unit <NUM>, an image sensor unit <NUM>, a tilting unit <NUM>, and an image control unit <NUM>.

The lighting unit <NUM> generates an incident light signal and emits the generated incident light signal to an object. In this case, the lighting unit <NUM> may generate and output an incident light signal in the form of a pulse wave or a continuous wave. The continuous wave may be a sinusoidal wave or a squared wave. By generating an incident light signal in the form of a pulse wave or a continuous wave, the ToF camera module <NUM> may detect a phase difference between an incident light signal output from the lighting unit <NUM> and a reflected light signal reflected from an object. Herein, incident light may refer to light that is output from the lighting unit <NUM> and incident on an object, and reflected light may refer to light that is output from the lighting unit <NUM> and then reflected from an object after reaching the object. From the position of the ToF camera module <NUM>, the incident light may be output light, and the reflected light may be incident light.

The lighting unit <NUM> may emit the generated incident light signal to the object during a predetermined integration time. Here, the integration time refers to one frame period. When a plurality of frames are generated, the predetermined integration time is repeated. For example, when the ToF camera module <NUM> captures the object at <NUM> fps, the integration time is <NUM>/<NUM> sec. Also, when <NUM> frames are generated, the integration time may be repeated <NUM> times.

The lighting unit <NUM> may generate a plurality of incident light signals having different frequencies. The lighting unit <NUM> may sequentially and repeatedly generate a plurality of incident light signals having different frequencies. Alternatively, the lighting unit <NUM> may generate a plurality of incident light signals having different frequencies at the same time.

<FIG> is a diagram illustrating a frequency of an incident light signal. According to an embodiment of the present invention, the lighting unit <NUM> may control the first half of the integration time to generate an incident light signal having a frequency f<NUM> and may control the other half of the integration time to generate an incident light signal having a frequency f<NUM>.

According to another embodiment, the lighting unit <NUM> may control some of a plurality of light-emitting diodes to generate an incident light signal having a frequency f<NUM> and may control the other light-emitting diodes to generate an incident light signal having a frequency f<NUM>.

To this end, the lighting unit <NUM> may include a light source <NUM> configured to generate light and a light modulating unit <NUM> configured to modulate light.

First, the light source <NUM> generates light. The light generated by the light source <NUM> may be infrared light having a wavelength of <NUM> to <NUM> or may be visible light having a wavelength of <NUM> to <NUM>. The light source <NUM> may use a light-emitting diode (LED) and may have a plurality of LEDs arranged in a certain pattern. In addition, the light source <NUM> may include an organic light-emitting diode (OLED) or a laser diode (LD).

The light source <NUM> is repeatedly turned on and off at predetermined time intervals to generate an incident light signal in the form of a pulse wave or a continuous wave. A predetermined time interval may be the frequency of the incident light signal. The turning-on and off of the light source may be controlled by the light modulating unit <NUM>.

The light modulating unit <NUM> controls the turning-on and off of the light source <NUM> to control the light source <NUM> to generate an incident light signal in the form of a continuous wave or a pulse wave. The light modulating unit <NUM> may control the light source <NUM> to generate an incident light signal in the form of a continuous wave or a pulse wave through frequency modulation or pulse modulation.

Meanwhile, the lens unit <NUM> collects a reflected light signal reflected from the object and forwards the reflected light signal to the image sensor unit <NUM>.

<FIG> is an example cross-sectional view of a camera module.

Referring to <FIG>, the camera module <NUM> includes a lens assembly <NUM>, an image sensor <NUM>, and a printed circuit board <NUM>. Here, the lens assembly <NUM> may correspond to the lens unit <NUM> of <FIG>, and the image sensor <NUM> may correspond to the image sensor unit <NUM> of <FIG>. Also, the image control unit <NUM> and the like of <FIG> may be implemented in the printed circuit board <NUM>. Although not shown, the lighting unit <NUM> of <FIG> may be disposed on the side of the image sensor <NUM> on the printed circuit board <NUM> or may be disposed outside the camera module <NUM>.

The lens assembly <NUM> includes a lens <NUM>, a lens barrel <NUM>, a lens holder <NUM>, and an IR filter <NUM>.

The lens <NUM> may include a plurality of lens and may include one lens. When the lens <NUM> includes a plurality of lens, the lens may be arranged with respect to a central axis to form an optical system. Here, the central axis may be the same as an optical axis of the optical system.

The lens barrel <NUM> is coupled to the lens holder <NUM> to provide a space for accommodating lens. The lens barrel <NUM> may be rotatably coupled to one or a plurality of lenses, but this is just an example. Therefore, the lens barrel <NUM> and the lenses may be coupled in another way, such as a scheme using an adhesive (e.g., an adhesive resin such as epoxy).

The lens holder <NUM> may be coupled to the lens barrel <NUM> to support the lens barrel <NUM> and may be coupled to the printed circuit board <NUM> equipped with the image sensor <NUM>. The lens holder <NUM> may form a space for attachment of the IR filter <NUM> under the lens barrel <NUM>. A helical pattern may be formed on an inner circumferential surface of the lens holder <NUM>, and similarly, a helical pattern may be formed on an outer circumferential surface of the lens barrel <NUM>. Thus, the lens holder <NUM> and the lens barrel <NUM> may be rotatably coupled to each other. However, this is just an example, and the lens holder <NUM> and the lens barrel <NUM> may be coupled to each other through an adhesive or may be integrally formed.

The lens holder <NUM> may include an upper holder <NUM>-<NUM> to be coupled to the lens barrel <NUM> and a lower holder <NUM>-<NUM> to be coupled to the printed circuit board <NUM> equipped with the image sensor <NUM>. The upper holder <NUM>-<NUM> and the lower holder <NUM>-<NUM> may be formed integrally with each other, may be separated but can be engaged with or coupled to each other, or may be separated and spaced apart from each other. In this case, the upper holder <NUM>-<NUM> may have a smaller diameter than the lower holder <NUM>-<NUM>.

The above example is just an embodiment, and the lens unit <NUM> may be configured in another structure capable of collecting a reflected light signal incident on the ToF camera module <NUM> and forwarding the reflected light signal to the image sensor unit <NUM>.

Referring to <FIG> again, the image sensor unit <NUM> generates an electric signal using the reflected light signal collected through the lens unit <NUM>.

The image sensor unit <NUM> may be synchronized with the turning-on and off period of the lighting unit <NUM> to absorb the reflected light signal. In detail, the image sensor unit <NUM> may absorb the light in phase or out of phase with the incident light signal output from the lighting unit <NUM>. That is, the image sensor unit <NUM> may repeatedly perform a step of absorbing a reflected light signal while the light source is turned on and a step of absorbing a reflected light signal while the light source is turned off.

Subsequently, the image sensor unit <NUM> may use a plurality of reference signals with different phase differences to generate an electric signal corresponding to each reference signal. The frequency of the reference signal may be set to be the same as the frequency of the incident light signal output from the lighting unit <NUM>. Accordingly, when the lighting unit <NUM> generates incident light signals using a plurality of frequencies, the image sensor unit <NUM> generates electric signals using a plurality of reference signals corresponding to the frequencies. The electric signals may include information regarding electric charge quantities or voltages corresponding to the reference signals.

<FIG> is a diagram illustrating an electric signal generation process according to an embodiment of the present invention.

As shown in <FIG>, the reference signal according to an embodiment of the present invention may include four reference signals C<NUM> to C<NUM>. The reference signals C<NUM> to C<NUM> may have the same frequency as the incident light signal and have a phase difference of <NUM> degrees from one another. The reference signal C<NUM>, which is one of the four reference signals, may have the same phase as the incident light signal. A reflected light signal has a phase delayed by a distance traveled by an incident light signal incident on and returned from an object. The image sensor unit <NUM> mixes the reflected light signal with each of the reference signals. Thus, the image sensor unit <NUM> may generate an electric signal corresponding to a shaded portion of <FIG> for each reference signal.

In another embodiment, when incident light signals are generated using a plurality of frequencies during an integration time, the image sensor unit <NUM> absorbs reflected light signals corresponding to the plurality of frequencies. For example, it is assumed that incident light signals having frequencies f<NUM> and f<NUM> are generated, and the plurality of reference signals have a phase difference of <NUM> degrees from one another. In this case, reflected light signals also have frequencies f<NUM> and f<NUM>. Thus, four electric signals may be generated using the reflected light signal with the frequency f<NUM> and corresponding four reference signals. Also, four electric signals may be generated using the reflected light signal with the frequency f<NUM> and corresponding four reference signals. Accordingly, a total of eight electric signals may be generated.

The image sensor unit <NUM> may be configured in a structure in which a plurality of pixels are arranged in a grid form. The image sensor unit <NUM> may be a complementary metal oxide semiconductor (CMOS) image sensor or a charged coupled device (CCD) image sensor. Also, the image sensor unit <NUM> includes a ToF sensor configured to receive infrared light reflected from a subject and measure a distance from the subject using a traveled time or a phase difference.

<FIG> is a diagram illustrating an image sensor <NUM> according to an embodiment of the present invention. For example, for an image sensor <NUM> with a resolution of <NUM>×<NUM> as shown in <FIG>, <NUM>,<NUM> pixels are arranged in a grid form. In this case, a predetermined interval, such as a shaded portion of <FIG>, may be formed between the plurality of pixels. In an embodiment of the present invention, one pixel refers to a pixel and a predetermined interval adjacent to the pixel.

According to an embodiment of the present invention, each pixel <NUM> may include a first light receiving unit <NUM>-<NUM> including a first photodiode and a first transistor and a second light receiving unit <NUM>-<NUM> including a second photodiode and a second transistor.

The first light receiving unit <NUM>-<NUM> receives a reflected light signal in the same phase as the waveform of the incident light. That is, while the light source is turned on, the first photodiode is turned on to absorb a reflected light signal. Also, while the light source is turned off, the first photodiode is turned off to stop absorbing a reflected light signal. The first photodiode converts the absorbed reflected light signal into an electric current and forwards the electric current to the first transistor. The first transistor converts the forwarded electric current into an electric signal and outputs the electric signal.

The second light receiving unit <NUM>-<NUM> receives a reflected light signal in the opposite phase to the waveform of the incident light. That is, while the light source is turned on, the second photodiode is turned off to absorb a reflected light signal. Also, while the light source is turned off, the second photodiode is turned on to stop absorbing a reflected light signal. The second photodiode converts the absorbed reflected light signal into an electric current and forwards the electric current to the second transistor. The second transistor converts the forwarded electric current into an electric signal.

Thus, the first light receiving unit <NUM>-<NUM> may be referred to as an in-phase receiving unit, and the second light receiving unit <NUM>-<NUM> may be referred to as an out-of-phase receiving unit. When, as described above, the first receiving unit <NUM>-<NUM> and the second light receiving unit <NUM>-<NUM> are activated with a time difference, the amount of light received may vary depending on the distance to the object. For example, when the object is in front of the ToF camera module <NUM> (i.e., the distance is equal to zero), the time it takes for light to be reflected from the object after the light is output from the lighting unit <NUM> is zero, and thus the turning-on and off period of the light source becomes a light receiving period with no changes. Accordingly, only the first light receiving unit <NUM>-<NUM> can receive light, and the second light receiving unit <NUM>-<NUM> cannot receive light. As another example, when the object is located a predetermined distance away from the ToF camera module <NUM>, it takes time for light to be reflected from the object after the light is output from the lighting unit <NUM>, and thus the turning-on and off period of the light source becomes different from a light receiving period. Accordingly, the amount of light received by the first light receiving unit <NUM>-<NUM> becomes different from that of the second light receiving unit <NUM>-<NUM>. That is, the distance to the object may be calculated using the difference between the amount of light input to the first light receiving unit <NUM>-<NUM> and the amount of light input to the second light receiving unit <NUM>-<NUM>. Referring to <FIG> again, the image control unit <NUM> calculates a phase difference between incident light and reflected light using an electric signal received from the image sensor unit <NUM> and calculates a distance between the object and the ToF camera module <NUM> using the phase difference.

In detail, the image control unit <NUM> calculates a phase difference between incident light and reflected light using information regarding electric charge quantity of the electric signal.

As described above, four electric signals may be generated for each frequency of the incident light signal. Therefore, the image control unit <NUM> may compute a phase difference td between the incident light signal and the reflected light signal using Equation <NUM> below: <MAT>
where Q<NUM> to Q<NUM> are electric charge quantities of four electric signals. Q<NUM> is an electric charge quantity of an electric signal corresponding to a reference signal having the same phase as the incident light signal. Q<NUM> is an electric charge quantity of an electric signal corresponding to a reference signal having a phase lagging by <NUM> degrees from the incident light signal. Q<NUM> is an electric charge quantity of an electric signal corresponding to a reference signal having a phase lagging by <NUM> degrees from the incident light signal. Q<NUM> is an electric charge quantity of an electric signal corresponding to a reference signal having a phase lagging by <NUM> degrees from the incident light signal.

Thus, the image control unit <NUM> may calculate a distance between the object and the ToF camera module <NUM> using the phase difference between the incident light signal and the reflected light signal. In this case, the image control unit <NUM> may compute a distance d between the object and the ToF camera module <NUM> using Equation <NUM> below: <MAT>
where c is the speed of light, and f is the frequency of incident light.

Meanwhile, in an embodiment of the present invention, a super resolution (SR) technique is used to increase the resolution of a depth map. The SR technique is a technique for obtaining a high-resolution image from a plurality of low-resolution images, and a mathematical model of the SR technique may be expressed using Equation <NUM> below:
<MAT>
where l≤k≤p, p is the number of low-resolution images, yk is a low-resolution image (=[yk,<NUM>, yk,<NUM>,. , yk,M]T; here, M=N<NUM>*Ny), Dk is a down sampling matrix, Bk is an optical blur matrix, Mk is an image warping matrix, x is a high-resolution image (=[x<NUM>, x<NUM>,. , xN]T; here, N=L<NUM>N<NUM>*L<NUM>N<NUM>), and nk is noise. That is, the SR technique refers to a technique for estimating x by applying the inverse function of the estimated resolution degradation factors to yk. The SR technique may be largely divided into a statistical scheme and a multi-frame scheme, and the multi-frame scheme may be largely divided into a space division scheme and a time division scheme. When the SR technique is used to acquire a depth map, the inverse function of Mk of Equation <NUM> is not present, and thus the statistical scheme may be tried. However, the statistical scheme requires a repeated computation process and thus has low efficiency.

In order to apply the SR technique to depth map extraction, the image control unit <NUM> may generate a plurality of low-resolution subframes using an electric signal received from the image sensor unit <NUM> and then may extract a plurality of low-resolution depth maps using the plurality of low-resolution subframes. Also, the image control unit <NUM> may rearrange pixel values of the plurality of low-resolution depth maps to extract a high-resolution depth map.

Here, the term "high resolution" has a relative meaning that represents a higher resolution than "low resolution.

Here, the term "subframe" may refer to image data generated from any integration time and an electric signal corresponding to a reference signal. For example, when an electric signal is generated using eight reference signals during a first integration time, i.e., one image frame, eight subframes may be generated, and one start frame may be further generated. Herein, a subframe may be used interchangeably with image data, subframe image data, etc..

Alternatively, in order to apply the SR technique according to an embodiment of the present invention to depth map extraction, the image control unit <NUM> may generate a plurality of low-resolution subframes using an electric signal received from the image sensor unit <NUM> and then may rearrange pixel values of the plurality of low-resolution subframes to generate a plurality of high-resolution subframes. Also, the image control unit <NUM> may extract a high-resolution depth map using the high-resolution subframes.

To this end, a pixel shift technique may be used. That is, the image control unit <NUM> may acquire several sheets of image data shifted by a subpixel for each subframe using the pixel shift technique, acquire a plurality of pieces of high-resolution subframe image data by applying the SR technique for each subframe, and extract a high-resolution depth map using the high-resolution subframe image data. In order to perform pixel shift, the ToF camera module <NUM> according to an embodiment of the present invention includes the tilting unit <NUM>.

Referring to <FIG> again, the tilting unit <NUM> changes an optical path of at least one of an incident light signal or a reflected light signal in units of subpixels of the image sensor unit <NUM>.

For each image frame, the tilting unit <NUM> changes an optical path of at least one of an incident light signal or a reflected light signal. As described above, one image frame may be generated at every integration time. Accordingly, when one integration time ends, the tilting unit <NUM> changes an optical path of at least one of an incident light signal or a reflected light signal.

The tilting unit <NUM> changes an optical path of an incident light signal or a reflected light signal in units of subpixels with respect to the image sensor unit <NUM>. In this case, the tilting unit <NUM> changes an optical path of at least one of an incident light signal or a reflected light signal upward, downward, leftward or rightward with respect to the current optical path.

<FIG> is a diagram illustrating that the tilting unit <NUM> changes an optical path of a reflected light signal.

In <FIG>, a portion indicated by solid lines indicates a current optical path of the reflected light signal, and a portion indicated by dotted lines indicates a changed optical path. When an integration time corresponding to the current optical path ends, the tilting unit <NUM> may change the optical path of the reflected light signal as represented by dotted lines. Thus, the path of the reflected light signal is shifted by a subpixel from the current optical path. For example, as shown in <FIG>, when the tilting unit <NUM> shifts the current optical path to the right by <NUM> degrees, the reflected light signal incident on the image sensor unit <NUM> may be shifted to the right by <NUM> pixels (subpixels).

According to an embodiment of the present invention, the tilting unit <NUM> may change an optical path of a reflected light signal clockwise with respect to a reference position. For example, as shown in <FIG>, after a first integration time ends, the tilting unit <NUM> shifts the optical path of the reflected light signal to the right by <NUM> pixels with respect to the image sensor unit <NUM> during a second integration time. Also, the tilting unit <NUM> shifts the optical path of the reflected light signal downward by <NUM> pixels with respect to the image sensor unit <NUM> during a third integration time. Also, the tilting unit <NUM> shifts the optical path of the reflected light signal leftward by <NUM> pixels with respect to the image sensor unit <NUM> during a fourth integration time. Also, the tilting unit <NUM> shifts the optical path of the reflected light signal upward by <NUM> pixels with respect to the image sensor unit <NUM> during a fifth integration time. That is, the tilting unit <NUM> may shift the optical path of the reflected light signal to its original position during four integration times. This can be applied in the same way even when an optical path of an incident light signal is shifted, and a detailed description thereof will be omitted. Also, the optical path change pattern being clockwise is just an example, and the optical path change pattern may be counterclockwise.

Meanwhile, the subpixel may be greater than zero pixels and smaller than one pixel. For example, the subpixel may have a size of <NUM> pixels and may have a size of <NUM>/<NUM> pixels. The size of the subpixel can be changed in design by a person skilled in the art.

<FIG> are diagrams illustrating an SR technique according to an embodiment of the present invention.

Referring to <FIG>, the image control unit <NUM> may extract a plurality of low-resolution depth maps using a plurality of low-resolution sub-frames generated during the same integration time, i.e., during the same frame. Also, the image control unit <NUM> may rearrange pixel values of the plurality of low-resolution depth maps to extract a high-resolution depth map. Here, optical paths of incident light signals or reflected light signals corresponding to the plurality of low-resolution depth maps may be different from each other.

For example, the image control unit <NUM> may generate low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> using a plurality of electric signals. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the first integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the second integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the third integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the fourth integration time. Thus, the image control unit <NUM> applies a depth map extraction technique to the plurality of low-resolution subframes generated for each integration time to extract low-resolution depth maps LRD-<NUM> to LRD-<NUM>. Low-resolution depth map LRD-<NUM> is a low-resolution depth map extracted using subframes <NUM>-<NUM> to <NUM>-<NUM>. Low-resolution depth map LRD-<NUM> is a low-resolution depth map extracted using subframes <NUM>-<NUM> to <NUM>-<NUM>. Low-resolution depth map LRD-<NUM> is a low-resolution depth map extracted using subframes <NUM>-<NUM> to <NUM>-<NUM>. Low-resolution depth map LRD-<NUM> is a low-resolution depth map extracted using subframes <NUM>-<NUM> to <NUM>-<NUM>. Also, the image control unit <NUM> rearranges pixel values of low-resolution depth maps LRD-<NUM> to LRD-<NUM> to extract high-resolution depth map HRD.

Alternatively, as described above, the image control unit <NUM> may rearrange pixel values of a plurality of subframes corresponding to the same reference signal to generate a high-resolution subframe. In this case, the plurality of subframes have different optical paths of corresponding incident light signals or reflected light signals. Also, the image control unit <NUM> may extract a high-resolution depth map using a plurality of high-resolution subframes.

For example, as shown in <FIG>, the image control unit <NUM> generates low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> using a plurality of electric signals. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the first integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the second integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the third integration time. Low-resolution subframes <NUM>-<NUM> to <NUM>-<NUM> are low-resolution subframes generated during the fourth integration time. Here, low-resolution subframes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> correspond to the same reference signal C<NUM> and different optical paths. Then, the image control unit <NUM> may rearrange pixel values of low-resolution subframes <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> to generate high-resolution subframe H-<NUM>. When high-resolution subframes H1 to H8 are generated through the rearrangement of the pixel values, the image control unit may apply the depth map extraction technique to high-resolution subframes H-<NUM> to H-<NUM> to extract a high-resolution depth map HRD.

<FIG> is a diagram illustrating a pixel value arrangement process according to an embodiment of the present invention.

Here, it is assumed that four low-resolution images having a size of <NUM>×<NUM> are used to generate one high-resolution image having a size of <NUM>×<NUM>. In this case, the high-resolution pixel grid has <NUM>×<NUM> pixels, which are the same as pixels of a high-resolution image. Here, the low-resolution image may have a meaning including a low-resolution subframe and a low-resolution depth map, and the high-resolution image may have a meaning including a high-resolution subframe and a high-resolution depth map.

In <FIG>, first to fourth low-resolution images are images captured when an optical path is shifted in units of a subpixel with a <NUM>-pixel size. The image control unit <NUM> arranges pixel values of the second to fourth low-resolution images to fit the high-resolution image in a direction in which the optical path is shifted with respect to the first low-resolution image in which the optical path is not shifted.

In detail, the second low-resolution image is an image shifted to the right by a subpixel from the first low-resolution image. Therefore, a pixel B of the second low-resolution image is arranged in a pixel located to the right of each pixel A of the first low-resolution image.

The third low-resolution image is an image shifted downward by a subpixel from the second low-resolution image. Therefore, a pixel C of the third low-resolution image is arranged in a pixel located under each pixel B of the second low-resolution image.

The fourth low-resolution image is an image shifted to the left by a subpixel from the third low-resolution image. Therefore, a pixel D of the fourth low-resolution image is arranged in a pixel located to the left of the pixel C of the third low-resolution image.

When all pixel values of the first to fourth low-resolution images are rearranged in a high-resolution pixel grid, a high-resolution image frame which has a resolution four times that of a low-resolution image is generated.

Meanwhile, the image control unit <NUM> may apply a weight value to an arranged pixel value. In this case, the weight value may be set differently depending on the size of the subpixel or the shift direction of the optical path and may be set differently for each low-resolution image.

To this end, the tilting unit <NUM> may change the optical path through software or hardware. The amount of calculation of the ToF camera module <NUM> increases when the tilting unit <NUM> changes the optical path through software, and the ToF camera module <NUM> becomes complicated in structure or increases in volume when the tilting unit <NUM> change the optical path through hardware.

According to an embodiment of the present invention, the tilting unit <NUM> obtains data shifted by a subpixel using a method of controlling the slope of a lens assembly, e.g., an IR filter <NUM> (see <FIG>) included in the lens assembly.

<FIG> and <FIG> are diagrams illustrating an effect of shifting an image frame input to an image sensor by controlling the slope of an IR filter. <FIG> shows a result of simulating a distance shifted for a tilting angle under the condition that the thickness of the IR filter is <NUM> and that the refractive index of IR is <NUM>.

Referring to <FIG> and the following Equation <NUM>, the shifted distance and the slope θ<NUM> of the IR filter <NUM> may have the following relationship.

When the slope of the IR filter <NUM> is controlled as described above, it is possible to obtain shifted image data without tilting the image sensor <NUM>.

According to an embodiment of the present invention, the tilting unit for controlling the slope of the IR filter may include a voice coil motor (VCM), and the IR filter <NUM> may be disposed between the image sensor and the VCM.

<FIG> is a perspective view of a VCM and an IR filter according to an embodiment of the present invention, <FIG> is a cross-sectional view of a ToF camera module including a VCM and an IR filter according to an embodiment of the present invention, <FIG> is a diagram showing a process of coupling an IR filter and a magnet assembly included in a VCM according to an embodiment of the present invention, <FIG> is a diagram showing a coupling process of a coil assembly included in a VCM according to an embodiment of the present invention, and <FIG> is a diagram showing a process of coupling a magnet assembly, an IR filter, and a coil assembly according to an embodiment of the present invention.

Referring to <FIG>, the tilting unit <NUM> includes a VCM <NUM>, and the VCM <NUM> includes a magnet assembly <NUM> and a coil assembly <NUM> and is coupled to, brought into contact with, or connected to the IR filter <NUM>.

In <FIG>, for convenience of description, it is shown that the VCM <NUM> is surrounded by the lens barrel <NUM> and the lens holder <NUM> and that the lens <NUM> and the IR filter <NUM> are omitted. However, the lens <NUM> and the IR filter <NUM> may be arranged as shown in <FIG>. That is, the lens <NUM> may be surrounded by the lens barrel <NUM> or may be accommodated in a space of the VCM <NUM>. Alternatively, the lens barrel <NUM> may be an element of the VCM <NUM>.

According to an embodiment of the present invention, the magnet assembly <NUM> includes a magnet holder <NUM> and a plurality of magnets <NUM>, and the plurality of magnets <NUM> may be spaced apart on the magnet holder <NUM> at predetermined intervals. For example, the magnet holder <NUM> may have a hollow circular ring shape or a quadrilateral ring shape, and a plurality of magnet guides <NUM> is formed to accommodate the plurality of magnets <NUM>. Here, the magnet holder <NUM> contains a magnetic material or a soft magnetic material, e.g., Fe.

Subsequently, the coil assembly <NUM> may include a coil holder <NUM>, a plurality of coils <NUM>, and a coil terminal <NUM>, and the plurality of coils <NUM> may be disposed on the coil holder <NUM> and spaced apart from one another at predetermined intervals to make pairs with the plurality of magnets. For example, the coil holder <NUM> may have a hollow circular ring shape or a quadrilateral ring shape, and a plurality of coil guides <NUM> may be formed to accommodate the plurality of coils <NUM>. The coil holder <NUM> may be the lens barrel <NUM>. The coil terminal <NUM> may be connected to the plurality of coils <NUM> and may apply power to the plurality of coils <NUM>.

The IR filter <NUM> includes a glass layer holder <NUM> and a glass layer <NUM> supported by the glass layer holder <NUM>. The glass layer holder <NUM> includes a first glass layer holder <NUM>-<NUM> disposed under the glass layer <NUM> and a second glass layer holder <NUM>-<NUM> disposed on an upper edge of the glass layer <NUM>. The second glass layer holder <NUM>-<NUM> may have a hollow circular ring shape or a quadrilateral ring shape and may be disposed in a hollow of the magnet holder <NUM> and surrounded by the magnet holder <NUM>. In this case, the second glass layer holder <NUM>-<NUM> includes a plurality of protrusions P1, P2, P3, and P4 corresponding to the plurality of magnet guides <NUM> of the magnet holder <NUM>. The plurality of protrusions P1, P2, P3, and P4 may be moved such that the protrusions are brought into contact with or spaced apart from the plurality of magnet guides <NUM>. The second glass layer holder <NUM>-<NUM> may contain a magnetic material or a soft magnetic material.

When power is applied to the plurality of coils <NUM> through the coil terminal <NUM>, an electric current flows through the plurality of coils <NUM>, and thus it is possible to generate a magnetic field between the plurality of coils <NUM> and the plurality of magnets <NUM>.

Thus, an electric driving force is generated between the plurality of magnet guides <NUM> and the plurality of protrusions P1, P2, P3, and P4 of the second glass layer holder <NUM>-<NUM>, and the glass layer <NUM> supported by the second glass layer holder <NUM>-<NUM> may be tilted at a predetermined angle.

For example, a slope formed between the protrusion P1 and the protrusion P3 or a slope formed between the protrusion P2 and the protrusion P4 may vary depending on a force applied between the plurality of magnet guides <NUM> and the plurality of protrusions P1, P2, P3, and P4. Also, the slope of the glass layer <NUM> may vary depending on the slope formed between the protrusion P1 and the protrusion P3 or the slope formed between the protrusion P2 and the protrusion P4. Here, the slope of the IR filter <NUM>, and particularly, the slope of the glass layer <NUM> varies depending on the positions of the plurality of protrusions P1, P2, P3, and P4 of the second glass layer holder <NUM>-<NUM>. Accordingly, the second glass layer holder <NUM>-<NUM> may be referred to herein as a shaper.

In this case, for the degree of freedom of tilting of the glass layer <NUM>, a spacer <NUM> may be further disposed between the magnet holder <NUM> and the first glass layer holder <NUM>-<NUM>.

Here, the glass layer <NUM> may be an IR-pass glass layer.

Alternatively, as shown in <FIG>, the glass layer <NUM> may be a general glass layer, and the IR filter <NUM> may further include an IR pass glass layer <NUM> spaced apart from the glass layer <NUM> and disposed on the image sensor <NUM>. When the IR pass glass layer <NUM> is disposed on the image sensor <NUM>, it is possible to reduce the possibility of moisture or foreign matter directly penetrating into the image sensor <NUM>.

Meanwhile, according to an embodiment of the present invention, the magnet assembly <NUM> may further include a magnet holder <NUM>. The magnet holder <NUM> may support upper portions of the plurality of magnets <NUM>, and thus the plurality of magnets <NUM> may move more stably and reliably.

As described above, according to an embodiment of the present invention, the slope of the IR filter <NUM> may be controlled according to the driving of the VCM <NUM>. To this end, the IR filter <NUM> should be disposed together with the VCM <NUM>, and thus the IR filter <NUM> needs to be spaced apart from the image sensor <NUM>.

Meanwhile, according to an embodiment of the present invention, the slope of the IR filter <NUM> needs to be frequently changed, and thus a free space for the movement of the IR filter <NUM> is required. In this case, the possibility of moisture, foreign matter, and the like penetrating into the free space for the movement of the IR filter <NUM> increases, and thus the image sensor <NUM> may be easily exposed to moisture or foreign matter.

In an embodiment of the present invention, a component for preventing the image sensor <NUM> from being exposed to moisture, foreign matter, and the like may be further included.

<FIG> is a cross-sectional view of a portion of a camera module according to an embodiment of the present invention. Here, for convenience of description, an upper portion of the camera module, e.g., the lens, the lens barrel, the VCM, and the like are omitted, but the description of <FIG> and <FIG> may be equally applied.

Referring to <FIG>, an image sensor <NUM> may be mounted on a printed circuit board <NUM> and accommodated in a housing <NUM>. Here, the housing may be a second lens holder <NUM>-<NUM>. The slope of the IR filter <NUM> may be controlled by the VCM <NUM> (see <FIG>). For example, when a first protrusion P1 of a first glass layer holder <NUM>-<NUM> faces upward and the third protrusion P3 faces downward due to the driving of the VCM <NUM>, a glass layer <NUM> of an IR filter <NUM> may be tilted.

According to an embodiment of the present invention, an elastic film <NUM> may be disposed between the IR filter <NUM> and the image sensor <NUM>. The elastic film <NUM> may be fastened to the housing <NUM>. In this case, one face of the elastic film <NUM> may be fastened to the housing <NUM>, and the other face of the elastic film <NUM> may be coupled to the tilting unit <NUM>. The elastic film <NUM> may be, for example, a reverse osmosis (RO) membrane, a nano filtration (NF) membrane, an ultra-filtration (UF) membrane, a micro filtration (MF) membrane, or the like. Here, the RO membrane is a membrane having a pore size of about <NUM> to <NUM> angstroms, the NF membrane is a membrane having a pore size of about <NUM> angstroms, the UF membrane is a membrane having a pore size of about <NUM> to <NUM> angstroms, and the MF membrane is a membrane having a pore size of about <NUM> to <NUM> angstroms. Accordingly, it is possible to prevent moisture, foreign matter, and the like from penetrating into a space between the IR filter <NUM> and the housing <NUM>, that is, the space arranged for the movement of the IR filter <NUM>.

In this case, the elastic film <NUM> may be a transparent and stretchable film with a thickness of <NUM> to <NUM>, and the IR filter <NUM> may be disposed on the elastic film <NUM> so that at least a portion of the IR filter <NUM> can be in direct contact with the elastic film <NUM>. That is, the shape of the elastic film <NUM> may be controlled by the tilting unit <NUM>. Thus, when the IR filter <NUM> is inclined, the elastic film <NUM> may be stretched or contracted together with the IR filter <NUM>. When the IR filter <NUM> returns to its original position, the elastic film <NUM> may be restored immediately along with the IR filter <NUM>. Accordingly, it is possible to stably support the movement of the IR filter <NUM>.

<FIG> show various examples of placing an elastic film.

Referring to <FIG>, the elastic film <NUM> may be adhered to the housing <NUM> for accommodating the image sensor <NUM> through an adhesive <NUM>.

Referring to <FIG>, the elastic film <NUM> may be fastened to the housing <NUM> for housing the image sensor <NUM> through an instrument <NUM>.

Referring to <FIG>, the elastic film <NUM> may be disposed to cover the outer circumferential surface of the housing <NUM> for accommodating the image sensor <NUM>. In order to fasten the elastic film <NUM>, an additional fastening member <NUM> may be disposed to surround the outer circumferential surface of the housing <NUM>.

Referring to <FIG>, the elastic film <NUM> may be disposed directly on the image sensor <NUM>.

Referring to <FIG>, the elastic film <NUM> may be disposed between the first glass layer holder <NUM>-<NUM> and the housing <NUM> and fastened by instructions <NUM> and <NUM>.

Referring to <FIG>, the elastic film <NUM> may be adhered to the first glass layer holder <NUM>-<NUM> and the housing <NUM> through adhesives <NUM> and <NUM>.

Claim 1:
A camera module (<NUM>) comprising:
a lighting unit (<NUM>) configured to output an incident light signal to be emitted to an object;
a lens unit (<NUM>) configured to collect a reflected light signal reflected from the object;
an image sensor unit (<NUM>) configured to generate an electric signal from the reflected light signal collected by the lens unit (<NUM>); and
an image control unit (<NUM>) configured to extract a depth information of the object using a phase difference between the incident light signal and the reflected light signal received by the image sensor unit (<NUM>),
wherein the lens unit (<NUM>) is disposed on the image sensor unit (<NUM>), and the lens unit (<NUM>) comprises an infrared, IR, filter (<NUM>) disposed on the image sensor unit (<NUM>) and at least one lens disposed on the IR filter (<NUM>),
characterized in that:
the camera module(<NUM>) further comprises
a tilting unit (<NUM>) configured to shift an optical path of the reflected light signal,
the image control unit (<NUM>) is configured to extract the depth information of the object from a plurality of frames corresponding to optical paths shifted by the tilting unit (<NUM>),
the tilting unit (<NUM>) controls a slope of the IR filter (<NUM>),
the tilting unit (<NUM>) comprises a voice coil motor, VCM (<NUM>), the IR filter (<NUM>) is disposed between the image sensor unit and the VCM (<NUM>),
the VCM (<NUM>) comprises: a magnet holder (<NUM>); a plurality of magnets disposed on the magnet holder (<NUM>) and spaced apart from one another at predetermined intervals; a coil holder (<NUM>); and a plurality of coils disposed on the coil holder (<NUM>) and spaced apart from one another at predetermined intervals to make pairs with the plurality of magnets,
the IR filter (<NUM>) comprises a glass layer and a glass layer holder (<NUM>) configured to support the glass layer,
at least a portion of the glass layer holder (<NUM>) is surrounded by the magnet holder (<NUM>),
the magnet holder (<NUM>) includes a plurality of magnet guides (<NUM>) for accommodating the plurality of magnets,
the glass layer holder (<NUM>) includes a plurality of protrusions corresponding to the plurality of magnet guides (<NUM>), and
the plurality of protrusions move using a force applied between the plurality of protrusions and the plurality of magnet guides (<NUM>) by a magnetic field between the plurality of coils and the plurality of magnets to change the slope of the IR filter.