Patent ID: 12219271

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described with reference to the attached drawings.

FIG.1is a block diagram illustrating an image system according to an embodiment.

Referring toFIG.1, an image sensing device1according to an embodiment includes an image sensor100and an application processor AP200. The image sensing device1may be implemented as a portable electronic device such as a digital camera, a camcorder, a mobile phone, a smart phone, a tablet personal computer (PC), a personal digital assistant (PDA), a mobile Internet device (MID), a wearable computer, an Internet of things (IoT) device, or an Internet of everything (IoE) device.

The image sensor100may sense an object101photographed via a lens103under the control of an application processor200. The image sensor100may convert an optical signal of the object101incident via the lens103into an electrical signal using a photo-sensing element (or a photoelectric conversion element), and generate and output image data based on the electrical signal.

The image sensor100may include a pixel array112, a row driver120(e.g., a driver circuit), a correlated double sampling (CDS) block130(e.g., a logic circuit), an analog-to-digital converter (ADC) block140, a ramp signal generator150, a timing generator160, a control register block170, a buffer180, a first image signal processor192, and a second image signal processor194.

The pixel array112may include a plurality of pixels arranged in a matrix form. Each of the plurality of pixels may sense light using a light sensing element (or device) and convert the sensed light into a pixel signal that is an electrical signal. Thus, the pixel array112may include a plurality of light sensing elements. For example, the light sensing element may be a photo diode, a photo transistor, a photo gate, a pinned photo diode (PPD), or a combination thereof. Each of the plurality of light sensing elements may have a 4-transistor structure including a photodiode, a transmission transistor, a reset transistor, an amplification transistor and a selection transistor. According to an embodiment, each of the plurality of light sensing elements may have a 1-transistor structure, a 3-transistor structure, a 5-transistor structure, or a structure in which a plurality of pixels share some transistors.

A color filter array114may be disposed on the pixel array112. The pixel array112and the color filter array114will be described in detail with reference toFIG.2.

The row driver120may activate each of the plurality of pixels under the control of the timing generator160. For example, the row driver120may drive pixels implemented in the pixel array112in units of rows. For example, the row driver120may generate control signals capable of controlling the operation of the plurality of pixels included in each of a plurality of rows.

According to the control signals, pixel signals output from each of the plurality of pixels are transmitted to the CDS block130.

The CDS block130may include a plurality of CDS circuits. Each of the plurality of CDS circuits may perform correlated double sampling on pixel values output from each of the plurality of column lines implemented in the pixel array112in response to at least one switch signal output from the timing generator160, and compare the correlated double sampled pixel values with ramp signals output from the ramp signal generator150to output a plurality of comparison signals.

The ADC block140may convert each of a plurality of the comparison signals output from the CDS block130into digital signals and output the plurality of digital signals to the buffer180.

The timing generator160may generate a signal that is a reference of operation timing of various components of the image sensor100. An operation timing reference signal generated by the timing generator160may be transmitted to the row driver120, the CDS block130, the ADC block140, and the ramp signal generator150.

The control register block170may control an entire operation of the image sensor100. The control register block170may control operations of the ramp signal generator150, the timing generator160and the buffer180.

The buffer180may output image row data and phase row data corresponding to the plurality of digital signals output from the ADC block140. The image row data may be generated from a normal pixel, and the phase row data may be generated from a phase detection pixel.

Each of the first image signal processor192and the second image signal processor194may output image data IDATA Gi and Ci and phase detection data Li and Ri by performing image processing on each of the image row data and the phase row data. For example, each of the first image signal processor192and the second image signal processor194may perform image processing to change a data format for each image data and each phase detection data (e.g., changing image data in a Bayer pattern to a YUV or RGB format), and image processing to improve image quality such as noise removal, brightness adjustment and sharpness adjustment.

The first image signal processor192and the second image signal processor194may be implemented with hardware of the image sensor100. Alternatively, the first image signal processor192and the second image signal processor194may be disposed outside the image sensor100or inside the application processor200.

The application processor200may receive the image data IDATA Gi and Ci and the phase detection data Li and Ri from the image sensor100via an interface. In an embodiment, the application processor200receives first image data IDATA from the image sensor100via a first channel VC0. The application processor200may receive the phase detection data Li and Ri and second image data Gi and Ci together from the image sensor100via a second channel VC1. In other words, the application processor200may receive a pair of phase detection data Li and Ri and second image data Gi and Ci via the second channel VC1. Hereinafter, it will be described in detail with reference toFIG.3.

The application processor200may perform post-processing on the image data IDATA Gi and Ci and the phase detection data Li and Ri. The post-processing may refer to an application of an image enhancement algorithm to remove artifacts. For example, the application processor200may perform an image post-processing operation of adjusting image parameters such as brightness, contrast, gamma, and luminance with respect to the image data IDATA Gi and Ci generated from a normal pixel NPX (seeFIG.2) as will be described below. For example, the image post-processing operation may include a variety of operations for improving image quality, such as a noise reduction operation, gamma correction, color filter array interpolation, a color matrix, color correction and color enhancement. Then, the application processor200may generate an image file by performing a compression operation and may restore the image data from the image file.

In an embodiment, the application processor200includes a calculating module210. The calculating module210may calculate a disparity (i.e., a phase difference) with respect to the phase detection data Li and Ri generated from phase detection pixels PD1and PD2as will be described below. The calculating module210may be implemented with, for example, software or firmware executed on the application processor200. Alternatively, the calculating module210may be implemented with hardware.

Based on the disparity calculation result, the application processor200may obtain a location of a focus, a direction of a focus, or a distance between the object101and the image sensor100. The application processor200may output the control signal to a lens driver to move a location or angle of a lens103based on the disparity calculation result.

The application processor200may be a central processing unit (CPU), a microprocessor or a microcontroller unit (MCU).

FIG.2is a view illustrating an embodiment of the pixel array (e.g.,112) and the color filter array (e.g.,114) inFIG.1.

Referring toFIG.2, a pixel array112aaccording to an embodiment includes a plurality of normal pixels NPX and a plurality of phase detection pixel groups PG. For example, the plurality of phase detection pixel groups PG may be regularly arranged along a first direction DR1and/or a second direction DR2.

Each of the phase detection pixel groups PG may include a plurality of phase detection pixels PD1and PD2disposed adjacent to each other. For example, each of the phase detection pixel groups PG may include a first phase detection pixel PD1and a second phase detection pixel PD2disposed adjacent to each other. The first phase detection pixel PD1and the second phase detection pixel PD2may be disposed adjacent to each other, for example, in the first direction DR1. However, the present disclosure is not limited thereto, and the first phase detection pixel PD1and the second phase detection pixel PD2may be disposed adjacent to each other in the second direction DR2. Alternatively, the first phase detection pixel and the second phase detection pixel included in a first phase detection pixel group PG may be disposed adjacent to each other in the first direction DR1, and the first phase detection pixel PD1and the second phase detection pixel PD2included in a second phase detection pixel group may be disposed adjacent to each other in the second direction DR2.

A color filter array114amay be disposed on the pixel array112a. For example, the color filter array114amay overlap the pixel array112ain a plan view. The color filter array114amay include a plurality of unit groups UG1, UG2, UG3and UG4. In an embodiment, each of the unit groups UG1, UG2, UG3and UG4include color filters of the same color arranged in an M×N matrix, where M and N are natural numbers. For example, each of the unit groups UG1, UG2, UG3and UG4may include color filters of a same color arranged in a 3×3 matrix. For example, each of the unit groups UG1, UG2, UG3and UG4may include unit color filters of a same color arranged into several rows and columns. Each of the color filters may be disposed to correspond to the normal pixel NPX, the first phase detection pixel PD1and the second phase detection pixel PD2, respectively. For example, each of the unit groups may be disposed on pixels arranged in a 3×3 matrix among the pixels including the normal pixel NPX, the first phase detection pixel PD1and the second phase detection pixel PD2.

The plurality of unit groups UG1, UG2, UG3and UG4may include, for example, a first unit group UG1and a second unit group UG2disposed adjacent to each other in the first direction DR1, and a third unit group UG3and a fourth unit group UG4disposed adjacent to each other in the first direction DR1. The first unit group UG1and the third unit group UG3may be disposed adjacent to each other in the second direction DR2, and the second unit group UG2and the fourth unit group UG4may be disposed adjacent to each other in the second direction DR2. The second direction DR2may intersect the first direction DR1. The first unit group UG1may include a first color filter R, the second unit group UG2may include a second color filter Gr, the third unit group UG3may include a third color filter Gb, and the fourth unit group UG4may include a fourth color filter B. The first color filter R may be a red color filter, the second and third color filters Gr and Gb may be green color filters, and the fourth color filter B may be a blue color filter.

In some embodiments, different unit groups UG1, UG2, UG3and UG4may be disposed on the first phase detection pixel PD1and the second phase detection pixel PD2. The color filter disposed on the first phase detection pixel PD1and the color filter disposed on the second phase detection pixel PD2may be color filters with different colors. For example, the color filter Gr disposed on the first phase detection pixel PD1may be a green color filter, and the color filter R disposed on the second phase detection pixel PD2may be a red color filter.

A plurality of first micro-lenses ML1may be disposed on the plurality of normal pixels NPX. Each of the first micro-lenses ML1may cover each of the normal pixels NPX. For example, the first micro-lenses ML1may overlap the normal pixels NPX in a plan view. A plurality of second micro-lenses ML2may be disposed on a plurality of phase detection pixel groups PG. Each of the second micro-lenses ML2may cover each of the phase detection pixel groups PG. That is, one second micro-lens ML2may cover the first phase detection pixel PD1and the second phase detection pixel PD included in one phase detection pixel group PG. For example, the second micro-lenses ML2may overlap the phase detection pixel groups PG in a plan view.

To increase the resolution of the image sensor100, the size of the pixel has been continuously reduced. In addition, to execute a phase difference calculation using light of the same wavelength band, a color filter of the same color may be disposed on the phase detection pixels PD1and PD2included in the phase detection pixel group PG. Accordingly, the color filter array includes, for example, a unit group including a color filter of the same color arranged in a concave-convex shape so as to arrange the color filter of the same color on the phase detection pixel group PG. That is, the unit group of the color filter array includes a portion protruding or indented from one side thereof. Accordingly, the difficulty of the manufacturing process of the image sensor increases.

On the other hand, in the image sensor according to an embodiment of the inventive concept, since a color filter of the same color does not need to be disposed on the phase detection pixels PD1and PD2, the color filter array114amay include a plurality of rectangular unit groups UG1, UG2, UG3and UG4. Accordingly, the difficulty of the manufacturing process of the image sensor may be reduced.

FIG.3is a view illustrating an operation of an image system according to an embodiment.

Referring toFIGS.1to3, the image sensor100may alternately transmit the image data and the phase detection data to the application processor200via the first channel VC0and the second channel VC1. The first channel VC0may be preset to transmit the image data according to an interface standard of the image sensor100and the application processor200. The second channel VC1may be preset to transmit the phase detection data according to the interface standard of the image sensor100and the application processor200. Accordingly, the application processor200may recognize the data provided to the first channel VC0as data for generating an image and may recognize the data provided to the second channel VC1as data for performing phase detection. The application processor200may perform the phase difference calculation based on the data provided to the second channel VC1.

In an embodiment, the first channel VC0and the second channel VC1refers to a virtual channel according to a mobile industry processor interface alliance (MIPI) standard.

In an embodiment, via the second channel VC1, the image sensor100provides phase detection data generated from a phase detection pixel, and image data generated from a normal pixel where the color filter of the same color as the color filter disposed on the phase detection pixel is disposed and which is included in the same unit group as the phase detection pixel. The color filter belonging to the same unit group may be disposed on the normal pixel and the phase detection pixel.

An example of the phase detection pixel group PG included in the third unit group UG3and the fourth unit group UG4will be described below.

The third unit group UG3with the third color filter Gb may be disposed on the first phase detection pixel PD1and a plurality of normal pixels NPX11to NP18. The fourth unit group UG4with the fourth color filter B may be disposed on the second phase detection pixel PD2and a plurality of normal pixels NPX21to NPX28. The first phase detection pixel PD1and the second phase detection pixel PD2may be disposed adjacent to each other. In an embodiment, the third color filter Gb disposed on the first phase detection pixel PD1differs from the fourth color filter B disposed on the second phase detection pixel PD2. For example, a color of the third color filter Gb may be different from a color of the fourth color filter B. The image sensor100may output, via the second channel VC1, a pair of the image data Gi generated from a normal pixel belonging to the same unit group as the first phase detection pixel PD1and a first phase data Li generated from the first phase detection pixel PD1, and a pair of a second phase detection data Ri generated from the second phase detection pixel PD2and the image data Ci generated from a normal pixel belonging to the same unit group as the second phase detection pixel PD2. It will be described with reference toFIGS.4to6.

FIGS.4to6are views illustrating the operation of the image system according to an embodiment. An example of the phase detection pixel group PG included in the third unit group UG3and the fourth unit group UG4inFIG.2will be described below.

Referring toFIG.4, in an embodiment, via the second channel VC1, the image sensor100provides the phase detection data Li and Ri generated from the phase detection pixel, and the image data Gi and Ci generated from the normal pixel where the color filter belonging to the same unit group as the color filter disposed on the phase detection pixel is disposed and which is disposed adjacent to the phase detection pixel. In an embodiment, the first phase detection data Li includes or indicates a first angle, the second phase detection data Ri includes or indicates a second angle, and a disparity can be determined from the first and second angles.

The image sensor100may output the image data Gi and the first phase detection data Li generated from the first phase detection pixel PD1via the second channel VC1. The image data Gi may be generated from one normal pixel NPX15among normal pixels NPX13, NPX15and NPX18where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the first phase detection pixel PD1is disposed and which are disposed adjacent to the first phase detection pixel PD1. The image sensor100may output the second phase detection data Ri generated from the second phase detection pixel PD2and the image data Ci via the second channel VC1. The image data Ci may be generated from one normal pixel NP24among normal pixels NPX21, NP24and NP26where the color filter B belonging to the same unit group UG4as the color filter B disposed on the second phase detection pixel PD2is disposed and which are disposed adjacent to the second phase detection pixel PD2.

The application processor200may calculate the disparity based on the first phase detection data Li, the image data Gi, the second phase detection data Ri and the image data Ci.

Referring toFIG.5, in an embodiment, the image sensor100provides, via the second channel VC1, the phase detection data Li and Ri generated from the phase detection pixel, and the image data Gi and Ci generated from the normal pixel where a color filter belonging to the same unit group as the color filter disposed on the phase detection pixel are disposed.

The image sensor100may output, via the second channel VC1, the image data Gi generated from one normal pixel NPX14among normal pixels NPX11to NPX18where the color filter Gb belonging to the same unit group UG3as the first phase detection pixel PD1is disposed, and the first phase detection data Li generated from the first phase detection pixel PD1. The image sensor100may output, via the second channel VC1, the second phase detection data Ri generated from the second phase detection pixel PD2, and the image data Ci generated from one normal pixel NPX23among normal pixels NPX21to NPX28where the color filter B belonging to the same unit group UG4as the second phase detection pixel PD2is disposed. Different fromFIG.4, a single normal pixel selected for a given unit group is not adjacent to the phase detection pixel of the given unit group.

Referring toFIG.6, in an embodiment, the image sensor100provides, via the second channel VC1, the phase detection data Li and Ri generated from the phase detection pixel, and average image data Gi and Ci of the image data generated from a plurality of normal pixels where a color filter belonging to the same unit group as the color filter disposed on the phase detection pixel are disposed.

The image sensor100may output the average image data Gi and the first phase detection data Li generated from the first phase detection pixel PD1via the second channel VC1. In an embodiment, the average image data Gi is an average of a plurality of image data generated from each of the plurality of normal pixels NPX11to NPX18where the color filter Gb belonging to the same unit group UG3as the first phase detection pixel PD1is disposed. The image sensor100may output the second phase detection data Ri generated from the second phase detection pixel PD2and the average image data Ci via the second channel VC1. In an embodiment, the average image data Ci is an average of a plurality of image data generated from each of the plurality of normal pixels NPX21to NPX28where the color filter B belonging to the same unit group UG4as the second phase detection pixel PD2is disposed.

FIG.7is a view illustrating the pixel array and the color filter array inFIG.1. For convenience of description, in which follows, the difference from the description usingFIG.2will be mainly described.

Referring toFIG.7, in an embodiment, a color filter array114bis disposed on a pixel array112b. The pixel array112bmay include a plurality of normal pixels NPX and a plurality of phase detection pixel groups PG. The pixel array112ofFIG.1may be implemented by the pixel array112b. The color filter array114ofFIG.1may be implemented by the color filter array114b.

The color filter array114bmay include the plurality of unit groups UG1, UG2, UG3and UG4including color filters of the same color arranged in an M×N matrix, where M and N are natural numbers. For example, each of the unit groups UG1, UG2, UG3and UG4may include color filters of the same color arranged in a 4×4 matrix. Each of the color filters may be disposed to correspond to the normal pixels NPX, the first phase detection pixel PD1, and the second phase detection pixel PD2, respectively. For example, each unit group may be disposed on pixels arranged in a 4×4 matrix among pixels including the normal pixel NPX, the first phase detection pixel PD1and the second phase detection pixel PD2.

FIG.8is a view illustrating the pixel array and the color filter array inFIG.1. For convenience of description, in which follows, the difference from that described usingFIG.7will be mainly described.

Referring toFIG.8, in an embodiment, the color filter array114bis disposed on the pixel array112c. The pixel array112ofFIG.1may be implemented by the pixel array112c. The pixel array112cmay include the plurality of normal pixels NPX, a plurality of first phase detection pixel groups PG1and a plurality of second phase detection pixel groups PG2. For example, the plurality of first phase detection pixel groups PG1and the plurality of second phase detection pixel groups PG2may be regularly arranged along the first direction DR1and/or the second direction DR2.

Each of the first phase detection pixel group PG1and the second phase detection pixel group PG2may include the plurality of phase detection pixels PD1and PD2disposed adjacent to each other. For example, each of the phase detection pixel groups PG may include the first phase detection pixel PD1and the second phase detection pixel PD2disposed adjacent to each other.

A first phase detection pixel group PG1may be disposed adjacent to a second phase detection pixel group PG2. For example, the first phase detection pixel group PG1may be disposed adjacent to the second phase detection pixel group PG2in the second direction DR2.

In an embodiment, the different unit groups UG1, UG2, UG3and UG4may be disposed on the first phase detection pixel PD1of the first phase detection pixel group PG1and the second phase detection pixel PD2of the first phase detection pixel group PG1. The different unit groups UG1, UG2, UG3and UG4may be disposed on the first phase detection pixel PD1of the second phase detection pixel group PG2and the second phase detection pixel PD2of the second phase detection pixel group PG2.

FIGS.9and10are views illustrating a method for calculating a disparity according to an embodiment.

An example of the first phase detection pixel PD1where the color filter Gr included in the second unit group UG2is disposed, the second phase detection pixel PD2where the color filter R included in the first unit group UG1is disposed, and the normal pixel NPX where the color filter Gr included in the same second unit group UG2as the color filter Gr disposed on the first phase detection pixel PD1is disposed is described.

Referring toFIG.9, the image sensor may sense light reflected from the object101(seeFIG.1) and condensed via the lens103. The condensed light may be incident on the first phase detection pixel PD1and the second phase detection pixel PD2via the second micro-lens ML2of the image sensor. First phase detection data A may be generated from light incident to the first phase detection pixel PD1, and second phase detection data B may be generated from light incident to the second phase detection pixel PD2.

In that case, since the color filter Gr on the first phase detection pixel PD1differs from the color filter R on the second phase detection pixel PD2, a disparity D1between the first phase detection data A and the second phase detection data B may be generated by not only a phase difference but also a sensitivity difference between colors. Therefore, since accurate auto-focusing is difficult to perform based on the disparity D1, it may be necessary to correct the first phase detection data A and the second phase detection data B due to the difference in the color filter.

Referring toFIG.10, the image sensor may sense the light reflected from the object101(seeFIG.1) and condensed via the lens103. The condensed light may be incident on the first phase detection pixel PD1via the second micro-lens ML2of the image sensor and may be incident on the normal pixel NPX via the first micro-lens ML1.

The first phase detection data A may be generated from the light incident on the first phase detection pixel PD1, and image data C may be generated from the light incident on the normal pixel NPX. The color filter Gr on the first phase detection pixel PD1is identical to the color filter Gr on the normal pixel NPX. A disparity D2between the first phase detection data A and the image data C may be approximately twice as large as the disparity D1inFIG.8. Accordingly, the disparity between the first phase detection data and the second phase detection data may be calculated using the image data. It will be described in detail with reference toFIGS.11to14.

FIG.11is a flowchart describing the method for calculating the disparity according to an embodiment.

Referring toFIGS.3and11, the pair of the first image data Gi and the first phase detection data Li and the pair of the second phase detection data Ri and the second image data Ci may be alternately provided via the second channel VC1. The calculating module210of the application processor200according to an embodiment may calculate the disparity Dt using Equations 1 to 3 where x means a reference location and d means a disparity, and d in which Equation 3 is minimized becomes the disparity Dt.

PD_left=∑L⁢i+∑Gi[Equation⁢1]PD_right=∑C⁢i+∑R⁢i[Equation⁢2]Dt=arg⁢mind(PD_leftx-PD_rightx-d)[Equation⁢3]

Specifically, a first value PD_left may be calculated by adding the sum (ΣLi) of the first phase detection data and the sum (ΣGi) of the first image data, while a second value PD_right may be calculated by adding the sum (ΣCi) of the second image data and the sum (ΣRi) of the second phase detection data (S110).

Then, the calculating module210may calculate the disparity Dt between the first value PD_left and the second value PD_right (S120). Accordingly, the calculating module210may calculate the disparity Dt based on the image data and the phase detection data. The application processor200may perform auto-focusing based on the calculated disparity Dt.

Alternatively, the calculating module210may calculate the first value PD_left of the phase detection data using Equation 4 and may calculate the second value PD_right of the phase detection data using Equation 5.
PD_left=Σ(Li+Gi)  [Equation 4]
PD_right=Σ(Ci+Ri)  [Equation 5]

Specifically, the first value PD_left may be calculated by adding the sum of the first phase detection data Li and the sum of the first image data Gi and the second value PD_right may be calculated by adding the sum of the second image data Ci and the sum of the second phase detection data Ri.

The first image data Gi and the second image data Ci may be generated according to one of the descriptions used with reference toFIGS.4to6.FIG.12is a flowchart describing a method for calculating a disparity according to an embodiment.

Referring toFIGS.3and12, the pair of the first image data Gi and the first phase detection data Li and the pair of the second phase detection data Ri and the second image data Ci may be alternately provided via the second channel VC1. The calculating module210of the application processor200according to an embodiment may calculate the disparity Dt using Equations 6 to 8 where x means a reference location and d means a disparity, and d in which Equation 4 is minimized becomes a first disparity Disparityeven, and d in which Equation 5 is minimized becomes a second disparity Disparityodd.

Disparitye⁢v⁢e⁢n=arg⁢mind(❘"\[LeftBracketingBar]"∑Li,x-∑Gi,x-d❘"\[RightBracketingBar]")[Equation⁢6]Disparityo⁢d⁢d=arg⁢mind(❘"\[LeftBracketingBar]"∑Ci,x-∑Ri,x-d❘"\[RightBracketingBar]")[Equation⁢7]Dt=Disparitye⁢v⁢e⁢n+Dispa⁢rityo⁢d⁢d[Equation⁢8]

Specifically, the first disparity Disparityevenwhich is the sum of the disparities of the first phase detection data Li and the first image data Gi may be calculated, and the second disparity Disparityoddwhich is the sum of the disparities of the second image data Ci and the second phase detection data Ri may be calculated.

Then, the calculating module210may calculate the disparity Dt by adding the first disparity Disparityevenand the second disparity Disparityodd(S220).

The first image data Gi and the second image data Ci may be generated according to one of the descriptions used with reference toFIGS.4to6.FIG.13is a flowchart describing a method for calculating a disparity according to an embodiment.

Referring toFIGS.3and13, the pair of the first image data Gi and the first phase detection data Li and the pair of the second phase detection data Ri and the second image data Ci may be alternately provided via the second channel VC1. The calculating module210of the application processor200according to an embodiments may calculate the disparity Dt using Equations 9 to 11 where x means a reference location and d means a disparity.

C⁢Ve⁢v⁢e⁢n(d)=(∑Li,x-∑Gi,x-d)*(∑Li,x-∑Gi,x-d)[Equation⁢9]C⁢Vo⁢d⁢d(d)=(∑Ci,x-∑Ri,x-d)*(∑Ci,x-∑Ri,x-d)[Equation⁢10]Disparity=arg⁢mind(C⁢Ve⁢v⁢e⁢n+C⁢Vo⁢d⁢d)[Equation⁢11]

Specifically, the first cost volume CVeven(d) of the sum of the disparities of the first phase detection data Li and the first image data Gi and the second cost volume CVodd(d) of the sum of the disparities of the second image data Ci and the second phase detection data Ri may be calculated (S310).

The first image data Gi and the second image data Ci may be generated according to one of the descriptions used with reference toFIGS.4to6. Then, the calculating module210may calculate the disparity Dt of the sum of the first cost volume CVeven(d) and the second cost volume CVodd(d) (S320).

FIG.14is a flowchart describing the method for calculating a disparity according to an embodiment.

Referring toFIGS.3and14, the pair of the first image data Gi and the first phase detection data Li and the pair of the second phase detection data Ri and the second image data Ci may be alternately provided via the second channel VC1. The calculating module210of the application processor200according to an embodiment may calculate the disparity Dt using Equations 12 and 13.

Cgain=∑Gi/∑Ci[Equation⁢12]Dt=arg⁢mind(❘"\[LeftBracketingBar]"∑Li,x-Cgain*∑Ri,x❘"\[RightBracketingBar]")[Equation⁢13]

Specifically, a ratio (Cgain) of the sum (ΣGi) of the first image data to the sum (ΣCi) of the second image data may be calculated (S410). That is, the second image data generated from a pixel with the second color filter may be converted into image data to be generated from a pixel with the first color filter. In that case, the first image data is generated from the pixel with the first color filter.

Then, the calculating module210may calculate calibration data (Cgain*ΣRi) by multiplying the sum (ΣRi) of the second phase detection data by the ratio (Cgain) calculated in S410(S420).

Then, the calculating module210may calculate the disparity Dt of the calibration data (Cgain*ΣRi) calculated in S420and the sum (ΣLi) of the first phase detection data (S430).

In an embodiment, the second image data Ci is generated from the normal pixel where the color filter is disposed instead of the green filter, while the first image data Gi is generated from the normal pixel where the green color filter is disposed. That is, the ratio may be a value in which the image data generated from the normal pixel with the color filter disposed instead of the green color filter is converted into the image data to be generated from the normal pixel where the green color filter is disposed.

The first image data Gi and the second image data Ci may be generated according to one of the descriptions used with reference toFIGS.4to6.FIGS.15to17are views describing an operation of an image system according to an embodiment. An example of the phase detection pixel group PG included in the third unit group UG3and the fourth unit group UG4inFIG.8will be described.

Referring toFIG.15, in an embodiment, the image sensor100may output image data Li and first-first phase detection data Gi generated from a first-first phase detection pixel PD11via the second channel VC1. The image data Gi may be generated from one normal NPX17among normal pixels NPX14and NPX17where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the first-first phase detection pixel PD11is disposed and which are disposed adjacent to the first-first phase detection pixel PD11. The image sensor100may output first-second phase detection data Ri generated from a first-second phase detection pixel PD12and the image data Ci via the second channel VC1. The image data Ci may be generated from one normal pixel NPX25among normal pixels NPX21and NPX25where the color filter B belonging to the same unit group UG4as the color filter B disposed on the first-second phase detection pixel PD12is disposed and which are disposed adjacent to the first-second phase detection pixel PD12. The image sensor100may output the image data Gi and second-first phase detection data Li generated from a second-first phase detection pixel PD21via the second channel VC1. The image data Gi may be generated from one normal pixel NPX20among normal pixels NPX20and NPX24where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the second-first phase detection pixel PD21is disposed and which are disposed adjacent to the second-first phase detection pixel PD21. The image sensor100may output the image data Ci and second-second phase detection data Ri generated from a second-second phase detection pixel PD22via the second channel VC1. The image data Ci may be generated from one normal pixel NPX28among normal pixels NPX28and NPX31where the color filter B belonging to the same unit group UG4as the color filter B disposed on the second-second phase detection pixel PD22is disposed and which are disposed adjacent to the second-second phase detection pixel PD22.

Referring toFIG.16, in an embodiment, the image sensor100may output the image data Gi and first-first phase detection data Li generated from the first-first phase detection pixel PD11via the second channel VC1. The image data Gi may be generated from one normal pixel NPX15among a plurality of normal pixels NPX11to NPX24where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the first-first phase detection pixel PD11is disposed. The image sensor100may output the first-second phase detection data Ri generated from the first-second phase detection pixel PD12and the image data Ci via the second channel VC1. The image data Ci may be generated from one NPX23among a plurality of normal pixels NPX21to NPX34where the color filter B belonging to the same unit group UG4as the color filter B disposed on the first-second phase detection pixel PD12is disposed. The image sensor100may output the image data Gi and the second-first phase detection data Li generated from the second-first phase detection pixel PD21via the second channel VC1. The image data Gi may be generated from one normal pixel NPX20among a plurality of normal pixels NPX11to NPX24where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the second-first phase detection pixel PD21is disposed. The image sensor100may output the second-second phase detection data Ri generated from the second-second phase detection pixel PD22and the image data Ci via the second channel VC1. The image data Ci may be generated from one normal pixel NPX28among a plurality of normal pixels NPX21to NPX34where the color filter B belonging to the same unit group UG4as the color filter B disposed on the second-second phase detection pixel PD22is disposed.

In that case, the image data Gi output together with the first-first phase detection data Li may be identical to the image data Gi output together with the second-first phase detection data Li. Furthermore, the image data Ci output together with the first-second phase detection data Ri may be identical to the image data Ci output together with the second-second phase detection data Ri.

Referring toFIG.17, in an embodiment, the image sensor100may output the average image data Gi and the first-first phase detection data Li generated from the first-first phase detection pixel PD11via the second channel VC1. The average image data Gi may be an average of a plurality of image data generated from each of the plurality of normal pixels NPX11to NPX24where the color filter Gb belonging to the same unit group UG3as the color filter Gb disposed on the first-first phase detection pixel PD11is disposed. The image sensor100may output the first-second phase detection data Ri generated from the first-second phase detection pixel PD12and the average image data Ci, via the second channel VC1. The average image data Ci may be an average of a plurality of image data generated from each of the plurality of normal pixels NPX21to NPX34where the color filter B belonging to the same unit group UG4as the color filter B disposed on the first-second phase detection pixel PD12is disposed. The image sensor100may output the average image data Gi and the second-first phase detection data Li generated from the second-first phase detection pixel PD21, via the second channel VC1. The image sensor100may output the second-second phase detection data Ri generated from the second-second phase detection pixel PD22and the average image data Ci, via the second channel VC1. The average image data Ci may be an average of a plurality of image data generated from each of the plurality of normal pixels NPX21to NPX34where the color filter B belonging to the same unit group UG4as the color filter B disposed on the second-second phase detection pixel PD22is disposed.

The average image data Gi output together with the first-first phase detection data Li generated from the first-first phase detection pixel PD11may be identical to the average image data Gi output together with the second-first phase detection data Li generated from the second-first phase detection pixel PD21. The average image data Ci output together with the second-first phase detection data Ri generated from the second-first phase detection pixel PD21may be identical to the average image data Ci output together with the second-second phase detection data Ri generated from the second-second phase detection pixel PD22.

FIG.18is a block diagram illustrating an image sensing device according to an embodiment. For convenience of description, in which follows, differences from the descriptions usingFIGS.1to17will be mainly described.

Referring toFIG.18, an image sensing device2according to an embodiment may include an image sensor100and an application processor200. The image sensor100may further include a calculating module196. The calculating module196may be implemented with, for example, software, firmware, or hardware executed on the image sensor100.

The calculating module196may operate in the same manner as the calculating module210described with reference toFIGS.1to16. That is, in some embodiments, the image sensor100may calculate the disparity D using the phase detection data Li and Ri and the image data Gi and Ci.

The application processor200may receive the image data IDATA output by the first image signal processor192via the first channel VC0. The application processor200may receive the disparity D calculated by the calculating module196via the second channel VC1. The application processor may perform the auto-focusing based on the disparity D.FIG.19is a block diagram illustrating the image sensing device according to some embodiments.

Referring toFIG.19, the image sensor100according to an embodiments may include a stacked first chip10and a stacked second chip20. The second chip20may be stacked on, for example, the first chip10in a third direction DR3. The first chip10and the second chip20may be electrically connected to each other. A pixel signal (data) transmitted from the first chip10may be transmitted to a logic area LC.

The first chip10may include the pixel array112(seeFIG.1). The second chip20may include the logic area LC and a memory area. The logic circuit area LC may include a plurality of elements for driving pixel signal (data). The logic circuit area LC may include, for example, the row driver120, the CDS block140, the ramp signal generator150, the timing generator160, the control register block170, the buffer180, the first image signal processor192and the second image signal processor194.

FIG.20is a block diagram illustrating an image sensor according to an embodiments. For convenience of explanation, in which follows, the difference from the description usingFIG.19will be mainly described.

Referring toFIG.20, an image sensor100′ may further include a third chip30. The third chip30, the second chip20, and the first chip10may be sequentially stacked in the third direction DR3. The third chip30may include a memory device. For example, the third chip30may include a volatile memory device such as DRAM and SRAM. The third chip30may receive a signal from the first chip10and the second chip20and process the signal through the memory device.

FIG.21is a block diagram of an electronic device including a multi-camera module.FIG.22is a detailed block diagram of the camera module inFIG.21.

Referring toFIG.21, an electronic device1000may include a camera module group1100, an application processor1200, a PMIC1300and an external memory1400.

The camera module group1100may include a plurality of camera modules1100a,1100band1100c. Although the drawing illustrates an embodiment where three camera modules1100a,1100band1100care arranged, the embodiments are not limited thereto. In some embodiments, the camera module group1100may be modified and implemented to include only two camera modules. In addition, in some embodiments, the camera module group1100may be modified and implemented to include n camera modules (where n is a natural number greater than 4).

Hereinafter, a detailed configuration of a camera module1100bwill be described in more detail with reference toFIG.22. However, the following description may be equally applied to the other camera modules1100aand1100caccording to one embodiment.

Referring toFIG.22, the camera module1100bmay include a prism1105, an optical path folding element (hereinafter referred to as “OPFE”)1110, an actuator1130, an image sensing device1140and a storage1150.

The prism1105may include a reflective surface1107of a light reflecting material to modify a path of light L incident from the outside.

In some embodiments, the prism1105may change the path of the light L incident in a first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism1105may change the path of the light L incident in the first direction X to the second direction Y perpendicular to the first direction X by rotating the reflective surface1107of the light reflecting material in a direction A around a central axis1106or rotating the reflective surface1107of the light reflecting material in a direction B around the central axis1106. In that case, the OPFE1110may also move in a third direction Z perpendicular to the first direction X and the second direction Y.

In some embodiments, as illustrated, a maximum rotation angle of the prism1105in the direction A may be 15 degrees or less in a positive (+) direction A and may be greater than 15 degrees in a negative (−) direction A, but the embodiments are not limited thereto.

In some embodiments, the prism1105may move at approximately 20 degrees, or between 10 and 20 degrees, or between 15 and 20 degrees, in a positive (+) or negative (−) direction B, where the prism1105may move at a moving angle which is the same angle in the positive (+) or negative (−) direction B or which is an almost similar angle in the range of about 1 degree.

In some embodiments, the prism1105may move a reflective surface1107of the light reflecting material in the third direction (e.g., the direction Z) parallel to the extending direction of the central axis1106.

The OPFE1110may include, for example, an optical lens consisting of m groups (where m is a natural number). The m lenses may move in the second direction Y and change an optical zoom ratio of the camera module1100b. For example, in the case where a basic optical zoom ratio of the camera module1100bis Z, when m optical lenses included in the OPFE1110are moved, the optical zoom ratio of the camera module1100bmay be changed to an optical zoom ratio of 3Z, 5Z, or 5Z or more.

The actuator1130may move the OPFE1110or the optical lens (hereinafter referred to as “an optical lens”) to a certain location. For example, the actuator1130may adjust the location of the optical lens such that an image sensor1142is located at a focal length of the optical lens for accurate sensing.

The image sensing device1140may include the image sensor1142, a control logic1144and a memory1146. The image sensor1142may sense an image of a sensing object using the light L provided via the optical lens. The control logic1144may control an overall operation of the camera module1100b. For example, the control logic1144may control the operation of the camera module1100baccording to the control signal provided via a control signal line CSLb.

The memory1146may store information necessary for the operation of the camera module1100b, such as calibration data1147. The calibration data1147may include information necessary for the camera module1100bto generate image data using the light L provided from the outside. The calibration data1147may include, for example, information on a degree of rotation, a focal length, an optical axis, as described above. When the camera module1100bis implemented in the form of a multi-state camera where the focal length changes according to the location of the optical lens, the calibration data1147may include a focal length value for each location (or each state) of the optical lens and information associated with the auto-focusing.

The storage1150may store the image data sensed via the image sensor1142. The storage1150may be disposed outside the image sensing device1140and implemented such that it is stacked with a sensor chip constituting the image sensing device1140. In some embodiments, the storage1150may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiments are not limited thereto.

Referring toFIGS.21and22together, in some embodiments, each of the plurality of camera modules1100a,1100band1100cmay include the actuator1130. Accordingly, each of the plurality of camera modules1100a,1100b, and1100cmay include the calibration data1147identical to or different from each other according to the operation of the actuator1130included therein.

In some embodiments, one camera module (e.g.,1100b) among the plurality of camera modules1100a,1100band1100cmay be a camera module in the form of a folded lens including the prism1105and the OPFE1110described above, while the remaining camera modules (e.g.,1100aand1100c) may be vertical camera modules that fail to include the prism1105and the OPFE1110, but the embodiments are not limited thereto.

In some embodiments, one camera module (e.g.,1100c) among the plurality of camera modules1100a,1100band1100cmay be, for example, a vertical type of depth camera that extracts depth information using an infrared ray IR. In that case, the application processor1200may merge image data provided from the depth camera with image data provided from another camera module (e.g.,1100aor1100b), thereby generating a 3D depth image.

In some embodiments, at least two camera modules (e.g.,1100aand1100b) among the plurality of camera modules1100a,1100band1100cmay have different field of views. In that case, for example, the optical lenses of at least two camera modules (e.g.,1100aand1100b) among the plurality of camera modules1100a,1100band1100cmay be different from each other, but the present disclosure is not limited thereto.

Furthermore, in some embodiments, field of views of each of the plurality of camera modules1100a,1100band1100cmay differ from each other. In that case, the optical lenses included in each of the plurality of camera modules1100a,1100band1100cmay also differ from each other, but the present invention is not limited thereto.

In some embodiments, each of the plurality of camera modules1100a,1100band1100cmay be physically separated from each other. In other words, instead of dividing and using a sensing area of one image sensor1142by the plurality of camera modules1100a,1100band1100c, an independent image sensor1142may be disposed inside each of the plurality of camera modules1100a,1100band1100c.

Referring back toFIG.21, the application processor1200may include an image processing device1210, a memory controller1220and an internal memory1230. The application processor1200may be implemented separately from the plurality of camera modules1100a,1100band1100c. For example, the application processor1200and the plurality of camera modules1100a,1100band1100cmay be separately implemented with separate semiconductor chips.

The image processing apparatus1210may include a plurality of sub-image processors1212a,1212band1212c, an image generator1214and a camera module controller1216.

The image processing apparatus1210may include a plurality of sub-image processors1212a,1212band1212ccorresponding to the number of the plurality of camera modules1100a,1100band1100c.

The image data generated from each of the camera modules1100a,1100band1100cmay be provided to the corresponding sub-image processors1212a,1212band1212cvia image signal lines ISLa, ISLb and ISLc separated from each other. For example, the image data generated from the camera module1100amay be provided to the sub-image processor1212avia the image signal line ISLa, the image data generated from the camera module1100bmay be provided to the sub-image processor1212bvia the image signal line ISLb, and the image data generated from the camera module1100cmay be provided to the sub-image processor1212cvia the image signal line ISLc. The image data transmission may be performed using, for example, a camera serial interface (CSI) based on a mobile industry processor interface (MIPI), but the embodiments are not limited thereto.

Meanwhile, in some embodiments, one sub-image processor may be disposed to correspond to the plurality of camera modules. For example, the sub-image processor1212aand the sub-image processor1212care not implemented separately from each other as illustrated in the drawing, but may be integrated into one sub-image processor, and the image data provided from the camera module1100aand the camera module1100cmay be selected through a selection element (e.g., a multiplexer) and then provided to the integrated sub-image processor.

The image data provided to each of the sub-image processors1212a,1212band1212cmay be provided to the image generator1214. The image generator1214may generate an output image using the image data provided from each of the sub-image processors1212a,1212band1212caccording to image generating information or a mode signal.

Specifically, the image generator1214may generate the output image by merging at least part of the image data generated from camera modules1100a,1100band1100chaving different field of views according to the image generating information or the mode signal. Furthermore, the image generator1214may generate the output image by selecting one of the image data generated from the camera modules1100a,1100band1100chaving different field of views according to the image generating information or the mode signal.

In some embodiments, the image generating information may include a zoom signal or a zoom factor. Furthermore, in some embodiments, the mode signal may be, for example, a signal based on a mode selected from a user.

When the image generating information is the zoom signal (zoom factor) and each of the camera modules1100a,1100band1100chas different field of views, the image generator1214may perform different operations according to the type of zoom signals. For example, when the zoom signal is a first signal, after merging the image data output from the camera module1100aand the image data output from the camera module1100c, the output image may be generated by using the merged image signal and the image data output from the camera module1100bnot used for the merging. When the zoom signal is a second signal different from the first signal, the image generator1214may generate the output image by selecting one of the image data output from each of the camera modules1100a,1100band1100cwithout conducting such image data merging. However, the embodiments are not limited thereto, and a method of processing the image data may be modified and implemented as necessary.

In some embodiments, the image generator1214may receive a plurality of image data with different exposure times from at least one of the plurality of sub-image processors1212a,1212band1212cand perform high dynamic range (HDR) processing on the plurality of image data, thereby generating merged image data with an increased dynamic range.

The camera module controller1216may provide control signals to each of the camera modules1100a,1100band1100c. The control signal generated from the camera module controller1216may be provided to the corresponding camera modules1100a,1100band1100cvia the control signal lines CSLa, CSLb and CSLc separated from each other.

One of the plurality of camera modules1100a,1100band1100cmay be designated as a master camera (e.g.,1100b) according to the image generating information including the zoom signal or the mode signal, while the remaining camera modules (e.g.,1100aand1100c) may be designated as slave cameras. The information may be included in the control signal and provided to the corresponding camera modules1100a,1100band1100cvia the control signal lines CSLa, CSLb and CSLc separated from each other.

The camera module operating as the master and slave cameras may be changed according to the zoom factor or the operation mode signal. For example, when the field of view of the camera module1100ais wider than the field of view of the camera module1100band the zoom factor exhibits a low zoom ratio, the camera module1100bmay operate as the master camera and the camera module1100amay operate as the slave camera. Conversely, when the zoom factor exhibits a high zoom ratio, the camera module1100acan operate as the master camera and the camera module1100bcan operate as the slave camera.

In some embodiments, the control signal provided from the camera module controller1216to each of the camera modules1100a,1100band1100cmay include a sync enable signal. For example, when the camera module1100bis the master camera and the camera modules1100aand1100care the slave cameras, the camera module controller1216may transmit the sync enable signal to the camera module1100b. The camera module1100bthat receives the sync enable signal may generate a sync signal based on the received sync enable signal and provide the generated sync signal to the camera modules1100aand1100cvia a sink signal line SSL. The camera module1100band the camera modules1100aand1100cmay be synchronized with the sync signal to transmit the image data to the application processor1200.

In an embodiment, the control signal provided from the camera module controller1216to the plurality of camera modules1100a,1100band1100cmay include mode information according to the mode signal. Based on the mode information, the plurality of camera modules1100a,1100band1100cmay operate in a first operation mode and a second operation mode in association with a sensing speed.

In the first operation mode, the plurality of camera modules1100a,1100band1100cmay generate an image signal (e.g., generate an image signal of a first frame rate) at a first speed, encode the image signal at a second speed that is higher than the first speed (e.g., encode an image signal of a second frame rate higher than the first frame rate), and transmit the encoded image signal to the application processor1200. In that case, the second speed may be less than or equal to 30 times the first speed.

The application processor1200may store the received image signal, i.e., the encoded image signal, in the memory1230formed inside or the storage1400formed outside the application processor1200, read and decode the encoded image signal from the memory1230or the storage1400, and display the image data generated based on the decoded image signal. For example, a corresponding sub-processor among the plurality of sub-processors1212a,1212band1212cof the image processing apparatus1210may perform decoding and may also perform image processing on the decoded image signal.

In the second operation mode, the plurality of camera modules1100a,1100band1100cmay generate an image signal at a third speed that is lower than the first speed (e.g., generate the image signal at a third frame rate lower than the first frame rate), and transmit the image signal to the application processor1200. The image signal provided to the application processor1200may be an unencoded signal. The application processor1200may perform the image processing on the received image signal or store the image signal in the memory1230or the storage1400.

The PMIC1300may provide power, for example, a power voltage, to each of a plurality of camera modules1100a,1100band1100c. For example, the PMIC1300may provide a first power to the camera module1100avia a power signal line PSLa, a second power to the camera module1100bvia a power signal line PSLb, and a third power to the camera module1100cvia a power signal line PSLc under the control of the application processor1200.

The PMIC1300may generate power corresponding to each of the plurality of camera modules1100a,1100band1100cin response to a power control signal PCON from the application processor1200and may also adjust a level of power. The power control signal PCON may include power adjustment signals for each operation mode of the plurality of camera modules1100a,1100band1100c. For example, the operation mode may include a low power mode, and in that case, the power control signal PCON may include information on a camera module that operates in the low power mode and a set level of power. Levels of power provided to each of the plurality of camera modules1100a,1100band1100cmay be identical to or different from each other. Furthermore, the level of power may be dynamically changed.

Although embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to the disclosed embodiments, but may be implemented in various different ways, and the present disclosure may be embodied in many different forms as will be understood by those skilled in the art. Therefore, embodiments set forth herein are exemplary only and not to be construed as a limitation.