IMAGE SENSOR, IMAGE PROCESSING SYSTEM INCLUDING THE SAME, AND OPERATION METHOD THEREOF

An image sensor, including a pixel array; a readout circuit configured to generate image data including first image data including phase information regarding a first phase, second image data including phase information regarding a second phase, and full image data including color information; and a signal processing unit including: a front end processing module configured to generate third image data including additional phase information regarding the first phase using the second image data and the full image data, and to generate fourth image data including additional phase information regarding the second phase using the first image data and the full image data and an auto-focusing processing module configured to calculate a phase difference between the first phase and the second phase using the first image data, the second image data, the third image data, and the fourth image data.

CROSS-REFERENCE TO RELATED APPLICATION

BACKGROUND

The disclosure relates to an image sensor, an image processing system, and an operation method of the image processing system, and more particularly, to an image sensor for performing an auto-focusing (AF) function, an image processing system, and an operation method of the image processing system.

2. Description of Related Art

Image sensors, which capture images and convert the captured images into electrical signals, are used not only in general consumer electronic devices, such as digital cameras, mobile phone cameras, and portable camcorders, but also in cameras mounted on cars, security devices, and robots. Image sensors as described above may include pixel arrays, and respective pixels included in the pixel arrays may include photodiodes. Image sensors may perform AF functions to accurately capture images in a short time.

SUMMARY

Provided is an image sensor having an auto-focusing (AF) function with improved accuracy, an image processing system including the same, and an operation method thereof.

In accordance with an aspect of the disclosure, an image sensor includes a pixel array including a plurality of pixel groups, wherein each pixel group of the plurality of pixel groups includes a plurality of pixels; a readout circuit configured to generate image data by reading pixel signals output from the pixel array, wherein the image data includes first image data including phase information regarding a first phase, second image data including phase information regarding a second phase, and full image data including color information; and a signal processing unit configured to process the image data, and including: a front end processing module configured to generate third image data including additional phase information regarding the first phase using the second image data and the full image data, and to generate fourth image data including additional phase information regarding the second phase using the first image data and the full image data; and an auto-focusing processing module configured to calculate a phase difference between the first phase and the second phase in a first direction using the first image data, the second image data, the third image data, and the fourth image data.

In accordance with an aspect of the disclosure, an image processing system includes an image sensor configured to generate image output data; and at least one processor configured to: perform image signal processing on the image output data, wherein the image sensor includes: a pixel array including a plurality of pixel groups, wherein each pixel group of the plurality of pixel groups includes a plurality of pixels; a readout circuit configured to generate image data by reading pixel signals output from the pixel array, wherein the image data includes first image data including phase information regarding a first phase, second image data including phase information regarding a second phase, and full image data including color information, and a signal processing unit configured to process the image data, and including: a front end processing module configured to generate third image data including additional phase information regarding the first phase using the second image data and the full image data, and to generate fourth image data including additional phase information regarding the second phase using the first image data and the full image data.

In accordance with an aspect of the disclosure, a method of operating an image processing system including an image sensor including a plurality of pixel groups, includes generating first image data including phase information regarding a first phase, second image data including phase information regarding a second phase, and full image data including color information; generating third image data including additional phase information regarding the first phase using the second image data and the full image data; generating fourth image data including additional phase information regarding the second phase using the first image data and the full image data; and calculating a phase difference between the first phase and the second phase in a first direction using the first image data, the second image data, the third image data, and the fourth image data.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements, and redundant or duplicative description thereof may be omitted.

The term “circuit” used herein may refer to software, or a hardware component such as an FPGA or an ASIC, and “circuit” performs certain roles. However, “circuit” is not limited to software or hardware. The “circuit” may be configured to be on a storage medium that may be addressed, or may be configured to play back one or more processors. Therefore, as an example, “circuit” may include components, such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

FIG.1is a block diagram illustrating an image processing system10according to an embodiment. In embodiments, the image processing system10may perform an auto-focusing (AF) function.

The image processing system10according to an embodiment may include an imaging unit11, an image sensor100, and a processor12. The image processing system10may include a focus detection function. The image sensor100and the imaging unit11may be components included in a camera module.

The image processing system10may be implemented as an electronic device that captures an image and displays the captured image or performs a captured image-based operation. The image processing system10may be implemented as, for example, a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may include a laptop computer, a mobile phone, a smartphone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an electronic book (e-book), a wearable device, and the like. In addition, the image processing system10may be mounted on an electronic device, such as a drone or an advanced drivers assistance system (ADAS), or an electronic device provided as a component in a vehicle, furniture, manufacturing equipment, a door, various types of measurement devices or the like.

The image processing system10may further include other components, such as a display and a user interface. The image processing system10may be implemented as a system on chip (SoC).

The overall operation of the image processing system10may be controlled by the processor12. The processor12may provide a lens driver11_2, a controller120, and the like with a control signal for an operation of each component. For example, the imaging unit11may further include an aperture driver for driving an aperture, and the processor12may provide a control signal for controlling the aperture driver. In an embodiment, the processor12may be an application processor (AP).

The imaging unit11may be a component for receiving light, and may include an optical lens11_1and the lens driver11_2. The optical lens11_1may include a plurality of lenses. The image sensor100may convert, into an electrical signal, a light signal reflected from an object20through the optical lens11_1, and may generate image data, for example image data IDT as shown inFIG.2, on the basis of electrical signals. AlthoughFIG.1illustrates the optical lens11_1as including one lens, embodiments are not limited thereto. For example, the optical lens11_1may also include a plurality of lenses.

The lens driver112may communicate information regarding focus detection with the processor12, and may adjust a position of the optical lens11_1according to a control signal provided by the processor12. The lens driver112may move the optical lens11_1in a direction in which a distance from the object20increases or decreases, and accordingly, a distance between the optical lens1I_1and the object20may be adjusted. A focus on the object20may be adjusted, which may cause an image of the object20to be focused or blurred, according to the position of the optical lens11_1.

For example, when the distance between the optical lens11_1and the object20is relatively close, the optical lens11_1may be out of an in-focus position for adjusting the focus on the object20, and a phase difference may occur between images captured by the image sensor100. Based on the control signal provided by the processor12, the lens driver1I_2may move the optical lens11_1in a direction which causes the distance from the object20to increase.

As another example, when the distance between the optical lens11_1and the object20is relatively far, the optical lens11_1may be out of the in-focus position, and a phase difference may occur between images formed on the image sensor100. Based on the control signal provided by the processor12, the lens driver11_2may move the optical lens11_1in a direction which causes the distance from the object20to decrease.

The image sensor100may convert incident light into an image signal. The image sensor100may include a pixel array110, a controller120, and a signal processing unit130. An optical signal transmitted through the optical lens11_1may reach a light receiving surface of the pixel array110and form an image of the object20.

The pixel array110may be a complementary metal oxide semiconductor (CMOS) image sensor (CIS) that converts an optical signal into an electrical signal. The sensitivity of the pixel array110, and/or other parameters of the pixel array110, may be adjusted by the controller120. The pixel array110may include a plurality of pixels that convert an optical signal into an electrical signal. Each of the plurality of pixels may generate a pixel signal according to a sensed intensity of light.

The image sensor100may provide output data to the processor12. The output data may include phase difference data including phase difference information, or may include image data including phase information such that the processor12may perform a phase difference calculation.

For example, the signal processing unit130may generate third image data including phase information regarding a first phase and fourth image data including phase information regarding a second phase based on first image data including the phase information regarding the first phase, second image data including the phase information regarding the second phase, and full image data including color information (i.e., image information). Here, each of the first image data and the second image data may include pixel data sampled according to a predefined sampling ratio. The signal processing unit130may generate phase difference data by calculating a phase difference between the first phase and the second phase using the first to fourth image data, and may transmit the phase difference data to the processor12.

As another example, the signal processing unit130may transmit, to the processor12as output data, first to fourth binning data obtained by binning the first to fourth image data, respectively, or the signal processing unit130may transmit, to the processor12, first merge data obtained by merging the first image data and the third image data that include the phase information regarding the first phase, and may transmit, to the processor12as output data, second merge data obtained by merging the second image data and the fourth image data that include the phase information regarding the second phase. The processor12may perform a phase difference calculation using output data.

For example, the phase difference calculation performed by the signal processing unit130of the image sensor100, or performed by the processor12, may be obtained by performing a correlation calculation between pieces of image data including different types of phase information. Based on the phase difference calculation, the processor12may obtain a position of the focus, a direction of the focus, a distance between the object20and the image sensor100, or the like. Based on the result of the phase difference calculation, the processor12may output a control signal to the lens driver11_2to move a position of the optical lens11_1.

Therefore, the image sensor100and the image processing system10according to embodiments may compensate for phase information, which may be lost by sampling only some of pixel data, using all of the first to fourth image data for the phase difference calculation. Accordingly, a signal-to-noise ratio (SNR) of an AF function of the image sensor100and the image processing system10may increase.

The processor12may reduce noise with respect to input data, and may perform, on the input image, imaging signal processing for image quality improvement, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. In addition, the processor12may generate an image file by compressing image data generated by performing image signal processing for image quality improvement, or may restore image data from the image file.

The pixel array110may include a color filter configured to allow sensing of various colors, and each of the plurality of pixels may sense a corresponding color. Accordingly, the image sensor100may generate output image data including color information. For example, the image sensor100may generate output image data having a Bayer pattern. The processor12may perform an operation of converting a format of the output image data into full color image data having each of a red color, a green color, and a blue color.

FIG.2is a block diagram illustrating a structure of an image sensor according to an embodiment.

Referring toFIG.2, an image sensor100may include a pixel array110, a controller120, a signal processing unit130, a row driver140, and a readout circuit150. The readout circuit150may include correlated double sampling (CDS) unit151, an analog-to-digital converter (ADC)153, and a buffer155.

The pixel array110may convert an optical signal into an electrical signal, and may include a plurality of pixels PX that are two-dimensionally arranged. The plurality of pixels PX may respectively generate pixel signals according to a sensed intensity of light. The pixel PX may be implemented as, for example, a photoelectric conversion device, such as a charge coupled device (CCD) or CMOS, and may be implemented as various types of photoelectric conversion devices. The pixel array110may include a color filter configured to allow sensing of various colors, and each of the plurality of pixels PX may sense a corresponding color.

In an embodiment, the pixel array110may include pixel groups in which four pixels arranged in two columns and two rows share one microlens. In an embodiment, the pixel array110may include pixel groups in which two adjacently arranged pixels share one microlens. Each of the pixel groups may include a corresponding color filter. A detailed structure of an example of the pixel array110is described below with reference toFIG.3.

The plurality of pixels PX may respectively output pixel signals to the CDS unit151through corresponding first to nthcolumn output lines CLO_0to CLO_n−1. The CDS unit151may sample and hold a pixel signal provided from the pixel array110. The CDS unit151may double-sample a level of particular noise (which may be referred to as a reset leve), and a level according to an image signal (which may be referred to as an image level), and may output a level corresponding to a difference thereof. In addition, the CDS unit151may receive a lamp signal generated by a lamp signal generator157, and may output a comparison result by comparing the lamp signal with the pixel signal.

The ADC153may convert an analog signal corresponding to a level received from the CDS unit151into a digital signal. The buffer155may latch a digital signal, and the latched digital signal may be sequentially output as image data IDT to the outside of the signal processing unit130or the image sensor100.

The controller120may control the row driver140so that the pixel array110absorbs light to accumulate electric charges, temporarily stores the accumulated electric charges, and outputs an electrical signal according to the stored electric charges to the outside of the pixel array110. In addition, the controller120may control the readout circuit150to measure a level of a pixel signal provided by the pixel array110.

The row driver140may generate signals (e.g., reset control signals RSs, transmission control signals TSs, and selection signals SELSs) for controlling the pixel array110and provide the signals (e.g., the reset control signals RSs, the transmission control signals TSs, and the selection signals SELSs) to the plurality of pixels PX. The row driver140may determine activation and deactivation timings of the reset control signals RSs, the transmission control signals TSs, and the selection signals SELSs provided to the pixels PX.

The signal processing unit130may perform signal processing on the received image data IDT output from the readout circuit150. For example, the signal processing unit130may generate third image data including phase information regarding a first phase and fourth image data including phase information regarding a second phase from first image data including the phase information regarding the first phase, second image data including the phase information regarding the second phase, and full image data including color information. The signal processing unit130may generate phase difference data by calculating a phase difference between the first phase and the second phase using the first to fourth image data, and may transmit the phase difference data as output data DO to the processor12.

In addition, the signal processing unit130may generate image output data including color information. For example, the signal processing unit130may generate image output data having a Bayer pattern by performing a re-mosaic processing operation.

FIG.3is a view illustrating a pixel array of an image sensor according to an embodiment. In embodiments, the view illustrated inFIG.3may be an example of a portion of the pixel array110ofFIG.2.

Referring toFIG.3, a pixel array110may include a plurality of pixel groups, for example, first to sixteenth pixel groups PG1to PG16. The first to eighth pixel groups PG1to PG8may be arranged in a first row group RG1, and the ninth to sixteenth pixel groups PG9to PG16may be arranged in a second row group RG2.

Each of the first to sixteenth pixel groups PG1to PG16may include four pixels PX arranged in two rows and two columns (2×2). In addition, each of the first to sixteenth pixel groups PG to PG16may include one microlens ML disposed on the four pixels PX. Accordingly, all of a plurality of pixels PX included in the pixel array110may be AF pixels capable of performing an AF function.

A pixel signal generated by each of four pixels PX included in one pixel group in which one microlens ML is arranged may vary due to a shape and refractive index of the microlens ML. For example, a phase of each of pixel signals generated by pixels PX included in one pixel group may be changed. Therefore, an image sensor (e.g., the image sensor100inFIGS.1and2) or a processor (e.g., the processor12inFIG.1) according to embodiments may perform an AF function according to the pixel signals. When the image sensor100or processor12performs an image function of capturing an image without performing an AF function, a high-definition image may be provided by correcting a phase difference in each of pixel signals generated by pixels PX included in one pixel group.

The pixel array110may include a color filter configured to allow sensing of various colors. Each of the first to sixteenth pixel groups PG1to PG16may include one of a green color filter GF, a red color filter RF, and a blue color filter BF. In an embodiment, an arrangement ratio of the red color filter RF, the green color filter GF, and the blue color filter BF in the pixel array110may be 1:2:1.

In an embodiment, from among a plurality of pixel groups (e.g., the first to sixteenth pixel groups PG1to PG16) included in the pixel array110, four pixel groups, which are arranged adjacent to one another, may include the same color filter. A color filter may be arranged to form a Bayer pattern in units of four pixel groups from among the first to sixteenth pixel groups PG1to PG16. For example, the first to fourth pixel groups PG1to PG4and the thirteenth to sixteenth pixel groups PG13to PG16may include green color filters GF, the fifth to eighth pixel groups PG5to PG8may include the red color filter RF, and the ninth to twelfth pixel groups PG9to PG12may include the blue color filter BF. However, embodiments are not limited thereto, and each of the first to sixteenth pixel groups PG1to PG16may include at least one of a white color filter, a yellow color filter, a cyan color filter, and a magenta color filter. As another example, each of the first to sixteenth pixel groups PG1to PG16may include one of a white color filter, a yellow color filter, a green color filter GF, a red color filter RF, and a blue color filter BF.

FIG.4is an example circuit diagram of the first pixel group PG1and the third pixel group PG3ofFIG.3.FIG.4illustrates an embodiment in which pixels included in the first pixel group PG1and the third pixel group PG3share a floating diffusion region, but embodiments are not limited thereto. For example, pixels included in the first pixel group PG1and the second pixel group PG2may share a floating diffusion region, pixels included in the first to fourth pixel groups PG1to PG4may share a floating diffusion region, or each of the first to fourth pixel groups PG1to PG4may include a separate floating diffusion region.

Referring toFIGS.3and4, a first pixel of the first pixel group PG1may include a first photodiode PD11and a first transmission transistor TX11, and a second pixel of the first pixel group PG1may include a second photodiode PD12and a second transmission transistor TX12. A third pixel of the first pixel group PG1may include a third photodiode PD13and a third transmission transistor TX13, and a fourth pixel of the first pixel group PG1may include a fourth photodiode PD14and a fourth transmission transistor TX14.

A first pixel of the third pixel group PG3may include a first photodiode PD31and a first transmission transistor TX31, and a second pixel of the third pixel group PG3may include a second photodiode PD32and a second transmission transistor TX32. A third pixel of the third pixel group PG3may include a third photodiode PD33and a third transmission transistor TX33, and a fourth pixel of the third pixel group PG3may include a fourth photodiode PD34and a fourth transmission transistor TX34.

Each of the first to fourth photodiodes PD11to PD14of the first pixel group PG1and the first to fourth photodiodes PD31to PD34of the third pixel group PG3may be a photoelectric conversion device that generates a photocharge that varies according to an intensity of light. For example, each of the first to fourth photodiodes PD11to PD14of the first pixel group PG1and the first to fourth photodiodes PD31to PD34of the third pixel group PG3may a P-N junction diode, and may generate electric charges, i.e., electrons that are negative electric charges and holes that are positive electric charges, in proportion to an amount of incident light. An example of a photoelectric conversion device may include at least one of a photo transistor, a photo gate, a pinned photo diode (PPD), and a combination thereof.

Each of the first to fourth transmission transistors TX11to TX14of the first pixel group PG1may transmit the generated photocharge to a floating diffusion region FD in response to a corresponding transmission control signal (e.g., one of the transmission control signals TSs). For example, the first to fourth transmission transistors TX11to TX14may be controlled according to transmission control signals TS11to TS14, respectively. Each of the first to fourth transmission transistors TX31to TX34of the third pixel group PG3may transmit the generated photocharge to the floating diffusion region FD in response to a corresponding transmission control signal (e.g., one of transmission control signals TSs). For example, the first to fourth transmission transistors TX31to TX34may be controlled according to transmission control signals TS31to TS34, respectively.

The first pixel group PG1and the third pixel group PG3may share the floating diffusion region FD, a selection transistor SX, a source follower SF, and a reset transistor RX with each other. However, embodiments are not limited thereto. For example, at least one of the selection transistor SX, the source follower SF, and the reset transistor RX may be omitted. Pixels included in the first pixel group PG1and the third pixel group PG3may output a pixel signal VOUT using the same column output line (e.g., an i+1thcolumn output line CL0_i). Here, the i+1thcolumn output line CLO_i (where i is an integer greater than or equal to 0 and less than n−1) may be, for example, one column output line from among first to nthcolumn output lines CLO_0to CLO_n−1 inFIG.2.

The reset transistor RX may periodically reset electric charges accumulated in the floating diffusion region FD. The reset transistor RX may include a source electrode connected to the floating diffusion region FD, and a drain electrode connected to a power supply voltage VPIX. When the reset transistor RX is turned on according to a reset control signal RS, the power supply voltage VPIX connected to the drain electrode of the reset transistor RX may be transmitted to the floating diffusion region FD. When the reset transistor RX is turned on, electric charges accumulated in the floating diffusion region FD may be discharged, and thus, the floating diffusion region FD may be reset.

The source follower SF may be controlled according to an amount of photocharges accumulated in the floating diffusion region FD. The source follower SF may be a buffer amplifier, and may buffer a signal according to electric charges charged in the floating diffusion region FD. The source follower SF may amplify a potential change in the floating diffusion region FD, and may output the amplified potential change as the pixel signal VOUT using the i+1thcolumn output line CLO_i.

The selection transistor SX may include a drain terminal that is connected to a source terminal of the source follower SF, and may output the pixel signal VOUT to a CDS unit (e.g., the CDS unit151inFIG.2) through the i+1thcolumn output line CLO_i in response to a selection signal SELS.

FIG.5is a flowchart illustrating an operation of an image processing system according to embodiments.FIG.6is a view illustrating image data generated by an image sensor, according to embodiments. Pixels (e.g., first to fourth pixels PX11to PX14, first to fourth pixels PX21to PX24, first to fourth pixels PX31to PX34, and first to fourth pixels PX41to PX44) inFIG.6may be arranged in one row group (e.g., the first row group RG1or the second row group RG2inFIG.3), and may be included in four pixel groups (e.g., the first to fourth pixel groups PG1to PG4inFIG.3, or the thirteenth to sixteenth pixel groups PG13to PG16inFIG.3) including the same color filer.

Referring toFIGS.5and6, in operation S10, an image processing system (e.g., the image processing system10inFIG.1) may generate first image data IDTL including phase information regarding a first phase, second image data IDTR including phase information regarding a second phase, and full image data IDTS including color information. The first image data IDTL, the second image data IDTR, and the full image data IDTS may be image data generated by a readout circuit (e.g., the readout circuit150inFIG.2) of an image sensor (e.g., the image sensor100inFIG.2). The first phase and the second phase may be phases which are opposite to each other, and an AF function in a first direction (e.g., an X-axis direction inFIG.3) may be performed by calculating a phase difference between the first phase and the second phase.

Each pixel group included in a pixel array may include four pixels. For example, a first pixel group may include the first to fourth pixels PX11to PX14, a second pixel group may include the first to fourth pixels PX21to PX24, a third pixel group may include the first to fourth pixels PX31to PX34, and a fourth pixel group may include first to fourth pixels PX41to PX44.

The full image data IDTS may be image data according to pixel signals output from all pixels of the first to fourth pixel groups, for example all of the first to fourth pixels PX11to PX14, the first to fourth pixels PX21to PX24, the first to fourth pixels PX31to PX34, and the first to fourth pixels PX41to PX44.

The first image data IDTL including the phase information regarding the first phase may be image data according to pixel signals output from the first pixel PX11of the first pixel group, the first pixel PX21of the second pixel group, the third pixel PX33of the third pixel group, and the third pixel PX43of the fourth pixel group. For example, the first image data IDTL may be image data generated by sampling pixel data corresponding to the first pixel PX11of the first pixel group, the first pixel PX21of the second pixel group, the third pixel PX33of the third pixel group, and the third pixel PX43of the fourth pixel group. In an embodiment, when compared to the full image data IDTS, the first image data IDTL may be image data sampled at a sampling ratio of 1/4.

The second image data IDTR including the phase information regarding the second phase may be image data according to pixel signals output from the fourth pixel PX14of the first pixel group, the fourth pixel PX24of the second pixel group, the second pixel PX32of the third pixel group, and the second pixel PX42of the fourth pixel group. For example, the second image data IDTR may be image data generated by sampling pixel data corresponding to the fourth pixel PX14of the first pixel group, the fourth pixel PX24of the second pixel group, the second pixel PX32of the third pixel group, and the second pixel PX42of the fourth pixel group. In an embodiment, when compared to the full image data IDTS, the second image data IDTR may be image data sampled at a sampling ratio of 1/4. A sampling ratio for generating the first image data IDTL and the second image data IDTR may be changed according to an AF mode.

In operation S20, the image processing system10may generate third image data IDTL′ including the phase information regarding the first phase using the second image data IDTR and the full image data IDTS. In operation S30, the image processing system10may generate fourth image data IDTR′ including the phase information regarding the second phase using the first image data IDTL and the full image data IDTS. For example, the image processing system10may generate the third image data IDTL′ by subtracting, from a data value of the full image data IDTS, a data value obtained by multiplying the second image data IDTR by a certain gain α. In addition, for example, the image processing system10may generate the fourth image data IDTR′ by subtracting, from the data value of the full image data IDTS, a data value obtained by multiplying the first image data IDTL by the certain gain α. The calculation of multiplying each of the first image data IDTL and the second image data IDTR by the certain gain α may be performed to adjust a brightness (or for example a data size) of the first image data IDTL and the second image data IDTR to correspond to a brightness (or for example a data size) of the full image data IDTS.

The third image data IDTL′ including the phase information regarding the first phase may be image data corresponding to the first pixel PX11and the third pixel PX13of the first pixel group, the first pixel PX21and the third pixel PX23of the second pixel group, the first pixel PX31and the third pixel PX33of the third pixel group, and the first pixel PX41and the third pixel PX43of the fourth pixel group. The fourth image data IDTR′ including the phase information regarding the second phase may be image data corresponding to the second pixel PX12and the fourth pixel PX14of the first pixel group, the second pixel PX22and the fourth pixel PX24of the second pixel group, the second pixel PX32and the fourth pixel PX34of the third pixel group, and the second pixel PX42and the fourth pixel PX44of the fourth pixel group.

In operation S40, the image processing system10may calculate a phase difference between the first phase and the second phase using the first image data IDTL, the second image data IDTR, the third image data IDTL′, and the fourth image data IDTR′. For example, the image processing system10may calculate a phase difference in a horizontal direction such as an X-axis direction, and may perform an AF operation of adjusting a position of an optical lens, according to the calculated phase difference. An example of a method of calculating the phase difference is described below with reference toFIG.9A.

Accordingly, even when each of the first image data IDTL and the second image data IDTR is generated by sampling only pixel data regarding some pixels from a pixel group, the image processing system10according to embodiments may additionally generate the third image data IDTL′ regarding the first phase based on the second image data IDTR, and may additionally generate the fourth image data IDTR′ regarding the second phase based on the first image data IDTL. Accordingly, phase information included in the third image data IDTL′ may be referred to as additional phase information regarding the first phase, and phase information included in the fourth image data IDTR′ may be referred to as additional phase information regarding the second phase. Therefore, the image processing system10may compensate for phase information, which may be lost by sampling only some pixel data, using all of the first to fourth image data for a phase difference calculation, and an SNR of an AF function may increase.

Operations S10to S30ofFIG.5may be performed by an image sensor of the image processing system10(e.g., the image sensor100inFIG.1), and operation S40may be performed by the image sensor100, or may be performed by the image sensor100and the processor12.

FIG.7is a diagram illustrating a structure and operation of a signal processing unit of an image sensor according to an embodiment.

Referring toFIG.7, a signal processing unit130may include a front end processing module131(illustrated as ISP_FE), an auto-focusing processing module133(illustrated as ISP_AF), and an image processing module132(illustrated as ISP_IMG). In embodiments, the structures of modules described below may be software blocks executed by a processor, or may be implemented by a combination of a dedicated hardware block and a processing unit.

The front end processing module131may receive first image data IDTL_RG1generated from pixel groups arranged in a first row group RG1of a pixel array (e.g., the pixel array110inFIG.3), and full image data IDTS_RG1generated from the first row group RG1. The front end processing module131may receive second image data IDTR_RG2generated from pixel groups arranged in a second row group RG2of the pixel array110, and full image data IDTS_RG2generated from the pixel groups arranged in the second row group RG2.

The front end processing module131may generate fourth image data IDTR′_RG1using the full image data IDTS_RG1and the first image data IDTL_RG1. For example, the front end processing module131may generate the fourth image data IDTR′_RG1having a data value of R′ by subtracting, from a data value S of the full image data IDTS_RG1, a data value α·L obtained by multiplying a data value L of the first image data IDTL_RG1by a certain gain α. Here, the first image data IDTL_RG1may be image data including the phase information regarding the first phase, and the fourth image data IDTR′_RG1may include the phase information regarding the second phase opposite to the first phase.

In addition, the front end processing module131may generate third image data IDTL′_RG2using the full image data IDTS_RG2and the second image data IDTR_RG2. For example, the front end processing module131may generate the third image data IDTL′_RG2having a data value of L′ by subtracting, from the data value S of the full image data IDTS_RG2, a data value α·R obtained by multiplying a data value R of the second image data IDTR_RG2by the certain gain α. Here, the second image data IDTR_RG2may be image data including the phase information regarding the second phase that is an opposite phase to the first phase, and the third image data IDTL′_RG2may include the phase information regarding the first phase.

The first image data IDTL_RG1in the first row group RG1may correspond to the first image data IDTL inFIG.6, the second image data IDTR_RG2in the second row group RG2may correspond to the second image data IDTR inFIG.6, and the full image data IDTS_RG1in the first row group RG1and the full image data IDTS_RG2in the second row group RG2may correspond to the full image data IDTS inFIG.6. In addition, the fourth image data IDTR′_RG1in the first row group RG1may correspond to the fourth image data IDTR′ inFIG.6, and the third image data IDTL′_RG2in the second row group RG2may correspond to the third image data IDTL′ inFIG.6.

FIG.7illustrates only image data generated from pixel groups arranged in the first row group RG1and the second row group RG2, but embodiments are not limited thereto. For example, other row groups arranged sequentially after the first row group RG1and the second row group RG2, e.g., pixel groups in a third row group, may generate second image data including the phase information regarding the second phase, and pixel groups arranged in a fourth row group may generate first image data including the phase information regarding the first phase.

The auto-focusing processing module133may receive the first image data IDTL_RG1, the second image data IDTR_RG2, the third image data IDTL′_RG2, and the fourth image data IDTR′_RG1from the front end processing module131. In an embodiment, the auto-focusing processing module133may calculate a phase difference between the first phase and the second phase using the first image data IDTL_RG1, the second image data IDTR_RG2, the third image data IDTL′_RG2, and the fourth image data IDTR′_RG1, and may generate phase difference data DD including phase difference information. In an embodiment, the auto-focusing processing module133may perform preprocessing for a phase difference calculation in a processor (e.g., the processor12inFIG.1) using the first image data IDTL_RG1, the second image data IDTR_RG2, the third image data IDTL′_RG2, and the fourth image data IDTR′_RG1. An example of a detailed structure and operation of the auto-focusing processing module133is described below with reference toFIG.9A.

The image processing module132may receive, from the front end processing module131, the full image data IDTS_RG1and IDTS_RG2. The full image data IDTS_RG1and IDTS_RG2may be image data generated by the pixel array110described with reference toFIG.3. For example, the full image data IDTS_RG1and IDTS_RG2may be image data generated by the pixel array110including a color filter array in which one pixel group includes four pixels arranged in 2×2, and four adjacently-arranged pixel groups include one color filter corresponding to the red color filter RF, the green color filter GF, and the blue color filter BF.

The image processing module132may generate image output data IOD having a Bayer pattern by re-mosaic processing the full image data IDTS_RG1and IDTS_RG2. In an embodiment, the image processing module132may convert the full image data IDTS_RG1and IDTS_RG2into red color data, green color data, and blue color data, and output the red color data, the green color data, and the blue color data by re-mosic processing and Bayer demosaic processing the full image data IDTS_RG1and the IDTS_RG2.

FIG.8is a view illustrating image data and fourth image data generated by an image sensor, according to embodiments. In the description ofFIG.8, description which is redundant or duplicative with respect to the description ofFIG.6may be omitted.

Referring toFIG.8, an image sensor (e.g., the image sensor100inFIG.2) may generate first image data IDTT including phase information regarding a third phase, second image data IDTB including phase information regarding a fourth phase, and full image data IDTS including color information. The third phase and the fourth phase may be phases that are opposite to each other, and an AF function in a second direction (e.g., a Y-axis direction inFIG.3) may be performed by calculating a phase difference between the third phase and the fourth phase.

The first image data IDTT including the phase information regarding the third phase may be image data according to pixel signals output from a first pixel PX11of a first pixel group, a second pixel PX22of a second pixel group, a first pixel PX31of a third pixel group, and a second pixel PX42of a fourth pixel group. For example, the first image data IDTT may be image data generated by sampling pixel data corresponding to the first pixel PX11of the first pixel group, the second pixel PX22of the second pixel group, the first pixel PX31of the third pixel group, and the second pixel PX42of the fourth pixel group. In an embodiment, when compared to the full image data IDTS, the first image data IDTT may be image data sampled at a sampling ratio of 1/4.

The second image data IDTB including the phase information regarding the fourth phase may be image data according to pixel signals output from a third pixel PX13of the first pixel group, a fourth pixel PX24of the second pixel group, a third pixel PX33of the third pixel group, and a fourth pixel PX44of the fourth pixel group. For example, the second image data IDTB may be image data generated by sampling pixel data corresponding to the third pixel PX13of the first pixel group, the fourth pixel PX24of the second pixel group, the third pixel PX33of the third pixel group, and the fourth pixel PX44of the fourth pixel group. In an embodiment, when compared to the full image data IDTS, the second image data IDTB may be image data sampled at a sampling ratio of 1/4.

Third image data IDTT′ including the phase information regarding the third phase may be generated using the second image data IDTB and the full image data IDTS. Fourth image data IDTB′ including the phase information regarding the fourth phase may be generated using the first image data IDTT and the full image data IDTS. In embodiments, the third image data IDTT′ and the fourth image data IDTB′ may be generated by the front end processing module131of the signal processing unit130illustrated inFIG.7. For example, the front end processing module131may generate the third image data IDTT′ by subtracting, from a data value of the full image data IDTS, a data value obtained by multiplying the second image data IDTB by a certain gain α. In addition, for example, the front end processing module131may generate the fourth image data IDTB′ by subtracting, from the data value of the full image data IDTS, a data value obtained by multiplying the first image data IDTT by the certain gain α.

The third image data IDTT′ including the phase information regarding the third phase may be image data including pixel data corresponding to the first pixel PX11and a second pixel PX12of the first pixel group, the first pixel PX21and a second pixel PX22of the second pixel group, the first pixel PX31and a second pixel PX32of the third pixel group, and the first pixel PX41and a second pixel PX42of the fourth pixel group. The fourth image data IDTB′ including the phase information regarding the fourth phase may be image data including pixel data corresponding to the third pixel PX13and a fourth pixel PX14of the first pixel group, the third pixel PX23and a fourth pixel PX24of the second pixel group, the third pixel PX33and a fourth pixel PX34of the third pixel group, and the third pixel PX43and a fourth pixel PX44of the fourth pixel group.

The auto-focusing processing module133of the signal processing unit130of the image sensor100may calculate a phase difference between the third phase and the fourth phase using the first image data IDTT, the second image data IDTB, the third image data IDTT′, and the fourth image data IDTB′. The third image data IDTT and the fourth image data IDTB′ may be generated by the front end processing module131of the signal processing unit130. For example, the image sensor100may calculate a phase difference in a vertical direction that is the Y-axis direction, and the image processing system10may perform an AF operation of adjusting a position of an optical lens (e.g., the optical lens111_1inFIG.1), according to the calculated phase difference.

FIGS.9A and9Bare block diagrams illustrating structures and operations of auto-focusing processing modules133and133aof an image sensor, according to embodiments.FIGS.9A and9Billustrate an example of data processing on the first image data IDTL, the third image data IDTL′, the second image data IDTR, and the fourth image data IDTR′ described with reference toFIG.6, but embodiments are not limited thereto. For example, similar data processing may be performed on the first image data IDTT, the third image data IDTT′, the second image data IDTB, and the fourth image data IDTB′ described with reference toFIG.8.

Referring toFIGS.6and9A, the auto-focusing processing module133may include a binning module1331, a first merge module1332, a second merge module1333, and a phase difference calculation module1334. The auto-focusing processing module133may generate phase difference data DD including phase difference information in an X-axis direction by receiving first image data IDTL, third image data IDTL′, second image data IDTR, and fourth image data IDTR′.

The binning module1331may generate first binning data BD1, third binning data BD3, second binning data BD2, and fourth binning data BD4by binning the first image data IDTL, the third image data IDTL′, and the second image data IDTR, and the fourth image data IDTR′, respectively. For example, the binning module1331may perform binning in units of pixel groups in which the same color filter is arranged and which are adjacently arranged, and may bin pixel data regarding pixels in first to fourth pixel groups (e.g., the first to fourth pixels PX11to PX14, the first to fourth pixels PX21to PX24, the first to fourth pixels PX31to PX34, and the first to fourth pixels PX41to PX44). For example, the binning module1331may perform binning in units of pixels arranged in a row group, and as a result, may include data values corresponding to the pixels arranged along the row group.

The first merge module1332may generate first merge data MD1by merging the first binning data BD1and the third binning data BD3regarding a first phase, and the second merge module1333may generate second merged data MD2by merging the second binning data BD2and the fourth binning data BD4regarding a second phase. The first merge module1332may generate the first merge data MD1based on Equation 1 below. The second merge module1333may also generate the second merge data MD2similarly to Equation 1 below.

Here, {circumflex over (L)}(x, y) may denote a data value corresponding to a pixel located at (x, y) in the first merge data MD1. L(x, y) may denote a data value corresponding to a pixel located at (x, y) in the first binning data BD1, and L′(x, y+i) may denote a data value corresponding to pixels arranged in a Y-axis direction perpendicular to a direction of a phase difference (e.g., the X-axis direction) based on a pixel located at (x, y) in the third binning data BD3. Therefore, L(x, y) may be a data value generated using a readout operation of a readout circuit of an image sensor, and L′(x, y+i) may be a data value generated using an operation of a signal processing unit of the image sensor. Further, a(i) may denote a correction coefficient (a weight) of the data value of L′(x,y+i). For example, a(i) may be a function in which a center value is 0 in a Gaussian function.

Accordingly, the first merge module1332may generate the first merge data MD1by adding a data value for a first pixel arranged at a position (x, y) in the first binning data BD1to values obtained by applying a weight to data values of respective pixels arranged in the Y-axis direction perpendicular to the X-axis direction, which may be the direction of the phase difference, based on a second pixel arranged at the position (x, y) in the third binning data BD3.

The phase difference calculation module1334may calculate a phase difference in the X-axis direction between the first phase and the second phase using the first merge data MD1and the second merge data MD2. According to a result of the calculation, the phase difference calculation module1334may output the phase difference data DD including the phase difference information.

Referring toFIG.9B, the auto-focusing processing module133amay include a binning module1331, a first merge module1332, and a second merge module1333. The auto-focusing processing module133amay output first merge data MD1including phase information regarding a first phase and second merge data MD2including phase information regarding a second phase by receiving first image data IDTL, third image data IDTL′, second image data IDTR, and fourth image data IDTR′.

The phase difference calculation module12_1may be included, for example, in the processor12inFIG.1. The phase difference calculation module12_1may calculate a phase difference in an X-axis direction between the first phase and the second phase using the first merge data MD1and the second merge data MD2. According to a result of the calculation, the phase difference calculation module12_1may output phase difference data DD including phase difference information.

Referring toFIGS.9A and9B, in an embodiment, the phase difference data DD generated to perform an AF operation in the Y-axis direction (which may be, for example, a vertical direction) may be generated by the auto-focusing processing module133, and the phase difference data DD generated to perform an AF operation in the X-axis direction (which may be, for example, a horizontal direction) may be generated by the phase difference calculation module12_1. In embodiments, it may be beneficial for an image processing system to have accuracy of the AF operation in the X-axis direction rather than accuracy of the AF operation in the Y-axis direction. Therefore, a phase difference calculation for performing the AF operation in the X-axis direction using a relatively large amount of data processing may be performed by the processor12. However, embodiments are not limited thereto.

FIGS.10A and10Bare block diagrams illustrating structures and operations of auto-focusing processing modules133band133cof an image sensor, according to embodiments.FIGS.10A and10Billustrate an example of data processing on the first image data IDTL, the third image data IDTL′, the second image data IDTR, and the fourth image data IDTR′ described with reference toFIG.6, but embodiments are not limited thereto. For example, similar data processing may be performed on the first image data IDTT, the third image data IDTT′, the second image data IDTB, and the fourth image data IDTB′ described with reference toFIG.8.

Referring toFIGS.6and10A, the auto-focusing processing module133bmay include a binning module1331, a first phase difference calculation module1335, a second phase difference calculation module1336, and a merge module1337. The auto-focusing processing module133bmay generate phase difference data DD including phase difference information in an X-axis direction by receiving the first image data IDTL, the second image data IDTR, the third image data IDTL′, and the fourth image data IDTR′. The binning module1331may perform operations which are similar to operations of the binning module1331inFIG.9A.

The first phase difference calculation module1335may calculate a phase difference in the X-axis direction between a first phase and a second phase using first binning data BD1and second binning data BD2. According to a result of the calculation, the first phase difference calculation module1335may output first phase difference data D1including the phase difference information.

The second phase difference calculation module1336may calculate the phase difference in the X-axis direction between the first phase and the second phase using third binning data BD3and fourth binning data BD4. According to a result of the calculation, the second phase difference calculation module1336may output second phase difference data D2including the phase difference information.

The merge module1337may generate phase difference data DD using the first phase difference data D1and the second phase difference data D2. For example, the merge module1337may generate the phase difference data DD by averaging the first phase difference data D1and the second phase difference data D2. For example, the merge module1337may calculate a data value (d=(d1+d2)/2) of the phase difference data DD by averaging a data value d1of the first phase difference data D1and a data value d2of the second phase difference data D2.

Referring toFIG.10B, the auto-focusing processing module133cmay include a binning module110_1. A first phase difference calculation module12_2, a second phase difference calculation module12_3, and a merge module12_4may perform operations which are similar to operations of the first phase difference calculation module1335, the second phase difference calculation module1336, and the merge module1337inFIG.10A. For example, the first phase difference calculation module12_2, the second phase difference calculation module12_3, and the merge module12_4may be, for example, included in the processor12inFIG.1.

Referring toFIGS.10A and10B, in an embodiment, the phase difference data DD generated to perform an AF operation in a Y-axis direction may be generated by the auto-focusing processing module133b, and the phase difference data DD generated to perform an AF operation in the X-axis direction may be generated by the merge module12_4. An image processing system may need accuracy of the AF operation in the X-axis direction rather than accuracy of the AF operation in the Y-axis direction. Therefore, a phase difference calculation for performing the AF operation in the X-axis direction using a relatively large amount of data processing may be performed by the processor12. However, embodiments are not limited thereto.

FIG.11is a flowchart illustrating an operation of an image processing system according to embodiments.FIG.12is a view illustrating image data generated by an image sensor. Pixels PX11to PX14, PX21to PX24, PX31to PX34, and PX41to PX44inFIG.12may be arranged in one row group (e.g., the first row group RG1or the second row group RG2inFIG.3), and may be included in four pixel groups (e.g., the first to fourth pixel groups PG1to PG4inFIG.3, or the thirteenth to sixteenth pixel groups PG13to PG16inFIG.3).

Referring toFIGS.11and12, in operation S50, an image processing system (e.g., the image processing system10inFIG.1) may change a sampling ratio according to a change in an AF mode, and may generate first image data IDTLa including phase information regarding a first phase, and full image data IDTS including color information.

The first image data IDTLa in operation S50may have a higher sampling ratio than the first image data IDTL in operation S10. For example, the first image data IDTLa may include pixel data regarding all pixels capable of generating the phase information regarding the first phase. The first image data IDTLa and the full image data IDTS may be image data generated by a readout circuit (e.g., the readout circuit150inFIG.2) of an image sensor (e.g., the image sensor100inFIG.2).

Each pixel group included in a pixel array may include four pixels. For example, a first pixel group may include first to fourth pixels PX11to PX14, a second pixel group may include first to fourth pixels PX21to PX24, a third pixel group may include first to fourth pixels PX31to PX34, and a fourth pixel group may include first to fourth pixels PX41to PX44.

The full image data IDTS may be image data according to pixel signals output from all pixels included in the first to fourth pixel groups, for example all of the first to fourth pixels PX11to PX14, the first to fourth pixels PX21to PX24, the first to fourth pixels PX31to PX34, and the first to fourth pixels PX41to PX44. The first image data IDTLa including the phase information regarding the first phase may be image data according to pixel signals output from the first pixel PX11and the third pixel PX13of the first pixel group, the first pixel PX21and the third pixel PX23of the second pixel group, the first pixel PX31and the third pixel PX33of the third pixel group, and the first pixel PX41and the third pixel PX43of the fourth pixel group.

In operation S60, the image processing system10may generate second image data IDTRa′ including phase information regarding a second phase, which may be a phase opposite to the first phase, using the first image data IDTLa and the full image data IDTS. For example, the image processing system10may generate the second image data IDTRa′ by subtracting a data value of the first image data IDTLa from a data value of the full image data IDTS.

The second image data IDTRa′ including the phase information regarding the second phase may be image data corresponding to the second pixel PX12and the fourth pixel PX14of the first pixel group, the second pixel PX22and the fourth pixel PX24of the second pixel group, the second pixel PX32and the fourth pixel PX34of the third pixel group, and the second pixel PX42and the fourth pixel PX44of the fourth pixel group.

In operation S70, the image processing system10may calculate a phase difference between the first phase and the second phase using the first image data IDTLa and the second image data IDTRa′. For example, the image processing system10may calculate a phase difference in a horizontal such as an X-axis direction, and may perform an AF operation of adjusting a position of an optical lens, according to the calculated phase difference. For example, the operation of calculating the phase difference may be performed by the phase difference calculation module1334and the phase difference calculation module12_1described with reference toFIGS.9A and9B.

When relatively higher accuracy of the AF operation is desired, the image processing system10may perform operations S50to S70inFIG.11instead of performing operations S10to S40inFIG.5by further increasing a sampling ratio when generating first image data according to a change in an AF mode. For example, operations S10to S40inFIG.5may be performed while the image processing system operates in a first AF mode, and operations S50to S70inFIG.11may be performed while the image processing system operates in a second AF mode.

FIG.13is a view illustrating a pixel array110aof an image sensor according to an embodiment, and is a view illustrating an example of a portion of the pixel array110inFIG.2.

Referring toFIG.13, the pixel array110amay include a plurality of pixel groups, for example, first to sixteenth pixel groups PG1to PG16. The first to eighth pixel groups PG1to PG8may be arranged in a first row group RG1, and the ninth to sixteenth pixel groups PG9to PG16may be arranged in a second row group RG2.

Each of the first to sixteenth pixel groups PG1to PG16may include two pixels PX arranged in one row and two columns (1×2). In addition, each of the first to sixteenth pixel groups PG1to PG16may include one microlens ML disposed on the two pixels PX. A pixel signal generated by each of two pixels PX included in one pixel group in which one microlens ML is arranged may vary due to a shape and refractive index of the microlens ML. Accordingly, all of a plurality of pixels PX included in the pixel array110amay be AF pixels capable of performing an AF function.

The pixel array110amay include a color filter configured to allow sensing of various colors. Each of the first to sixteenth pixel groups PG1to PG16may include one of a green color filter GF, a red color filter RF, and a blue color filter BF. In an embodiment, an arrangement ratio of the red color filter RF, the green color filter GF, and the blue color filter BF in the pixel array110amay be 1:2:1.

In an embodiment, from among a plurality of pixel groups (e.g., the first to sixteenth pixel groups PG1to PG16) included in the pixel array110a, four pixel groups, which are arranged adjacent to one another, may include the same color filter. A color filter may be arranged to form a Bayer pattern in units of four pixel groups from among the first to sixteenth pixel groups PG1to PG16. For example, the first to fourth pixel groups PG1to PG4and the thirteenth to sixteenth pixel groups PG13to PG16may include the green color filters GF, the fifth to eighth pixel groups PG5to PG8may include the red color filter RF, and the ninth to twelfth pixel groups PG9to PG12may include the blue color filter BF.

FIG.14is a view illustrating image data generated by an image sensor according to embodiments. Pixels PX11A, PX12A, PX21A, PX22A, PX31A, PX32A, PX41A, and PX42A inFIG.14may be arranged in one row group (e.g., the first row group RG1or the second row group RG2inFIG.13), and may be included in four pixel groups (e.g., the first to fourth pixel groups PG1to PG4inFIG.13or the thirteenth to sixteenth pixel groups PG13to PG16inFIG.13) including the same color filter.

Referring toFIG.14, each pixel group included in a pixel array may include two pixels. For example, a first pixel group may include a first pixel PX11A and a second pixel PX12A arranged adjacent to each other in an X-axis direction, a second pixel group may include a first pixel PX21A and a second pixel PX22A arranged adjacent to each other in the X-axis direction, a third pixel group may include a first pixel PX31A and a second pixel PX32A arranged adjacent to each other in the X-axis direction, and a fourth pixel group may include a first pixel PX41A and a second pixel PX42A arranged adjacent to each other in the X-axis direction.

Full image data IDTSb may be image data according to pixel signals output from all pixels of the first to fourth pixel groups, for example all of the first pixel PX11A and the second pixel PX12A of the first pixel group, the first pixel PX21A and the second pixel PX22A of the second pixel group, the first pixel PX31A and the second pixel PX32A of the third pixel group, and the first pixel PX41A and the second pixel PX42A of the fourth pixel group.

First image data IDTLb including phase information regarding a first phase may be image data according to pixel signals output from the first pixel PX11A of the first pixel group and the first pixel PX41A of the fourth pixel group. For example, the first image data IDTLb may be image data generated by sampling pixel data corresponding to the first pixel PX11A of the first pixel group and the first pixel PX41A of the fourth pixel group. In an embodiment, when compared to the full image data IDTSb, the first image data IDTLb may be image data sampled at a sampling ratio of 1/4.

Second image data IDTRb including phase information regarding a second phase may be image data according to pixel signals output from the second pixel PX22A of the second pixel group and the second pixel PX32A of the third pixel group. For example, the second image data IDTRb may be image data generated by sampling pixel data corresponding to the second pixel PX22A of the second pixel group and the second pixel PX32A of the third pixel group. In an embodiment, when compared to the full image data IDTSb, the second image data IDTRb may be image data sampled at a sampling ratio of 1/4.

The image processing system10may generate third image data IDTLb′ including the phase information regarding the first phase using the second image data IDTRb and the full image data IDTSb, and the image processing system10may generate fourth image data IDTRb′ including the phase information regarding the second phase using the first image data IDTLb and the full image data IDTSb. For example, the image processing system10may generate the third image data IDTLb′ by subtracting, from a data value of the full image data IDTSb, a data value obtained by multiplying the second image data IDTRb by a certain gain α′. In addition, for example, the image processing system10may generate the fourth image data IDTRb′ by subtracting, from the data value of the full image data IDTSb, a data value obtained by multiplying the first image data IDTLb by the certain gain α′. The calculation of multiplying each of the first image data IDTLb and the second image data IDTRb by the certain gain α′ may be performed to adjust a brightness of the first image data IDTLb and the second image data IDTRb to correspond to a brightness of the full image data IDTSb.

The third image data IDTLb′ including the phase information regarding the first phase may be image data corresponding to the first pixel PX11A of the first pixel group, the first pixel PX21A of the second pixel group, the first pixel PX31A of the third pixel group, and the first pixel PX41A of the fourth pixel group. The fourth image data IDTRb′ including the phase information regarding the second phase may be image data corresponding to the second pixel PX12A of the first pixel group, the second pixel PX22A of the second pixel group, the second pixel PX32A of the third pixel group, and the second pixel PX42A of the fourth pixel group.

The image processing system10may calculate a phase difference between the first phase and the second phase using the first image data IDTLb, the second image data IDTRb, the third image data IDTLb′, and the fourth image data IDTRb′. For example, the image processing system10may calculate a phase difference in a horizontal direction such as an X-axis direction, and may perform an AF operation of adjusting a position of an optical lens, according to the calculated phase difference.

FIG.15is a view illustrating image data generated by an image sensor according to embodiments. In the description ofFIG.15, description which is redundant or duplicative with respect to the description ofFIG.14may be omitted.

Referring toFIG.15, an image sensor (e.g., the image sensor100inFIG.2) may generate first image data IDTTc including phase information regarding a third phase, second image data IDTBc including phase information regarding a fourth phase, and full image data IDTSc including color information. The third phase and the fourth phase may be opposite phases to each other, and an AF function in a Y-axis direction may be performed by calculating a phase difference between the third phase and the fourth phase.

Each pixel group included in a pixel array may include two pixels. For example, a first pixel group may include a first pixel PX11B and a second pixel PX12B arranged adjacent to each other in a Y-axis direction, a second pixel group may include a first pixel PX21B and a second pixel PX22B arranged adjacent to each other in the Y-axis direction, a third pixel group may include a first pixel PX31B and a second pixel PX32B arranged adjacent to each other in the Y-axis direction, and a fourth pixel group may include a first pixel PX41B and a second pixel PX42B arranged adjacent to each other in the Y-axis direction.

The full image data IDTSc may be image data according to pixel signals output from all pixels of the first to fourth pixel groups, for example all of the first pixel PX11B and the second pixel PX12B of the first pixel group, the first pixel PX21B and the second pixel PX22B of the second pixel group, the first pixel PX31B and the second pixel PX32B of the third pixel group, and the first pixel PX41B and the second pixel PX42B of the fourth pixel group.

The first image data IDTTc including the phase information regarding the third phase may be image data according to pixel signals output from the first pixel PX11B of the first pixel group and the first pixel PX41B of the fourth pixel group. For example, the first image data IDTTc may be image data generated by sampling pixel data corresponding to the first pixel PX11B of the first pixel group and the first pixel PX41B of the fourth pixel group. In an embodiment, when compared to the full image data IDTSc, the first image data IDTTc may be image data sampled at a sampling ratio of 1/4.

The second image data IDTBc including the phase information regarding the fourth phase may be image data according to pixel signals output from the second pixel PX22B of the second pixel group and the second pixel PX32B of the third pixel group. For example, the second image data IDTBc may be image data generated by sampling pixel data corresponding to the second pixel PX22B of the second pixel group and the second pixel PX32B of the third pixel group. In an embodiment, when compared to the full image data IDTSc, the second image data IDTBc may be image data sampled at a sampling ratio of 1/4.

The image processing system10may generate third image data IDTTc′ including the phase information regarding the third phase using the second image data IDTBc and the full image data IDTSc, and the image processing system10may generate fourth image data IDTBc′ including the phase information regarding the fourth phase using the first image data IDTTc and the full image data IDTSc. For example, the image processing system10may generate the third image data IDTTc′ by subtracting, from a data value of the full image data IDTSc, a data value obtained by multiplying the second image data IDTBc by a certain gain α′. In addition, for example, the image processing system10may generate the fourth image data IDTBc′ by subtracting, from the data value of the full image data IDTSc, a data value obtained by multiplying the first image data IDTTc by the certain gain α′.

The third image data IDTTc′ including the phase information regarding the third phase may be image data corresponding to the first pixel PX11B of the first pixel group, the first pixel PX21B of the second pixel group, the first pixel PX31B of the third pixel group, and the first pixel PX41B of the fourth pixel group. The fourth image data IDTBc′ including the phase information regarding the fourth phase may be image data corresponding to the second pixel PX12B of the first pixel group, the second pixel PX22B of the second pixel group, the second pixel PX32B of the third pixel group, and the second pixel PX42B of the fourth pixel group.

The image processing system10may calculate a phase difference between the third phase and the fourth phase using the first image data IDTTc, the second image data IDTBc, the third image data IDTTc′, and the fourth image data IDTBc′. For example, the image processing system10may calculate a phase difference in the vertical direction that is the Y-axis direction, and may perform an AF operation of adjusting a position of an optical lens, according to the calculated phase difference.

FIG.16is a view illustrating a pixel array110bof an image sensor according to an embodiment, and is a view illustrating an example of a portion of the pixel array110inFIG.2.

Referring toFIG.16, the pixel array110bmay include a plurality of pixel groups, for example, first to fourth pixel groups PG1to PG4. The first pixel group PG1and the second pixel group PG2may be arranged in a first row group RG1, and the third pixel group PG3and the fourth pixel group PG4may be arranged in a second row group RG2.

Each of the first to fourth pixel groups PG1to PG4may include four pixels PX arranged in two rows and two columns 2×2. In addition, each of the first to fourth pixel groups PG1to PG4may include one microlens ML disposed on the four pixels PX. Accordingly, all of a plurality of pixels PX included in the pixel array110bmay be AF pixels capable of performing an AF function.

The pixel array110bmay include a color filter configured to allow sensing of various colors. Each of the first to fourth pixel groups PG1to PG4may include one of a green color filter GF, a red color filter RF, and a blue color filter BF. In an embodiment, a color filter may be arranged in the first to fourth pixel groups PG1to PG4to form a Bayer pattern in units of pixel groups.

An image sensor and an image processing system including the pixel array110billustrated inFIG.16may perform operations S10to S40described with reference toFIG.5. When performing an AF operation by sampling only some pixel data, the image sensor and the image processing system may generate third image data regarding a first phase, which is an opposite phase to a second phase, from second image data regarding the second phase, and may generate fourth image data regarding the second phase from first image data regarding the first phase. By using all of the first to fourth image data for a phase difference calculation, phase information, which may be lost by sampling only some pixel data, may be compensated for, and accuracy of an AF function may increase.

FIG.17is a schematic view illustrating an image sensor IS according to an embodiment.

Referring toFIG.17, the image sensor IS may be a stacked image sensor including a first chip CP1and a second chip CP2stacked in a vertical direction. The image sensor IS may be an implementation of the image sensor100described with reference toFIG.2. AlthoughFIG.17illustrates the image sensor IS as a structure in which two chips are stacked, embodiments are not limited thereto. For example, in embodiments the image sensor IS may have a structure in which three chips are stacked.

The first chip CP1may include a pixel region PR and a pad region PR1, and the second chip CP2may include a peripheral circuit region PR3and a lower pad region PR2. A pixel array in which a plurality of pixels PX are arranged may be formed in the pixel region PR, and may include the pixel arrays110,110a, and110bdescribed with reference toFIGS.3to16.

The peripheral circuit region PR3of the second chip CP2may include a logic circuit block LC, and may include a plurality of transistors. The peripheral circuit region PR3may provide a constant signal to each of the plurality of pixels PX included in the pixel region PR, and may read a pixel signal output from each of the plurality of pixels PX. The readout circuit150inFIG.2may be arranged in the peripheral circuit region PR3. The logic circuit block LC may include a signal processing unit SPU. The signal processing unit SPU may correspond to the signal processing unit130inFIG.2.

The lower pad region PR2of the second chip CP2may include a lower conductive pad PAD′. A plurality of lower conductive pads PAD′ may be included, and may correspond to conductive pads PAD, respectively. The lower conductive pad PAD′ may be electrically connected to the conductive pad PAD of the first chip CP1by a via structure VS.