Patent ID: 12207007

DETAILED DESCRIPTIONS

Terms used in the present specification will be briefly described, and embodiments will be described in detail.

The terms used in the disclosed embodiments have been selected as currently widely used general terms as possible by considering the functions in the disclosed embodiments, which may change depending on intention of a person skilled in the art or a precedent, emergence of a new technology, and so on. In addition, in a certain case, there are also terms randomly selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the disclosed embodiments should be defined based on meaning of the terms and contents of the present specification, rather than simple names of the terms.

Terms including an ordinal number, such as first, second, and so on, may be used to describe various components, but the components are not limited to the terms. Terms are used only for the purpose of distinguishing one component from another component. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component without departing from the scope of the disclosed embodiments. The term “and/or” includes a combination of a plurality of related items or any one of the plurality of related items.

Hereinafter, descriptions will be made in detail so as to be easily implemented by those skilled in the art to which the disclosed embodiments pertain with reference to the accompanying drawings. However, embodiments may be implemented in several different forms and are not limited to the embodiments described herein. In order to clearly describe the embodiments in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

FIG.1is a block diagram illustrating an image sensor inspection system1according to some embodiments, andFIG.2is a block diagram illustrating an image sensor100according to some embodiments.

Referring toFIGS.1and2, the image sensor inspection system1according to the embodiment includes the image sensor100and a test device200.

Referring toFIG.2, the image sensor100includes a pixel array110, a row driver120, an analog-to-digital conversion circuit130(hereinafter, referred to as an ADC circuit), a ramp signal generator140, a timing controller150, a processor160, and an interface170.

The pixel array110includes a plurality of pixels PX arranged in a matrix, a plurality of row lines RL, and a plurality of column lines CL respectively connected to the plurality of pixels PX. Each of the plurality of pixels PX may include at least one photoelectric conversion element (or referred to as a photo-sensing element). The photoelectric conversion element may sense light and convert the sensed light into photo-charges. For example, the photoelectric conversion element may be a photo-sensing element, which is composed of an organic material or an inorganic material, such as an inorganic photodiode, an organic photodiode, a perovskite photodiode, a phototransistor, a photogate or a pinned photodiode. In some embodiments, each of the plurality of pixels PX may include a plurality of photoelectric conversion elements. The plurality of photo-sensing elements may be arranged on the same layer or may also be stacked in directions perpendicular to each other.

A micro lens for light collection may be arranged over each of the plurality of pixels PX or each of pixel groups including adjacent pixels PX. Each of the plurality of pixels PX may detect light in a spectrum region from light received through the micro lens.

The pixel array110according to some embodiments may include pixels having a red, green, and blue (RGB) pattern) or a red, green, blue, white color (RGBWC) pattern. For example, the pixel array110may include red pixels that convert light in a red spectrum region into an electrical signal, green pixels that convert light in a green spectrum region into an electrical signal, and blue pixels that convert light in a blue spectrum region into an electrical signal. In addition, the pixel array110may include white pixels that convert light having all components in the red spectrum region, the green spectrum region, and the blue spectrum region into electrical signals. A color filter through which light is transmitted in a certain spectrum region may be over each of the plurality of pixels PX. However, the inventive concept is not limited thereto, and the pixel array110may include pixels that convert lights in spectrum regions other than red, green, and blue into electrical signals.

The plurality of pixels PX according to some embodiments may also have a multi-layer structure. The pixel PX having a multi-layer structure includes stacked photo-sensing elements that convert lights in different spectrum regions into electrical signals, and electrical signals corresponding to different colors may be generated from the photo-sensing elements. In other words, electrical signals corresponding to a plurality of colors may be output from one pixel PX.

In addition, the pixel array110may include a plurality of row lines and a plurality of column lines. The plurality of row lines RL may each extend in a row direction and may each be connected to pixels PX in the same row. For example, each of the plurality of row lines RL may transmit a control signal output from the row driver120to elements included in the pixels PX, for example, each of a plurality of transistors in the row.

The plurality of column lines CL may extend in a column direction and may be connected to the pixels PX in the same column. Each of the plurality of column lines CL may transmit pixel signals output from the pixels PX, for example, reset signals and sensing signals, to the ADC circuit130in units of columns of the pixel array110.

The row driver120generates control signals for driving the pixel array110under the control of the timing controller150and provide the control signals to each of the plurality of pixels PX of the pixel array110through the plurality of row lines RL. The row driver120may control the plurality of pixels PX of the pixel array110to sense light incident at the same time or in units of rows. In addition, the row driver120may select the pixels PX in units of rows or in units of at least two rows from among the plurality of pixels PX, and the selected pixels PX output pixel signals through the plurality of column lines CL.

The ADC circuit130may receive a plurality of pixel signals read out from the plurality of pixels PX in a row selected by the row driver120among the plurality of pixels PX and may convert the plurality of pixel signals into a plurality of pixel values which are digital data.

The ADC circuit130may convert the plurality of pixel signals received from the pixel array110through the plurality of column lines CL into digital data based on a ramp signal RAMP from the ramp signal generator140, and thus, first image data, for example, raw image data, may be generated and output in units of rows.

The ADC circuit130may include a plurality of ADCs corresponding to the plurality of column lines CL, and each of the plurality of ADCs may compare a pixel signal received through a corresponding column line CL with the ramp signal RAMP and generate a pixel value based on a result of the comparison. For example, the ADC may remove a reset signal from a sensing signal by using a CDS method and generate a pixel value indicating the amount of light sensed by the pixel PX.

The ramp signal generator140may generate the ramp signal RAMP that increases or decreases with a predetermined slope and provide the ramp signal RAMP to the ADC circuit130.

The timing controller150may control timings of other components of the image sensor100, for example, timings of the row driver120, the ADC circuit130, the ramp signal generator140, and the processor160. The timing controller150may provide timing signals indicating operation timing to each of the row driver120, the ADC circuit130, the ramp signal generator140, and the processor160. Embodiments of the present disclosure may be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as “circuitry,” “a module” or variants thereof.

The processor160may process data for a plurality of pixel values input from the ADC circuit130. The processor160may perform image quality compensation, binning, downsizing, etc. on the image data. Accordingly, image-processed output image data IDT1may be generated and output in a predetermined unit.

For example, the processor160may process image data for each color. For example, when the image data includes a red pixel value, a green pixel value, and a blue pixel value, the processor160may process red pixels, green pixels, and blue pixels in parallel and/or serially to each other. In addition, the processor160may perform color-specific processing on image data in parallel and may also include a plurality of processing circuits.

The processor160may perform an operation of generating test image data to evaluate performance of an image sensor, according to some embodiments, which will be described below.

The interface170connects the test device200to the image sensor100. An interface according to some embodiments may include a physical layer such as C-PHY or D-PHY. In addition, the interface of the inventive concept may include a physical layer based on a mobile industry processor interface (MIPI). In addition, the test device200may include electrical die sorting (EDS) equipment.

The image sensor100may be mounted in an electronic device having an image sensing function or a light sensing function. For example, the image sensor100may be mounted in an electronic device, such as a camera, a smartphone, a wearable device, an Internet of things (IoT) device, a home appliance, a tablet personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation, a drone, an advanced driver assistance systems (ADAS), or so on. In addition, the image sensor100may be mounted in an electronic device that is provided as a component in a vehicle, furniture, manufacturing equipment, a door, various measurement devices, or so on.

FIG.3is a block diagram illustrating an image sensor100and a test device200according to some embodiments.

The image sensor100according to some embodiments includes a test image generator161and an interface170. The test image generator161may be included in the processor160.

The test image generator161receives pixel data from the pixel array110and generates test image data. Here, the test image generator161receives the pixel data in units of row regions and divides a row region of the input pixel data into a column region. A description of how the test image generator161receives pixel data will be made in detail with reference toFIGS.9A and9B.

Here, the test image generator161may receive input pixel data in units of channels. A channel refers to a unit in which pixel data input to the processor160is grouped. For example, when pixel data is input through n channels, (the number of pixels input through each channel and included in each channel)×n may be equal to a size of the input row region. In addition, n may be any natural number.

For example, the test image generator161may divide the input pixel data into a 2n-th column region and a 2n−1-th column region according to the order of input. In this case, the 2n-th columns may be defined as a first column region, and the 2n−1-th columns may be defined as a second column region. In addition, the test image generator161may divide the input pixel data into a 4n-th column region, a 4n−1-th column region, a 4n−2-th column region, and a 4n−3-th column region in the order of input. In this case, 4n-th columns may be defined as a first column region, and 4n−1-th columns may be defined as a second column region. In addition, 4n−2-th columns may be defined as a third column region, and 4n−3-th columns may be defined as a fourth column region. That is, the test image generator161may divide the input pixel data into a column region of 2 units or a column region of 4 units.

Embodiments for a method of dividing pixel data into predetermined column regions will be described in detail with reference toFIGS.7A to14. In addition, a method of dividing pixel data is not limited thereto.

The test image generator161according to some embodiments may generate test image data based on first column regions among the divided column regions. For example, when the 2n-th columns are defined as the first column region, the test image generator161may transmit the first column regions to the test device200, and in this case, the first column regions may be combined to each other to generate a test image. In addition, when the 4n-th columns are defined as the first column region, the test image generator161may transmit the first column regions to the test device200, and in this case, the first column regions may be combined to each other to generate a test image. However, the column regions for generating the test image are not limited to the first column region, and test image data may be generated based on the second column regions. In addition, when the column regions are divided based on the 4n column region, test image data may be generated based on the second column region, the third column region, or the fourth column region.

Residual pixels may be generated depending on sizes of row regions of the pixel data input to the test image generator161. For example, in a case in which pixel data is divided based on 2n or 4n columns in the image processing method, when it is assumed that the sizes of row regions of the input pixel data are referred to as (the number of pixels included in each channel)×n, and when the number of pixels included in each channel is not divided by 2 or 4, there may be unprocessed residual pixels. For example, when 8 pieces of pixel data are included in one channel and pixel data is input through 8 channels, a size of the input row region may include 64 pieces of pixel data but is not limited thereto. Here, when there may be residual pixels as a result of dividing column regions of the input pixel data, the test image generator161may additionally arrange the residual pixels after the last column among the column regions, and thus, test image data may be generated.

The interface170transmits the divided pixel data to the test device200. As described above, the interface170according to the disclosed embodiment may include a physical layer such as C-PHY or D-PHY and may include a physical layer based on a Mobile Industry Processor Interface (MIPI).

As described above, when image data input from the pixel array110is divided into 2n or 4n column units, a size of the image data is reduced to ½ or ¼ of the original size. When the capacity of the image data is reduced, the capacity of the image data transmitted to the test device200through the interface170is also reduced, and the transmission speed of the test image data may be lower than the input speed of the pixel data input to the test image generator. For example, when the size of image data is reduced to ½ or ¼, a speed at which test image data is transmitted to the test device200through the interface170may be ½ or ¼ of a speed at which the pixel data is input to the test image generator161. However, a speed at which test image data is transmitted to the test device200through the interface170is not limited thereto.

The test device200merges the input test image data and evaluates performance of the image sensor100. The performance of the image sensor100evaluated here may include image quality, resolution, chromatic aberration, binning degree, or sensitivity of the image sensor100but is not limited thereto and may include other factors for evaluating the performance of the image sensor100. In addition, the test device200may include EDS equipment.

FIG.4is a flowchart illustrating an image processing method of the image sensor100, according to some embodiments.

Referring toFIG.4, the test image generator161of the image sensor100according to some embodiments receives image data of the pixel array110(S110).

Here, pixel data may be input in units of row regions of respective pixels.

When the image data of the pixel array110is received, the test image generator161divides the pixel data into column regions (S120).

For example, the test image generator161may divide the input pixel data into a 2n-th column region and a 2n−1-th column region in the order of input. In this case, 2n-th columns may be defined as a first column region, and 2n−1-th columns may be defined as a second column region. In addition, the test image generator161may divide the input pixel data into a 4n-th column region, a 4n−1-th column region, a 4n−2-th column region, and a 4n−3-th column region in the order of input.

When the pixel data is divided into column regions, the interface170transmits the test image data to the test device200(S130).

As described above, when the image data input from the pixel array110is divided into 2n or 4n column units, a size of the image data is reduced to ½ or ¼ of the original image. When the size of the image data is reduced, a size of the image data transmitted to the test device200through the interface170is also reduced, and a transmission speed of the test image data is less than an input speed of the pixel data input to the test image generator161. For example, when the size of the image data is reduced by ½ or ¼, a speed at which the test image data is transmitted to the test device200through the interface170may be ½ or ¼ of a speed at which pixel data is input to the test image generator161.

FIG.5is a flowchart illustrating a method of processing residual pixels of the image sensor100, according to some embodiments.

The test image generator161divides pixel data into column regions (S210).

When the pixel data is divided into column regions, the test image generator161determines whether the number of divided column regions is an odd number (S220). For example, it may be determined whether the number of pixels included in each channel is divided by 2 or 4 when a size of a row region corresponds to (the number of pixels included in each channel)×n. Here, (the number of pixels included in each channel)×n may be equal to a total size of row regions of the input pixel data. When the number of divided column regions is an odd number, the test image generator161adds the residual pixel data to the last-divided column region (a 2n-th column region or a 4n-th column region) (S230).

For example, when a value of {(the number of pixels included in each channel)×n} for input data is not divided by 2 or 4, the test image generator161processes the unprocessed residual pixel data as the last valid pixel data. For example, when a size of a row region of pixel data input to the test image generator161is (the number of pixels included in each channel)×n (when the number of pixels included in the channel is 7), and when the test image generator161generates a test image by using columns corresponding to the 2n-th column, test image data may be generated based on pixels in the 2n-th column. Here, seventh pixel data may be processed as data separate from the pixel data in the second column, the fourth column, and the sixth column.

In order to prevent unintentional pixel data from being added to the test image data, when residual pixels are generated, the test image generator161may process the residual pixel data from pixel data in the 2n-th column. For example, the test image generator161adds residual pixel data to the last-divided column region (a 2n-th column region). An image processing method related to the residual pixel processing will be described in detail with reference toFIGS.10B,12B,12C, and12D.

When the residual pixel data is added to the last column region of the pixel data, the test image generator161generates test image data (S240).

However, when the number of pixels included in each channel is divided into 2 or 4 in a relationship in which the size of the row region corresponds to “(the number of pixels included in each channel)×n”, which are described above, are established, the test image generator161may generate test image data without processing the residual pixel data.

FIG.6is a flowchart illustrating an image processing method when pixel data is discontinuously input, according to some embodiments.

Referring toFIG.6, pixel data of the image sensor100according to some embodiments is input to a receiver of the test image generator161(S310).

When the pixel data is input, the test image generator161determines whether valid pixel data is discontinuously input (S320). The valid pixel data exists in a 2n-th column region or a 4n-th column region and refers to pixel data that is read out and extracted to generate test image data.

When the valid pixel data is continuously input, an input speed of the pixel data is constant, and the test image generator161divides the pixel data into column regions and generates test image data.

However, when the valid pixel data is discontinuously input, the input speed of the pixel data is not constant, and the test image generator161may generate a clock signal corresponding to the valid pixel data (S330). The test image generator161may divide columns of the pixel data based on the clock signal corresponding to the valid pixel data. When the clock signal corresponding to the valid pixel data is generated, the test image generator161may divide the columns of the input pixel data even when the valid pixel data is discontinuously input.

When a clock signal corresponding to an input speed of pixel data is generated, an overlapping region of discontinuous pixel data may be read out (S340).

For example, when an input speed of first to sixth row regions is different from an input speed of seventh and eighth row regions in the pixel data input from the 8×8 pixel array110, the test image generator161may adjust a clock signal for dividing columns in the seventh and eighth row regions. A method of dividing a column region by adjusting a clock signal will be described in detail with reference toFIGS.13and14.

FIGS.7A,7B,7C,8A, and8Bare diagrams illustrating an image processing method of an image sensor, according to some embodiments. Specifically,FIGS.7A,7B, and7Cillustrate a case in which the image processing method divides pixel data based on 2n column regions, andFIGS.8A and8Billustrate a case in which the image processing method divides pixel data based on 4n column regions.

Referring toFIGS.7A,7B,7C,8A, and8B, pixel data PIX is input to the processor160of the image sensor100. Specifically, the pixel data PIX may be input to the test image generator161of the processor160.

Referring toFIGS.7A and7B, the test image generator161receives pixel data in units of n row regions and divides the input pixel data into column regions. For example, the processor160may receive corresponding pixel data PIX among the pixel data PIX in the order of row region through eight channels. Here, the pixel data PIX corresponding to one row region may be input in units of each channel, and the row region corresponding to the input may be divided into first to eighth row regions.

When the row regions of pixel data are sequentially input, the test image generator161may divide the input pixel data into a 2n-th column region and a 2n−1-th column region in the order of input. In this case, 2n-th columns may be defined as a first column region, and 2n−1-th columns may be defined as a second column region. When the column regions are divided, the test image generator161may transmit test image data TID generated based on each column region to the test device200through the interface170.

The processor160receives the pixel data PIX through n channels. When the pixel data PIX is input, the test image generator161of the processor160may generate a readout start signal Line Start to extract pixel data of a column region in each row region. When the pixel data starts to be read out, the test image generator161selects valid pixel data Data Valid). For example, the test image generator161divides pixel data in each row region into odd-numbered valid pixel data Odd Data Valid and even-numbered valid pixel data Even Data Valid in the order of input and reads out the divided valid pixel data. That is, the test image generator161may generate output data Output by classifying pixel data in the row region of the input pixel data into 2n-th pixel data and 2n−1-th pixel data. Here, in this case, 2n-th columns may be defined as a first column region, and 2n−1-th columns may be defined as a second column region. However, units in which the processor160divides the row region are not limited to a 2n-th pixel or a 2n−1-th pixel, and 4n-th, 4n−1-th, 4n−2-th, and 4n−3-th pixels may also be classified in the same manner as in the above-described example embodiment.

FIG.7Cillustrates in more detail that the processor160derives output data based on input data. Referring toFIG.7C, by dividing pixel data into a first column region and a second column region, the processor160may reduce a size of the transmitted image data by half compared to a case in which original image data is transmitted as it is. As a result of reduction in size of the transmitted image data, the processor160may reduce an image data transmission speed by half compared to a case in which the original image data is transmitted as it is.

In addition, referring toFIGS.8A and8B, the test image generator161receives pixel data in units of row regions and divides a row region of the input pixel data into column regions. The test image generator161may divide the input pixel data into a 4n-th column region, a 4n−1-th column region, a 4n−2-th column region, and a 4n−3-th column region in the order of input. In this case, 4n-th columns may be defined as a first column region, and 4n−1-th columns may be defined as a second column region. In addition, 4n−2-th columns may be defined as a third column region, and 4n−3-th columns may be defined as a fourth column region. When the column regions are divided, the test image generator161may transmit test image data TID generated based on each column region to the test device200through the interface170. In addition, in the same manner as in the embodiment described with reference toFIGS.7A and7B, the processor160may receive corresponding pixel data PIX among the pixel data PIX in the order of row region through n channels. Here, a row region corresponding to the input may be divided into first to eighth row regions. In addition, the test image generator161may divide the input pixel data into a column region of 2 units or a column region of 4 units, but the method of dividing the pixel data is not limited thereto.

The processor160receives the pixel data PIX through n channels. When the pixel data PIX is input, the test image generator161of the processor160may generate a readout start signal Line Start to extract pixel data of a column region in each row region. When the pixel data starts to be read out, the test image generator161selects valid pixel data Data Valid. For example, the test image generator161may read out the pixel data in each row region by classifying into 4n-th, 4n−1-th, 4n−2-th, and 4n−3-th pixels in the order of input. That is, the test image generator161may generate output data Output by classifying the pixel data in a row region of the input pixel data into 4n-th pixel data, 4n−1-th pixel data, 4n−2-th pixel data, and 4n−3-th pixel data. Here, in this case, 4n-th columns may be defined as a first column region, and 4n−1-th columns may be defined as a second column region. In addition, 4n−2-th columns may be defined as a third column region, and 4n−3-th columns may be defined as a fourth column region.

FIG.9Aillustrates a method of reading out pixel data to perform an image processing method, according to some embodiments.FIG.9Billustrates that pixel data is input in the order of row region, according to some embodiments. Specifically,FIG.9Aillustrates a case in which pixel data input to the test image generator161is divided into units of 2n columns.

Referring toFIGS.9A and9B, respective pixel data An, Bn, Cn, Dn, En, Fn, Gn, and Hn may be input in an 8×8 format. Hereinafter, for the sake of convenience, pixel data in the same row region is classified in the same alphabet, and pixel data in the same column region is classified in the same number. However, a shape of the pixel data is not limited thereto.

Referring toFIG.9A, a vertical axis may be defined as a row region of pixel data, and a horizontal axis may be defined as a column region thereof. Referring toFIG.9B, pixel data is input in units of row regions.

According to some embodiments, the test image generator161may receive pixel data in units of row regions and divide a zeroth column (Pixel Data 0), a second column (Pixel Data 2), a fourth column (Pixel Data 4), and sixth column (Pixel Data 6) into valid pixel data for generating test image data TID. Here, assuming that N is 2, pixel data input through a pixel array may be arranged in the order of the row regions A0, A1, A2, . . . , H0, H1, H2, H3, H4, H5, H6, and H7 to be input to a first input row region row 1 inFIG.9B. In addition, the pixel data input through the pixel array may be arranged in the order of row regions A′0, A′1, A′2, . . . , H′0, H′1, H′2, H′3, H′4, H′5, H′6, and H′7 to be input to a second input row region row 2.

Referring back toFIG.9A, the processor160may extract a column region in each row region among the row regions of pixel data input in the form illustrated inFIG.9B. For example, when a zeroth column (Pixel Data 0), a second column (Pixel Data 2), a fourth column (Pixel Data 4), and sixth column (Pixel Data 6) are divided into pixel data for generating test image data TID, a first channel may include pixel data A0 to A7 corresponding to the first column region, and pixel data of the divided column region may be A0, A2, A4, and A6.

Hereinafter, a result of reading out pixel data according to the method ofFIGS.9A and9Bwill be described with reference toFIGS.10A and10B.

FIGS.10A and10Billustrate a result of reading out pixel data according to an image processing method of an image sensor according to some embodiments. Specifically,FIG.10Aillustrates a case in which the number of columns of pixel data is 2n, andFIG.10Billustrates a case in which the number of columns of pixel data is 2n−1.

Referring toFIG.10A, as a result of dividing the zeroth column (Pixel Data 0), the second column (Pixel Data 2), the fourth column (Pixel Data 4), and the sixth column (Pixel Data 6) are divided into valid pixel data for generating the test image data TID, zeroth to sixth columns of each row are selected as pixel data for generating the test image data TID. When the number of columns of pixel data is 2n, even when the column region is divided, there are no residual pixels, and thus, a process of processing residual pixels are not required. Here, the processor160may pack in advance data A0, A2, A4, and A6 to present the data A0, A2, A4, and A6 as start data of valid pixel data, and the data A0, A2, A4, and A6 may be overlapped with the beginning of a row region.

Referring toFIG.10B, when the number of columns of input pixel data is 2n−1, pixels H0, H2, H4, and H6 may be additionally arranged in the test image data unlike the case ofFIG.10A. Here, the processor160may pack in advance the data H0, H2, H4, and H6 to present the data H0, H2, H4, and H6 as the last data of the valid pixel data, and the data H0, H2, H4, and H6 may be overlapped with the last portion of a row region.

FIG.11is a diagram illustrating a method of reading out pixel data to perform an image processing method, according to some embodiments. Specifically,FIG.11illustrates a case in which pixel data input to the test image generator161is divided into 4n column units.

Referring toFIG.11, the test image generator161may receive pixel data in units of row regions and divide a zeroth column (Pixel Data 0) and fourth column (Pixel Data 4) into valid pixel data for generating the test image data TID. Hereinafter, a result of reading out pixel data according to the method illustrated inFIG.11will be described with reference toFIGS.12A to12D.

FIGS.12A to12Billustrate results of reading out pixel data according to an image processing method of an image sensor, according to some embodiments.

Specifically,FIG.12Aillustrates the test image data TID when there are 4n column regions of pixel data, andFIGS.12B to12Dillustrate the test image data TID when there are no 4n column regions of pixel data.

Referring toFIG.12A, as a result of dividing a zeroth column (Pixel Data 0) and a fourth column (Pixel Data 4) into valid pixel data for generating the test image data TID, a zeroth columns and a fourth column of each row are selected as pixel data for generating the test image data TID. When the sum of the number of pixels included in each channel is not divided by 4 in a relationship in which the size of the total input row regions is (the number of pixels included in each channel)×n, residual pixels are not generated even when the column region is divided, and thus, a process of processing the residual pixels is not required.

Referring toFIG.12B, the sum of the number of pixels included in each channel is not divided by 4 in a relationship in which the size of the total input row regions is (the number of pixels included in each channel)×n, and thus, pixels F0, F4, G0, G4, H0, and H4 may be additionally arranged in the test image data. Here, the processor160may pack in advance the pixel data of F0, F4, G0, G4, H0, and H4 to present the pixel data as the last data of the valid pixel data, and the pixels F0, F4, G0, G4, H0, and H4 may be overlapped with the last portion of a row region.

Referring toFIG.12C, the sum of the number of pixels included in each channel is not divided by 4 in a relationship in which the size of the total input row regions is (the number of pixels included in each channel)×n, and thus, pixels G0, G4, H0, and H4 may be additionally arranged in the test image data. Here, the processor160may pack in advance the pixel data of F0, G0, G4, H0, and H4 to present the pixel data as the last data of the valid pixel data, and the pixels G0, G4, H0, and H4 may be overlapped with the last portion of a row region.

Referring toFIG.12D, the sum of the number of pixels included in each channel is not divided by 4 in a relationship in which the size of the total input row regions is (the number of pixels included in each channel×n), and thus, pixels H0 and H4 may be additionally arranged in the test image data. Here, the processor160may pack in advance the pixel data H0 and H4 to present the pixel data as the last data of the valid pixel data, and the pixel data H0 and H4 may be overlapped with the last portion of a row region.

FIG.13illustrates a method of reading out pixel data to perform an image processing method when pixel data is irregularly input, according to some embodiments, andFIG.14illustrates a result of reading out pixel data according to an image processing method when pixel data is irregularly input, according to some embodiments

Referring toFIG.13, pixel data may be irregularly input to the test image generator161. For example, input speeds of a G row and an H row ofFIG.13are different from input speeds of an A row to an F row. That is, when a row region of pixel data according to the disclosed embodiment is input, input patterns of valid pixel data may be different from each other. When the input patterns of valid pixel data of each pixel data is different from each other, the test image generator161may determine an input pattern of valid pixel data of pixel data and adjust a size of valid pixel data for reading out the pixel data. According to some embodiments, when the input patterns of the G row and the H row have a difference of ½ pixel size from the input patterns of other pixel data, a valid pixel data readout signal for reading out the pixel data of the G row and the H row at once may be generated.

Referring toFIG.14, the test image generator161may divide columns of all row regions despite irregularly input pixel data in the embodiment ofFIG.13by adjusting a readout signal corresponding to valid pixel data. For example, in order to generate the test image data TID corresponding to a difference in input pixel interval between pixel data G0, G4, H0, and H4, an arrangement of a sixth column (Pixel Data 6) and a seventh column (Pixel data 7) may be adjusted, and as a result, pixel data may be read out by a clock signal corresponding to input patterns of the pixel data G0, G4, H0, and H4.

FIG.15is a block diagram of an electronic device including multiple camera modules to which some embodiments according to the inventive concept may be applied.FIG.17is a detailed block diagram of the camera module ofFIG.15.

Referring toFIG.15, an electronic device1000may include a camera module group1100, an application processor1200, a power management integrated circuit (PMIC)1300, and an external memory1400.

The camera module group1100may include a plurality of camera modules1100a,1100b, and1100c. AlthoughFIG.15illustrates some embodiments in which three camera modules1100a,1100b, and1100care arranged, embodiments are not limited thereto. In some embodiments, the camera module group1100may be changed to include only two camera modules or n (n is a natural number greater than or equal to 4) camera modules.

Hereinafter, a detailed configuration of the camera module1100bwill be described in more detail with reference toFIG.17, but the following description may be equally applied to other camera modules1100aand1100caccording to some embodiments.

Referring toFIG.17, the camera module1100bmay include a prism1105, an optical path folding element (hereinafter, referred to as an “OPFE”)1110, an actuator1130, an image sensing device1140, and a storage1150

The prism1105may include a reflective surface1107of a light reflective material to change a path of light L incident from the outside.

In some embodiments, the prism1105may change a path of the light L incident in a first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism1105may change the path of the light L incident in the first direction X to the second direction Y perpendicular to the first direction X by rotating the reflective surface1107of the light reflective material in an A direction about a central axis1106or rotating the central axis1106in a B direction. In this case, the OPFE1110may also move in a third direction Z perpendicular to the first and second directions X and Y.

In some embodiments, as illustrated, the greatest rotation angle of the prism1105in the A direction may be less than or equal to 15 degrees in a positive (+) A direction and greater than 15 degrees in a negative (−) A direction, but embodiments are not limited thereto.

In some embodiments, the prism1105is movable about 20 degrees in a positive (+) or negative (−) B direction, or between 10 degrees and 20 degrees, or between 15 degrees and 20 degrees, wherein the prism1105may move at the same angle in the positive (+) or negative (−) direction B or may move up to an almost similar angle within a range of about 1 degree.

In some embodiments, the prism1105may move the reflective surface1107of the light reflective material in a third direction (for example, the Z direction) parallel to an extension direction of the central axis1106.

In some embodiments, the camera module1100bmay include two or more prisms to variously change a path of the light L incident in the first direction X to the second direction Y perpendicular to the first direction X, again to the first direction X or the third direction Z, and again to the second direction Y, and so on by using the prisms.

The OPFE1110may include an optical lens including a group of, for example, m (here, m is a natural number) lenses. The m lenses may move in the second direction Y to change an optical zoom ratio of the camera module1100b. For example, when a basic optical zoom ratio of the camera module1100bis referred to as Z, and when m optical lenses included in the OPFE1110are moved, the optical zoom ratio of the camera module1100bmay be changed to an optical zoom ratio of 3Z or 5Z or higher.

The actuator1130may move the OPFE1110or an optical lens (hereinafter, referred to as an optical lens) to a preset position. For example, the actuator1130may adjust a position of the optical lens such that the image sensor1142is located at a focal length of the optical lens for accurate sensing.

The image sensing device1140may include an image sensor1142, a control logic1144, and a memory1146. The image sensor1142may sense an image of a sensing target by using light L provided through an optical lens. The control logic1144may control all operations of the camera module1100band process the sensed image. For example, the control logic1144may control an operation of the camera module1100bin response to a control signal provided through a control signal line CSLb and extract image data (for example, the face, arms, legs, and so on of a person in an image) corresponding to a preset image in the sensed image.

In some embodiments, the control logic1144may perform image processing, such as encoding and noise reduction of the sensed image.

The memory1146may store information necessary for the operation of the camera module1100b, such as calibration data1147. The calibration data1147is information necessary for the camera module1100bto generate image data by using the light L provided from the outside and may include, for example, information on a degree of rotation, information on a focal length, information on an optical axis, and so on. When the camera module1100bis implemented in the form of a multi-state camera in which a focal length is changed depending on positions of an optical lens, the calibration data1147may include a focal length value for each position (or state) of the optical lens and information on auto focusing.

The storage1150may store image data sensed by the image sensor1142. The storage1150may be outside the image sensing device1140and may be implemented in a stacked form with a sensor chip included in the image sensing device1140. In some embodiments, the image sensor1142is configured as a first chip, and the control logic1144, the storage1150, and the memory1146are configured as a second chip such that one of the two chips may be stacked on the other.

In some embodiments, the storage1150may include an electrically erasable programmable read-only memory (EEPROM), but embodiments are not limited thereto. In some embodiments, the image sensor1142may include a pixel array, and the control logic1144may include an analog-to-digital converter and an image signal processing unit for processing a sensed image.

Referring toFIGS.15and17, in some embodiments, each of the plurality of camera modules1100a,1100b, and1100cmay include the actuator1130. Accordingly, each of the plurality of camera modules1100a,1100b, and1100cmay include the same or different calibration data1147according to an operation of the actuator1130included in the respective camera module.

In some embodiments, one camera module (for example,1100b) of the plurality of camera modules1100a,1100b, and1100cis a folded-lens-type camera module including the prism1105and the OPFE1110described above, and the other camera modules (for example,1100aand1100c) may be a vertical-type camera module that does not include the prism1105and the OPFE1110, but are not limited thereto.

In some embodiments, one camera module (for example,1100c) of the plurality of camera modules1100a,1100b, and1100cmay be a vertical-type depth camera that extracts depth information by using, for example, infrared ray (IR). In this case, the application processor1200may merge image data received from the depth camera and image data received from another camera module (for example,1100aor1100b) to generate a three-dimensional (3D) depth image.

In some embodiments, at least two camera modules (for example,1100aand1100b) of the plurality of camera modules1100a,1100b, and1100cmay have different fields of view (viewing angles). In this case, for example, optical lenses of at least two camera modules (for example,1100aand1100b) of the plurality of camera modules1100a,1100b, and1100cmay be different from each other, but embodiments are not limited thereto.

In addition, in some embodiments, viewing angles of the plurality of camera modules1100a,1100b, and1100cmay be different from each other. For example, the camera module1100amay be an ultrawide camera, the camera module1100bmay be a wide camera, and the camera module1100cmay be a tele camera, but embodiments are not limited thereto. In this case, optical lenses respectively included in the plurality of camera modules1100a,1100b, and1100cmay also be different from each other, but embodiments are not limited thereto.

In some embodiments, the plurality of camera modules1100a,1100b, and1100cmay be physically separated from each other. That is, a sensing region of one image sensor1142is not divided by the plurality of camera modules1100a,1100b, and1100cto be used, but the plurality of camera modules1100a,1100b, and1100cmay each include an independent image sensor1142.

Referring back toFIG.15, the application processor1200may include an image processing device1210, a memory controller1220, and an internal memory1230. The application processor1200may be separated from the plurality of camera modules1100a,1100b, and1100cas, for example, a separate semiconductor chip.

The image processing device1210may include a plurality of sub image processors1212a,1212b, and1212c, an image generator1214, and a camera module controller1216.

The image processing device1210may include the number of sub image processors1212a,1212b, and1212ccorresponding to the number of camera modules1100a,1100b, and1100c.

Image data generated by the camera module1100amay be provided to the sub image processor1212athrough an image signal line ISLa, image data generated by the camera module1100bmay be provided to the sub image processor1212bthrough an image signal line ISLb, and image data generated by the camera module1100cmay be provided to the sub image processor1212cthrough an image signal line ISLc. The image data may be transmitted by using, for example, a camera serial interface (CSI) based on a MIPI, but embodiments are not limited thereto.

In some embodiments, one sub image processor may also be arranged to correspond to a plurality of camera modules. For example, the sub image processor1212aand the sub image processor1212care not separated from each other as illustrated but may be integrated into one sub image processor, and the image data provided from the camera modules1100aand1100cmay be selected by a selection element (for example, a multiplexer) or so on, and then may be provided to an integrated sub image processor. In this case, the sub image processor1212bmay receive image data from the camera module1100bwithout being integrated.

In some embodiments, image data generated by the camera module1100amay be provided to the sub image processor1212athrough an image signal line ISLa, image data generated by the camera module1100bmay be provided to the sub image processor1212bthrough an image signal line ISLb, and image data generated by the camera module1100cmay be provided to the sub image processor1212cthrough an image signal line ISLc. The image data processed by the sub image processor1212bmay be directly provided to the image generator1214, but image data processed by the sub image processor1212aand image data processed by the sub image processor1212cmay be selected by a selection element (for example, a multiplexer) or so on and then may be provided to the image generator1214.

Each of the sub image processors1212a,1212b, and1212cmay perform image processing, such as bad pixel correction, 3A (auto-focus correction, auto-white balance, and auto-exposure) adjustment, noise reduction, sharpening, gamma control, and/or remosaic for image data provided from the camera modules1100a,1100b, and1100c.

In some embodiments, remosaic signal processing may be performed by each of the camera modules1100a,1100b, and1100c, and then may also be provided to the sub image processors1212a,1212b, and1212c.

Image data processed by each of the sub image processors1212a,1212b, and1212cmay be provided to the image generator1214. The image generator1214may generate an output image by using the image data provided from each of the sub image processors1212a,1212b, and1212cin response to image generating information or a mode signal.

Specifically, the image generator1214may generate the output image by merging at least a part of the image data generated by the image processors1212a,1212b, and1212cin response to the image generation information or the mode signal. In addition, the image generator1214may generate the output image by selecting any one of the image data generated by the image processors1212a,1212b, and1212cin response to the image generation information or the mode signal.

In some embodiments, the image generation information may include a zoom signal or a zoom factor. In addition, in some embodiments, the mode signal may be based on, for example, a mode selected by a user.

When an image generation information is a zoom signal (a zoom factor), and when the respective camera modules1100a,1100b, and1100chave different fields of view (viewing angles), the image generator1214may perform different operations depending on the type of the zoom signal. For example, when the zoom signal is a first signal, an output image may be generated by using image data output from the sub image processor1212aand image data output from the sub image processor1212bamong image data output from the sub image processor1212aand image data output from the sub image processor1212c. When the zoom signal is a second signal different from the first signal, the image generator1214may generate an output image by using the image data output from the sub image processor1212cand the image data output from the sub image processor1212bamong the image data output from the sub image processor1212aand the image data output from the sub image processor1212c. When the zoom signal is a third signal different from the first and second signals, the image generator1214may generate an output image by selecting any one from among image data output from the respective sub image processors1212a,1212b, and1212cwithout merging the image data. However, embodiments are not limited thereto, and a method of processing image data may be changed as needed.

Referring toFIG.16, in some embodiments, the image processing device1210may further include a multiplexor or selector1213that selects outputs of the sub image processors1212a,1212b, and1212cand transmits the selected outputs to the image generator1214.

In this case, the selector1213may perform different operations according to a zoom signal or a zoom factor. For example, when the zoom signal is a fourth signal (for example, a zoom ratio is a first ratio), the selector1213may select any one of the outputs of the sub image processors1212a,1212b, and1212cand transmit the selected output to the image generator1214.

In addition, when the zoom signal is a fifth signal different from the fourth signal (for example, the zoom ratio is a second ratio), the selector1213may sequentially transmit p (p is a natural number greater than or equal to 2) outputs among the outputs of the sub image processors1212a,1212b, and1212cto the image generator1214. For example, the selector1213may sequentially transmit the output of the sub image processor1212band the output of the sub image processor1212cto the image generator1214. In addition, the selector1213may sequentially transmit the output of the sub image processor1212aand the output of the sub image processor1212bto the image generator1214. The image generator1214may generate one output image by merging the sequentially provided p outputs.

Here, the sub image processors1212a,1212b, and1212cperforms image processing, such as demosaic, down-scaling to a video/preview resolution size, gamma correction, and high dynamic range (HDR) processing, and the processed image data is transmitted to the image generator1214. Accordingly, even when the processed image data is provided to the image generator1214through the selector1213and one signal line, an image merging operation of the image generator1214may be performed at high speed.

In some embodiments, the image generator1214may receive a plurality of image data having different exposure times from at least one of the plurality of sub image processors1212a,1212b, and1212cand perform a high dynamic range (HDR) for the plurality of image data, and thus, merged image data having an increased dynamic range may be generated.

The camera module controller1216may provide control signals respectively to the camera modules1100a,1100b, and1100c. The control signals generated by the camera module controller1216may be respectively provided to the camera modules1100a,1100b, and1100cthrough the control signal lines CSLa, CSLb, and CSLc separated from each other.

Any one of the plurality of camera modules1100a,1100b, and1100cmay be designated as a master camera (for example,1100b) according to either image generation information including a zoom signal or a mode signal, and the other camera modules (for example,1100aand1100c) may be designated as slave cameras. The information may be included in the control signals and respectively provided to the camera modules1100a,1100b, and1100cthrough the control signal lines CSLa, CSLb, and CSLc separated from each other.

Camera modules operating as a master and a slave may be changed according to a zoom factor or an operation mode signal. For example, when a viewing angle of the camera module1100ais wider than a viewing angle of the camera module1100band a zoom factor indicates a low zoom ratio, the camera module1100amay operate as a master, and the camera module1100bmay operate as a slave. In contrast to this, when the zoom factor indicates a high zoom ratio, the camera module1100bmay operate as a master and the camera module1100amay operate as a slave.

In some embodiments, the control signals provided from the camera module controller1216to the camera modules1100a,1100b, and1100cmay include a sync enable signal. For example, when the camera module1100bis a master camera and the camera modules1100aand1100care slave cameras, the camera module controller1216may transmit the sync enable signal to the camera module1100b. The camera module1100breceiving the sync enable signal may generate a sync signal based on the received sync enable signal and transmit the generated sync signal to the camera modules1100aand1100c. The camera module1100band the camera modules1100aand1100cmay be synchronized to the sync signal to transmit image data to the application processor1200.

In some embodiments, the control signals provided from the camera module controller1216to the plurality of camera modules1100a,1100b, and1100cmay each include mode information according to a mode signal. Based on the mode information, the plurality of camera modules1100a,1100b, and1100cmay operate in a first operation mode and a second operation mode in relation to a sensing speed.

The plurality of camera modules1100a,1100b, and1100cmay generate (for example, generate an image signal of a first frame rate) an image signal at a first speed in the first operation mode, encode (for example, encode an image signal of a second frame rate higher than the first frame rate) the image signal at a second speed higher than the first speed, and transmit the encoded image signal to the application processor1200. In this case, the second speed may be 30 times or less of the first speed.

The application processor1200may store the received image signal, that is, the encoded image signal in the internal memory1230provided therein or the external memory1400outside the application processor1200, and then read the encoded image signal from the internal memory1230or the external memory1400and decode the read image signal, and display image data generated based on the decoded image signal may be displayed. For example, one of the plurality of sub image processors1212a,1212b, and1212cof the image processing device1210may perform decoding, and also perform image processing for the decoded image signal.

The plurality of camera modules1100a,1100b, and1100cmay generate (for example, generate an image signal of a third frame rate lower than the first frame rate) an image signal at a third speed lower than the first speed in the second operation mode and may transmit the image signal to the application processor1200. The image signal provided to the application processor1200may be an unencoded signal. The application processor1200may perform image processing for the received image signal or store the image signal in the internal memory1230or the external memory1400.

The PMIC1300may supply power, for example, a power supply voltage, to each of the plurality of camera modules1100a,1100b, and1100c. For example, the PMIC1300may supply first power to the camera module1100athrough a power signal line PSLa, second power to the camera module1100bthrough a power signal line PSLb, and third power to the camera module1100cthrough a power signal line PSLc, by the control of the application processor1200.

The PMIC1300may generate power corresponding to each of the plurality of camera modules1100a,1100b, and1100cin response to a power control signal PCON from the application processor1200and adjust a level of the power. The power control signal PCON may include a power adjustment signal for each operation mode of the plurality of camera modules1100a,1100b, and1100c. For example, an operation mode may include a low power mode, and in this case, the power control signal PCON may include information on a camera module operating in a low power mode and a set power level. Levels of powers respectively provided to the plurality of camera modules1100a,1100b, and1100cmay be the same as or different from each other. In addition, the levels of powers may be changed dynamically.

While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the following claims.