Control method, camera assembly, and mobile terminal

A control method, a camera assembly, and a mobile terminal are provided. The control method includes: obtaining original image data by controlling exposure of the 2D pixel array, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels; and outputting target image data according to the original image data.

TECHNICAL FIELD

This disclosure relates to the field of imaging, and more specifically, to a control method, a camera assembly, and a mobile terminal.

BACKGROUND

With the development of electronic technology, terminals with camera functions have been popularized in daily life. At present, types of data outputted by mobile phones are mainly based on an image sensor with Bayer pixel arrangement or an image sensor with QuadBayer pixel arrangement. For image sensors with other pixel arrangements, the data output needs to be improved.

SUMMARY

In a first aspect, a control method for an image sensor is provided. The image sensor includes a two-dimensional (2D) pixel array. The 2D pixel array includes multiple minimal repeating units, where each minimal repeating unit includes multiple subunits. Each subunit includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The control method includes the following. Original image data is obtained by controlling exposure of the 2D pixel array, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels. Target image data is outputted according to the original image data.

In a second aspect, a camera assembly is provided. The camera assembly includes an image sensor and a processor. The image sensor includes a 2D pixel array. The 2D pixel array includes multiple minimal repeating units, where each minimal repeating unit includes multiple subunits. Each subunit includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The processor is configured to obtain original image data by controlling exposure of the 2D pixel array, and to output target image data according to the original image data, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.

In a third aspect, a mobile terminal is provided. The mobile terminal includes a housing and a camera assembly received in the housing. The camera assembly includes an image sensor and a processor. The image sensor includes a 2D pixel array. The 2D pixel array includes multiple minimal repeating units, where each minimal repeating unit includes multiple subunits. Each subunit includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The processor is configured to obtain original image data by controlling exposure of the 2D pixel array, and to output target image data according to the original image data, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.

DETAILED DESCRIPTION

Implementations of the disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, where the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The implementations described below with reference to the accompanying drawings are exemplary, only used to explain the disclosure, and should not be construed as a limitation on the disclosure.

With the development of electronic technology, terminals with camera functions have been popularized in daily life. At present, types of data outputted by mobile phones is mainly based on an image sensor with Bayer pixel arrangement or an image sensor with QuadBayer pixel arrangement. For image sensors with other pixel arrangements, the data output needs to be improved.

For the above reasons, referring toFIG.1,FIG.16,FIG.17, andFIG.31, implementations of the disclosure provide a control method, a camera assembly40, and a mobile terminal90.

Referring toFIG.1andFIG.16, the control method in implementations of the disclosure is applied to an image sensor10. The image sensor10includes a two-dimensional (2D) pixel array11. The 2D pixel array11includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The 2D pixel array11includes minimal repeating units, where each minimal repeating unit includes multiple subunits102. Each subunit102includes multiple color pixels and multiple panchromatic pixels. The control method includes operations01and02.01: Original image data is obtained by controlling exposure of the 2D pixel array11, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.02: Target image data is outputted according to the original image data.

Referring toFIG.17andFIG.18, in some implementations, the target image data is outputted according to the original image data (i.e., operation02) as follow.021: In a first data-reading mode, the target image data is outputted by reading the original image data row-by-row.

Referring toFIG.17andFIG.21, in some implementations, the target image data includes first target image data and second target image data. The target image data is outputted according to the original image data (i.e., operation02) as follow.022: In a second data-reading mode, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102, and the second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17andFIG.23, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0221: The first target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0222: Pixel addition is performed on the color original image data generated by exposure of the multiple color pixels in each subunit102, and pixel addition is performed on the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.0223: The second target image data is obtained by performing pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition.

Referring toFIG.17andFIG.24, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0224: The first target image data is obtained by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0225: The second target image data is obtained by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17andFIG.26, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0226: The first target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0227: The second target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17andFIG.28, in some implementations, the original image data is obtained by controlling exposure of the 2D pixel array (i.e., operation01) as follows.011: First original image data is obtained by controlling multiple color pixels and multiple panchromatic pixels in a (2n−1)-th row to be exposed for a first exposure duration, and second original image data is obtained by controlling multiple color pixels and multiple panchromatic pixels in a 2n-th row to be exposed for a second exposure duration, where n is a natural number greater than or equal to 1, the first exposure duration is different from the second exposure duration, the first original image data includes first color original image data generated by exposure of the color pixels and first panchromatic original image data generated by exposure of the panchromatic pixels, and the second original image data includes second color original image data generated by exposure of the color pixels and second panchromatic original image data generated by exposure of the panchromatic pixels.

The target image data includes first target image data, second target image data, third target image data, and fourth target image data. The target image data is outputted according to the original image data (i.e., operation02) as follows.023: In a third data-reading mode, the first target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, the second target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, the third target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and the fourth target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Referring toFIG.17andFIG.30, in some implementations, the third target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration as follows.0231: The third target image data is obtained by performing pixel addition on the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration.

The fourth target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration as follows.0232: The fourth target image data is obtained by performing pixel addition on the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Referring toFIG.1andFIG.17, the camera assembly40in implementations of the disclosure includes an image sensor10and a processor20. The image sensor10includes a two-dimensional (2D) pixel array11. The 2D pixel array11includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The 2D pixel array11includes minimal repeating units, where each minimal repeating unit includes multiple subunits102. Each subunit102includes multiple color pixels and multiple panchromatic pixels. The processor20is configured to obtain original image data by controlling exposure of the 2D pixel array11, and to output target image data according to the original image data, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.

In some implementations, the processor20may be configured to, in a first data-reading mode, output the target image data by reading the original image data row-by-row.

In some implementations, the processor20may be configured to, in a second data-reading mode, obtain the first target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain first original image data by controlling multiple color pixels and multiple panchromatic pixels in a (2n−1)-th row to be exposed for a first exposure duration, and obtain second original image data by controlling multiple color pixels and multiple panchromatic pixels in a 2n-th row to be exposed for a second exposure duration, where n is a natural number greater than or equal to 1. The first exposure duration is different from the second exposure duration, the first original image data includes first color original image data generated by exposure of the color pixels and first panchromatic original image data generated by exposure of the panchromatic pixels, and the second original image data includes second color original image data generated by exposure of the color pixels and second panchromatic original image data generated by exposure of the panchromatic pixels. The target image data includes first target image data, second target image data, third target image data, and fourth target image data. The processor may be further configured to, in a third data-reading mode, obtain the first target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, obtain the second target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, obtain the third target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

In some implementations, the processor20may be configured to obtain the third target image data by performing pixel addition on the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data by performing pixel addition on the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Referring toFIG.1,FIG.17, andFIG.31, the mobile terminal90according to implementations of the disclosure includes a housing80and a camera assembly40installed in the housing80. The camera assembly40includes an image sensor10and a processor20. The image sensor10includes a two-dimensional (2D) pixel array11. The 2D pixel array11includes multiple color pixels and multiple panchromatic pixels. The color pixels have a narrower spectral response than the panchromatic pixels. The 2D pixel array11includes minimal repeating units, where each minimal repeating unit includes multiple subunits102. Each subunit102includes multiple color pixels and multiple panchromatic pixels. The processor20is configured to obtain original image data by controlling exposure of the 2D pixel array11, and to output target image data according to the original image data, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.

In some implementations, the processor20may be configured to, in a first data-reading mode, output the target image data by reading the original image data row-by-row.

In some implementations, the processor20may be configured to, in a second data-reading mode, obtain the first target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

In some implementations, the processor20may be configured to obtain first original image data by controlling multiple color pixels and multiple panchromatic pixels in a (2n−1)-th row to be exposed for a first exposure duration, and obtain second original image data by controlling multiple color pixels and multiple panchromatic pixels in a 2n-th row to be exposed for a second exposure duration, where n is a natural number greater than or equal to 1, the first exposure duration is different from the second exposure duration, the first original image data includes first color original image data generated by exposure of the color pixels and first panchromatic original image data generated by exposure of the panchromatic pixels, and the second original image data includes second color original image data generated by exposure of the color pixels and second panchromatic original image data generated by exposure of the panchromatic pixels. The target image data includes first target image data, second target image data, third target image data, and fourth target image data. The processor may be further configured to, in a third data-reading mode, obtain the first target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, obtain the second target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, obtain the third target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

In some implementations, the processor20is configured to obtain the third target image data by performing pixel addition on the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data by performing pixel addition on the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

According to the control method, the camera assembly40, and the mobile terminal90in implementations of the disclosure, multiple panchromatic pixels are added in the 2D pixel array11. Compared with the general image sensor based on Bayer pixel arrangement or based on QuadBayer pixel arrangement, luminous flux can be increased, allowing a better signal to noise ratio (SNR). According to the control method, the camera assembly40, and the mobile terminal90in the implementations of the disclosure, exposure of the 2D pixel array is controlled to obtain the original image data, and the target image data is outputted according to the original image data, so that a data output structure of the output target image data can be matched with back-end algorithms and hardware functions.

In the following, a basic structure of the image sensor10will be introduced. Referring toFIG.1,FIG.1is a schematic diagram of the image sensor10in implementations of the disclosure. The image sensor10includes a 2D pixel array11, a filter array16, and a lens array17. Along a light-receiving direction of the image sensor10, the lens array17, the filter array16, and the 2D pixel array11are arranged in sequence.

The image sensor10may use a complementary metal oxide semiconductor (CMOS) photosensitive element or a charge-coupled device (CCD) photosensitive element.

The 2D pixel array11includes multiple pixels101arranged in a 2D array. The 2D pixel array11includes minimal repeating units, and each minimal repeating unit includes multiple subunits102.

The filter array16includes multiple filters160, and each filter160covers a corresponding pixel101. The spectral response of each pixel101(i.e., the color of light that the pixel101can receive) is determined by the color of the filter160corresponding to that pixel101.

The lens array17includes multiple lenses170, and each lens170covers a corresponding subunit102(as illustrated inFIG.1). Optionally, each lens170may cover a corresponding pixel101.

FIG.2is a schematic diagram of a pixel circuit110in implementations of the disclosure. The operation principle of the pixel circuit110is described below in conjunction withFIG.1andFIG.2.

As illustrated inFIG.1andFIG.2, the pixel circuit110includes a photoelectric conversion element117(for example, a photodiode PD), an exposure control circuit116(for example, a transfer transistor112), a reset circuit (for example, a reset transistor113), an amplifying circuit (for example, an amplifying transistor114), and a selecting circuit (for example, a selecting transistor115). In implementations of the disclosure, the transfer transistor112, the reset transistor113, the amplifying transistor114, and the selecting transistor115are each, for example, a MOS transistor, but are not limited thereto.

For example, referring toFIG.1andFIG.2, the gate TG of the transfer transistor112is connected to a vertical drive unit (not illustrated) of the image sensor10through the exposure control line (not illustrated). The gate RG of the reset transistor113is connected to the vertical drive unit through a reset control line (not illustrated). The gate SEL of the selecting transistor115is connected to the vertical drive unit through a selecting line (not illustrated). The exposure control circuit116(such as transfer transistor112) in each pixel circuit110is electrically connected with the photoelectric conversion element117and is configured to transfer a potential accumulated by the photoelectric conversion element117after illumination. For example, the photoelectric conversion element117includes the photodiode PD, and the anode of the photodiode PD is connected to ground, for example. The photodiode PD converts the received light into charges. The cathode of the photodiode PD is connected to a floating diffusion unit FD through the exposure control circuit116(for example, the transfer transistor112). The floating diffusion unit FD is connected to the gate of the amplifying transistor114and the source of the reset transistor113.

For example, the exposure control circuit116is the transfer transistor112, and the control terminal TG of the exposure control circuit116is the gate of the transfer transistor112. When a pulse of an effective level (for example, VPIX level) is transmitted to the gate of the transfer transistor112through the exposure control line, the transfer transistor112is turned on. The transfer transistor112transmits the charges generated from photoelectric conversion by the photodiode PD to the floating diffusion unit FD.

For example, the drain of the reset transistor113is connected to a pixel power supply VPIX. The source of the reset transistor113is connected to the floating diffusion unit FD. Before the charges are transferred from the photodiode PD to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor113through the reset line, and the reset transistor113is turned on. The reset transistor113resets the floating diffusion unit FD to the pixel power supply VPIX.

For example, the gate of the amplifying transistor114is connected to the floating diffusion unit FD. The drain of the amplifying transistor114is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor113, the amplifying transistor114outputs a reset level through an output terminal OUT via the selecting transistor115. After the charges of the photodiode PD are transferred by the transfer transistor112, the amplifying transistor114outputs a signal level through the output terminal OUT via the selecting transistor115.

For example, the drain of the selecting transistor115is connected to the source of the amplifying transistor114. The source of selecting transistor115is connected to a column processing unit (not illustrated) of the image sensor10through the output terminal OUT. When a pulse of an effective level is transmitted to the gate of selecting transistor115through the selecting line, the selecting transistor115is turned on. The signal outputted from the amplifying transistor114is transmitted to the column processing unit through the selecting transistor115.

It should be noted that the pixel structure of the pixel circuit110in the implementations of the disclosure is not limited to the structure illustrated inFIG.2. For example, the pixel circuit110may have a three-transistor pixel structure, in which the functions of the amplifying transistor114and the selecting transistor115are realized by a single transistor. For example, the exposure control circuit116is also not limited to one transfer transistor112, and other electronic elements or structures with control terminals to control the conduction function can be used as the exposure control circuit in the implementations of the disclosure. The implementation of a single transfer transistor112is simple, low cost, and easy to control.

In an image sensor including multiple types of color pixels, pixels of different colors receive different exposure amounts per unit time. While some colors are saturated, other colors have not yet been exposed to an ideal state. For example, exposure to 60%-90% of a saturated exposure amount may have a relatively good SNR and accuracy, but the implementations of the disclosure are not limited thereto.

FIG.3illustrates RGBW (red, green, blue, panchromatic) as an example. Referring toFIG.3, the horizontal axis represents an exposure duration, the vertical axis represents an exposure amount, Q represents a saturated exposure amount, LW represents an exposure curve of the panchromatic pixel W, LG represents an exposure curve of the green pixel G, LR represents an exposure curve of the red pixel R, and LB represents an exposure curve of the blue pixel.

As can be seen fromFIG.3, the slope of the exposure curve LW of the panchromatic pixel W is the steepest, which means that the panchromatic pixel W can obtain more exposure per unit time and reaches saturation at time t1. The slope of the exposure curve LG of the green pixel G is the second steepest, and the green pixel G reaches saturation at time t2. The slope of the exposure curve LR of the red pixel R is the third steepest, and the red pixel R reaches saturation at time t3. The slope of the exposure curve LB of the blue pixel B is the least steep, and the blue pixel B reaches saturation at time t4. As can be seen fromFIG.3, the exposure amount received by the panchromatic pixel W per unit time is greater than the exposure amount received by the color pixel per unit time, that is, the sensitivity of the panchromatic pixel W is higher than that of the color pixel.

In this regard, for the image sensor10according to the implementations of the disclosure, multiple panchromatic pixels are added in the 2D pixel array11. Compared with the general image sensor based on Bayer pixel arrangement or based on QuadBayer pixel arrangement, luminous flux can be increased, allowing a better SNR.

It should be noted that the spectral response of each pixel101(i.e., the color of light that the pixel101can receive) is determined by the color of the filter160corresponding to the pixel101. Color pixels and panchromatic pixels throughout this disclosure refer to pixels101that are capable of responding to light of the same color as the corresponding filter160.

FIG.4toFIG.15illustrate multiple examples of arrangements of pixels101in the image sensor10(illustrated inFIG.1). Referring toFIG.4toFIG.15, multiple pixels101in the 2D pixel array11include multiple panchromatic pixels W and multiple color pixels (for example, multiple first color pixels A, multiple second color pixels B, and multiple color pixels C). The color and panchromatic pixels are distinguished by the wavelengths of light that can pass through the filter160(illustrated inFIG.1) overlying the corresponding pixel. The color pixel has a narrower spectral response than the panchromatic pixel. A response spectrum of a color pixel is, for example, a part of a response spectrum of a panchromatic pixel W. The 2D pixel array11includes minimal repeating units (FIG.4toFIG.15illustrate various examples of the minimal repeating units of pixels in image sensors10). The minimal repeating unit is repeated and arranged in rows and columns. Each minimal repeating unit includes multiple subunits102, and each subunit102includes multiple monochromatic pixels and multiple panchromatic pixels. For example, each minimal repeating unit includes four subunits102, where one subunit102includes multiple monochromatic pixels A (i.e., first color pixels A) and multiple panchromatic pixels W, and two subunits102include multiple monochromatic pixels B (i.e., second color pixels B) and multiple panchromatic pixels W, and the remaining one subunit102includes multiple monochromatic pixels C (i.e., third color pixels C) and multiple panchromatic pixels W.

For example, the minimal repeating unit has the same number of pixels101in rows and columns. For example, the minimal repeating unit has, but is not limited to, 4 rows and 4 columns, 6 rows and 6 columns, 8 rows and 8 columns, or 10 rows and 10 columns. For example, the subunit102has the same number of pixels101in rows and columns. For example, the subunit102includes, but is not limited to, 2 rows and 2 columns, 3 rows and 3 columns, 4 rows and 4 columns, or 5 rows and 5 columns. Such arrangement helps to balance resolution and color performance of the image in the row and column directions, thus improving the display effect.

In an example, in the minimal repeating unit, the panchromatic pixels W are arranged in a first diagonal direction D1, and the color pixels are arranged in a second diagonal direction D2different from the first diagonal direction D1.

For example,FIG.4is a schematic diagram of an arrangement of pixels101in a minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 16 pixels in 4 rows and 4 columns, and a subunit102has 4 pixels in 2 rows and 2 columns. The 16 pixels are arranged as follow:W A W BA W B WW B W CB W C W
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.4, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper left corner and the lower right corner inFIG.4). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the lower left corner and the upper right corner inFIG.4). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

It should be noted that the first diagonal direction D1and the second diagonal direction D2are not limited to the diagonal lines, but also include directions parallel to the diagonal lines. The “direction” herein is not a single direction, but can be understood as the concept of a “straight line” indicating the arrangement, and can be a two-way direction indicated at both ends of the straight line.

For example,FIG.5is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 16 pixels101in 4 rows and 4 columns, and a subunit102has 4 pixels101in 2 rows and 2 columns. The 16 pixels are arranged as follow:A W B WW A W BB W C WW B W C
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.5, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper right corner and the lower left corner inFIG.5). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the upper left corner and the lower right corner inFIG.5). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

For example,FIG.6is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure.FIG.7is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The implementations ofFIG.6andFIG.7correspond to the arrangements of pixels101inFIG.4andFIG.5respectively, where the first color pixel A is a red pixel R, the second color pixel B is a green pixel G, and the third color pixel C is a blue pixel Bu.

It should be noted that, in some implementations, a response waveband of the panchromatic pixel W is a visible band (e.g., 400 nm-760 nm). For example, an infrared filter may be employed on the panchromatic pixel W to filter out infrared lights. In some implementations, the response waveband of the panchromatic pixel is a visible band and a near infrared band (e.g., 400 nm-1000 nm), and is matched with a response waveband of the photoelectric conversion element (such as the photodiode PD) in the image sensor10. For example, the panchromatic pixel W may not be provided with a filter, and the response waveband of the panchromatic pixel W is determined by the response waveband of the photodiode, and thus the response waveband of the panchromatic pixel W matches the response waveband of the photodiode. The implementations of the disclosure include but are not limited to the above waveband.

In some implementations, in the minimal repeating units illustrated inFIG.4andFIG.5, the first color pixel A may be a red pixel R, the second color pixel B may be a yellow pixel Y, and the third color pixel C may be a blue pixel Bu.

In some implementations, in the minimal repeating units illustrated inFIG.4andFIG.5, the first color pixel A may be a magenta pixel M, the second color pixel B may be a cyan pixel Cy, and the third color pixel C may be a yellow pixel Y.

For example,FIG.8is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 36 pixels101in 6 rows and 6 columns, and a subunit102has 9 pixels101in 3 rows and 3 columns. The 36 pixels are arranged as follow:W A W B W BA W A W B WW A W B W BB W B W C WW B W C W CB W B W C W
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.8, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper left corner and the lower right corner inFIG.8). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the lower left corner and the upper right corner inFIG.8). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

For example,FIG.9is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 36 pixels101in 6 rows and 6 columns, and the subunit102has 9 pixels101in 3 rows and 3 columns. The 36 pixels are arranged as follow:A W A W B WW A W B W BA W A W B WW B W C W CB W B W C WW B W C W C
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.9, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper right corner and the lower left corner inFIG.9). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the upper left corner and the lower right corner inFIG.9). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

For example, in the minimal repeating units illustrated inFIG.8andFIG.9, the first color pixel A is may be red pixel R, the second color pixel B may be a green pixel G, and the third color pixel C may be a blue pixel Bu. Optionally, in the minimal repeating units illustrated inFIG.8andFIG.9, the first color pixel A may be a red pixel R, the second color pixel B may be a yellow pixel Y, and the third color pixel C may be a blue pixel Bu. Optionally, in the minimal repeating units illustrated inFIG.8andFIG.9, the first color pixel A may be a magenta pixel M, the second color pixel B may be a cyan pixel Cy, and the third color pixel C may be a yellow pixel Y.

For example,FIG.10is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 64 pixels101in 8 rows and 8 columns, and a subunit102has 16 pixels101in 4 rows and 4 columns. The 64 pixels are arranged as follow:W A W A W B W BA W A W B W B WW A W A W B W BA W A W B W B WW B W B W C W CB W B W C W C WW B W B W C W CB W B W C W C W
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.10, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper left corner and the lower right corner inFIG.10). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the lower left corner and the upper right corner inFIG.10). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

For example,FIG.11is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 64 pixels101in 8 rows and 8 columns, and a subunit102has 16 pixels101in 4 rows and 4 columns. The 64 pixels are arranged as follow:A W A W B W B WW A W A W B W BA W A W B W B WW A W A W B W BB W B W C W C WW B W B W C W CB W B W C W C WW B W B W C W C
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.11, the panchromatic pixels W are arranged in a first diagonal direction D1(that is, a direction connecting the upper right corner and the lower left corner inFIG.11). The color pixels are arranged in a second diagonal direction D2(such as a direction connecting the upper left corner and the lower right corner inFIG.11). The first diagonal direction D1is different from the second diagonal direction D2. For example, the first diagonal line is perpendicular to the second diagonal line.

In the examples ofFIG.4toFIG.11, in each subunit102, adjacent panchromatic pixels W are arranged diagonally, and adjacent color pixels are also arranged diagonally. In another example, in each subunit102, adjacent panchromatic pixels are arranged horizontally and adjacent color pixels are also arranged horizontally, or adjacent panchromatic pixels are arranged vertically and adjacent color pixels are also arranged vertically. Panchromatic pixels in adjacent subunits102may be arranged horizontally or vertically, and color pixels in adjacent subunits102may also be arranged in horizontally or vertically.

For example,FIG.12is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 16 pixels101in 4 rows and 4 columns, and a subunit102has 4 pixels101in 2 rows and 2 columns. The 16 pixels are arranged as follow:W A W BW A W BW B W CW B W C where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.12, in each subunit102, adjacent panchromatic pixels W are arranged vertically, and adjacent color pixels are also arranged vertically.

For example,FIG.13is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. The minimal repeating unit has 16 pixels101in 4 rows and 4 columns, and a subunit102has 4 pixels101in 2 rows and 2 columns. The 16 pixels are arranged as follow:A A B BW W W WB B C CW W W W
where W represents a panchromatic pixel, A represents a first color pixel in multiple color pixels, B represents a second color pixel in the multiple color pixels, and C represents a third color pixel in the multiple color pixels.

As illustrated inFIG.13, in each subunit102, adjacent panchromatic pixels W are arranged horizontally, and adjacent color pixels are also arranged horizontally.

For example,FIG.14is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure.FIG.15is a schematic diagram of an arrangement of pixels101in another minimal repeating unit in implementations of the disclosure. Implementations ofFIG.14andFIG.15correspond to arrangements of pixels inFIG.12andFIG.13, where the first color pixel A may be red pixel R, the second color pixel B may be a green pixel G, and the third color pixel C may be a blue pixel Bu.

In some implementations, in the minimal repeating units illustrated inFIG.12andFIG.13, the first color pixel A may be a red pixel R, the second color pixel B may be a yellow pixel Y, and the third color pixel C may be a blue pixel Bu.

In some implementations, in the minimal repeating units illustrated inFIG.12andFIG.13, the first color pixel A may be a magenta pixel M, the second color pixel B may be a cyan pixel Cy, and the third color pixel C may be a yellow pixel Y.

The multiple panchromatic pixels and multiple color pixels in the 2D pixel array11of any arrangement illustrated inFIG.4toFIG.15can be controlled by different exposure control lines, so as to realize independent control of exposure duration of the panchromatic pixels and exposure duration of color pixels. Control terminals of the exposure control circuits of at least two panchromatic pixels adjacent in the first diagonal direction are electrically connected to the first exposure control line, and control terminals of at least two color pixels adjacent in the second diagonal direction are electrically connected to the second exposure control line. A first exposure signal may be transmitted through the first exposure control line to control the first exposure duration of the panchromatic pixels, and a second exposure signal may be transmitted through the second exposure control line to control the second exposure duration of the color pixels.

When the exposure duration for the panchromatic pixels and the exposure duration for the color pixels are independently controlled, the first exposure duration for the panchromatic pixels may be shorter than the second exposure duration for the color pixels. For example, a ratio of the first exposure duration to the second exposure duration may be one of 1:2, 1:3, and 1:4. For example, in a dark environment, the color pixels are more likely to be underexposed. Therefore, the ratio of the first exposure duration to the second exposure duration can be set to be 1:2, 1:3, or 1:4 according to ambient brightness. When the exposure ratio is the above integer ratio or close to the integer ratio, it is advantageous for the setting of timing and the setting and control of signals.

In some implementations, a relative relationship between the first exposure duration and the second exposure duration may be determined according to ambient brightness. For example, in case that the ambient brightness is less than or equal to a brightness threshold, the panchromatic pixels are exposed for the first exposure duration that is equal to the second exposure duration. In case that the ambient brightness is greater than the brightness threshold, the panchromatic pixels are exposed for the first exposure duration that is less than the second exposure duration. The relative relationship between the first exposure duration and the second exposure duration may be determined according to a brightness difference between the ambient brightness and the brightness threshold in case that the ambient brightness is greater than the brightness threshold. For example, the greater the brightness difference, the smaller the ratio of the first exposure duration to the second exposure duration. For example, when the brightness difference is within a first range [a,b), the ratio of the first exposure duration to the second exposure duration is 1:2; when the brightness difference is within a second range [b,c), the ratio of the first exposure duration to the second exposure duration is 1:3; and when the brightness difference is greater than or equal to c, the ratio of the first exposure duration to the second exposure duration is 1:4.

Referring toFIG.1andFIG.16, the control method in implementations of the disclosure may be applied to the image sensor10in any of the implementations above. The control method includes operations01and02.01: Original image data is obtained by controlling exposure of the 2D pixel array11, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.02: Target image data is outputted according to the original image data.

Referring toFIG.1andFIG.17, the control method of the implementations of the disclosure can be implemented by the camera assembly40. The camera assembly40includes the image sensor10of any of the above-described implementations and a processor20. The processor20may be integrated within the image sensor10or independently provided outside the image sensor10. Operations01and02may be implemented by the processor20. That is, the processor20can be configured to obtain original image data by controlling exposure of the 2D pixel array11, and to output target image data according to the original image data, where the original image data includes color original image data generated by exposure of the color pixels and panchromatic original image data generated by exposure of the panchromatic pixels.

According to the control method and the camera assembly40in implementations of the disclosure, multiple panchromatic pixels are added in the 2D pixel array11. Compared with the general image sensor based on Bayer pixel arrangement or based on QuadBayer pixel arrangement, luminous flux can be increased, allowing a better SNR. According to the control method and the camera assembly40in implementations of the disclosure, exposure of the 2D pixel array is controlled to obtain the original image data, and the target image data is outputted according to the original image data, so that a data output structure of the output target image data can be matched with back-end algorithms and hardware functions.

The original image data obtained by the exposure of the 2D pixel array11may be completely corresponding the pixels101in the 2D pixel array11in terms of arrangement. That is, the color original image data is obtained by exposure of the color pixels in the 2D pixel array11, and the panchromatic original image data is obtained by exposure of the panchromatic pixels in the 2D pixel array11.

Referring toFIG.17andFIG.18, in some implementations, the target image data is outputted according to the original image data (i.e., operation02) as follow.021: In a first data-reading mode, the target image data is outputted by reading the original image data row-by-row.

Referring toFIG.17, in some implementations, operation021may be implemented by the processor20. That is, the processor20may be configured to, in a first data-reading mode, output the target image data by reading the original image data row-by-row.

Specifically, the first data-reading mode is suitable for scenarios requiring full-size resolution or high resolution, or scenarios with high ambient brightness. When the processor20reads out the original image data row-by-row to output the target image data, the data output structure of the target image data completely corresponds to the arrangement of the pixels101in the 2D pixel array11. The data-reading method of this implementation of the disclosure has advantages of less calculation and high speed.

Referring toFIG.19, in an example, the minimal repeating unit includes 4 subunits102, which are respectively denoted as subunit U1, subunit U2, subunit U3, and subunit U4from left to right and from top to bottom. The minimal repeating unit includes 16 pixels101, which are denoted from left to right and from top to bottom as: color pixel R11, panchromatic pixel W12, color pixel G13, panchromatic pixel W14, panchromatic pixel W21, color pixel R22, panchromatic pixel W23, color pixel G24, color pixel G31, panchromatic pixel W32, color pixel B33, panchromatic pixel W34, panchromatic pixel W41, color pixel G42, panchromatic pixel W43, color pixel B44. That is, subunit U1includes color pixel R11, panchromatic pixel W12, panchromatic pixel W21, and color pixel R22, subunit U2includes color pixel G13, panchromatic pixel W14, panchromatic pixel W23, and color pixel G24, subunit U3includes color pixel G31, panchromatic pixel W32, panchromatic pixel W41, and color pixel G42, and subunit U4includes color pixel B33, panchromatic pixel W34, panchromatic pixel W43, and color pixel B44.

In the first data-reading mode, the processor20reads out the original image data row-by-row to output the target image data. For example, when the 2D pixel array11includes 4 minimal repeating units and the 4 minimal repeating units are arranged horizontally, the target image data outputted by the processor20reading the original image data row-by-row is: color original image data of color pixel R11, panchromatic original image data of panchromatic pixel W12, color original image data of color pixel G13, panchromatic original image data of panchromatic pixel W14. . . panchromatic original image data of panchromatic pixel W21, color original image data of color pixel R22, panchromatic original image data of panchromatic pixel W23, color original image data of color pixel G24. . . color original image data of color pixel G31, panchromatic original image data of panchromatic pixel W32, color original image data of color pixel B33, panchromatic original image data of panchromatic pixel W34. . . panchromatic original image data of panchromatic pixel W41, color original image data of color pixel G42, panchromatic original image data of panchromatic pixel W43, color original image data of color pixel B44, and so on. In the implementation of the disclosure, in each subunit102, adjacent panchromatic pixels are arranged diagonally, and adjacent color pixels are also arranged diagonally.

Referring toFIG.20, in an example, the minimal repeating unit includes 4 subunits102, which are respectively denoted as subunit U1, subunit U2, subunit U3, and subunit U4from left to right and from top to bottom. The minimal repeating unit includes 16 pixels101, which are denoted from left to right and from top to bottom as: color pixel R11, color pixel R12, color pixel G13, color pixel G14, panchromatic pixel W21, panchromatic pixel W22, panchromatic pixel W23, panchromatic pixel W24, color pixel G31, color pixel G32, color pixel B33, color pixel B34, panchromatic pixel W41, panchromatic pixel W42, panchromatic pixel W43, panchromatic pixel W44. That is to say, subunit U1includes color pixel R11, color pixel R12, panchromatic pixel W21, and panchromatic pixel W22, subunit U2includes color pixel G13, color pixel G14, panchromatic pixel W23, and panchromatic pixel W24, subunit U3includes color pixel G31, color pixel G32, panchromatic pixel W41, and panchromatic pixel W42, and subunit U4includes color pixel B33, color pixel B34, panchromatic pixel W43, and panchromatic pixel W44.

In the first data-reading mode, the processor20reads out the original image data row-by-row to output the target image data. For example, when the 2D pixel array11includes 4 minimal repeating units and the 4 minimal repeating units are arranged horizontally, the target image data outputted by the processor20reading the original image data row-by-row is: color original image data of color pixel R11, color original image data of color pixel R12, color original image data of color pixel G13, color original image data of color pixel G14. . . panchromatic original image data of panchromatic pixel W21, panchromatic original image data of panchromatic pixel W22, panchromatic original image data of panchromatic pixel W23, panchromatic original image data of panchromatic pixel W24. . . color original image data of color pixel G31, color original image data of color pixel G32, color original image data of color pixel B33, color original image data of color pixel B34. . . panchromatic original image data of panchromatic pixel W41, panchromatic original image data of panchromatic pixel W42, panchromatic original image data of panchromatic pixel W43, panchromatic original image data of panchromatic pixel W44. . . . In the implementation of the disclosure, in each subunit102, in each subunit102, adjacent panchromatic pixels are arranged horizontally, and adjacent color pixels are also arranged horizontally (or in other words, in the 2D pixel array11, the color pixels and panchromatic pixels are arranged row-by-row, that is, one row of color pixels, one row of panchromatic pixels, and so on; or one row of panchromatic pixels, one row of color pixels, and so on).

In other examples, the arrangement of the pixels101of the minimal repeating unit may also be as follows. Adjacent panchromatic pixels may be arranged vertically, and adjacent color pixels may also be arranged vertically (or in other words, in the 2D pixel array11, color pixels and panchromatic pixels are arranged column-by-column, that is, a column of color pixels a column of panchromatic pixels, and so on; or a column of panchromatic pixels, a column of color pixels, and so on). In this case, in the first data-reading mode, the processor20may also read the original image data row-by-row to output the target image data, which will not be described in detail one by one herein.

It should be noted that, since in the examples illustrated inFIG.19andFIG.20the original image data obtained by the exposure of the 2D pixel array11completely corresponds to the pixels101in the 2D pixel array11in terms of arrangement, illustrations of the original image data are omitted inFIG.19andFIG.20.

Referring toFIG.17andFIG.21, in some implementations, the target image data includes first target image data and second target image data. The target image data is outputted according to the original image data (i.e., operation02) as follow.022: In a second data-reading mode, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102, and the second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17, in some implementations, the target image data includes first target image data and second target image data. Operation022may be implemented by the processor20. That is, the processor20may be configured to, in a second data-reading mode, obtain the first target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Specifically, the second data-reading mode is suitable for most scenarios, and is especially suitable for scenarios with low ambient brightness.

Referring toFIG.22, in an example, the minimal repeating unit includes 4 subunits102, and the minimal repeating unit includes 16 pixels101. The 4 subunits102and the 16 pixels101inFIG.22are denoted similarly to the 4 subunits102and the 16 pixels101inFIG.19, which will not be described in detail herein.

In the second data-reading mode, the processor20obtains the first target image data according to the color original image data generated by the exposure of the multiple color pixels in each subunit102, and obtains the second target image data according to the color original image data generated by the exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by the exposure of multiple panchromatic pixels in each subunit102. Specifically, in the example ofFIG.22, the processor20may obtain the first target image data according to the color original image data generated by the exposure of multiple color pixels arranged diagonally in each subunit102, and obtains the second target image data according to the color original image data generated by the exposure of the multiple color pixels arranged diagonally in each subunit102and the panchromatic original image data generated by the exposure of multiple panchromatic pixels arranged diagonally in each subunit102. In an example of another arrangement of pixels101, the processor20may obtain the first target image data according to the color original image data generated by the exposure of the multiple color pixels arranged horizontally in each subunit102, and obtain the second target image data according to the color original image data generated by the exposure of the multiple color pixels arranged horizontally in each subunit102and the panchromatic original image data generated by the exposure of multiple panchromatic pixels arranged horizontally in each subunit102. Optionally, the processor20may obtain the first target image data according to the color original image data generated by the exposure of the multiple color pixels arranged vertically in each subunit102, and obtain the second target image data according to the color original image data generated by the exposure of the multiple color pixels arranged vertically in each subunit102and the panchromatic original image data generated by the exposure of multiple panchromatic pixels arranged vertically in each subunit102.

For example, when the 2D pixel array11includes 4 minimal repeating units and the 4 minimal repeating units are arranged horizontally, the target image data outputted by the processor20according to the original image data is: the first target image data obtained according to the color original image data of the color pixel R11and the color original image data of the color pixel R22, the first target image data obtained according to the color original image data of the color pixel G13and the color original image data of the color pixel G24. . . the second target image data obtained according to the color original image data of the color pixel R11, the color original image data of the color pixel R22, the panchromatic original image data of the panchromatic pixel W12and the panchromatic original image data of the panchromatic pixel W21, the second target image data obtained according to the color original image data of the color pixel G13, the color original image data of the color pixel G24, the panchromatic original image data of the panchromatic pixel W14, and the panchromatic original image data of the panchromatic pixel W23. . . the first target image data obtained according to the color original image data of the color pixel G31and the color original image data of the color pixel G42, the first target image data obtained according to the color original image data of the color pixel B33and the color original image data of the color pixel B44. . . the second target image data obtained according to the color original image data of the color pixel G31, the color original image data of the color pixel G42, the panchromatic original image data of the panchromatic pixel W32, and the panchromatic original image data of the panchromatic pixel W41, the second target image data obtained according to the color original image data of the color pixel B33, the color original image data of the color pixel B44, the panchromatic original image data of the panchromatic pixel W34, and the panchromatic original image data of the panchromatic pixel W43, and so on. In the implementation of the disclosure, one row of target image data includes the first target image data obtained according to color original image data of multiple color pixels, and one row of target image data include the second target image data obtain according to color original image data of multiple color pixels and panchromatic original image data of multiple panchromatic pixels. Since the panchromatic original image data of multiple panchromatic pixels is fused into the second target image data, the SNR of the image sensor10can be improved.

In other examples, when the ambient brightness is extremely low, the second target image data may be obtained as follows instead. The processor20obtains the second target image data according to the panchromatic original image data generated by exposure of multiple panchromatic pixels in each subunit102. For example, the processor20performs pixel addition on the panchromatic original image data generated by exposure of multiple panchromatic pixels in each subunit102to obtain the second target image data.

It should be noted that, in the example ofFIG.22, since the original image data obtained by the exposure of the 2D pixel array11completely corresponds to the pixels101in the 2D pixel array11in terms of arrangement, illustrations of the original image data are omitted inFIG.22.

Referring toFIG.17andFIG.23, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0221: The first target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0222: Pixel addition is performed on the color original image data generated by exposure of the multiple color pixels in each subunit102, and pixel addition is performed on the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.0223: The second target image data is obtained by performing pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition.

Referring toFIG.17, in some implementations, operation0221, operation0222, and operation0223may be implemented by the processor20. That is, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, perform pixel addition on the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition.

Specifically, still referring toFIG.22, the first target image data may include, in the right image ofFIG.22, the first target image data at the color pixel R, the first target image data at the color pixel G, and the first target image data at the color pixel B. The second target image data may include, in the right image ofFIG.22, the second target image data at the color pixel R′, the second target image data at the color pixel G′, and the second target image data at the color pixel B′.

For subunit U1, the processor20performs pixel addition (also referred to as Analog sum) on the color original image data of color pixel R11and the color original image data of color pixel R22in the left image ofFIG.22, to obtain the first target image data at color pixel R11in the right image ofFIG.22. The processor20performs pixel addition on the color original image data of color pixel R11and the color original image data of color pixel R22in the left image ofFIG.22, performs pixel addition on the panchromatic original image data of panchromatic pixel W12and the panchromatic original image data of panchromatic pixel W21, and then performs pixel averaging (also referred to as Digital avg) on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition, to obtain the second target image data at color pixel R′21in the right image ofFIG.22.

For subunit U2, the processor20performs pixel addition on the color original image data of color pixel G13and the color original image data of color pixel G24in the left image ofFIG.22, to obtain the first target image data at color pixel G12in the right image ofFIG.22. The processor20performs pixel addition on the color original image data of color pixel G13and the color original image data of color pixel G24in the left image ofFIG.22, performs pixel addition on the panchromatic original image data of panchromatic pixel W14and the panchromatic original image data of panchromatic pixel W23, and then performs pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition, to obtain the second target image data at the color pixel G′22in the right image ofFIG.22.

For subunit U3, the processor20performs pixel addition on the color original image data of color pixel G31and the color original image data of color pixel G42in the left image ofFIG.22, to obtain the first target image data at color pixel G31in the right image ofFIG.22. The processor20performs pixel addition on the color original image data of color pixel G31and the color original image data of color pixel G42in the left image ofFIG.22, performs pixel addition on the panchromatic original image data of panchromatic pixel W32and the panchromatic original image data of panchromatic pixel W41, and then performs pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition, to obtain the second target image data at color pixel G′41in the right image ofFIG.22.

For subunit U4, the processor20performs pixel addition on the color original image data of color pixel B33and the color original image data of color pixel B44in the left image ofFIG.22, to obtain the first target image data at color pixel B32in the right image ofFIG.22. The processor20performs pixel addition on the color original image data of color pixel B33and the color original image data of color pixel B44in the left image ofFIG.22, performs pixel addition on the panchromatic original image data of panchromatic pixel W34and the panchromatic original image data of panchromatic pixel W43, and then performs pixel averaging on the color original image data subjected to pixel addition and the panchromatic original image data subjected to pixel addition, to obtain the second target image data at color pixel B′42in the right image ofFIG.22.

Referring toFIG.17andFIG.24, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0224: The first target image data is obtained by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0225: The second target image data is obtained by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17, in some implementations, operation0224and operation0225may be implemented by the processor20. That is, the processor20may be configured to obtain the first target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel averaging on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.25, in an example, the minimal repeating unit includes 4 subunits102, and the minimal repeating unit includes 16 pixels101. The 4 subunits102and the 16 pixels101inFIG.25are denoted similarly to the 4 subunits102and the 16 pixels101inFIG.19, and will not be described in detail herein.

The first target image data may include the first target image data at the color pixel R, the first target image data at the color pixel G, and the first target image data at the color pixel B in the right image ofFIG.25. The second target image data may include the second target image data at the color pixel R′, the second target image data at the color pixel G′, and the second target image data at the color pixel B′ in the right image ofFIG.25.

For subunit U1, the processor20performs pixel averaging on the color original image data of color pixel R11and the color original image data of color pixel R22in the left image ofFIG.25, to obtain the first target image data at color pixel R11in the right image ofFIG.25. The processor20performs pixel averaging on the color original image data of color pixel R11, the color original image data of color pixel R22, the panchromatic original image data of panchromatic pixel W12, and the panchromatic original image data of panchromatic pixel W21in the left image ofFIG.25, to obtain the second target image data at color pixel R′21in the right image ofFIG.25.

For subunit U2, the processor20performs pixel averaging on the color original image data of color pixel G13and the color original image data of color pixel G24in the left image ofFIG.25, to obtain the first target image data at color pixel G12in the right image ofFIG.25. The processor20performs pixel averaging on the color original image data of color pixel G13, the color original image data of color pixel G24, the panchromatic original image data of panchromatic pixel W14, and the panchromatic original image data of panchromatic pixel W23in the left image ofFIG.25, to obtain the second target image data at color pixel G′22in the right image ofFIG.25.

For subunit U3, the processor20performs pixel averaging on the color original image data of color pixel G31and the color original image data of color pixel G42in the left image ofFIG.25, to obtain the first target image data at color pixel G31in the right image ofFIG.25. The processor20performs pixel averaging on the color original image data of color pixel G31, the color original image data of color pixel G42, the panchromatic original image data of panchromatic pixel W32, and the panchromatic original image data of panchromatic pixel W41in the left image ofFIG.25, to obtain the second target image data at color pixel G′41in the right image ofFIG.25.

For subunit U4, the processor20performs pixel averaging on the color original image data of color pixel B33and the color original image data of color pixel B44in the left image ofFIG.25, to obtain the first target image data at color pixel B32in the right image ofFIG.25. The processor20performs pixel averaging on the color original image data of color pixel B33, the color original image data of color pixel B44, the panchromatic original image data of panchromatic pixel W34, and the panchromatic original image data of panchromatic pixel W43in the left image ofFIG.25, to obtain the second target image data at color pixel B′42in the right image ofFIG.25.

Referring toFIG.17andFIG.26, in some implementations, the first target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102as follows.0226: The first target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102.

The second target image data is obtained according to the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102as follows.0227: the second target image data is obtained by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.17, in some implementations, operation0226and operation0227may be implemented by the processor20. That is, the processor20may be configured to obtain the first target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102, and obtain the second target image data by performing pixel addition on the color original image data generated by exposure of the multiple color pixels in each subunit102and the panchromatic original image data generated by exposure of the multiple panchromatic pixels in each subunit102.

Referring toFIG.27, in an example, the minimal repeating unit includes 4 subunits102, and the minimal repeating unit includes 16 pixels101. The 4 subunits102and the 16 pixels101inFIG.27are denoted similarly to the 4 subunits102and the 16 pixels101inFIG.19, and will not be described in detail herein.

The first target image data includes the first target image data at the color pixel R, the first target image data at the color pixel G, and the first target image data at the color pixel B in the right image ofFIG.27. The second target image data may include the second target image data at the color pixel R′, the second target image data at the color pixel G′, and the second target image data at the color pixel B′ in the right image ofFIG.27.

For subunit U1, the processor20performs pixel addition on the color original image data of color pixel R11and the color original image data of color pixel R22in the left image ofFIG.27, to obtain the first target image data at color pixel R11in the right image ofFIG.27. The processor20performs pixel addition on the color original image data of color pixel R11, the color original image data of color pixel R22, the panchromatic original image data of panchromatic pixel W12, and the panchromatic original image data of panchromatic pixel W21in the left image ofFIG.27, to obtain the second target image data at color pixel R′21in the right image ofFIG.27.

For subunit U2, the processor20performs pixel addition on the color original image data of color pixel G13and the color original image data of color pixel G24in the left image ofFIG.27, to obtain the first target image data at color pixel G12in the right image ofFIG.27. The processor20performs pixel addition on the color original image data of color pixel G13, the color original image data of color pixel G24, the panchromatic original image data of panchromatic pixel W14, and the panchromatic original image data of panchromatic pixel W23in the left image ofFIG.27, to obtain the second target image data at color pixel G′22in the right image ofFIG.27.

For subunit U3, the processor20performs pixel addition on the color original image data of color pixel G31and the color original image data of color pixel G42in the left image ofFIG.27, to obtain the first target image data at color pixel G31in the right image ofFIG.27. The processor20performs pixel addition on the color original image data of color pixel G31, the color original image data of color pixel G42, the panchromatic original image data of panchromatic pixel W32, and the panchromatic original image data the panchromatic pixel W41in the left image ofFIG.27, to obtain the second target image data at color pixel G′41in the right image ofFIG.27.

For subunit U4, the processor20performs pixel addition on the color original image data of color pixel B33and the color original image data of color pixel B44in the left image ofFIG.27, to obtain the first target image data at color pixel B32in the right image ofFIG.27. The processor20performs pixel addition on the color original image data of color pixel B33, the color original image data of color pixel B44, the panchromatic original image data of panchromatic pixel W34, and the panchromatic original image data of panchromatic pixel W43in the left image ofFIG.27, to obtain the second target image data at color pixel B′42in the right image ofFIG.27.

Referring toFIG.17andFIG.28, in some implementations, the original image data is obtained by controlling exposure of the 2D pixel array11(i.e., operation01) as follows.011: First original image data is obtained by controlling multiple color pixels and multiple panchromatic pixels in a (2n−1)-th row to be exposed for a first exposure duration, second original image data is obtained by controlling multiple color pixels and multiple panchromatic pixels in a 2n-th row to be exposed for a second exposure duration, where n is a natural number greater than or equal to 1, the first exposure duration is different from the second exposure duration, the first original image data includes first color original image data generated by exposure of the color pixels and first panchromatic original image data generated by exposure of the panchromatic pixels, and the second original image data includes second color original image data generated by exposure of the color pixels and second panchromatic original image data generated by exposure of the panchromatic pixels.

The target image data includes first target image data, second target image data, third target image data, and fourth target image data. The target image data is outputted according to the original image data (i.e., operation02) as follows.023: In a third data-reading mode, the first target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, the second target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, the third target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and the fourth target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Referring toFIG.17, in some implementations, operation011and operation023may be implemented by the processor20. That is, the processor20may be configured to obtain first original image data by controlling multiple color pixels and multiple panchromatic pixels in a (2n−1)-th row to be exposed for a first exposure duration, and obtain second original image data by controlling multiple color pixels and multiple panchromatic pixels in a 2n-th row to be exposed for a second exposure duration, where n is a natural number greater than or equal to 1, the first exposure duration is different from the second exposure duration, the first original image data includes first color original image data generated by exposure of the color pixels and first panchromatic original image data generated by exposure of the panchromatic pixels, and the second original image data includes second color original image data generated by exposure of the color pixels and second panchromatic original image data generated by exposure of the panchromatic pixels. The target image data includes first target image data, second target image data, third target image data, and fourth target image data. The processor may be further configured to, in a third data-reading mode, obtain the first target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, obtain the second target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, obtain the third target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Specifically, the first exposure duration may be longer than the second exposure duration, or the first exposure duration may be shorter than the second exposure duration. The third data-reading mode is suitable for high-dynamic-range (HDR) scenarios.

Referring toFIG.29, in an example, the minimal repeating unit includes 4 subunits102, and the minimal repeating unit includes 16 pixels101. The 4 subunits102and the 16 pixels101inFIG.29are denoted similarly to the 4 subunits102and the 16 pixels101inFIG.19, and will not be described in detail herein.

Color pixel R11, panchromatic pixel W12, color pixel G13, and panchromatic pixel W14are exposed for the first exposure duration. Panchromatic pixel W21, the color pixel R22, the panchromatic pixel W23, and the color pixel G24are exposed for the second exposure duration. Color pixel G31, panchromatic pixel W32, color pixel B33, and panchromatic pixel W34are exposed for the first exposure duration. Panchromatic pixel W41, color pixel G42, panchromatic pixel W43, and color pixel B44are exposed for the second exposure duration.

In the third data-reading mode, the processor obtains the first target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration, obtains the second target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration, obtains the third target image data according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtains the fourth target image data according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration. In the example ofFIG.29, the processor20may obtain the third target image data according to the first color original image data generated by exposure of the color pixels arranged horizontally in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels arranged horizontally in each subunit102for the first exposure duration, and obtains the fourth target image data according to the second color original image data generated by exposure of the color pixels arranged horizontally in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels arranged horizontally in each subunit102for the second exposure duration. In an example of another arrangement of pixels101, the processor20may obtain the third target image data according to the first color original image data generated by exposure of the color pixels arranged vertically in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels arranged vertically in each subunit102for the first exposure duration, and obtains the fourth target image data according to the second color original image data generated by exposure of the color pixels arranged vertically in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels arranged vertically in each subunit102for the second exposure duration.

For example, when the 2D pixel array11includes two minimal repeating units and the two minimal repeating units are arranged horizontally, the target image data outputted by the processor20according to the original image data is: first target image data obtained according to the first color original image data of color pixel R11exposed for the first exposure duration, third target image data obtained according to the first color original image data of color pixel R11exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W12exposed for the first exposure duration, first target image data obtained according to the first color original image data of color pixel G13exposed for the first exposure duration, third target image data obtained according to the first color original image data of color pixel G13exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W14exposed for the first exposure duration . . . second target image data obtained according to the second color original image data of color pixel R22exposed for the second exposure duration, fourth target image data obtained according to the second color original image data of color pixel R22exposed for the second exposure duration and the second panchromatic original image data of the panchromatic pixel W21exposed for the second exposure duration, second target image data obtained according to the second color original image data of color pixel G24exposed for the second exposure duration, fourth target image data obtained according to the second color original image data of color pixel G24exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W23exposed for the second exposure duration . . . first target image data obtained according to the first color original image data of color pixel G31exposed for the first exposure duration, third target image data obtained according to the first color original image data of color pixel G31exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W32exposed for the first exposure duration, first target image data obtained according to the first color original image data of color pixel B33exposed for the first exposure duration, third target image data obtained according to the first color original image data of color pixel B33exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W34exposed for the first exposure duration . . . second target image data obtained according to the second color original image data of color pixel G42exposed for the second exposure duration, fourth target image data obtained according to the second color original image data of color pixel G42exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W41exposed for the second exposure duration, second target image data obtained according to the second color original image data of color pixel B44exposed for the second exposure duration, fourth target image data obtained according to the second color original image data of color pixel B44exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W34exposed for the second exposure duration . . . . In implementations of the disclosure, the first target image data, the third target image data, the second target image data, and the fourth target image data form data subunits, and one data subunit corresponds to one subunit102. One column of target image data includes the first target image data obtained according to the first color original image data of the color pixel exposed for the first exposure duration and the second target image data obtained according to the second color original image data of the color pixel exposed for the second exposure duration. One column of target image data includes the third target image data obtained according to the first color original image data of the color pixel exposed for the first exposure duration and the first panchromatic original image data of the panchromatic pixel exposed for the first exposure duration and the fourth target image data obtained according to the second color original image data of the color pixel exposed for the second exposure duration and the second panchromatic original image data of the panchromatic pixel exposed for the second exposure duration. Since the panchromatic original image data of the panchromatic pixels exposed for the first exposure duration or the panchromatic pixels exposed for the second exposure duration is fused into the second target image data and the fourth target image data, the SNR of the image sensor10can be improved.

It should be noted that, since in the example illustrated inFIG.29, the original image data obtained by the exposure of the 2D pixel array11completely corresponds to the pixels101in the 2D pixel array11in terms of arrangement, illustrations of the original image data are omitted inFIG.29.

Referring toFIG.17andFIG.30, in some implementations, the third target image data is obtained according to the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration as follows.0231: The third target image data is obtained by performing pixel addition on the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration.

The fourth target image data is obtained according to the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.0232: The fourth target image data is obtained by performing pixel addition on the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Referring toFIG.17, in some implementations, operation0231and operation0232may be implemented by the processor20. That is, the processor20may be configure to obtain the third target image data by performing pixel addition on the first color original image data generated by exposure of the color pixels in each subunit102for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the first exposure duration, and obtain the fourth target image data by performing pixel addition on the second color original image data generated by exposure of the color pixels in each subunit102for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixels in each subunit102for the second exposure duration.

Specifically, still referring toFIG.29, the first target image data may include the first target image data at the color pixel R corresponding to S, the first target image data at the color pixel G corresponding to S, and the first target image data at the color pixel B corresponding to S in the right image ofFIG.29. The second target image data may include the second target image data at the color pixel R corresponding to L, the second target image data at the color pixel G corresponding to L, and the second target image data at the color pixel B corresponding to L in the right image ofFIG.29. The third target image data may include the third target image data at the color pixel R corresponding to S′, the third target image data at the color pixel G corresponding to S′, and the third target image data at the color pixel B corresponding to S′ in the right image ofFIG.29. The fourth target image data may include the fourth target image data at the color pixel R corresponding to L′, the fourth target image data at the color pixel G corresponding to L′, and the fourth target image data at the color pixel B corresponding to L′ in the right image ofFIG.29.

For the first target image data and the second target image data, the processor20may directly read the first color original image data generated by exposure of the color pixels for the first exposure duration to output the first target image data, and directly read the second color original image data generated by exposure of the color pixels for the second exposure duration to output the second target image data.

For example, for subunit U1, the processor20directly reads the first color original image data of color pixel R11exposed for the first exposure duration to output the first target image data, and directly reads the second color original image data of color pixel R22exposed for the second exposure duration to output the second target image data. For subunit U2, the processor20directly reads the first color original image data of color pixel G13exposed for the first exposure duration to output the first target image data, and directly reads the second color original image data of color pixel G24exposed for the second exposure duration to output the second target image data. For subunit U3, the processor20directly reads the first color original image data of color pixel G31exposed for the first exposure duration to output the first target image data, and directly reads the second color original image data of color pixel G42exposed for the second exposure duration to output the second target image data. For subunit U4, the processor20directly reads the first color original image data of color pixel B33exposed for the first exposure duration to output the first target image data, and directly reads the second color original image data of color pixel B44exposed for the second exposure duration to output the second target image data.

In the implementations of the disclosure, for the third target image data and the fourth target image data, the processor20may perform pixel addition on the first color original image data generated by exposure of the color pixel for the first exposure duration and the first panchromatic original image data generated by exposure of the panchromatic pixel for the first exposure duration, to obtain the third target image data, and perform pixel addition on the second color original image data generated by exposure of the color pixel for the second exposure duration and the second panchromatic original image data generated by exposure of the panchromatic pixel for the second exposure duration, to obtain the fourth target image data.

For example, for subunit U1, the processor20performs pixel addition on the first color original image data of color pixel R11exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W12exposed for the first exposure duration to obtain the third target image data, and performs pixel addition on the second color original image data of color pixel R22exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W21exposed for the second exposure duration to obtain the fourth target image data.

For subunit U2, the processor20performs pixel addition on the first color original image data of color pixel G13exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W14exposed for the first exposure duration to obtain the third target image data, and performs pixel addition on the second color original image data of color pixel G24exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W23exposed for the second exposure duration to obtain the fourth target image data.

For subunit U3, the processor20performs pixel addition on the first color original image data of color pixel G31exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W32exposed for the first exposure duration to obtain the third target image data, and performs pixel addition on the second color original image data of color pixel G42exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W41exposed for the second exposure duration to obtain the fourth target image data.

For subunit U4, the processor20performs pixel addition on the first color original image data of color pixel B33exposed for the first exposure duration and the first panchromatic original image data of panchromatic pixel W34exposed for the first exposure duration to obtain the third target image data, and performs pixel addition on the second color original image data of color pixel B44exposed for the second exposure duration and the second panchromatic original image data of panchromatic pixel W43exposed for the second exposure duration to obtain the fourth target image data.

In some implementations, after the target image data is obtained through any of the above implementations, a corresponding algorithm may be further selected from the back-end algorithms for the camera assembly40according to the data output structure of the target image data, so as to process the target image data (the back-end algorithm can be compatible with the data output structure). The target image data processed using the algorithm can be matched with hardware such as the back-end image processor.

Referring toFIG.31, a mobile terminal90according to implementations of the disclosure includes a housing80and the camera assembly40according to any of the above implementations. The camera assembly40is received in the housing80. Specifically, the camera assembly40can be installed on the front of the housing80as a front camera assembly, or the camera assembly40can be installed on the back of the housing80as a rear camera assembly. The mobile terminal90may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, smart glasses, a smart helmet, etc.), a head-mounted display device, a virtual reality device, etc., which are not limited herein.

According to the mobile terminal90in implementations of the disclosure, multiple panchromatic pixels are added in the 2D pixel array11. Compared with the general image sensor based on Bayer pixel arrangement or based on QuadBayer pixel arrangement, luminous flux can be increased, allowing a better SNR. According to the mobile terminal90in implementations of the disclosure, exposure of the 2D pixel array11is controlled to obtain the original image data, and the target image data is outputted according to the original image data, so that a data output structure of the outputted target image data can be matched with the back-end algorithms and hardware functions.

In the description of this specification, the terms “one implementation”, “some implementations”, “exemplary implementation”, “example”, “specific example”, “some examples” and the like, mean that a particular feature, structure, material, or characteristic described in conjunction with the implementations is included in at least one implementation or example of the disclosure. In this specification, schematic representations of the above terms do not necessarily refer to the same implementation or example. Furthermore, the particular feature, structure, material or characteristic described may be combined in any suitable manner in any one or more implementations or examples. Furthermore, those skilled in the art may combine and merge different implementations or examples described in this specification, as well as the features of the different implementations or examples, without conflicting each other.

Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code including one or more executable instructions for implementing a specified logical function or operation of the process, and the scope of the implementations of the disclosure includes alternative implementations in which the functions may be performed out of the order illustrated or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should be understood by those skilled in the art to which the implementations of the disclosure belong.

Although the implementations of the disclosure have been illustrated and described above, it should be understood that the above implementations are exemplary and should not be construed as limitations to the disclosure. Variations, modifications, substitutions, and alterations may be made to the above implementations by those skilled in the art within the scope of the disclosure.