Patent Description:
An image sensor may include pixels for detection of a phase difference. The image sensor may generate phase difference information by using the pixels. <CIT> discloses an imaging apparatus performing phase detection autofocus and noise reduction. <CIT>discloses dual-core focusing image sensor.

Accordingly, an aspect of the disclosure is to provide an image sensor that designates some (e.g., <NUM>%) of all the pixels for detecting a phase difference and generates depth data. However, data that is obtained from the designated pixels may be used only to generate depth data. In other words, the data obtained from the designated pixels may not be used as image data.

In a dual pixel or dual photodiode (2PD) image sensor, each pixel includes two photodetectors, with one color filter covering the two photodetectors, and at least one micro lens covering the color filter. The 2PD image sensor generates image data as well as depth data, by using data obtained from two photodetectors.

For example, the image sensor may generate depth data based on a difference in an incident light quantity between the two photodetectors, may add the incident light quantities of the two photodetectors, and may generate image data based on the added incident light quantity. The 2PD image sensor may generate an image of a resolution corresponding to the half of the total number of photodetectors. For example, the 2PD image sensor including <NUM> photodetectors may generate image data of a <NUM> resolution.

Accordingly, an aspect of the disclosure is to provide an electronic device, which include a phase difference detecting pixel capable of generating image data of a resolution corresponding to the total number of photodetectors (sub-pixels) as set out in independent claim <NUM>.

According to embodiments of the disclosure, it may be possible to generate image data of a resolution corresponding to the overall number of photodetectors (or sub-pixels). Besides, a variety of effects directly or indirectly understood through this disclosure may be provided.

<FIG> illustrates an electronic device capable of adjusting a focus according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> according to an embodiment may generate a preview image based on image data obtained by using a camera module <NUM> and may output the generated preview image to a display <NUM>. The camera module <NUM> may include, for example, a lens assembly including one or more lenses and an image sensor. Each of pixels of the image sensor may include a plurality of sub-pixels. Some sub-pixels of the plurality of sub-pixels may be sub-pixels sharing a micro lens, and the remaining sub-pixels may be sub-pixels not sharing the micro lens.

In screen <NUM>, the electronic device <NUM> may generate phase difference data (or depth data) by a path difference of a light passing through the micro lens of sub-pixels that share the micro lens.

In screen <NUM>, in operation AF, the electronic device <NUM> may adjust a focus on an external object by moving the one or more lenses included in the camera module <NUM> based on the generated phase difference data. As such, the electronic device <NUM> may display a preview image focusing on the external object to the display <NUM>.

An example is illustrated in <FIG> as the camera module <NUM> is a front camera of the electronic device <NUM>. However, the disclosure is not limited thereto. For example, the camera module <NUM> may be disposed on at least one of a back surface or a side surface of the electronic device <NUM>.

<FIG> illustrates a configuration diagram of an image sensor including one pixel according to an embodiment of the disclosure.

Referring to <FIG>, according to an embodiment, an image sensor <NUM> may include at least one pixel <NUM> and a control circuit <NUM>. For convenience of description, an example is illustrated in <FIG> as the image sensor <NUM> includes one pixel. However, the disclosure is not limited thereto. For example, the image sensor <NUM> may include a plurality of pixels that are formed to have a specified channel pattern.

According to an embodiment, the at least one pixel <NUM> may include a first micro lens ML1, a second micro lens ML2, a third micro lens ML3, a color filter CF1, a first sub-pixel PD1, a second sub-pixel PD2, a third sub-pixel PD3, and a fourth sub-pixel PD4.

The first micro lens ML1 may cover the first sub-pixel PD1 and may adjust a path of an incident light such that a light incident from the outside arrives at the first sub-pixel PD1. The second micro lens ML2 may cover the second sub-pixel PD2 and may adjust a path of an incident light such that a light incident from the outside arrives at the second sub-pixel PD2. The third micro lens ML3 may cover the third sub-pixel PD3 and the fourth sub-pixel PD4 and may adjust a path of an incident light such that a light incident from the outside arrives at the third sub-pixel PD3 and the fourth sub-pixel PD4.

The color filter CF1 is disposed between the first to third micro lenses ML1, ML2, and ML3 and the first to fourth sub-pixels PD1, PD2, PD3, and PD4 and may transmit a light in a specified wavelength range (e.g., a wavelength range corresponding to a green color). The color filter CF1 may block a light being out of the specified wavelength range such that only a light, which belongs to the specified wavelength range, of the light passing through the first to third micro lenses ML1, ML2, and ML3 arrives at the first to fourth sub-pixels PD1, PD2, PD3, and PD4.

The first sub-pixel PD1 may include a first photodetector (e.g., a photo diode) that is able to detect a light passing through the first micro lens ML1 and the color filter CF1. The second sub-pixel PD2 may include a second photodetector (e.g., a photo diode) that is able to detect a light passing through the second micro lens ML2 and the color filter CF1. The third sub-pixel PD3 may include a third photodetector (e.g., a photo diode) that is able to detect a light passing through the third micro lens ML3 and the color filter CF1. The fourth sub-pixel PD4 may include a fourth photodetector (e.g., a photo diode) that is able to detect a light passing through the third micro lens ML3 and the color filter CF1. Due to a characteristic of the third micro lens ML3, a path difference may occur between lights incident onto the third sub-pixel PD3 and the fourth sub-pixel PD4 through the third micro lens ML3. The third sub-pixel PD3 and the fourth sub-pixel PD4 may be disposed to be adjacent to each other in a transverse direction, a longitudinal direction, or a diagonal direction of a pixel, in which the third sub-pixel PD3 and the fourth sub-pixel PD4 are included, of the at least one pixel <NUM>.

The control circuit <NUM> generates phase difference data (or depth data) by using data (e.g., an incident light quantity) obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4. For example, the control circuit <NUM> may verify a difference between data obtained from the third sub-pixel PD3 and data obtained from the fourth sub-pixel PD4 and may generate phase difference data corresponding to the verified difference.

The control circuit <NUM> may process the pieces of data respectively obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4 and may generate pixel data on the at least one pixel <NUM>. A scheme in which the control circuit <NUM> processes obtained data may vary depending on an output mode of the at least one pixel <NUM> (or the image sensor <NUM>). According to an embodiment, in a first output mode, the control circuit <NUM> may generate the pixel data on the at least one pixel <NUM>, by performing binning (e.g., adding or averaging) on the data obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4. In this case, one pixel data (e.g., pixel data of a pixel unit) may be generated with respect to one pixel (e.g., <NUM>). According to the invention, in a second output mode, the control circuit <NUM> generates four pixel data (e.g., pixel data of a sub-pixel unit) on the at least one pixel <NUM> by using the data obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4.

According to various embodiments, the data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 may have a phase difference due to the third micro lens ML3 and thus may be different from actual image data. To solve an error of image data due to the difference between the obtained data and the actual image data, the control circuit <NUM> may generate phase difference data based on the data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 and may restore the data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 based on the data obtained from the first and second sub-pixels PD1 and PD2.

According to the invention, the control circuit <NUM> restores data on the third sub-pixel PD3 and data on the fourth sub-pixel PD4 by summing the data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 and dividing the summed data by a ratio of the data obtained from the first sub-pixel PD1 to the data obtained from the second sub-pixel PD2. For another example, the data obtained from the first sub-pixel PD1 and the data obtained from the second sub-pixel PD2 may be <NUM> and <NUM>, respectively, and the data obtained from the third sub-pixel PD3 and the data obtained from the fourth sub-pixel PD4 may be <NUM> and <NUM>, respectively. In this case, the control circuit <NUM> may generate restored pixel data of <NUM> (= <NUM>*(<NUM>/<NUM>)) with regard to the third sub-pixel PD3 and restored pixel data of <NUM> (= <NUM>*(<NUM>/<NUM>)) with regard to the fourth sub-pixel PD4, by dividing "<NUM>" being a result of summing the data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 by "<NUM>:<NUM>" being a ratio of the data obtained from the first sub-pixel PD1 to the data obtained from the second sub-pixel PD2.

The control circuit <NUM> may transmit the generated phase difference data and the pixel data to an external processor. In the second output mode, the control circuit <NUM> may rearrange the pixel data and may transmit the rearranged pixel data to the external processor.

According to various embodiments, the at least one pixel <NUM> may include sub-pixels, the number of which exceeds "<NUM>". In this case, the first sub-pixel PD1 and the second sub-pixel PD2 that are used for restoration of the third sub-pixel PD3 and the fourth sub-pixel PD4 may be sub-pixels the closest to the third sub-pixel PD3 and the fourth sub-pixel PD4. For example, the at least one pixel <NUM> may include a <NUM> X <NUM> array of sub-pixels (a total of <NUM> sub-pixels), and the third sub-pixel PD3 and the fourth sub-pixel PD4 may be sub-pixels at positions (<NUM>, <NUM>) and (<NUM>, <NUM>) of the <NUM> X <NUM> array. In this case, the first sub-pixel PD1 and the second sub-pixel PD2 may be sub-pixels at positions (<NUM>, <NUM>) and (<NUM>, <NUM>).

According to various embodiments, the at least one pixel <NUM> may include a plurality of pixels. In this case, a plurality of sub-pixels included in each pixel (e.g., <NUM>) may be arranged to be contiguous to each other in at least one of a transverse direction, a longitudinal direction, or a diagonal direction of each pixel <NUM>. Also, the plurality of pixels may be arranged along a specified channel pattern.

According the above embodiment, the image sensor <NUM> may detect phase difference data by using the third sub-pixel PD3 and the fourth sub-pixel PD4 for phase difference detection and may also generate pixel data on the third sub-pixel PD3 and pixel data on the fourth sub-pixel PD4.

<FIG> illustrate array structures of sub-pixels included in at least one pixel according to various embodiments of the disclosure. For convenience of description, the color filter CF1 covering the first to fourth sub-pixels PD1, PD2, PD3, and PD4 is omitted, and the first to fourth sub-pixels PD1, PD2, PD3, and PD4 and the first to third micro lenses ML1, ML2, and ML3 are illustrated in <FIG>.

Referring to <FIG>, the first to fourth unit sub-pixels PD1, PD2, PD3, and PD4 included in the at least one pixel <NUM> may be arranged in the shape of a rectangle. In this case, the third sub-pixel PD3 and the fourth sub-pixel PD4 may be disposed to be contiguous to each other in a transverse direction as illustrated in <FIG>. Referring to <FIG>, the third sub-pixel PD3 and the fourth sub-pixel PD4 may be disposed to be contiguous to each other in a longitudinal direction. Referring to <FIG>, the third sub-pixel PD3 and the fourth sub-pixel PD4 may be disposed to be contiguous to each other in a diagonal direction.

According to an embodiment, an image sensor (e.g., the image sensor <NUM> of <FIG>) may include at least one pixel that includes a first sub-pixel (e.g., the first sub-pixel PD1 of <FIG>), a second sub-pixel (e.g., the second sub-pixel PD2 of <FIG>), a third sub-pixel (e.g., the third sub-pixel PD3 of <FIG>), and a fourth sub-pixel (e.g., the fourth sub-pixel PD4 of <FIG>) covered by a color filter (e.g., the color filter CF1 of <FIG>) and includes a first micro lens (e.g., the first micro lens ML1 of <FIG>) covering the first sub-pixel, a second micro lens (e.g., the second micro lens ML2 of <FIG>) covering the second sub-pixel, and a third micro lens (e.g., the third micro lens ML3 of <FIG>) covering the third sub-pixel and the fourth sub-pixel, and a control circuit (e.g., the control circuit <NUM> of <FIG>) that is electrically connected with the at least one pixel. The control circuit may obtain a light of an external object by using the at least one pixel, may generate depth data associated with the external object, by using data obtained from the third sub-pixel and the fourth sub-pixel through the third micro lens, may generate pixel data on the at least one pixel by processing data obtained from the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel with regard to the light, and may transmit the pixel data and the depth data to an external processor.

The color filter may be formed to transmit a light in a wavelength range corresponding to a green light.

The third sub-pixel and the fourth sub-pixel may be disposed to be contiguous to each other in a transverse direction in the at least one pixel.

The third sub-pixel and the fourth sub-pixel may be disposed to be contiguous to each other in a longitudinal direction in the at least one pixel.

The third sub-pixel and the fourth sub-pixel may be disposed to be contiguous to each other in a diagonal direction in the at least one pixel.

The at least one pixel may include a plurality of pixels, and the third sub-pixel and the fourth sub-pixel included in each of the plurality of pixels may be disposed to be contiguous to each other in a plurality of directions of a transverse direction of each of the pixels, a longitudinal direction of each of the pixels, or a diagonal direction of each of the pixels.

The control circuit may restore third data on the third sub-pixel and fourth data on the fourth sub-pixel, based at least on first data obtained from the first sub-pixel and second data obtained from the second sub-pixel and may generate pixel data on the at least one pixel, including the first data, the second data, the third data, and the fourth data.

The control circuit may perform binning on the data respectively obtained from the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel to generate pixel data on the at least one pixel.

<FIG> illustrates an image sensor including a plurality of pixels according to an embodiment of the disclosure. For convenience of description, a <NUM> X <NUM> pixel array of a plurality of pixels is illustrated in <FIG>.

Referring to <FIG>, an image sensor <NUM> (e.g., the image sensor <NUM> of <FIG>) may include a plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> and a control circuit <NUM> (e.g., the control circuit <NUM> of <FIG>). In an embodiment, the image sensor <NUM> may not include some of the above components or may further include any other components. For example, the image sensor <NUM> may further include a memory (not illustrated). In an embodiment, some of the components of the image sensor <NUM> may be combined to form one entity, which may identically perform functions of the some components before the combination.

The plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> may be arranged along a specified channel pattern. The specified channel pattern may be, for example, a Bayer channel pattern in which "R" image data and "G" image data appear alternately at an odd-numbered row line and "G" image data and "B" image data appear alternately at an even-numbered row line. Each pixel (e.g., the pixel <NUM> or the pixel <NUM> of <FIG>) may include four sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) (e.g., the first to fourth sub-pixels PD1, PD2, PD3, and PD4 of <FIG>), four or three micro lenses ML1, ML2, and ML3 (e.g., the first to third micro lenses ML1, ML2, and ML3 of <FIG>) covering the four sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4), and a color filter CF1, CF2, CF3, or CF4 (e.g., the color filter CF1, CF2, CF3, or CF4 of <FIG>) disposed between the four sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) and micro lenses (e.g., the micro lenses ML4, ML5, ML6, and ML7). The color filter CF1, CF2, CF3, or CF4 covering the four sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) may be an "R" channel color filter CF1 formed to transmit a red light, or may be a "G" channel color filter CF2 or CF3 formed to transmit a green light or a "B" color filter CF4 formed to transmit a blue light. However, the color filter CF1, CF2, CF3, or CF4 included in each pixel (e.g., the pixel <NUM>) may be a color filter complying with the specified channel pattern.

The pixels <NUM> and <NUM> (or at least some pixels) corresponding to the "G" channel from among the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> may include three micro lenses ML1, ML2, and ML3, the number of which is less than the number of four sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) and may include sub-pixels (hereinafter referred to as "phase difference sub-pixels") sharing one micro lens (e.g., the micro lens ML3) and sub-pixels (hereinafter referred to as "adjacent sub-pixels") not sharing a micro lens ML1, ML2, or ML3 contiguous to the phase difference sub-pixels (e.g., the sub-pixels PD3 and PD4). The phase difference sub-pixels (e.g., sub-pixels PD3 and PD4) may be disposed to be contiguous to each other in at least one of a transverse direction, a longitudinal direction, or a diagonal direction of each pixel <NUM>.

The pixel <NUM> corresponding to the "R" channel from among the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> may include four micro lenses (e.g., the micro lenses ML4, ML5, ML6, and ML7) respectively corresponding to sub-pixels (e.g., the sub-pixels PD5, PD6, PD7, and PD8). Also, the pixel <NUM> corresponding to the "B" channel may include four micro lenses respectively corresponding to four sub-pixels.

The control circuit <NUM> may include a timing controller <NUM>, a row selector <NUM>, a column selector <NUM>, a readout circuit <NUM>, an analog-to-digital converter (ADC) <NUM>, an red-green-blue (RGB) converter <NUM>, and an output buffer <NUM>. The timing controller <NUM> may generate a control signal for controlling an operation of at least one of the row selector <NUM>, the column selector <NUM>, the readout circuit <NUM>, the ADC <NUM>, the RGB converter <NUM>, and the output buffer <NUM>. The row selector <NUM> may selectively activate one of row lines of the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> depending on the control signal of the timing controller <NUM>. The column selector <NUM> may selectively activate one of column lines of the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> depending on the control signal of the timing controller <NUM>. The readout circuit <NUM> may read out (or obtain) data from selected pixels of the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> depending on the control signal of the timing controller <NUM>. The ADC <NUM> may convert analog pixel data obtained from the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> into digital pixel data. The RGB converter <NUM> may rearrange the digital pixel data, which does not correspond to a specified channel pattern, to correspond to the specified channel pattern and may generate image data having the specified channel pattern as a result of the rearrangement. The output buffer <NUM> may buffer the image data corresponding to the specified channel pattern, for example, in units of a frame. The control circuit <NUM> described below indicates the timing controller <NUM> and each component controlled by the timing controller <NUM>.

The control circuit <NUM> may generate phase difference data associated with an external object by using data (e.g., an incident light quantity) that is obtained from phase difference sub-pixels (e.g., the sub-pixels PD3 and PD4). For example, the control circuit <NUM> may verify (e.g., calculate) a difference between pieces of data respectively obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4, which share the third micro lens ML3, and may generate phase difference data based on the verified difference. The control circuit <NUM> may generate phase difference data corresponding to all phase difference sub-pixels included in the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM>.

The control circuit <NUM> may generate phase difference data based on data obtained from sub-pixels included in each pixel (e.g., the pixel <NUM>) and may process the obtained data to generate pixel data for each pixel. The control circuit <NUM> may differently process data obtained from each pixel (e.g., the pixel <NUM>) depending on an output mode and may generate pixel data according to the output mode.

According to an embodiment, the control circuit <NUM> may perform binning (e.g., summing or averaging) on data obtained from sub-pixels (e.g., the sub-pixels PD1, PD2, PD3, and PD4) included in each pixel (e.g., the pixel <NUM>) in the first output mode (e.g., the binning mode) and may generate one pixel data for each pixel (e.g., the pixel <NUM>). According to an embodiment, the control circuit <NUM> may generate pixel data for each pixel (e.g., the pixel <NUM>), based on data obtained from sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) included in each pixel (e.g., the pixel <NUM>) in the second output mode (e.g., the rearrangement mode). As data obtained from the phase difference sub-pixels (e.g., the sub-pixels PD3 and PD4) have a phase difference due to one micro lens (e.g., the micro lens ML3) covering the phase difference sub-pixels, a difference may exist between the obtained data and actual image data. To solve an error of image data due to the difference, the control circuit <NUM> may restore data on the phase difference sub-pixels based on the data obtained from adjacent sub-pixels (e.g., the sub-pixels PD1 and PD2) contiguous to the phase difference sub-pixels (e.g., the sub-pixels PD3 and PD4) in the second output mode. For example, the control circuit <NUM> may sum the data respectively obtained from the phase difference sub-pixels PD3 and PD4 to generate summed data and may verify a ratio between the adjacent sub-pixels PD1 and PD2. The control circuit <NUM> may restore data on the phase difference sub-pixels PD3 and PD4 by dividing the summed data to correspond to the verified ratio. In the above embodiment, because data obtained from phase difference sub-pixels are restored based on data obtained from adjacent sub-pixels corresponding to a color channel in which the phase difference sub-pixels are included, the control circuit <NUM> may restore pixel data on the phase difference sub-pixels to be similar to actual image data. According to various embodiments, the control circuit <NUM> may restore data of phase difference sub-pixels only when outputting image data exceeding a specified resolution. The specified resolution may be a resolution requiring pixel data of a phase difference sub-pixel when the image data are generated. For example, in the case where the image sensor <NUM> is composed of <NUM> pixels each including sub-pixels, the specified resolution may be <NUM>.

The control circuit <NUM> may combine the generated pixel data to generate image data having a specified channel pattern. In the second output mode, the control circuit <NUM> may rearrange pixel data of a sub-pixel unit depending on a specified channel pattern (re-mosaic) and may generate image data having the specified channel pattern. The control circuit <NUM> may generate image data having the specified channel pattern by exchanging left and right pixel data, top and bottom pixel data, or diagonal pixel data in rearranging pixel data of a sub-pixel unit. The control circuit <NUM> may transmit the generated phase difference data and the pixel data (including pixel data) to an external processor.

According to various embodiments, the image sensor <NUM> may generate image data of some pixels of the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> and may transmit the generated image data to the external processor. For example, in the case where information about some pixels corresponding to a zoom region from among the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> is received from the external processor due to adjustment of a zoom ratio, the image sensor <NUM> may generate pixel data based on data obtained from the some pixels and may transmit the image data including the pixel data to the external processor.

According to various embodiments, the image sensor <NUM> may include phase difference detection pixels in a pixel corresponding to the "R" channel and a pixel corresponding to the "B" channel. In this case, phase difference data may be denser.

According to the above embodiment, the image sensor <NUM> may generate image data of a resolution corresponding to the overall number of sub-pixels (e.g., photodetectors). Also, the image sensor <NUM> may restore data on phase difference sub-pixels by using data obtained from sub-pixels adjacent to the phase difference sub-pixels, thus preventing a decrease in the quality of image due to the phase difference sub-pixels.

According to an embodiment, an image sensor (e.g., the image sensor <NUM> of <FIG>) may include a plurality of pixels (e.g., the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>) arranged depending on a specified channel pattern, wherein each of the plurality of pixels includes one color filter (e.g., the color filter CF1, CF2, CF3, or CF4 of <FIG>), and a first sub-pixel (e.g., the first sub-pixel PD1 or PD5 of <FIG>), a second sub-pixel (e.g., the second sub-pixel PD2 or PD6 of <FIG>), a third sub-pixel (e.g., the third sub-pixel PD3 or PD7 of <FIG>, and a fourth sub-pixel (e.g., the fourth sub-pixel PD4 or PD8 of <FIG>) covered by the color filter, and a control circuit (e.g., the control circuit <NUM> of <FIG>) that controls the plurality of pixels. Each of at least some pixels (e.g., at least some pixels <NUM> of <FIG>) of the plurality of pixels may further include a first micro lens (e.g., the first micro lens ML1 of <FIG>) covering the first sub-pixel (e.g., the first sub-pixel PD1 of <FIG>), a second micro lens (e.g., the second micro lens ML2 of <FIG>) covering the second sub-pixel (e.g., the second sub-pixel PD2 of <FIG>), and a third micro lens (e.g., the third micro lens ML3 of <FIG>) covering the third sub-pixel (e.g., the third sub-pixel PD3 of <FIG>) and the fourth sub-pixel (e.g., the fourth sub-pixel PD4 of <FIG>). The control circuit may obtain a light of an external object by using the plurality of pixels, may generate depth data associated with the external object, by using data obtained from the third sub-pixel and the fourth sub-pixel included in each of the at least some pixels, may generate pixel data on each of the pixels by using data obtained from the first to fourth sub-pixels included in each of the plurality of pixels, and may transmit the pixel data and the depth data to an external processor.

The color filter included in the at least some pixels may be formed to transmit a light in a wavelength range corresponding to a green light.

The third sub-pixel and the fourth sub-pixel may be disposed to be contiguous to each other in at least one of a transverse direction of each of the at least some pixels, a longitudinal direction of each of the at least some pixels, or a diagonal direction of each of the at least some pixels.

The control circuit may restore data on the third sub-pixel and data on the fourth sub-pixel, based on data obtained from the first sub-pixel and data obtained from the second sub-pixel and may generate pixel data on each of the at least some pixels based on the restored data and the data obtained from the first sub-pixel and the second sub-pixel.

The control circuit may perform binning on the data respectively obtained from the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel to generate pixel data on each of the at least some pixels.

<FIG> illustrates a configuration diagram of an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may include a camera module <NUM> (e.g., the camera module <NUM> of <FIG>), a display <NUM> (e.g., the display <NUM> of <FIG>), and a processor <NUM>. In an embodiment, the electronic device <NUM> may not include some of the above components or may further include any other components. For example, the electronic device <NUM> may further include a memory <NUM>. In an embodiment, some of the components of the electronic device <NUM> may be combined to form one entity, which may identically perform functions of the some components before the combination.

According to an embodiment, the camera module <NUM> may include a lens assembly <NUM> including one or more lenses and an image sensor <NUM>.

The lens assembly <NUM> may be disposed on/over the image sensor <NUM>, and may collect a light output from or reflected by an external object to be transferred to pixels of the image sensor <NUM>.

The image sensor <NUM> (e.g., the image sensor <NUM> of <FIG>) may be disposed under the lens assembly <NUM> and may generate image data on the external object based on a light passing through the one or more lenses included in the lens assembly <NUM>. The image sensor <NUM> may include a plurality of pixels (e.g., the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>) and a control circuit (e.g., the control circuit <NUM> of <FIG>) for controlling the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM>. The configurations of the plurality of pixels <NUM>, <NUM>, <NUM>, and <NUM> and the control circuit <NUM> are described with reference to <FIG>, and thus, additional description will be omitted to avoid redundancy in <FIG>.

The display <NUM> may display, for example, various kinds of content (e.g., a text, an image, a video, an icon, and/or a symbol). The display <NUM> may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, or an electronic paper display. For example, the display <NUM> may be a touchscreen display.

For example, the memory <NUM> may store instructions or data associated with at least one other component(s) of the electronic device <NUM>. The memory <NUM> may be a volatile memory (e.g., a random access memory (RAM) or the like), a nonvolatile memory (e.g., a read only memory (ROM), a flash memory, or the like), or a combination thereof. For example, the memory <NUM> may store a lookup table including phase difference data corresponding to differences between pieces of data obtained from phase difference sub-pixels (e.g., sub-pixels PD3 and PD4).

The processor <NUM> may perform data processing or an operation associated with a control and/or a communication of at least one other component(s) of the electronic device <NUM> by using the instructions stored in the memory <NUM>. The processor <NUM> may include at least one of a graphic processing unit (GPU), an application processor, or an image signal processor.

According to an embodiment, the processor <NUM> may generate pixel data on each pixel (e.g., the pixel <NUM>) by using the image sensor <NUM> and may generate image data based on the pixel data. For example, the processor <NUM> may obtain data from sub-pixels included in each pixel (e.g., the pixel <NUM>) by using the image sensor <NUM> and may process the obtained data to generate pixel data on each pixel. The processor <NUM> may differently generate pixel data on each pixel by controlling an output mode of the image sensor <NUM>. According to an embodiment, in the first output mode (e.g., the binning mode) of the image sensor <NUM>, the processor <NUM> may perform binning (e.g., summing or averaging) on data obtained from sub-pixels (e.g., sub-pixels PD1, PD2, PD3, and PD4) included in each pixel (e.g., the pixel <NUM>) and may generate one pixel data for each pixel (e.g., the pixel <NUM>). According to an embodiment, in the second output mode (e.g., the rearrangement mode) of the image sensor <NUM>, the processor <NUM> may generate pixel data of a sub-pixel unit based on data from sub-pixels included in each pixel. In this case, the image sensor <NUM> may generate four pixel data for each pixel. In the second output mode, the image sensor <NUM> may restore data on phase difference sub-pixels based on data obtained from adjacent sub-pixels contiguous to the phase difference sub-pixels. For example, by using the image sensor <NUM>, the processor <NUM> may sum data respectively obtained from the phase difference sub-pixels PD3 and PD4 to generate summed data and may verify a ratio between data obtained from the adjacent sub-pixels PD1 and PD2. By using the image sensor <NUM>, the processor <NUM> may restore data on the phase difference sub-pixels PD3 and PD4 by dividing the summed data to correspond to the verified ratio. According to various embodiments, the image sensor <NUM> may restore data on phase difference sub-pixels only in the case of outputting image data exceeding a specified resolution. The specified resolution may be a resolution requiring pixel data of a phase difference sub-pixel when image data are generated. For example, in the case where the image sensor <NUM> is composed of <NUM> pixels each including sub-pixels, the specified resolution may be <NUM>.

The processor <NUM> may generate image data having a specified channel pattern by combining the generated pixel data by using the image sensor <NUM>. In the second output mode, by using the image sensor <NUM>, the processor <NUM> may rearrange pixel data of a sub-pixel unit depending on a specified channel pattern (re-mosaic) and may generate image data having the specified channel pattern. For example, the image sensor <NUM> may generate image data having the specified channel pattern by exchanging left and right pixel data, top and bottom pixel data, or diagonal pixel data in rearranging pixel data of a sub-pixel unit depending on a command of the processor <NUM>. For another example, in the second output mode, the processor <NUM> may receive pixel data of a sub-pixel unit from the image sensor <NUM>, may rearrange the received pixel data of the sub-pixel unit depending on a specified channel pattern (re-mosaic), and may generate image data having the specified channel pattern.

The processor <NUM> may display an image generated based on the image data through the display <NUM>. For example, the processor <NUM> may perform color interpolation on image data such that each pixel of the image data includes "R" information, "G" information, and "B" information and may convert the color-interpolated image data so as to correspond to a specified format (e.g., a YUV format). The processor <NUM> may generate an image based on the converted image data and may display the generated image through the display <NUM>. Additionally or alternatively, the processor <NUM> may store the generated image in the memory <NUM>. For example, the processor <NUM> may store the generated image in the memory <NUM> in response to an input associated with image photographing.

By using the image sensor <NUM>, the processor <NUM> may obtain phase difference data associated with an external object based on data obtained from phase difference sub-pixels. The processor <NUM> may adjust a focus on the external object by moving the lens assembly <NUM> or the one or more lenses included in the lens assembly <NUM> based on the phase difference data associated with the external object. For example, the processor <NUM> may verify a position of the external object to be focused (e.g., a face position of an image in which a character is included), from the image data. When the position of the external object is verified, the processor <NUM> may determine phase difference sub-pixels, which will be used to adjust a focus, from among phase difference sub-pixels included in the image sensor <NUM>. In this regard, the processor <NUM> may select phase difference sub-pixels, which are at a position of the external object or are the closest to the position of the external object and will be used to adjust a focus, from among phase difference sub-pixels included in the image sensor <NUM>. The processor <NUM> may determine phase difference data on the external object by using the determined phase difference sub-pixels and may adjust a focus on the external object by moving the one or more lenses included in the lens assembly <NUM> based on the determined phase difference data. Additionally or alternatively, the processor <NUM> may determine distance information of an area corresponding to the determined phase difference sub-pixels based on the determined phase difference data.

According to various embodiments, each pixel (e.g., the pixel <NUM>) may include sub-pixels, the number of which exceeds "<NUM>". In the case where the number of sub-pixels changes, the number of micro lenses ML1, ML2, and ML3 may be adjusted to correspond to the number of sub-pixels. However, in the specification, for convenience of description, the case where each pixel (e.g., the pixel <NUM>) includes four sub-pixels is described as an example.

According to the above embodiment, the electronic device <NUM> may restore data obtained from phase difference sub-pixels based on data obtained from adjacent sub-pixels belonging to a channel in which phase difference sub-pixels included in the image sensor <NUM> are included, thus preventing a decrease in the quality of image due to the phase difference sub-pixels.

According to an embodiment, an electronic device (e.g., the electronic device <NUM> of <FIG>) may include a camera module (e.g., the camera module <NUM> of <FIG>), wherein the camera module includes a lens assembly (e.g., the lens assembly <NUM> of <FIG>) including one or more lenses and an image sensor (e.g., the image sensor <NUM> of <FIG>) including at least one pixel (e.g., the pixels <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>), a display (e.g., the display <NUM> of <FIG>), and a processor (e.g., the processor <NUM> of <FIG>) that is electrically connected with the display and the camera module. The at least one pixel may include a first sub-pixel (e.g., sub-pixel PD1 of <FIG>), a second sub-pixel (e.g., sub-pixel PD2 of <FIG>), a third sub-pixel (e.g., sub-pixel PD3 of <FIG>), and a fourth sub-pixel (e.g., sub-pixel PD4 of <FIG>) covered by one color filter (e.g., the color filter CF1 of <FIG>) and may include a first micro lens covering the first sub-pixel, a second micro lens covering the second sub-pixel, and a third micro lens covering the third sub-pixel and the fourth sub-pixel. The processor may obtain a light of an external object passing through the one or more lenses by using the at least one pixel, may generate depth data (phase difference data) associated with the external object, based on data obtained from the third sub-pixel and the fourth sub-pixel through the third micro lens, by using the image sensor, may generate pixel data on the at least one pixel based on the data obtained from the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel, by using the image sensor, may display an image generated based on the pixel data through the display, and may adjust a focus on the external object by moving the one or more lenses based on the depth data.

The processor may restore third data on the third sub-pixel and fourth data on the fourth sub-pixel, based at least on first data obtained from the first sub-pixel and second data obtained from the second sub-pixel, by using the image sensor, and may generate pixel data on the at least one pixel, including the first data, the second data, the third data, and the fourth data, by using the image sensor.

The processor may perform binning on the data respectively obtained from the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel to generate pixel data on the at least one pixel, by using the image sensor.

<FIG> illustrates a binning and rearrangement of pixel data according to an embodiment, <FIG> illustrates an example of data restoration of phase difference sub-pixels according to an embodiment, and <FIG> illustrates another example of data restoration of phase difference sub-pixels according to an embodiment.

Referring to <FIG>, an image sensor (e.g., the image sensor <NUM> of <FIG>) may include a plurality of pixels <NUM> to <NUM> arranged in a <NUM> X <NUM> array depending on a specified channel pattern. Each of the pixels <NUM> to <NUM> may include <NUM> X <NUM> sub-pixels.

According to an embodiment, in the first output node, the image sensor <NUM> may generate one pixel data <NUM> for each pixel by performing binning on data obtained sub-pixels included in each pixel (e.g., a pixel corresponding to the "R" channel, the "G" channel, or the "B" channel) in units of a pixel. For example, the image sensor <NUM> may generate one "R" pixel data <NUM> by performing binning on data obtained from four sub-pixels corresponding to an "R" channel <NUM> and may generate one "G" pixel data <NUM> by performing binning on data obtained from four sub-pixels corresponding to a "G" channel <NUM>. In this manner, the image sensor <NUM> may generate pixel data <NUM> to <NUM> by performing binning on data obtained from four sub-pixels included in each of the remaining pixels <NUM> to <NUM>. Because the plurality of pixels <NUM> to <NUM> are arranged depending on the specified channel pattern, the image sensor <NUM> may generate image data <NUM> having the specified channel pattern by sequentially arranging the pixel data <NUM> to <NUM> to be combined into a single frame.

In the second output mode, the image sensor <NUM> may generate pixel data <NUM> (e.g., pixel data corresponding to the number of sub-pixels) of a sub-pixel unit based on data obtained from respective sub-pixels. For example, the image sensor <NUM> may generate the image data <NUM> by performing analog-to-digital conversion on data obtained from respective sub-pixel, rearranging the converted data to correspond to the specified channel pattern to generate pixel data corresponding to the number of sub-pixels, and combining the generating pixel data into a single frame. For example, the image sensor <NUM> may process image data to have the specified channel pattern, by exchanging positions of pixel data of sub-pixels or using a pixel value (e.g., averaging) of pixel data on the same channel close thereto.

Before rearranging the converted data to correspond to the specified channel pattern, the image sensor <NUM> may restore data on phase difference sub-pixels (e.g., sub-pixels PD3 and PD4) based on data obtained from adjacent sub-pixels (e.g., sub-pixels PD1 and PD2) placed adjacent to the phase difference sub-pixels (e.g., sub-pixels PD3 and PD4) and may generate the image data <NUM> based on the restored data.

Referring to <FIG>, the image sensor <NUM> may sum data obtained from phase difference sub-pixels G3 and G4 corresponding to a first channel <NUM> to generate summed data and may verify a ratio between adjacent sub-pixels G1 and G2 corresponding to the first channel <NUM>. The image sensor <NUM> may restore data on the phase difference sub-pixels G3 and G4 by dividing the summed data to correspond to the verified ratio.

Referring to <FIG>, the image sensor <NUM> may use data on adjacent sub-pixels 561a and 561b of phase difference sub-pixels 561c and 561d corresponding to a first channel <NUM> and data on adjacent sub-pixels 562a, 562b, 563a, 563b, 564a, and 564b corresponding to peripheral channels <NUM>, <NUM>, and <NUM> of the same color as the first channel <NUM> for the purpose of restoring data on phase difference sub-pixels 561c and 561d corresponding to a first channel <NUM>. For example, the image sensor <NUM> may restore data on the phase difference sub-pixels 561c and 561d corresponding to the first channel <NUM> by using a ratio between first summing data obtained from first sub-pixels 561a, 562a, 563a, and 564a corresponding to the first channel <NUM> and the peripheral channels <NUM>, <NUM>, and <NUM> and second summing data of summing data obtained from second sub-pixels 561b, 562b, 563b, and 564b corresponding to the first channel <NUM> and the peripheral channels <NUM>, <NUM>, and <NUM>. For another example, the image sensor <NUM> may apply a weight to each of pieces of data obtained from the first sub-pixels 561a, 562a, 563a, and 564a and the second sub-pixels 561b, 562b, 563b, and 564b and may restore data on phase difference sub-pixels 561c and 561d corresponding to the first channel <NUM> based on a ratio between summing data of the weight-applied data of the first sub-pixels 561a, 562a, 563a, and 564a and summing data of the weight-applied data of the second sub-pixels 561b, 562b, 563b, and 564b. A magnitude of a weight to be applied may be greater based on a distance between the phase difference sub-pixels 561c and 561d to be restored and the first sub-pixels 561a, 562a, 563a, and 564a or the second sub-pixels 561b, 562b, 563b, and 564b.

<FIG> is a diagram illustrating sub-pixels of an image sensor that sense a light corresponding to first and second external objects according to an embodiment of the disclosure.

Below, a method of adjusting a focus based on phase difference data will be described with reference to <FIG> is a diagram illustrating sub-pixels of an image sensor (e.g., the image sensor <NUM> of <FIG>), at which a light corresponding to first and second external objects is sensed. In <FIG>, each quadrangle (e.g., 1PD) may indicate a non-phase difference sub-pixel, and a hatched rectangle (e.g., 2PD) may illustrate a phase difference sub-pixel.

Referring to <FIG>, based on data obtained from at least some phase difference sub-pixels of phase difference sub-pixels included in the image sensor <NUM>, the processor <NUM> may generate phase difference data associated with a partial area (or pixels) <NUM> (hereinafter referred to as a "first external object area <NUM>") of the image sensor <NUM> corresponding to a first external object and a partial area (or pixels) <NUM> (hereinafter referred to as a "second external object area <NUM>") of the image sensor <NUM> corresponding to a second external object. For example, the processor <NUM> may obtain phase difference data on the first external object area <NUM> based on data obtained from phase difference sub-pixels <NUM> and <NUM> at which a light corresponding to the first external object area <NUM> is sensed and may move the one or more lenses included in the lens assembly <NUM> to a position focusing on the first external object area <NUM> based on the obtained phase difference data. In this regard, the processor <NUM> may determine phase difference data on the first external object area <NUM> by using data obtained at least from the phase difference sub-pixel <NUM> and the phase difference sub-pixel <NUM> and may adjust a focus on the first external object area <NUM> (or a first external object) by moving the one or more lenses included in the lens assembly <NUM> by using the phase difference data on the first external object area <NUM>. The processor <NUM> may determine distance information of the first external object area <NUM> (or the first external object) by using the phase difference data on the first external object area <NUM>. For another example, the processor <NUM> may determine phase difference data on the second external object area <NUM> by using data obtained at least from the phase difference sub-pixels <NUM> and <NUM> corresponding to the second external object area <NUM>. The processor <NUM> may determine distance information of the second external object area <NUM> (or the second external object) by using the phase difference data on the second external object area <NUM> thus determined.

<FIG> illustrates a flowchart of a method for operating an image sensor according to an embodiment of the disclosure.

Referring to <FIG>, in operation <NUM>, a control circuit (e.g., the control circuit <NUM> of <FIG>) may obtain a light associated with an external object by using at least one pixel (e.g., the at least one pixel <NUM> of <FIG>). For example, the control circuit <NUM> may activate the at least one pixel <NUM> and may sequentially read out charges accumulated at sub-pixels included in the at least one pixel <NUM>.

In operation <NUM>, the control circuit <NUM> may generate depth data associated with an external object by using data obtained from the third sub-pixel (e.g., the third sub-pixel PD3 of <FIG>) and the fourth sub-pixel (e.g., the fourth sub-pixel PD4 of <FIG>) through a third micro lens (e.g., the third micro lens ML3 of <FIG>). For example, the control circuit <NUM> may verify (e.g., calculate) a difference between data (e.g., an incident light quantity) obtained from the third sub-pixel PD3 and data (e.g., an incident light quantity) obtained from the fourth sub-pixel PD4 and may generate depth data of an external object corresponding to the third sub-pixel PD3 and the fourth sub-pixel PD4 based on the verified difference.

In operation <NUM>, the control circuit <NUM> may process pieces of data respectively obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4 with regard to a light and may generate pixel data on the at least one pixel <NUM>. For example, in the first output mode, the control circuit <NUM> may generate pixel data on the at least one pixel <NUM>, by binning the data obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4 and performing analog-to-digital conversion on the binned data. For another example, in the second output mode, the control circuit <NUM> may generate pixel data of a sub-pixel unit by performing analog-to-digital conversion on the data respectively obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4.

In operation <NUM>, the control circuit <NUM> may output the generated pixel data and the generated depth data to an external processor (e.g., the processor <NUM> of <FIG>). In the case of the pixel data of the sub-pixel unit, the control circuit <NUM> may rearrange the pixel data and may transmit the rearranged pixel data to the external processor <NUM>.

<FIG> illustrates a flowchart of a method for adjusting focus according to an embodiment of the disclosure.

Referring to <FIG>, in operation <NUM>, a processor (e.g., the processor <NUM> of <FIG>) may obtain a light for an external object passing through the lens (e.g., one or more lenses included in the lens assembly <NUM> of <FIG>) by using the at least one pixel <NUM>. For example, the processor <NUM> may control an image sensor (e.g., the image sensor <NUM> of <FIG>) to activate the at least one pixel <NUM> and may sequentially read out charges accumulated at sub-pixels included in the at least one pixel <NUM>.

In operation <NUM>, the processor <NUM> may generate depth data associated with an external object based on data obtained from the third sub-pixel PD3 and the fourth sub-pixel PD4 through the third micro lens ML3, by using the image sensor <NUM>. For example, the processor <NUM> may verify (e.g., calculate) a difference between data (e.g., an incident light quantity) obtained from the third sub-pixel PD3 and data (e.g., an incident light quantity) obtained from the fourth sub-pixel PD4 and may generate depth data (e.g., phase difference data) of an external object corresponding to the third sub-pixel PD3 and the fourth sub-pixel PD4 based on the verified difference.

In operation <NUM>, the processor <NUM> may generate pixel data on the at least one pixel based on the pieces of data respectively obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4, by using the image sensor <NUM>. For example, in the first output mode, by using the image sensor <NUM>, the processor <NUM> may generate pixel data on the at least one pixel <NUM>, by binning the data obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4 and performing analog-to-digital conversion on the binned data. For another example, in the second output mode, by using the image sensor <NUM>, the processor <NUM> may generate pixel data on a sub-pixel unit, by performing analog-to-digital conversion on the data obtained from the first sub-pixel PD1, the second sub-pixel PD2, the third sub-pixel PD3, and the fourth sub-pixel PD4.

In operation <NUM>, the processor <NUM> may display an image generated based on the pixel data through the display (e.g., the display <NUM> of <FIG>). For example, the processor <NUM> may perform color interpolation on image data such that each pixel of the image data includes "R" information, "G" information, and "B" information and may convert the color-interpolated image data to correspond to a specified format (e.g., a YUV format). The processor <NUM> may generate an image based on the converted image data and may display the generated image through a display (e.g., the display <NUM> of <FIG>).

In operation <NUM>, the processor <NUM> may adjust a focus on the external object by moving the one or more lenses based on the depth data associated with the external object. For example, the processor <NUM> may verify a position of the external object to be focused (e.g., a face position of an image in which a character is included), from the image data. When the position of the external object is verified, the processor <NUM> may determine phase difference sub-pixels, which will be used to adjust a focus on the external object, from among phase difference sub-pixels included in the image sensor <NUM>. The processor <NUM> may obtain depth data (phase difference data) on the external object by using data obtained from the determined phase difference sub-pixels and may adjust a focus on the external object by moving the one or more lenses based on the depth data.

<FIG> is a block diagram of an electronic device in a network environment according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> in a network environment <NUM> may communicate with an electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or an electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). According to an embodiment, the electronic device <NUM> may include a processor <NUM> (e.g., the processor <NUM> of <FIG>), memory <NUM> (e.g., the memory <NUM> of <FIG>), an input device <NUM>, a sound output device <NUM>, a display device <NUM> (e.g., the display <NUM> of <FIG>), an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a haptic module <NUM>, a camera module <NUM> (e.g., the image sensor <NUM> and the lens assembly <NUM> of <FIG>), a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module(SIM) <NUM>, or an antenna module <NUM>.

According to one embodiment, as at least part of the data processing or computation, the processor <NUM> may load a command or data received from another component (e.g., the sensor module <NUM> or the communication module <NUM>) in a volatile memory <NUM>, process the command or the data stored in the volatile memory <NUM>, and store resulting data in a non-volatile memory <NUM>. According to an embodiment, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor <NUM> (e.g., a GPU, an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor <NUM>.

According to an embodiment, the audio module <NUM> may obtain the sound via the input device <NUM>, or output the sound via the sound output device <NUM> or an external electronic device (e.g., an electronic device <NUM>) (e.g., speaker of headphone) directly (e.g., wiredly) or wirelessly coupled with the electronic device <NUM>.

The communication module <NUM> may include one or more communication processors that are operable independently from the processor <NUM> (e.g., the AP) and supports a direct (e.g., wired) communication or a wireless communication. A corresponding one of these communication modules may communicate with the external electronic device via the first network <NUM> (e.g., a short-range communication network, such as Bluetooth <IMG>, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network <NUM> (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)).

For example, when the electronic device <NUM> should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device <NUM>, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service.

<FIG> is a block diagram illustrating a camera module according to an embodiment of the disclosure.

Referring to <FIG>, the camera module <NUM> may include a lens assembly <NUM> (e.g., the lens assembly <NUM> of <FIG>), a flash <NUM>, an image sensor <NUM> (e.g., the image sensor <NUM> of <FIG>), an image stabilizer <NUM>, a memory <NUM> (e.g., a buffer memory), or an image signal processor <NUM> (e.g., the processor <NUM> of <FIG>). The lens assembly <NUM> may collect a light emitted from a subject targeted for image photographing. The lens assembly <NUM> may include one or more lenses. According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In this case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. Some of the plurality of lens assemblies <NUM> may have the same lens attributes (e.g., a view angle, a focal length, autofocusing, an f number, or an optical zoom), or at least one lens assembly may have one or more lens attributes different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens.

The flash <NUM> may emit light that is used to reinforce light emitted or reflected from a subject. According to an embodiment, the flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a RGB LED, a white LED, an IR LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor <NUM> may obtain an image corresponding to the subject by converting a light transmitted through the lens assembly <NUM> after being emitted or reflected from the subject, into an electrical signal. According to an embodiment, the image sensor <NUM> may include one selected from image sensors having different attributes, such as an RGB sensor, a black-and-white (BW) sensor, an IR sensor, or an UV sensor, a plurality of image sensors having the same attributes, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented by using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

In response to the movement of the camera module <NUM> or the electronic device <NUM> including the camera module <NUM>, the image stabilizer <NUM> may move the image sensor <NUM> or at least one lens included in the lens assembly <NUM> in a specific direction or may control operation attributes of the image sensor <NUM> (e.g., may adjust the readout timing). This makes it possible to compensate for at least part of a negative influence of the movement on an image being photographed. According to an embodiment, the image stabilizer <NUM> may sense such a movement of the camera module <NUM> or the electronic device <NUM> by using a gyro sensor (not illustrated) or an acceleration sensor (not illustrated) disposed inside or outside the camera module <NUM>. According to an embodiment, the image stabilizer <NUM> may be implemented, for example, with an optical image stabilizer. The memory <NUM> may at least temporarily store at least a portion of an image obtained through the image sensor <NUM> for a next image processing task. For example, when image photographing is delayed due to a shutter operation or multiple images are quickly photographed, a raw (or original) image obtained (e.g., a Bayer-patterned image or a high-resolution image) may be stored in the memory <NUM>, and a copy image (e.g., a low-resolution image) corresponding to the raw image may be previewed through the display device <NUM>. Afterwards, when a specified condition is satisfied (e.g., when a user's input or system command is received), at least a portion of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be implemented with at least a portion of the memory <NUM> or with a separate memory that operates independently of the memory <NUM>.

The image signal processor <NUM> may perform one or more image processing on an image obtained through the image sensor <NUM> or an image stored in the memory <NUM>. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor <NUM> may control at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM> (e.g., may control an exposure time or a readout timing). An image processed by the image signal processor <NUM> may be again stored in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) of the camera module <NUM>. According to an embodiment, the image signal processor <NUM> may be implemented with at least a portion of the processor <NUM>, or with a separate processor that operates independently of the processor <NUM>. In the case where the image signal processor <NUM> is implemented with a processor independent of the processor <NUM>, under control of the processor <NUM>, at least one image processed by the image signal processor <NUM> may be displayed through the display device <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include a plurality of camera modules <NUM> having different attributes or functions. In this case, at least one of the plurality of camera modules <NUM> may be, for example, a wide-angle camera, and at least another of the plurality of camera modules <NUM> may be a telephoto camera. Alternatively, at least one of the plurality of camera modules <NUM> may be, for example, a front camera, and at least another of the plurality of camera modules <NUM> may be a rear camera.

The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., PlayStore <IMG>), or between two user devices (e.g., smart phones) directly.

Claim 1:
An electronic device (<NUM>, <NUM>) comprising:
a camera (<NUM>, <NUM>), wherein the camera (<NUM>, <NUM>) includes a lens assembly (<NUM>) including one or more lenses, an image sensor (<NUM>, <NUM>) including at least one pixel,
wherein each of the at least one pixel includes (<NUM>) a first sub-pixel, a second sub-pixel, a third sub-pixel, and a fourth sub-pixel, wherein the first sub-pixel, the second sub-pixel, the third sub-pixel, and the fourth sub-pixel are covered by a color filter, and
wherein the first sub-pixel is covered by a first micro lens, the second sub-pixel is covered by a second micro lens, and the third sub-pixel and the fourth sub-pixel are covered by a third micro lens;
a display (<NUM>, <NUM>); and
a processor (<NUM>) configured to:
control the image sensor to obtain a light reflected from an external object and permeating through the lens assembly (<NUM>) by using the at least one pixel (<NUM>),
control the image sensor to obtain first data from the first sub-pixel, second data from the second sub-pixel, third data from the third sub-pixel and fourth data from the fourth sub-pixel,
generate depth data associated with the external object using the third data and the fourth data based on light permeating through the third micro lens,
restore the third data and the fourth data by summing the third data and the fourth data and dividing the summed data by a ratio between the first data and the second data, such that a ratio between the third data and the fourth data is equal to the ratio between the first data and the second data, and generate pixel data using the first data, the second data, the restored third data and the restored fourth data on each pixel of the at least one pixel (<NUM>),
generate an image based on the pixel data,
control the display (<NUM>, <NUM>) to output the image, and
move one or more lenses based on the depth data.