Patent Description:
Various types of electronic devices (e.g., smartphones, tablet PCs, and camera devices) capable of capturing images have come to the market. The electronic device may recognize an object to perform auto focus (AF).

A phase auto focus (PAF) scheme for detecting a phase difference between two pixels adjacent to each other and determining a focus is used as a method for performing auto focus (AF). The PAF scheme may be compared with a continuous AF scheme for measuring a focus depending on a change in planar frequency characteristics of an image to obtain more accurate and fast result in determining directionality of the focus.

In the PAF scheme, two-dimensional image pixels arranged consecutively and periodically on adjacent positions have a plurality of photo diodes (hereinafter, PDs) sharing one microlens. When PDs which are electrically separated from each other and optically have the same characteristic are not accurately focused, they may indicate different detection characteristics. The PAF scheme may analyze pieces of information among the plurality of PDs, which are detected when they are not focused, and may determine the consistency of AF. <CIT> relates to an image capture device, pixel, and method of determining a focus setting for an image capture device. <CIT> relates to convolutional image processing accomplished using a computer, where the computer can include a multilayered analysis engine, e.g. a convolutional neural network (CNN).

An image sensor of an electronic device according to an existing technology may perform AF by detecting a phase difference between <NUM> PDs arranged in a horizontal direction. <NUM> PDs separated horizontally may share one microlens and may be separated by a pixel wall. Lights incident through the microlens may be differently refracted along a path to be separated from each other. Because the image is formed horizontally by the same lens, when the focus is matched and there is a subject in an arrangement direction of pixels, characteristics and level values of pixels located horizontally may be varied. In this case, an edge separated in a vertical direction causes a problem incapable of detecting an ideal value irrespective of whether the focus is matched.

When using a 4PD image sensor including <NUM> (2x2) PDs separated in a horizontal direction and a vertical direction, an AF detection circuit performs calculation in the horizontal direction and calculation in the vertical direction at the same time, thus needing relatively many buffers therein to check a correlation between pixels adjacent to each other. Furthermore, the calculation process becomes complicated and there may occur a problem due to an increase in power consumption or cost.

Various embodiments of the disclosure may simultaneously or sequentially process AF data in a plurality of directions using data obtained by a 4PD image sensor.

Further aspects of the present invention are outlined in the dependent claims. When the term "embodiment" is used for describing an unclaimed combination of features, it has to be understood as referring to examples useful for understanding the present invention. In accordance with an aspect of the disclosure, an electronic device is provided. The electronic device may include an image sensor including a plurality of pixels, an AF processing unit that performs calculation for performing auto focus based on a phase difference, a memory, and a controller that delivers data obtained by the image sensor to the AF processing unit or the memory. Each pixel included in the plurality of pixels may include a plurality of photo diodes and a microlens that covers the plurality of photo diodes. The controller may provide the AF processing unit with at least one of first AF data of photo diodes arranged in a first direction among the plurality of photo diodes or second AF data of photo diodes arranged in a second direction among the plurality of photo diodes. The AF processing unit may perform first phase auto focus (PAF) calculation based on the first AF data or may perform second PAF calculation based on the second AF data.

The electronic device according to embodiments disclosed in the disclosure may simultaneously or sequentially process AF data in a plurality of directions using data obtained by an image sensor.

The electronic device according to embodiments disclosed in the disclosure may detect directionality of an edge and may reorder AF data in an edge direction, thus simplifying calculation of an AF detection circuit.

The electronic device according to embodiments disclosed in the disclosure may reduce a calculation time and may enhance AF accuracy by means of rotation or reordering of the AF data of the image sensor.

With regard to description of drawings, the same or similar denotations may be used for the same or similar components.

Hereinafter, various embodiments of the disclosure are described with reference to the accompanying drawings.

With regard to description of drawings, similar denotations may be used for similar components.

<FIG> illustrates an electronic device capable of capturing an image according to an embodiment.

Referring to <FIG>, an electronic device <NUM> according to an embodiment may generate a preview image based on image data obtained using a camera module (or a camera device) <NUM> and may output the generated preview image to a display <NUM>.

The camera module <NUM> may include, for example, a lens assembly including one or more lens and an image sensor. Each pixel of the image sensor may include a plurality of photo diodes.

On a first screen <NUM>, the electronic device <NUM> may generate phase difference data (or depth data) by optical path differences generated by the plurality of photo diodes (hereinafter, PDs) which share a microlens.

On a second screen <NUM>, the electronic device <NUM> may adjust a focus on an external object <NUM> by moving the one or more lenses included in the camera module <NUM>, based on the generated phase difference data. Thus, the electronic device <NUM> may display a preview image focusing on the external object on the display <NUM>.

It is illustratively shown that the camera module <NUM> is the front camera of the electronic device <NUM> in <FIG>, but not limited thereto. For example, the camera module <NUM> may be disposed on at least one of a rear surface or a side surface of the electronic device <NUM>.

<FIG> illustrates a configuration of an image sensor according to an embodiment.

Referring to <FIG>, an image sensor <NUM> included in a camera module (e.g., a camera module <NUM> of <FIG>) may include a plurality of pixels. It is illustratively shown that the image sensor <NUM> outputs a Bayer-patterned image based on a signal generated by a 4PD image sensor in <FIG>, but not limited thereto.

According to an embodiment, one <NUM> of a plurality of pixels may include a microlens <NUM>, a color filter <NUM>, a first PD (or a first sub-pixel) (PD1) <NUM>, a second PD (or a second sub-pixel) (PD2) <NUM>, a third PD (or a third sub-pixel) (PD3) <NUM>, and a fourth PD (or a fourth sub-pixel) (PD4) <NUM>.

The microlens <NUM> may cover the first to fourth PDs <NUM>, <NUM>, <NUM>, and <NUM>. The microlens <NUM> may adjust a path of an incident light such that a light incident from the outside may arrive at the first to fourth PDs <NUM>, <NUM>, <NUM>, and <NUM>.

The color filter <NUM> may be disposed between the microlens <NUM> and the first to fourth PDs <NUM>, <NUM>, <NUM>, and <NUM> to pass a light of a specified wavelength range (e.g., a wavelength range corresponding to a green light). The color filter <NUM> may allow only the light of the specified wavelength range in the light passing through the microlens <NUM> and may limit a light except from the specified wavelength range.

Each of the first to fourth PDs <NUM>, <NUM>, <NUM>, and <NUM> may convert the light passing through the microlens <NUM> and the color filter <NUM> into an electrical signal.

The light introduced from the outside may be refracted by the microlens <NUM> to be changed in progress path. The light passing through the microlens <NUM> may be directly introduced into the PD or may be reflected from a pixel wall W between the PDs to be introduced into the PD.

<FIG> is a drawing illustrating describing an edge direction and AF detection of an object various embodiments.

Referring to <FIG>, an image sensor <NUM> may include a plurality of pixels. One <NUM> of the plurality of pixels may be covered by one microlens <NUM>. For example, the one pixel <NUM> may include first to fourth PDs <NUM>, <NUM>, <NUM>, and <NUM>.

A first object <NUM> disposed in a vertical direction may form an edge <NUM> in a horizontal direction. The two pixels (e.g., the first PD <NUM> and the second PD <NUM>) separated horizontally may be separated by a pixel wall <NUM>. Lights incident through the microlens <NUM> may be differently refracted along a path to be separated from each other. Because the image is formed by the same microlens <NUM>, characteristics and level values of pixels (e.g., the first PD <NUM> and the second PD <NUM>) located horizontally may be varied for the first object <NUM> which has an identical focus and is present in an arrangement direction of pixels.

A second object <NUM> disposed in the horizontal direction may form an edge <NUM> in the vertical direction. The two pixels (e.g., the first PD <NUM> and the second PD <NUM>) separated vertically may be separated by a pixel wall <NUM>. Lights incident through the microlens <NUM> may be differently refracted along a path to be separated from each other. Because the image is formed by the same microlens, characteristics and level values of pixels (e.g., the first PD <NUM> and the third PD <NUM>) located vertically may be varied for the second object <NUM> which has an identical focus and is present in an arrangement direction of pixels.

<FIG> illustrates a camera module according to various embodiments.

Referring to <FIG>, a camera module <NUM> may include an image sensor <NUM>, a controller <NUM>, a memory <NUM>, an image processing unit <NUM>, and an AF processing unit <NUM>. <FIG> is separated according to a function, but not limited thereto. For example, the image sensor <NUM> and the controller <NUM> may be integrated into one, or the controller <NUM> and the AF processing unit <NUM> may be integrated into one.

The image sensor <NUM> may convert a light incident from the outside into an electrical signal. The image sensor <NUM> may be composed of a plurality of pixels. Each pixel may include a plurality of PDs. Hereinafter, a description will be given of, but not limited to, the case where each pixel includes four PDs.

According to an embodiment, the image sensor <NUM> may generate image data for taking a picture or a moving image or image data for displaying a preview screen. Furthermore, the image sensor <NUM> may generate data (or a signal) (hereinafter, AF data) for performing AF.

According to an embodiment, when the image sensor <NUM> is composed of a plurality of PDs sharing the same one microlens, data of the plurality of PDs may be output in an arranged order (e.g., a Bayer pattern). According to another embodiment, the image sensor <NUM> may generate and output a signal of a format processed through an output mode such as binning to reduce output data. The image sensor <NUM> may output a signal of a format processed by performing image processing in a horizontal/vertical direction or a diagonal direction depending on the arrangement of pixels and if necessary.

The controller (or a preprocessing unit or a data conversion unit) <NUM> may process image data or AF data received from the image sensor <NUM>. The controller <NUM> may provide the image data to the image processing unit (e.g., an ISP) <NUM>. The controller <NUM> may temporarily store the AF data in the memory <NUM> or may deliver the AF data to the AF processing unit <NUM>.

The controller <NUM> may detect directionality from the AF data of the image sensor <NUM> for accurate and simple AF calculation with respect to horizontal and vertical or more directions and may reorder data in a dominant direction, such that the AF processing unit <NUM> performs AF calculation.

For example, the controller <NUM> may maintain AF data in a first direction (e.g., a horizontal direction) and may rotate or reorder AF data in a second direction (e.g., a vertical direction) and may deliver the AF data to the AF processing unit <NUM>. As a result, the AF processing unit <NUM> may consistently perform PAF calculation irrespective of directionality.

According to an embodiment, the controller <NUM> may convert and deliver the image data or the AF data, received from the image sensor <NUM>, to the memory <NUM>, the image processing unit <NUM>, or the AF processing unit <NUM> or may deliver the image data or the AF data to the memory <NUM>, the image processing unit <NUM>, or the AF processing unit <NUM> without a separate conversion task.

According to various embodiments, the controller <NUM> may include a direction filter <NUM> or a rotator <NUM>. The direction filter <NUM> may be used to determine a dominant direction among edge directions of an object. The rotator <NUM> may change the direction of the AF data to change the AF data in the form of data capable of being easily processed by the AF processing unit <NUM>.

The storage unit <NUM> may store the AF data (or the converted AF data). For example, the storage unit <NUM> may be a DRAM. The stored AF data may be delivered to the AF processing unit <NUM> depending on directionality of an edge.

The image processing unit (e.g., the ISP) <NUM> may process image data. For example, the received image data may be processed to be displayed as a preview image.

The AF processing unit <NUM> may perform AF (PAF) using a phase difference based on the received AF data. The AF processing unit <NUM> may include at least one or more PAF circuits <NUM>. It is illustratively shown that the one PAF circuit is included in <FIG>, but not limited thereto.

The PAF circuit <NUM> may perform PAF calculation in a specified direction. For example, the PAF circuit <NUM> may process AF data of some (e.g., two) of a plurality of photo diodes (e.g., first to fourth PDs) included in one pixel of the image sensor <NUM>.

According to an embodiment, the PAF circuit <NUM> may perform PAF calculation using AF data extracted from pixels (e.g., the first PD and the second PD or the third PD and the fourth PD) separated horizontally or may perform PAF calculation using AF data extracted from pixels (e.g., the first PD and the third PD or the second PD and the fourth PD) separated vertically. Hereinafter, a description will be given of a 2PD PAF circuit, but not limited thereto.

According to an embodiment, the camera module <NUM> may include a plurality of PAF circuits respectively corresponding to a plurality of edge directions capable of being detected by the image sensor <NUM>. For example, in case of edge detection in a vertical/horizontal direction, two PAF circuits may be included. For another example, when it is possible to detect edges in eight directions, eight PAF circuits may be included.

According to another embodiment, the camera module <NUM> may perform PAF calculation for the plurality of edge directions using one PAF circuit. For example, in case of the edge detection in the vertical/horizontal direction, PAF calculation of the edge in the horizontal direction may be primarily performed and PAF calculation of the edge in the vertical direction may be secondarily performed.

According to another embodiment, the camera module <NUM> may determine a dominant direction among directions of edges using one PAF circuit, perform PAF for one dominant direction, and may fail to, or subsidiarily, proceed with PAF for other directions.

According to various embodiments, at least some of the functions performed by the controller <NUM>, the image processing unit <NUM>, or the AF processing unit <NUM> may be performed by a processor (e.g., an AP) of an electronic device (e.g., an electronic device <NUM> of <FIG>) including the camera module <NUM>.

According to various embodiments, at least some of the controller <NUM>, the image processing unit <NUM>, or the AF processing unit <NUM> may be components (e.g., chips) separated in hardware. Alternatively, at least some of the controller <NUM>, the image processing unit <NUM>, or the AF processing unit <NUM> may be operations separated in software in the same calculation element.

<FIG> is an output timing diagram of image data and AF data according to various embodiments.

Referring to <FIG>, when a state of an enable signal <NUM> of an image sensor <NUM> is changed, the image sensor <NUM> may output image data (e.g., a Bayer signal) <NUM>. The image sensor <NUM> may output AF data <NUM> output at a longer time interval than the image data (e.g., the Bayer signal) <NUM>, together with the image data (e.g., the Bayer signal) <NUM>.

The image sensor <NUM> may process AF data in a direction to obtain AF from the image data (e.g., the Bayer signal) <NUM> and may output the AF data <NUM> with smaller resolution than the image data (e.g., the Bayer signal) <NUM> by means of binning or the like.

It is illustratively shown that the AF data AF_LR <NUM> in the horizontal direction and the AF data AF_TB <NUM> in the vertical direction are output at the same timing in <FIG>, but not limited thereto. For example, the AF data AF_LR <NUM> in the horizontal direction and the AF data AF_TB <NUM> in the vertical direction are output at different timings.

<FIG> illustrates AF processing using two 2PD PAF circuits in a Bayer pattern according to various embodiments. Two 2PD PAF circuits are illustratively shown in <FIG>, but not limited thereto.

Referring to <FIG>, a 4PD image sensor <NUM> may include four PDs in one pixel. The 4PD image sensor <NUM> may sequentially read out PDs included in each pixel to generate a multi-PD signal (or raw data) <NUM>.

An image generator <NUM> may integrate values of four PDs included in one pixel to generate image data <NUM>. The image generator <NUM> may transmit the image data <NUM> to an image processing unit (e.g., an ISP) <NUM>.

A first AF data generator <NUM> may deliver AF data <NUM> in a first direction (e.g., a horizontal direction) in the multi-PD signal <NUM> to a first PAF circuit <NUM>.

A second AF data generator <NUM> may deliver AF data <NUM> in a second direction (e.g., a vertical direction) in the multi-PD signal <NUM> to a second PAF circuit <NUM>.

Each of the first PAF circuit <NUM> and the second PAF circuit <NUM> may perform PAF calculation for a different direction. The first PAF circuit <NUM> and the second PAF circuit <NUM> may perform AF calculation at the same time. As a result, quick and accurate AF may be performed.

It is illustratively shown that the two PAF circuits are included in <FIG>, but not limited thereto. For example, AF data for eight directions may be simultaneously and separately processed by eight PAF circuits.

<FIG> illustrates AF processing using one 2PD PAF circuit in a Bayer pattern according to various embodiments.

An image generator <NUM> may integrate values of four PDs included in one pixel to generate main image data <NUM>. The image generator <NUM> may transmit the image data <NUM> to an image processing unit (e.g., an ISP) <NUM>.

An AF data generator <NUM> may convert the multi-PD signal <NUM> into AF data <NUM> capable of being processed by a PAF circuit <NUM> and may store the AF data <NUM> in a memory (e.g., a DRAM) <NUM>. For example, vertical data 703a and horizontal data 703b may be stored.

The memory <NUM> may store the AF data <NUM>. A storage unit <NUM> may be a memory buffer included in a camera module or a memory area formed independently of the camera module.

The PAF circuit <NUM> may process the AF data <NUM> stored for each direction from the memory <NUM>. According to an embodiment, the PAF circuit <NUM> may sequentially process the AF data <NUM> depending on a direction. For example, the PAF circuit <NUM> may primarily process the horizontal data 703b and may secondarily process the vertical data 703a.

According to an embodiment, the rotator <NUM> may rotate and deliver signals in the other directions except for a first direction (e.g., a horizontal direction) to the PAF circuit <NUM>.

For example, the vertical data 703a may rotate at <NUM> degrees to be provided to the PAF circuit <NUM> (703a-<NUM>). As a result, the horizontal data 703b and the converted vertical data 703a-<NUM> may have a data structure similar to each other, and the calculation process of the PAF circuit <NUM> may be simplified.

<FIG> is an output timing diagram of processing image data and AF data according to various embodiments.

Referring to <FIG>, in response to a timing signal SENSOR_VSYNC of an image sensor <NUM>, the image sensor <NUM> may store AF data SAVE_LR and SAVE_TB at a specified time interval.

A PAF circuit <NUM> may start PAF using the horizontal data SAVE_LR at a time when the storage of the horizontal data SAVE_LR is completed. PAF (AF_LR) for a horizontal direction may be completed at the same time that the storage of the horizontal data SAVE_LR is ended (or as a portion of the storage of the horizontal data SAVE_LR is delayed).

After the PAF for the horizontal direction is completed, the PAF circuit <NUM> may start PAF (AF_TB) using the stored vertical data SAVE_TB. PAF using the vertical data SAVE_TB may proceed in a state where the vertical data SAVE_TB in stored (a state where a timing signal SENSOR_VSYNC is low).

AF data may be converted into a size smaller than image data to be output. It is possible to process PAF (AF_TB) within a relatively short time using the PAF (AF_LR) for the horizontal direction or the vertical data SAVE_TB. It may be possible to proceed with PAF (AF_TB), after the readout of all frames is completed because of having a different readout direction, using the vertical data SAVE_TB.

According to various embodiments, the PAF circuit <NUM> may perform PAF calculation for a plurality of directions in the state where the timing signal SENSOR_VSYNC is low. For example, the PAF circuit <NUM> may perform PAF calculation for the plurality of directions in a vertical direction, a <NUM>-degree direction, and a <NUM>-degree direction in the state where the timing signal SENSOR_VSYNC is low.

<FIG> illustrates AF processing using a direction filter in a Bayer pattern according to various embodiments.

An AF data generator <NUM> may convert the multi-PD signal <NUM> into AF data <NUM> capable of being processed by a PAF circuit <NUM> and may store the AF data <NUM> in a memory (e.g., a DRAM) <NUM>. For example, vertical data 803a and horizontal data 803b may be stored.

A direction filter <NUM> may detect an edge direction, using the AF data <NUM>. The PAF circuit <NUM> may process AF data for one direction among the AF data <NUM> stored for each direction from the memory <NUM>. The PAF circuit <NUM> may perform one PAF calculation for one direction determined by means of the direction filter <NUM>.

For example, the vertical data 803a which is one direction according to an arrangement direction of an edge (or an arrangement direction of an object) may be selected between the horizontal data 703a and the vertical data 703b. The PAF circuit <NUM> may rotate the vertical data 803a by means of a rotator <NUM> (803a-<NUM>) to perform one PAF calculation.

According to an embodiment, the rotator <NUM> may rotate and deliver signals in the other directions except for a first direction (e.g., a horizontal direction) to the PAF circuit <NUM>. For example, the vertical data 803a may rotate at <NUM> degrees to be delivered to the PAF circuit <NUM> (803a-<NUM>). As a result, the horizontal data 803b and the converted vertical data 803a-<NUM> may have a data structure similar to each other, and the calculation process of the PAF circuit <NUM> may be simplified.

<FIG> is an output timing diagram of selective AF processing using a direction filter according to various embodiments.

While the storage of the horizontal data SAVE_LR and the horizontal data SAVE_LR proceed, a PAF circuit <NUM> may perform edge detection (or directionality determination of an image) using a direction filter <NUM>.

After the storage of the horizontal data SAVE_LR and the horizontal data SAVE_LR is ended, the PAF circuit <NUM> may proceed with PAF for one direction selected by means of the direction filter <NUM>. In a state where the timing signal SENSOR_VSYNC is low, one PAF may proceed.

<FIG> illustrates AF processing using two 2PD PAF circuits in a processing pattern according to various embodiments. The two 2PD PAF circuits are illustratively shown in <FIG>, but not limited thereto.

Referring to <FIG>, a 4PD image sensor <NUM> may include four PDs in one pixel. The 4PD image sensor <NUM> may integrate values of four PDs included in one pixel to generate image data <NUM>. The 4PD image sensor <NUM> may transmit the image data <NUM> to an image processing unit (e.g., an ISP) <NUM>.

The 4PD image sensor <NUM> may deliver first processing data <NUM> obtained by processing AF data in a first direction (e.g., a horizontal direction) to a first PAF circuit <NUM>.

The 4PD image sensor <NUM> may deliver second processing data <NUM> obtained by processing AF data in a second direction (e.g., a vertical direction) to a second PAF circuit <NUM>.

<FIG> illustrates AF processing using one 2PD PAF circuit in a processing pattern according to various embodiments.

The 4PD image sensor <NUM> may store first processing data <NUM> obtained by processing AF data in a first direction (e.g., a horizontal direction) in a memory (e.g., a DRAM) <NUM>. For example, vertical data 1003a and horizontal data 1003b may be stored.

A PAF circuit <NUM> may process the AF data <NUM> stored for each direction from the memory <NUM>. According to an embodiment, the PAF circuit <NUM> may sequentially process the AF data <NUM> depending on a direction. For example, the PAF circuit <NUM> may primarily process the horizontal data 1003b and may secondarily process the vertical data 1003a.

According to an embodiment, a rotator <NUM> may rotate and deliver signals in the other directions except for the first direction (e.g., the horizontal direction) to the PAF circuit <NUM>. For example, the vertical data 1003a may rotate at <NUM> degrees to be provided to the PAF circuit <NUM> (1003a-<NUM>). As a result, the horizontal data 1003b and the converted vertical data 1003a-<NUM> may have a data structure similar to each other, and the calculation process of the PAF circuit <NUM> may be simplified.

<FIG> illustrates AF processing using a direction filter in a processing pattern according to various embodiments.

Referring to <FIG>, a 4PD image sensor <NUM> may include <NUM> PDs in one pixel. The 4PD image sensor <NUM> may integrate values of four PDs included in one pixel to generate image data <NUM>. The 4PD image sensor <NUM> may transmit the image data <NUM> to an image processing unit (e.g., an ISP) <NUM>.

The 4PD image sensor <NUM> may store first processing data <NUM> obtained by processing AF data in a first direction (e.g., a horizontal direction) in a memory (e.g., a DRAM) <NUM>. For example, vertical data 1103a and horizontal data 1103b may be stored.

For example, the vertical data 1103a which is one direction according to an arrangement direction of an edge (or an arrangement direction of an object) may be selected between the horizontal data 1103a and the vertical data 1103b. The PAF circuit <NUM> may rotate the vertical data 1103a by means of a rotator <NUM> (1103a-<NUM>) to perform one PAF calculation.

According to an embodiment, the rotator <NUM> may rotate and deliver signals in the other directions except for the first direction (e.g., the horizontal direction) to the PAF circuit <NUM>. For example, the vertical data 1103a may rotate at <NUM> degrees to be provided to the PAF circuit <NUM> (1103a-<NUM>). As a result, the horizontal data 1103b and the converted vertical data 1103a-<NUM> may have a data structure similar to each other, and the calculation process of the PAF circuit <NUM> may be simplified.

<FIG> and <FIG> are drawings illustrating detecting an edge in a plurality of directions by means of a direction filter and performing PAF calculation according to various embodiments.

A direction filter <NUM> may compare detection values for the plurality of directions, in AF data using the multi-PD signal (or the raw data), to detect directionality (an edge direction) of an image. For example, when data of a first portion and a second portion are changed to a specified value or more by an external object among all pixels of the AF data, it may be determined that a boundary (edge) of the first portion and the second portion is formed.

For example, in <FIG>, data of a first portion 1201a1 and a second portion 1201a2 of the 4PD image sensor <NUM> may be changed to the specified value or more by a first object <NUM>. The direction filter <NUM> may compare values of image data to detect an edge direction 1251a perpendicular to a boundary of the first portion 1201al and the second portion 1201a2 as a fifth direction (<NUM> degree).

For another example, data of a first portion 1201b1 and a second portion 1201b2 of the 4PD image sensor <NUM> may be changed to the specified value or more by a second object <NUM>. The direction filter <NUM> may compare values of image data to detect an edge direction 1252a perpendicular to a boundary of the first portion 1201b1 and the second portion 1201b2 as a third direction (<NUM> degrees).

For example, data of a first portion 1201c1 and a second portion 1201c2 of the 4PD image sensor <NUM> may be changed to the specified value or more by a third object <NUM>. The direction filter <NUM> may compare values of image data to detect an edge direction 1253a perpendicular to a boundary of the first portion 1201c1 and the second portion 1201c2 as a first direction (<NUM> degrees).

For another example, data of a first portion 1201d1 and a second portion 1201d2 of the 4PD image sensor <NUM> may be changed to the specified value or more by a fourth object <NUM>. The direction filter <NUM> may compare values of image data to detect an edge direction 1254a perpendicular to a boundary of the first portion 1201d1 and the second portion 1201d2 as a seventh direction (<NUM> degrees).

A reordering unit <NUM> may rotate or reorder AF data for each direction (or a direction where an edge is mainly disposed). The reordering unit <NUM> may rotate or reorder AF data in an edge direction 1251a, 1252a, 1253a, or 1254a perpendicular to the boundary of the first portion 1201a1, 1201b1, 1201c1, or 1201d1 and the second portion 1201a2, 1201b2, 1201c2, or 1201d2. As a result, the PAF circuit <NUM> may perform PAF calculation depending on directionality of an edge.

Although the PAF circuit <NUM> performs determined calculation irrespective of information about directionality, it may perform PAF calculation depending on directionality of an edge to obtain a PAF result with high reliability. The PAF circuit <NUM> may extract a statistical value for each direction and may perform AF determination.

<FIG> is a block diagram of an electronic device <NUM> in a network environment <NUM> according to various embodiments of the present disclosure. The electronic device according to various embodiments disclosed in this document may be a device of various types. The electronic device includes, for example, at least one of a portable communication device (e.g., smartphone), a computer device (e.g., a personal digital assistant; PDA), a tablet PC, a laptop PC (a desktop PC, a workstation, or a server), a portable multimedia device (e.g., an e-book reader or an MP3 player), a portable medical device devices (e.g., heartbeat measuring devices, blood glucose monitoring devices, blood pressure measuring devices, and body temperature measuring devices), a camera, or a wearable device. The wearable device may include at least one of an accessory type (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lens, or head-mounted-devices (HMDs)), a fabric or garment-integrated type (e.g., an electronic apparel), a body-attached type (e.g., a skin pad or tattoos), or a bio-implantable type (e.g., an implantable circuit). According to various embodiments, the electronic device may include at least one of, for example, televisions (TVs), digital versatile disk (DVD) players, audios, audio accessory devices (e.g., speakers, headphones, or headsets), refrigerators, air conditioners, cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, home automation control panels, security control panels, game consoles, electronic dictionaries, electronic keys, camcorders, or electronic picture frames.

In another embodiment, the electronic device may include at least one of navigation devices, satellite navigation system (e.g., Global Navigation Satellite System (GNSS)), event data recorders (EDRs) (e.g., black box for a car, a ship, or a plane), vehicle infotainment devices (e.g., head-up display for vehicle), industrial or home robots, drones, automatic teller's machines (ATMs), points of sales (POSs), measuring instruments (e.g., water meters, electricity meters, or gas meters), or internet of things (e.g., light bulbs, sprinkler devices, fire alarms, thermostats, or street lamps). The electronic device according to an embodiment of this disclosure may not be limited to the above-described devices, and may provide functions of a plurality of devices like smartphones which has measurement function of personal biometric information (e.g., heart rate or blood glucose). In this disclosure, the term "user" may refer to a person who uses an electronic device or may refer to a device (e.g., an artificial intelligence electronic device) that uses the electronic device.

<FIG> is a block diagram illustrating an electronic device <NUM> (e.g., the foldable electronic device <NUM> of <FIG>) in a network environment <NUM> according to various embodiments. Referring to <FIG>, the electronic device <NUM> in the network environment <NUM> may communicate with an electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or at least one of an electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input module <NUM>, a sound output module <NUM>, a display module <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a connecting terminal <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module (SIM) <NUM>, or an antenna module <NUM>. In some embodiments, at least one of the components (e.g., the connecting terminal <NUM>) may be omitted from the electronic device <NUM>, or one or more other components may be added in the electronic device <NUM>. In some embodiments, some of the components (e.g., the sensor module <NUM>, the camera module <NUM>, or the antenna module <NUM>) may be implemented as a single component (e.g., the display module <NUM>).

<FIG> is a block diagram <NUM> illustrating the camera module <NUM> according to various embodiments.

Referring to <FIG>, the camera module <NUM> may include a lens assembly <NUM>, a flash <NUM>, an image sensor <NUM>, an image stabilizer <NUM>, memory <NUM> (e.g., buffer memory), or an image signal processor <NUM>. The lens assembly <NUM> may collect light emitted or reflected from an object whose image is to be taken. The lens assembly <NUM> may include one or more lenses. According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In such a case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. Some of the plurality of lens assemblies <NUM> may have the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may have one or more lens attributes different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens.

The flash <NUM> may emit light that is used to reinforce light reflected from an object. According to an embodiment, the flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor <NUM> may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted via the lens assembly <NUM> into an electrical signal. According to an embodiment, the image sensor <NUM> may include one selected from image sensors having different attributes, such as a RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer <NUM> may move the image sensor <NUM> or at least one lens included in the lens assembly <NUM> in a particular direction, or control an operational attribute (e.g., adjust the read-out timing) of the image sensor <NUM> in response to the movement of the camera module <NUM> or the electronic device <NUM> including the camera module <NUM>. This allows compensating for at least part of a negative effect (e.g., image blurring) by the movement on an image being captured. According to an embodiment, the image stabilizer <NUM> may sense such a movement by the camera module <NUM> or the electronic device <NUM> using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module <NUM>. According to an embodiment, the image stabilizer <NUM> may be implemented, for example, as an optical image stabilizer. The memory <NUM> may store, at least temporarily, at least part of an image obtained via the image sensor <NUM> for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a Bayer-patterned image, a high-resolution image) may be stored in the memory <NUM>, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display device <NUM>. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be configured as at least part of the memory <NUM> or as a separate memory that is operated independently from the memory <NUM>.

The image signal processor <NUM> may perform one or more image processing with respect to an image obtained via the image sensor <NUM> or an image stored in the memory <NUM>. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor <NUM> may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM>. An image processed by the image signal processor <NUM> may be stored back in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) outside the camera module <NUM>. According to an embodiment, the image signal processor <NUM> may be configured as at least part of the processor <NUM>, or as a separate processor that is operated independently from the processor <NUM>. If the image signal processor <NUM> is configured as a separate processor from the processor <NUM>, at least one image processed by the image signal processor <NUM> may be displayed, by the processor <NUM>, via the display device <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include a plurality of camera modules <NUM> having different attributes or functions. In such a case, at least one of the plurality of camera modules <NUM> may form, for example, a wide-angle camera and at least another of the plurality of camera modules1380 may form a telephoto camera. Similarly, at least one of the plurality of camera modules <NUM> may form, for example, a front camera and at least another of the plurality of camera modules1380 may form a rear camera.

An electronic device (e.g., an electronic device <NUM> of <FIG> or an electronic device <NUM> of <FIG>) according to various embodiments may include an image sensor (e.g., an image sensor <NUM> of <FIG> or an image sensor <NUM> of <FIG>) including a plurality of pixels, an AF processing unit (e.g., an AF processing unit <NUM> of <FIG>) for performing calculation for performing auto focus based on a phase difference, a memory (e.g., a memory <NUM> of <FIG>, a memory <NUM> of <FIG>, or a memory <NUM> of <FIG>), and a controller (e.g., a controller <NUM> of <FIG>) for delivering data obtained by the image sensor (e.g., the image sensor <NUM> of <FIG> or the image sensor <NUM> of <FIG>) to the AF processing unit or the memory (e.g., the memory <NUM> of <FIG>, the memory <NUM> of <FIG>, or the memory <NUM> of <FIG>). Each pixel included in the plurality of pixels may include a plurality of photo diodes and a microlens for covering the plurality of photo diodes. The controller (e.g., the controller <NUM> of <FIG>) may provide the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) with at least one of first AF data of photo diodes arranged in a first direction among the plurality of photo diodes or second AF data of photo diodes arranged in a second direction among the plurality of photo diodes. The AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may perform first phase auto focus (PAF) calculation based on the first AF data or may perform second PAF calculation based on the second AF data.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may include a first AF circuit and a second AF circuit. The second AF circuit may perform the second phase auto focus (PAF) calculation, while the first AF circuit performs the first phase auto focus (PAF) calculation.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may include one AF circuit. The AF circuit may perform the first phase auto focus (PAF) calculation and may perform the second PAF, when the first phase auto focus (PAF) calculation is ended.

According to various embodiments, the controller (e.g., the controller <NUM> of <FIG>) may rotate the second AF data to correspond to the first direction and may provide the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) with the second AF data.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may select one of the first direction or the second direction by means of a direction filter and may perform PAF calculation for AF data corresponding to the selected direction.

According to various embodiments, the AF controller (e.g., the controller <NUM> of <FIG>) may store the first AF data or the second AF data in the memory (e.g., the memory <NUM> of <FIG>, the memory <NUM> of <FIG>, or the memory <NUM> of <FIG>). The AF controller (e.g., the controller <NUM> of <FIG>) may rotate or reorder and store at least one of the first AF data or the second AF data in the memory (e.g., the memory <NUM> of <FIG>, the memory <NUM> of <FIG>, or the memory <NUM> of <FIG>).

According to various embodiments, the AF controller (e.g., the controller <NUM> of <FIG>) may provide the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) with the first AF data or the second AF data with a Bayer pattern or a processing pattern.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may perform the first PAF calculation during a process where the first AF data and the second AF data are stored in the memory (e.g., the memory <NUM> of <FIG>, the memory <NUM> of <FIG>, or the memory <NUM> of <FIG>) and may calculate the second PAF calculation after the process is ended.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may perform the first PAF calculation while the first AF data and the second AF data are read out by means of the image sensor (e.g., the image sensor <NUM> of <FIG> or the image sensor <NUM> of <FIG>) and may perform the second PAF calculation after the readout is ended.

According to various embodiments, the image sensor (e.g., the image sensor <NUM> of <FIG> or the image sensor <NUM> of <FIG>) may be formed such that one microlens covers four photo diodes. The AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may compare measurement values of the two photo diodes to perform PAF calculation.

An electronic device (e.g., an electronic device <NUM> of <FIG> or an electronic device <NUM> of <FIG>) according to various embodiments may include an image sensor (e.g., an image sensor <NUM> of <FIG> or an image sensor <NUM> of <FIG>) including a plurality of pixels, wherein each pixel included in the plurality of pixels includes a plurality of photo diodes and a microlens for covering the plurality of photo diodes, an AF processing unit (e.g., an AF processing unit <NUM> of <FIG>) for performing calculation for performing auto focus based on a phase difference, a memory (e.g., a memory <NUM> of <FIG>, a memory <NUM> of <FIG>, or a memory <NUM> of <FIG>), and a controller (e.g., a controller <NUM> of <FIG>) for delivering data obtained by the image sensor (e.g., the image sensor <NUM> of <FIG> or the image sensor <NUM> of <FIG>) to the AF processing unit or the memory (e.g., the memory <NUM> of <FIG>, the memory <NUM> of <FIG>, or the memory <NUM> of <FIG>). The controller (e.g., the controller <NUM> of <FIG>) may set priorities based on measurement values of photo diodes corresponding to a plurality of directions. The AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may perform PAF calculation, based on the priorities.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may include a plurality of PAF circuits respectively corresponding to the plurality of directions. Each of the plurality of PAF circuits may start PAF calculation at the same time.

According to various embodiments, the AF processing unit (e.g., the AF processing unit <NUM> of <FIG>) may include one AF circuit. The AF circuit may sequentially perform the PAF calculation in the priorities.

A method for performing auto focus (AF) performed in an electronic device (e.g., an electronic device <NUM> of <FIG> or an electronic device <NUM> of <FIG>) according to various embodiments may include obtaining image data by means of an image sensor (e.g., an image sensor <NUM> of <FIG> or an image sensor <NUM> of <FIG>) including a plurality of pixels, obtaining first AF data of photo diodes arranged in a first direction among a plurality of photo diodes included in each pixel included in the plurality of pixels based on the image data, obtaining second AF data of photo diodes arranged in a second direction among the plurality of photo diodes based on the image data, and performing first phase auto focus (PAF) calculation based on the first AF data or performing second PAF calculation based on the second AF data.

According to various embodiments, the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) may include a first AF circuit and a second AF circuit. The performing of the first PAF calculation or the second PAF calculation may include performing the second phase auto focus (PAF) calculation by means of the second AF circuit, while performing the first phase auto focus (PAF) by means of the first AF circuit.

According to various embodiments, the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) may include one AF circuit. The performing of the first PAF calculation or the second PAF calculation may include performing the first phase auto focus (PAF) using the AF circuit and performing the second PAF calculation, when the first phase auto focus (PAF) calculation is ended.

According to various embodiments, the obtaining of the second AF data may include rotating the second AF data to correspond to the first direction.

According to various embodiments, the performing of the first PAF calculation or the second PAF calculation may include selecting one of the first direction or the second direction by means of a direction filter of the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) and performing PAF calculation for AF data corresponding to the selected direction.

According to various embodiments, the method for performing the AF may further include storing the first AF data or the second AF data in a memory (e.g., a memory <NUM> of <FIG>, a memory <NUM> of <FIG>, or a memory <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>).

It should be appreciated that various embodiments of the present disclosure and the terms used therein are not intended to limit the technological features set forth herein to particular embodiments and include various changes, or replacements for a corresponding embodiment. As used herein, each of such phrases as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases. As used herein, such terms as "1st" and "2nd", or "first" and "second" may be used to simply distinguish a corresponding component from another, and does not limit the components in other aspect (e.g., importance or order). It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with", "coupled to", "connected with", or "connected to" another element (e.g., a second element), it means that the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.

Claim 1:
An electronic device, comprising:
an image sensor including a plurality of pixels;
an auto focus, AF, processing unit configured to perform calculation for performing auto focus based on a phase difference;
a memory; and
at least one processor configured to deliver data obtained by the image sensor to the AF processing unit or the memory,
wherein each pixel included in the plurality of pixels includes a plurality of photo diodes and a microlens configured to cover the plurality of photo diodes,
wherein the at least one processor is configured to provide the AF processing unit with first AF data of photo diodes arranged in a first direction among the plurality of photo diodes,
wherein the at least one processor is configured to provide the AF processing unit with second AF data of photo diodes arranged in a second direction among the plurality of photo diodes by rotating or reordering the second AF data to have a data structure in the first direction, and
wherein the AF processing unit performs first phase auto focus, PAF, calculation based on the first AF data and performs second PAF calculation based on the second AF data.