Patent Description:
An electronic device, such as a smartphone or a tablet personal computer (PC), may perform various functions. For example, the electronic device may perform functions such as call, web search, video playback, and music playback. Furthermore, the electronic device may provide a security function through user authentication. Furthermore, the electronic device may perform user authentication through face recognition and unlock a screen or may log in to a banking application depending on the result of performing the user authentication.

<CIT> appears to disclose a face authentication device and a face authentication method which utilize an image sensor which can obtain two-dimensional image data and imaging plane phase differential information of a face of an authentication target. <CIT> appears to disclose an image sensor that includes an array of pixels arranged in rows and columns, where each pixel may include a number of adjacent sub-pixels covered by a single microlens. Image signals from the sub-pixels may be used to calculate phase information in each pixel in the array. <CIT> appears to disclose systems and methods for providing biometrically authenticated access using a mobile device. <CIT> appears to disclose technologies relating to biometric authentication such as methods that include obtaining images of a subject including a view of an eye and determining a behavioral metric based on detected movement of the eye.

An electronic device according to an existing technology may release a lock or may execute a security function, based on face recognition. To defend a spoofing attack of defrauding a face recognition system using a fake face, the electronic device may use a liveness detection technology which uses an RGB image.

Accordingly, an aspect of the disclosure is to provide an electronic device for recognizing an object using a 2PD image sensor.

In accordance with an aspect of the disclosure, an electronic device is provided. The electronic device includes a camera configured to include an image sensor including a plurality of pixels, a memory, and a processor configured to control the camera. Each pixel included in the plurality of pixels may include a plurality of photodiodes and a microlens covering the plurality of photodiodes. The processor may be configured to obtain phase images and image data for an external object using the plurality of photodiodes of the image sensor and authenticate the external object using the phase images and the image data.

The electronic device according to various embodiments disclosed in the disclosure may implement a face recognition system using a <NUM> photodiode (2PD) image sensor of a single sensor (a single camera module).

The electronic device according to various embodiments disclosed in the disclosure may identify liveness for an external object at high precision using depth information about a distance from an object, which is calculated using the 2PD image sensor.

<FIG> is a drawing illustrating an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> may recognize an object (e.g., a face) <NUM> based on image data obtained using a camera module (for example, a camera device, or a camera) <NUM>. The electronic device <NUM> may authenticate the recognized object (e.g., the face) <NUM> and may provide a security function (e.g., execute an application which needs user authentication). For example, the electronic device <NUM> may unlock a screen or may log in to a banking application, through authentication of the object (e.g., a face) <NUM>. Hereinafter, a description will be given of, but not limited to, a case where the object <NUM> is a face of a person.

The camera module <NUM> may include, for example, a lens assembly including one or more lens and an image sensor. Each pixel of the image sensor may include a plurality of photodiodes (PDs).

On screen <NUM>, the electronic device <NUM> may generate phase images by optical path differences generated by a plurality of PDs which share a microlens. The plurality of PDs may be arranged at an adjacent location on a continuous or periodic basis. The plurality of PDs may be electrically separated from each other and may optically have the same characteristic.

The electronic device <NUM> may recognize the object (e.g., a face) <NUM> based on the generated phase images. The electronic device <NUM> may determine a binning mode (or a binning level) for the phase images and may determine a depth in RGB data or the phase images. The electronic device <NUM> may determine liveness for the object (e.g., a face) <NUM> using the RGB data (or an RGB image) or the phase images to perform user authentication.

On screen <NUM>, when the user authentication is completed, the electronic device <NUM> may execute a specified application or service associated with the user authentication (e.g., display another screen (activity) different from a user authentication screen on a display <NUM>). For example, when the user authentication is completed, the electronic device <NUM> may unlock a lock state of the display <NUM> or may log in to the banking application.

It is illustratively shown that the camera module <NUM> is the front camera of the electronic device <NUM> in <FIG>, but not limited thereto. For example, the camera module <NUM> may be positioned at least one of a rear surface or a side surface of the electronic device <NUM>.

<FIG> is a drawing illustrating a configuration of an image sensor included in a camera module according to an embodiment of the disclosure.

Referring to <FIG>, an image sensor <NUM> included in a camera module (e.g., a camera module <NUM> of <FIG>) may include a plurality of pixels. It is illustratively shown that the image sensor <NUM> outputs a Bayer-patterned image based on a signal generated by a 2PD image sensor in <FIG>, but not limited thereto.

Hereinafter, a description will be given of, but not limited to, a case where the image sensor <NUM> generates left/right phase images (or 2PD raw data) and Bayer-patterned RGB data.

According to an embodiment, one <NUM> of a plurality of pixels may include a microlens <NUM>, a color filter <NUM>, a first PD (or a first sub-pixel) (PD1) <NUM>, and a second PD (or a second sub-pixel) (PD2) <NUM>.

The microlens <NUM> may cover the first PD <NUM> and the second PD <NUM>. The microlens <NUM> may adjust a path of an incident light such that light incident from the outside may arrive at the first PD <NUM> and the second PD <NUM>.

The color filter <NUM> may be positioned between the microlens <NUM> and PDs (the first PD <NUM> and the second PD <NUM>) to pass light of a specified wavelength range (e.g., a wavelength range corresponding to green light). The color filter <NUM> may allow only the light of the specified wavelength range in light passing through the microlens <NUM> to arrive at the first PD <NUM> and the second PD <NUM> and may limit light except from the specified wavelength range.

Each of the first PD <NUM> and the second PD <NUM> may convert light passing through the microlens <NUM> and the color filter <NUM> into an electrical signal. As light introduced from the outside (e.g., light reflected from an object) may be reflected by the microlens <NUM>, a path thereof may be changed. Light passing through the microlens <NUM> may be directly introduced into PDs, or may be reflected from a pixel wall W between the PDs to be introduced into the PDs.

For example, when light reflected from the same point (or an adjacent point) of a face object (e.g., an object <NUM> of <FIG>) is incident to the first PD <NUM> and the second PD <NUM>, an optical path difference may be generated by refraction through the microlens <NUM> or reflection by the pixel wall W. Thus, a phase difference may occur between first data of the first PD <NUM> and second data of the second PD <NUM>. An electronic device (e.g., an electronic device <NUM> of <FIG>) may generate a 2PD disparity map by the phase difference and may use the 2PD disparity map in a process of recognizing the object <NUM>.

<FIG> is a block diagram illustrating a configuration of an electronic device according to an embodiment of the disclosure. <FIG> is separated according to a function, but not limited thereto. For example, an image sensor <NUM> and an image processing unit <NUM> may be integrated into one, or the image processing unit <NUM> and an object recognition unit <NUM> may be integrated into one. Furthermore, an operation of the image processing unit <NUM> and the object recognition unit <NUM> may be an operation by calculation of a processor in an electronic device <NUM>.

Referring to <FIG>, the electronic device <NUM> may include the image sensor <NUM>, the image processing unit <NUM>, and the object recognition unit <NUM>.

The image sensor <NUM> may include a plurality of pixels. The image sensor <NUM> may generate left/right phase images and Bayer-patterned RGB data based on a signal generated by a 2PD image sensor.

The image processing unit <NUM> may process RGB data collected by the image sensor <NUM>. The image processing unit <NUM> may deliver information necessary for object recognition to the object recognition unit <NUM>.

The object recognition unit <NUM> may recognize and authenticate a face object (e.g., an object <NUM> of <FIG>). The object recognition unit <NUM> may recognize and authenticate the object <NUM> using all the left/right phase images and the RGB data. According to an embodiment, the object recognition unit <NUM> may include a binning determining unit <NUM>, a depth calculating unit <NUM>, and a liveness calculating unit <NUM>.

The binning determining unit <NUM> may determine a binning mode (or a binning level or a binning model) for the left/right phase images. The binning determining unit <NUM> may determine a binning mode of the left/right phase images based on depth information of the object <NUM> or illumination information around the object <NUM>.

The depth calculating unit <NUM> may calculate depth information with the object <NUM> based on the left/right phase images or the RGB data. The depth information may be used to determine a binning level of the object <NUM> or determine liveness.

The liveness calculating unit <NUM> may determine liveness of the object <NUM> (whether the object <NUM> is alive) based on the left/right phase images or the RGB data. The liveness calculating unit <NUM> may use liveness models of various algorithms. The liveness calculating unit <NUM> may determine liveness using depth information, a 2PD disparity map, and edge information.

<FIG> is a flowchart illustrating a method for recognizing an object according to an embodiment of the disclosure.

Referring to <FIG>, at operation <NUM>, an image sensor (e.g., an image sensor <NUM> of <FIG>) may collect left/right phase images and RGB data for a face object (e.g., an object <NUM> of <FIG>). For example, an electronic device (e.g., an electronic device <NUM> of <FIG>) may capture the object <NUM> using a camera module (e.g., a camera module <NUM> of <FIG>) in a screen lock state. The image sensor <NUM> in the camera module <NUM> may output the left/right phase images and the RGB data.

At operation <NUM>, a processor of the electronic device <NUM> may determine liveness of the object <NUM> using the left/right phase images and the RGB data. For example, the processor may generate a 2PD disparity map using the left/right phase images. The processor may detect an edge using the RGB data or may calculate liveness of the RGB data. The processor may reflect information about a distance from the object <NUM>, information about illumination around the object <NUM>, or the like in an overall manner to determine liveness of the object <NUM>. The processor may determine whether the object <NUM> is a real face of the user or a fake face using a photo or an image, depending on the calculated liveness level.

<FIG> is a drawing illustrating a method for recognizing an object according to an embodiment of the disclosure. <FIG> is illustrative, but not limited thereto.

Referring to <FIG>, an image sensor (e.g., an image sensor <NUM> of <FIG>) may collect left/right (L/R) phase images <NUM> and RGB data <NUM> for a face object (e.g., an object <NUM> of <FIG>). A processor of an electronic device (e.g., an electronic device <NUM> of <FIG>) may determine liveness of the object <NUM> using all the L/R phase images <NUM> and the RGB data <NUM>.

According to an embodiment, the processor may generate a 2PD disparity map <NUM> using the L/R phase images <NUM>. The processor may calculate a liveness score by applying a first liveness model <NUM> based on the 2PD disparity map <NUM>. The first liveness model <NUM> may be a model which calculates liveness with respect to a region with a large phase difference on the 2PD disparity map <NUM>.

According to an embodiment, the processor may calculate a liveness score by applying a second liveness model <NUM> in the RGB data <NUM>. The second liveness model <NUM> may be a model which detects an edge of the RGB data <NUM> (a boundary of the object <NUM>) to calculate liveness.

According to an embodiment, the processor may calculate a liveness score by applying a third liveness model <NUM> for the RGB data <NUM> itself. The third liveness model <NUM> may be a model which calculates liveness depending to locations of feature points of the object <NUM> detected from the RGB data <NUM>, a mutual arrangement relationship between the feature points, or a change degree over time in the feature points. Additional information about the first liveness model <NUM>, the second liveness model <NUM>, or the third liveness model <NUM> may be provided in <FIG>.

The processor may consider information, such as information about a distance from the face object <NUM> or information about illumination around the object <NUM>, in an overall manner to adjust a weight for each model. The processor may compare the calculated liveness score with a predetermined reference value to determine whether the object <NUM> is a real face of a user or a fake face using a photo or an image.

<FIG> is a drawing illustrating a binning mode according to an embodiment of the disclosure. <FIG> is illustrative, but not limited thereto.

Referring to <FIG>, a binning determining unit (e.g., a binning determining unit <NUM> of <FIG>) may determine a binning mode (or a binning level or a binning model) <NUM>, <NUM>, or <NUM> for left/right phase images <NUM>. A data size may be reduced by selecting some of pixel data of the left/right phase images <NUM> depending on the binning mode. For example, for <NUM>*<NUM> binning, data for one pixel selected among <NUM> pixel regions may be maintained, and data for the other pixels may be excluded. The higher the binning level, the larger the size of a pixel region proceeding with binning and the smaller the data size.

When the binning level is reduced, memory occupancy and a data processing time of the left/right phase images <NUM> may be increased and precision for object recognition using the left/right phase images <NUM> may be enhanced. On the other hand, when the binning level is increased, the memory occupancy and the data processing time may be reduced and the precision for the object recognition using the left/right phase images <NUM> may be lowered.

For example, separate binning may fail to be performed in the first binning mode <NUM>. In this case, the left/right phase images <NUM> may be maintained without selection of separate pixel data (e.g., the same resolution of <NUM>*<NUM>*(L, R) as the left/right phase images <NUM> may be kept and the same data capacity of <NUM> MB as the left/right phase images <NUM> may be kept). Thus, memory occupancy may increase, and a data processing speed may decrease (a data processing time may increase). On the other hand, precision for object recognition using the left/right phase images may be enhanced.

For another example, binning of a middle level may be performed in the second binning mode <NUM> (e.g., resolution of <NUM>*<NUM>*(L, R) may be kept and data capacity of <NUM> MB may be kept by <NUM>*<NUM> binning). In this case, memory occupancy may more decrease than the first binning mode <NUM> and a data processing speed may more increase than the first binning mode <NUM>. On the other hand, precision for object recognition using the left/right phase images <NUM> may be more lowered than the first binning mode <NUM>.

For another example, binning of the highest level may be performed in the third binning mode <NUM> (e.g., resolution of <NUM>*<NUM>*(L, R) may be kept and data capacity of <NUM> MB may be kept by <NUM>*<NUM> binning). In this case, memory occupancy may more decrease than the first binning mode <NUM> or the second binning mode <NUM> and a data processing speed may more increase than the first binning mode <NUM> or the second binning mode <NUM>. On the other hand, precision for object recognition using the left/ right phase images <NUM> may be more lowered than the first binning mode <NUM> or the second binning mode <NUM>.

According to various embodiments, the binning determining unit <NUM> may determine a binning level (or a binning mode) based on parameter information received from an image processing unit (e.g., an image processing unit <NUM> of <FIG>). For example, the parameter information may be, for example, camera exposure information, sensitivity information, or gain information, and the binning determining unit <NUM> may determine an illumination value at the time of image capture based on the parameter information. According to an embodiment, the binning determining unit <NUM> may determine an illumination value at the time of image capture based on information received via a separate illumination sensor.

For example, when the left/right phase images <NUM> are collected in a low illumination environment, the binning determining unit <NUM> may lower a binning level of the left/ right phase images <NUM> to enhance precision of object recognition using the left/right phase images <NUM>. As a result, a problem where performance of determining liveness based on RGB data is degraded in a dark environment may be supplemented.

For another example, when the left/right phase images <NUM> are collected in a general illumination environment or a high illumination environment, the binning determining unit <NUM> may enhance a binning level of the left/right phase images <NUM> to lower precision of object recognition using the left/right phase images <NUM>. As a result, memory occupancy and a data processing speed may decrease.

According to various embodiments, the binning determining unit <NUM> may determine the binning mode <NUM>, <NUM>, or <NUM> based on distance information (or depth information) received from a depth calculating unit (e.g., a depth calculating unit <NUM> of <FIG>). For example, when a distance from a face object (e.g., an object <NUM> of <FIG>) is greater than or equal to (or is greater than) a predetermined first reference value, the binning determining unit <NUM> may lower a binning level depending to the first binning mode <NUM> or may fail to proceed with binning. As a result, precision of object recognition using the left/right phase images <NUM> may be enhanced. When the distance is distant from the object <NUM>, because a 2PD disparity characteristic calculated in the left/right phase images <NUM> is degraded, the binning determining unit <NUM> may lower a binning level to enhance precision of object recognition.

For another example, when the distance from the object <NUM> is less than (or is less than or equal to) the predetermined first reference value and is greater than or equal to (is greater than) a predetermined second reference value, the binning determining unit <NUM> may proceed with binning of a middle level depending on the second binning mode <NUM>. As a result, precision of object recognition using the left/right phase images <NUM> may be reflected as a suitable level.

For another example, when the distance from the object <NUM> is less than or equal to (or is less than) the predetermined second reference value, the binning determining unit <NUM> may proceed with binning of the highest level depending on the third binning mode <NUM>. As a result, precision of object recognition using the left/right phase images may be lowered. When the distance is close to the object <NUM>, because the 2PD disparity characteristic calculated in the left/right phase images <NUM> is improved, it may be easy to perform object recognition although a binning level is enhanced.

According to various embodiments, the binning determining unit <NUM> may provide the depth calculating unit <NUM> with the binned left/right phase images <NUM> to be used. Furthermore, the binning determining unit <NUM> may provide a liveness calculating unit (e.g., a liveness calculating unit <NUM> of <FIG>) with the binned left/right phase images to be used to determine liveness.

<FIG> is a drawing illustrating calculating a depth by detecting a face region according to an embodiment of the disclosure. <FIG> is illustrative, but not limited thereto.

Referring to <FIG>, a depth calculating unit (e.g., a depth calculating unit <NUM> of <FIG>) may calculate depth information with a face object (e.g., an object <NUM> of <FIG>) based on left/right phase images or RGB data. The calculated depth information may be used to determine a binning level of the object <NUM> or determine liveness.

According to various embodiments, the depth calculating unit <NUM> may calculate depth information based on a face region (or a face size) <NUM> of the object <NUM> detected from the RGB data. Furthermore, the depth calculating unit <NUM> may calculate a depth of the object <NUM> by additionally using a 2PD disparity map <NUM> detected from the left/ right phase images. For example, a real distance between a lens and a face may fail to be distant from a calculated value in an attack situation using a screen of a smartphone with a small display. In this case, when depth information is calculated using the 2PD disparity map <NUM>, precision of the depth information may be enhanced. According to an embodiment, the depth calculating unit <NUM> may calculate depth information with respect to a face region 710a on the 2PD disparity map <NUM>.

According to various embodiments, the depth calculating unit <NUM> may calculate depth information based on Equation <NUM> below.

Herein, D denotes the depth information, d1 denotes the depth detected on the 2PD disparity map, d2 denotes the depth detected from the RGB image (based on the face size), and W denotes the predefined weight.

According to various embodiments, the depth calculating unit <NUM> may provide a liveness calculating unit (e.g., a liveness calculating unit <NUM> of <FIG>) with the calculated depth information to be used to calculate a liveness score.

According to various embodiments, the depth calculating unit <NUM> may provide a binning determining unit (e.g., a binning determining unit <NUM>) with the calculated depth information. The binning determining unit <NUM> may determine a binning mode depending on the depth information.

<FIG> is a drawing illustrating applying a liveness model according to an embodiment of the disclosure. <FIG> is illustrative, but not limited thereto.

Referring to <FIG>, a liveness identifying unit (e.g., a liveness calculating unit <NUM> of <FIG>) may determine whether a face object (e.g., an object <NUM> of <FIG>) is a living object, based on left/right phase images or RGB data. The liveness calculating unit <NUM> may use liveness models of various algorithms.

According to various embodiments, the liveness calculating unit <NUM> may calculate a first liveness score S1 by applying a 2PD liveness model <NUM> based on a 2PD disparity map using left/right phase images. The 2PD disparity map may be generated in the liveness calculating unit <NUM> or may be generated via a separate component rather than the liveness calculating unit <NUM>.

According to various embodiments, the liveness calculating unit <NUM> may calculate a second liveness score S2 by applying an edge liveness model (or an edge detection model) <NUM> by detection of an edge (a boundary of the object <NUM>) in RGB data The edge detection may be performed in the liveness calculating unit <NUM> or may be performed via a separate component rather than the liveness calculating unit <NUM>.

According to various embodiments, the liveness calculating unit <NUM> may calculate a third liveness score S3 by applying an RGB liveness model <NUM> for RGB data itself.

According to various embodiments, the liveness calculating unit <NUM> may consider information, such as information about a distance from the object <NUM> or information about illumination around the object <NUM>, in an overall manner to determine weights W1, W2, and W3 for liveness models <NUM>, <NUM>, and <NUM>, respectively, and calculate a liveness score. For example, the liveness calculating unit <NUM> may calculate a final liveness score using Equation <NUM> below.

Herein, S denotes the final liveness score, S1 denotes the 2PD liveness model score, S2 denotes the edge liveness model score, S3 denotes RGB liveness model score, and W1, W2 or W3 denotes the weight.

According to various embodiments, the liveness calculating unit <NUM> may adjust the weight W1 of the 2PD liveness model <NUM> or the weight W2 of the edge liveness model <NUM>, based on depth information provided from a depth calculating unit (e.g., a depth calculating unit <NUM> of <FIG>).

For example, when the depth information is greater than or equal to (or is greater than) a reference value, the liveness calculating unit <NUM> may lower the weight W1 of the 2PD liveness model <NUM> by reflecting that the more distant the distance from the object <NUM>, the smaller the phase difference between a first PD and a second PD.

For another example, when the depth information is less than (or is less than or equal to) the reference value, the liveness calculating unit <NUM> may lower the weight W2 of the edge liveness model <NUM> by reflecting that the more close the distance from the object <NUM>, the more the edge region is not included in RGB data. On the other hand, the liveness calculating unit <NUM> may enhance the weight W1 of the 2PD liveness model <NUM> by reflecting that the closer the distance from the object <NUM>, the larger the phase difference between the first PD and the second PD.

The liveness calculating unit <NUM> may compare the calculated liveness score with a predetermined reference value to determine whether the object <NUM> is a real face of a user or a fake face using a photo or an image.

<FIG> illustrates a block diagram of an electronic device in a network environment, according to an embodiment of the disclosure. Electronic devices according to various embodiments disclosed in the disclosure may be various types of devices. An electronic device may include at least one of, for example, a portable communication device (e.g., a smartphone, a computer device (e.g., a PDA: personal digital assistant), a tablet PC, a laptop PC, a desktop PC, a workstation, or a server), a portable multimedia device (e.g., e-book reader or MP3 player), a portable medical device (e.g., heart rate, blood sugar, blood pressure, or body temperature measuring device), a camera, or a wearable device. The wearable device may include at least one of an accessory type device (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head wearable device head-mounted-device (HMD)), a fabric or clothing integral device (e.g., an electronic clothing), a body-attached device (e.g., skin pads or tattoos), or an bio implantable circuit. In some embodiments, the electronic device may include at least one of, for example, a television, a DVD (digital video disk) player, an audio device, an audio accessory device (e.g., a speaker, headphones, or a headset), a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set top box, a home automation control panel, a security control panel, a game console, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include at least one of a navigation device, GNSS (global navigation satellite system), an EDR (event data recorder (e.g., black box for vehicle/ship/airplane), an automotive infotainment device (e.g., vehicle head-up display), an industrial or home robot, a drone, ATM (automated teller machine), a POS (point of sales) instrument, a measurement instrument (e.g., water, electricity, or gas measurement equipment), or an Internet of Things device (e.g. bulb, sprinkler device, fire alarm, temperature regulator, or street light). The electronic device according to the embodiment of the disclosure is not limited to the above-described devices. Further, for example, as in a smart phone equipped with measurement of biometric information (e.g., a heart rate or blood glucose) of an individual, the electronic device may have a combination of functions of a plurality of devices. In the disclosure, the term "user" may refer to a person using the electronic device or a device (e.g., an artificial intelligence electronic device) using the electronic device.

Referring to <FIG>, the electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) in the network environment <NUM> may communicate with an electronic device <NUM> over a first network <NUM> (e.g., a short range wireless communication network) or may communicate with an electronic device <NUM> or a server <NUM> over a second network <NUM> (e.g., a long distance wireless communication network), according to an embodiment of the disclosure. According to an embodiment, the electronic device <NUM> may communicate with the electronic device <NUM> through the server <NUM>. According to an embodiment, the electronic device <NUM> may include a processor <NUM>, a memory <NUM>, an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module <NUM>, or an antenna module <NUM>. In any embodiment, at least one (e.g., the display device <NUM> or the camera module <NUM>) of the components may be omitted from the electronic device <NUM>, or one or more other components may be further included in the electronic device <NUM>. In any embodiment, some of the components may be implemented with a single integrated circuit. For example, the sensor module <NUM> (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be embedded in the display device <NUM> (e.g., a display).

The processor <NUM> may execute, for example, software (e.g., a program <NUM>) to control at least one other component (e.g., a hardware or software component) of the electronic device <NUM> connected to the processor <NUM>, and may perform various data processing or operations. According to an embodiment, as at least a part of the data processing or operations, the processor <NUM> may load a command or data received from any other component (e.g., the sensor module <NUM> or the communication module <NUM>) to a volatile memory <NUM>, may process the command or data stored in the volatile memory <NUM>, and may store processed data in a nonvolatile memory <NUM>. According to an embodiment, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit or an application processor) and an auxiliary processor <NUM> (e.g., a graphic processing device, an image signal processor, a sensor hub processor, or a communication processor), which may be operated independently of or together with the main processor <NUM>. Additionally or alternatively, the auxiliary processor <NUM> may be configured to use lower power than the main processor <NUM> or to be specialized for a specified function. The auxiliary processor <NUM> may be implemented separately from the main processor <NUM> or may be implemented as a part of the main processor <NUM>.

The auxiliary processor <NUM> may control at least a part of a function or states associated with at least one component (e.g., the display device <NUM>, the sensor module <NUM>, or the communication module <NUM>) of the electronic device <NUM>, for example, instead of the main processor <NUM> while the main processor <NUM> is in an inactive (e.g., sleep) state and together with the main processor <NUM> while the main processor <NUM> is in an active (e.g., an application execution) state. According to an embodiment, the auxiliary processor <NUM> (e.g., an image signal processor or a communication processor) may be implemented as a part of any other component (e.g., the camera module <NUM> or the communication module <NUM>) which is functionally (or operatively) associated with the auxiliary processor <NUM>.

The memory <NUM> may store various data which are used by at least one component (e.g., the processor <NUM> or the sensor module <NUM>) of the electronic device <NUM>. The data may include, for example, software (e.g., the program <NUM>), or input data or output data associated with a command of the software. The memory <NUM> may include the volatile memory <NUM> or the nonvolatile memory <NUM>.

The program <NUM> may be stored in the memory <NUM> as software, and may include, for example, an operating system <NUM>, a middleware <NUM>, or an application <NUM>.

The input device <NUM> may receive a commands or data which will be used by a component (e.g., the processor <NUM>) of the electronic device <NUM>, from the outside (e.g., a user) of the electronic device <NUM>.

The sound output device <NUM> may output a sound signal to the outside of the electronic device <NUM>. The speaker may be used for a general purpose such as multimedia play or recording play, and the receiver may be used to receive an incoming call. According to an embodiment, the receiver may be implemented separately from the speaker or may be implemented as a part of the speaker.

The display device <NUM> may visually provide information to the outside (e.g., the user) of the electronic device <NUM>. The display device <NUM> may include, for example, a display, a hologram device, or a control circuit for controlling a projector and a corresponding device. According to an embodiment, the display device <NUM> may include a touch circuitry configured to sense a touch, or a sensor circuitry (e.g., a pressure sensor) configured to measure the strength of force generated by the touch.

The audio module <NUM> may convert sound to an electrical signal, or reversely, may convert an electrical signal to sound. According to an embodiment, the audio module <NUM> may obtain sound through the input device <NUM>, or may output sound through the sound output device <NUM>, or through an external electronic device (e.g., the electronic device <NUM>) (e.g., a speaker or a headphone) directly or wirelessly connected with the electronic device <NUM>.

The sensor module <NUM> may sense an operation state (e.g., power or a temperature) of the electronic device <NUM> or an external environment state (e.g., a user state), and may generate an electrical signal or a data value corresponding the sensed state. According to an embodiment, the sensor module <NUM> may include, for example, a gesture sensor, a grip sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

The interface <NUM> may support one or more specified protocols that may be used to directly and wirelessly connect the electronic device <NUM> with an external electronic device (e.g., the electronic device <NUM>).

A connection terminal <NUM> may include a connector that may allow the electronic device <NUM> to be physically connected with an external electronic device (e.g., the electronic device <NUM>). According to an embodiment, the connection terminal <NUM> may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module <NUM> may convert an electrical signal to a mechanical stimulation (e.g., vibration or movement) or an electrical stimulation which the user may perceive through the sense of touch or the sense of movement. According to an embodiment, the haptic module <NUM> may include, for example, a motor, a piezoelectric sensor, or an electrical stimulation device.

The camera module <NUM> may photograph a still image and a video. According to an embodiment, the camera module <NUM> may include one or more lenses, image sensors, image signal processors, or flashes (or electrical flashes).

The power management module <NUM> may manage the power which is supplied to the electronic device <NUM>. According to an embodiment, the power management module <NUM> may be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

The battery <NUM> may power at least one component of the electronic device <NUM>. According to an embodiment, the battery <NUM> may include, for example, a primary cell not recharged, a secondary cell rechargeable, or a fuel cell.

The communication module <NUM> may establish a direct (or wired) communication channel or a wireless communication channel between the electronic device <NUM> and an external electronic device (e.g., the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) or may perform communication through the established communication channel. The communication module <NUM> may include one or more communication processors which is operated independently of the processor <NUM> (e.g., an application processor) and supports direct (or wired) communication or wireless communication. According to an embodiment, the communication module <NUM> may include a wireless communication module <NUM> (e.g., a cellular communication module, a short range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module <NUM> (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding communication module of such communication modules may communicate with an external electronic device over the first network <NUM> (e.g., a short range communication network such as Bluetooth, Wi-Fi direct, or infrared data association (IrDA)) or the second network <NUM> (e.g., a long distance communication network such as a cellular network, an Internet, or a computer network (e.g., LAN or WAN)). The above-described kinds of communication modules may be integrated in one component (e.g., a single chip) or may be implemented with a plurality of components (e.g., a plurality of chips) which are independent of each other. The wireless communication module <NUM> may verify and authenticate the electronic device <NUM> within a communication network, such as the first network <NUM> or the second network <NUM>, by using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module <NUM>.

The antenna module <NUM> may transmit a signal or a power to the outside (e.g., an external electronic device) or may receive a signal or a power from the outside. According to an embodiment, the antenna module <NUM> may include one or more antennas, and at least one antenna which is suitable for a communication scheme used in a computer network such as the first network <NUM> or the second network <NUM> may be selected, for example, by the communication module <NUM> from the one or more antennas. The signal or power may be exchanged between the communication module <NUM> and an external electronic device through the selected at least one antenna or may be received from the external electronic device through the selected at least one antenna and the communication module <NUM>.

At least some of the components may be connected to each other through a communication scheme (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) between peripheral devices and may exchange signals (e.g., commands or data) with each other.

According to an embodiment, a command or data may be transmitted or received (or exchanged) between the electronic device <NUM> and the external electronic device <NUM> through the server <NUM> connecting to the second network <NUM>. Each of the electronic devices <NUM> and <NUM> may be a device, the kind of which is the same as or different from a kind of the electronic device <NUM>. According to an embodiment, all or a part of operations to be executed in the electronic device <NUM> may be executed in one or more external devices of the external electronic devices <NUM>, <NUM>, or <NUM>. For example, in the case where the electronic device <NUM> should perform any function or service automatically or in response to a request from the user or any other device, the electronic device <NUM> may request one or more external electronic devices to perform at least a part of the function or service, instead of internally executing the function or service or additionally. The one or more external electronic devices which receive the request may execute at least a part of the function or service thus requested or an additional function or service associated with the request, and may provide a result of the execution to the electronic device <NUM>. The electronic device <NUM> may process received result as it is or additionally, and may provide a result of the processing as at least a part of the response to the request. To this end, for example, a cloud computing, distributed computing, or client-server computing technology may be used.

<FIG> is a block diagram illustrating a camera module according to an embodiment of the disclosure.

Referring to <FIG>, the block diagram <NUM> including the camera module (e.g., the camera module <NUM> of <FIG>) <NUM> may include a lens assembly <NUM>, a flash <NUM>, an image sensor <NUM>, an image stabilizer <NUM>, memory <NUM> (e.g., buffer memory), or an image signal processor <NUM>. The lens assembly <NUM> may collect light emitted or reflected from an object whose image is to be taken. The lens assembly <NUM> may include one or more lenses. According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In such a case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. Some of the plurality of lens assemblies <NUM> may have the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may have one or more lens attributes different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens.

The flash <NUM> may emit light that is used to reinforce light reflected from an object. According to an embodiment, the flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor <NUM> may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted via the lens assembly <NUM> into an electrical signal. According to an embodiment, the image sensor <NUM> may include one selected from image sensors having different attributes, such as a RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer <NUM> may move the image sensor <NUM> or at least one lens included in the lens assembly <NUM> in a particular direction, or control an operational attribute (e.g., adjust the read-out timing) of the image sensor <NUM> in response to the movement of the camera module <NUM> or the electronic device <NUM> including the camera module <NUM>. This allows compensating for at least part of a negative effect (e.g., image blurring) by the movement on an image being captured. According to an embodiment, the image stabilizer <NUM> may sense such a movement by the camera module <NUM> or the electronic device <NUM> using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module <NUM>. According to an embodiment, the image stabilizer <NUM> may be implemented, for example, as an optical image stabilizer. The memory <NUM> may store, at least temporarily, at least part of an image obtained via the image sensor <NUM> for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a Bayer-patterned image, a high-resolution image) may be stored in the memory <NUM>, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display device <NUM>. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be configured as at least part of the memory <NUM> or as a separate memory that is operated independently from the memory <NUM>.

The image signal processor <NUM> may perform one or more image processing with respect to an image obtained via the image sensor <NUM> or an image stored in the memory <NUM>. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor <NUM> may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM>. An image processed by the image signal processor <NUM> may be stored back in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display device <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) outside the camera module <NUM>. According to an embodiment, the image signal processor <NUM> may be configured as at least part of the processor <NUM>, or as a separate processor that is operated independently from the processor <NUM>. If the image signal processor <NUM> is configured as a separate processor from the processor <NUM>, at least one image processed by the image signal processor <NUM> may be displayed, by the processor <NUM>, via the display device <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include a plurality of camera modules <NUM> having different attributes or functions. In such a case, at least one of the plurality of camera modules <NUM> may form, for example, a wide-angle camera and at least another of the plurality of camera modules <NUM> may form a telephoto camera. Similarly, at least one of the plurality of camera modules <NUM> may form, for example, a front camera and at least another of the plurality of camera modules <NUM> may form a rear camera.

The electronic device according to various embodiments disclosed in the disclosure may be various types of devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a mobile medical appliance, a camera, a wearable device, or a home appliance. The electronic device according to an embodiment of the disclosure should not be limited to the above-mentioned devices.

According to various embodiments, an electronic device (e.g., the electronic device <NUM> of <FIG>) may include a housing including an opening, and a camera module including at least a portion exposed to an outside through the opening. The camera module may include a lens unit, and an image sensor that converts light introduced through the lens unit into an electrical signal, a magnetic member, a coil unit disposed on a first surface of the magnetic member to face the magnetic member, a magnetic substance unit attached to a second surface of the magnetic member, and a position sensor disposed to be adjacent to the magnetic member to face at least a portion of the magnetic substance part.

According to various embodiments, the position sensor may be disposed to be adjacent to a side surface of the magnetic member perpendicular to the first surface or the second surface.

According to various embodiments, the lens unit may reciprocate in a direction parallel to the first surface or the second surface by electromagnetic force generated by the magnetic member and the coil unit.

According to an embodiment, the magnetic substance unit may be in the planar shape. According to another embodiment, the magnetic substance unit may include a steeped structure. The stepped structure may be formed in an area, which corresponds to a space between the magnetic member and the position sensor, of the magnetic substance unit. The stepped structure may be formed to allow the at least a portion of the magnetic substance unit to protrude toward the position sensor.

According to various embodiments, the position sensor may sense a magnetic flux diverging through the magnetic member or the magnetic substance unit. The position sensor may include a sensing surface disposed in parallel to the first surface or the second surface.

According to various embodiments, the magnetic substance unit may include a first part attached to a first pole of the magnetic member, and a second part attached to a second pole of the second magnetic member. The first part may include a first stepped structure, and the second part may include a second stepped structure. The first stepped structure and the second stepped structure may have equal heights.

According to various embodiments, the position sensor may be a hall sensor.

According to various embodiments, an electronic device may include a housing including an opening and a camera module including at least a portion exposed to an outside through the opening. The camera module may include a lens unit, an image sensor that converts light introduced through the lens unit into an electrical signal, a first magnetic member, a second magnetic member, a first coil unit disposed on a first surface of the first magnetic member to face the first magnetic member, a second coil unit disposed on a first surface of the second magnetic member to face the second magnetic member, a magnetic substance unit attached to a second surface of the first magnetic member and a second surface of the second magnetic member, and a position sensor interposed between the first magnetic member and the second magnetic member to face at least a portion of the magnetic substance unit.

According to various embodiments, the magnetic substance unit may include a first part attached to a first pole of the first magnetic member and a first pole of the second magnetic member and a second part attached to a second pole of the first magnetic member and a second pole of the second magnetic member. The first part may include a first protruding structure, and the second part may include a second protruding structure.

According to various embodiments, the at least a portion of the magnetic substance unit may be in the first protruding structure or the second protruding structure.

According to various embodiments, the camera module may include a lens unit, an image sensor that converts light introduced through the lens unit into an electrical signal, a magnetic member, a coil unit disposed on a first surface of the magnetic member to face the magnetic member, a magnetic substance unit attached to a second surface of the magnetic member, and a position sensor disposed to be adjacent to the magnetic member to face at least a portion of the magnetic substance part.

According to various embodiments, the magnetic substance unit may include a first part attached to a first pole of the magnetic member, and a second part attached to a second pole of the magnetic member.

It should be understood that various embodiments of the disclosure and terms used in the embodiments do not intend to limit technical features disclosed in the disclosure to the particular embodiment disclosed herein; rather, the disclosure should be construed to cover various modifications, equivalents, or alternatives of embodiments of the disclosure. With regard to description of drawings, similar or related components may be assigned with similar reference numerals. As used herein, singular forms of noun corresponding to an item may include one or more items unless the context clearly indicates otherwise. In the disclosure disclosed herein, each of the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "one or more of A, B, and C", or "one or more of A, B, or C" and the like used herein may include any and all combinations of one or more of the associated listed items. The expressions, such as "a first", "a second", "the first", or "the second" may be used merely for the purpose of distinguishing a component from the other components, but do not limit the corresponding components in other aspect (e.g., the importance or the order).

The term "module" used in the disclosure may include a unit implemented in hardware, software, or firmware and may be interchangeably used with the terms "logic", "logical block", "part" and "circuit". The "module" may be a minimum unit of an integrated part or may be a part thereof. The "module" may be a minimum unit for performing one or more functions or a part thereof. For example, according to an embodiment, the "module" may include an application-specific integrated circuit (ASIC).

Various embodiments of the disclosure may be implemented by software (e.g., the program <NUM>) including an instruction stored in a machine-readable storage medium (e.g., an internal memory <NUM> or an external memory <NUM>) readable by a machine (e.g., the electronic device <NUM>). For example, the processor (e.g., the processor <NUM>) of a machine (e.g., the electronic device <NUM>) may call the instruction from the machine-readable storage medium and execute the instructions thus called. This means that the machine may perform at least one function based on the called at least one instruction. The one or more instructions may include a code generated by a compiler or executable by an interpreter. The machine-readable storage medium may be provided in the form of non-transitory storage medium. Here, the term "non-transitory", as used herein, means that the storage medium is tangible, but does not include a signal (e.g., an electromagnetic wave). The term "non-transitory" does not differentiate a case where the data is permanently stored in the storage medium from a case where the data is temporally stored in the storage medium.

According to an embodiment, the method according to various embodiments disclosed in the disclosure may be provided as a part of a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)) or may be directly distributed (e.g., download or upload) online through an application store (e.g., a Play Store<IMG>) or between two user devices (e.g., the smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or generated in a machine-readable storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., the module or the program) of the above-described components may include one or plural entities. According to various embodiments, at least one or more components of the above components or operations may be omitted, or one or more components or operations may be added. Alternatively or additionally, some components (e.g., the module or the program) may be integrated in one component. In this case, the integrated component may perform the same or similar functions performed by each corresponding components prior to the integration. According to various embodiments, operations performed by a module, a programming, or other components may be executed sequentially, in parallel, repeatedly, or in a heuristic method, or at least some operations may be executed in different sequences, omitted, or other operations may be added.

An electronic device (e.g., an electronic device <NUM> of <FIG> or an electronic device <NUM> of <FIG>) according to various embodiments may include a camera module (e.g., a camera module <NUM> of <FIG> or a camera module <NUM> of <FIG>) configured to include an image sensor (e.g., an image sensor <NUM> of <FIG> or an image sensor <NUM> of <FIG>) including a plurality of pixels, a memory (e.g., a memory <NUM> of <FIG>), and a processor (e.g., an object recognition unit <NUM> of <FIG> or a processor <NUM> of <FIG>) configured to control the camera module (e.g., the camera module <NUM> of <FIG> or the camera module <NUM> of <FIG>). Each pixel included in the plurality of pixels may include a plurality of photodiodes and a microlens covering the plurality of photodiodes. The processor may obtain phase images and image data for an external object using the plurality of photodiodes of the image sensor (e.g., the image sensor <NUM> of <FIG> or the image sensor <NUM> of <FIG>) and may authenticate the external object using the phase images and the image data.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may determine a binning mode for the phase images based on depth information about a distance from the external object or parameter information associated with image capture of the image data. The processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may change a size of a pixel region for selecting data of the phase images depending on the binning mode. The parameter information may include at least one of exposure information, sensitivity information or gain information of the camera module (e.g., the camera module <NUM> of <FIG> or the camera module <NUM> of <FIG>).

According to various embodiments, the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) may further include an illumination sensor. The processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may determine a binning mode for the phase images based on illumination information at the time of image capture of the image data, the illumination information being collected by the illumination sensor.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may calculate depth information about the external object based on the phase images and the image data and may determine liveness for the external object based on the calculated depth information.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may calculate the depth information based on a disparity map calculated using the phase images and arrangement information of the external object, the arrangement information being calculated using the image data.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may determine the liveness for the external object based on a first liveness score calculated based on the phase images and a second liveness score and a third liveness score calculated based on the image data.

According to various embodiments, the first liveness score may be calculated using a disparity map calculated using the phase images. The second liveness score may be calculated using edge information of the external object, the edge information being detected from the image data. The third liveness score may be calculated using information about a feature point of the external object, the feature point being detected from the image data.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may determine a first weight for the first liveness score and a second weight for the second liveness score based on the depth information.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may set the first weight to be lower than the second weight, when the depth information is greater than or equal to a predetermined reference value, and may set the first weight to be higher than the second weight, when the depth information is less than the predetermined reference value.

According to various embodiments, the processor (e.g., the object recognition unit <NUM> of <FIG> or the processor <NUM> of <FIG>) may compare authentication information about a user, the authentication information being stored in the memory (e.g., the memory <NUM> of <FIG>), with user information extracted based on the phase images or the image data to authenticate the external object, when the external object is a living object based on determining the liveness.

According to various embodiments, the plurality of pixels may include a first pixel and a second pixel. The first pixel may include a first pixel wall, a first photodiode positioned in a first direction of the first pixel wall, and a second photodiode positioned in a second direction of the first pixel wall. The second pixel may include a second pixel wall, a third photodiode positioned in the first direction of the second pixel wall, and a fourth photodiode positioned in the second direction of the second pixel wall. A first phase image among the phase images may be obtained by the first photodiode and the third photodiode, and a second phase image among the phase images may be obtained by the second photodiode and the fourth photodiode.

A method for recognizing an object in an electronic device (e.g., an electronic device <NUM> of <FIG> or an electronic device <NUM> of <FIG>) according to various embodiments may include obtaining phase images and image data for an external object using an image sensor (e.g., an image sensor <NUM> of <FIG> or an image sensor <NUM> of <FIG>), including a plurality of photodiodes, of the electronic device (e.g., the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) and authenticating the external object using the phase images and the image data.

According to various embodiments, the authenticating may include determining a binning mode for a level selecting data of the phase images based on depth information about the external object or parameter information associated with image capture of the image data.

According to various embodiments, the authenticating may include calculating depth information about the external object based on the phase images and the image data and determining liveness for the external object based on the calculated depth information.

According to various embodiments, the authenticating may include calculating a disparity map using the phase images, calculating arrangement information of the external object using the image data, and calculating the depth information based on the disparity map and the arrangement information.

Claim 1:
An electronic device (<NUM>), comprising:
a camera (<NUM>) comprising an image sensor (<NUM>) including a plurality of pixels (<NUM>);
a memory; and
a processor configured to control the camera,
wherein each pixel (<NUM>) included in the plurality of pixels (<NUM>) includes a plurality of photodiodes (<NUM>)and a microlens (<NUM>) covering the plurality of photodiodes (<NUM>), the plurality of photodiodes (<NUM>) of a pixel (<NUM>) comprising at least a first photodiode (<NUM>) and a second photodiode (<NUM>) in a defined position with respect to the first photodiode (<NUM>), and
wherein the processor is further configured to:
obtain (<NUM>) phase images (<NUM>, <NUM>) and image data for an external object (<NUM>) using the plurality of photodiodes (<NUM>) of the image sensor (<NUM>), wherein the phase images (<NUM>, <NUM>) comprise a first phase image captured by the first photodiodes (<NUM>) of the plurality of pixels (<NUM>) and a second phase image captured by the second photodiodes (<NUM>) of the plurality of pixels (<NUM>),
calculate depth information about the external object (<NUM>) based on a disparity map (<NUM>, <NUM>) calculated using the phase images (<NUM>, <NUM>) and arrangement information of the external object (<NUM>), the arrangement information being calculated using the image data, and
determine (<NUM>) liveness of the external object (<NUM>) based on the calculated depth information.