Patent Description:
In a user authentication system, a computing device may determine whether to allow a user to access the computing device based on authentication information provided by the user. For example, the authentication information may include a password input by the user or biometric information of the user. The biometric information includes information related to a fingerprint, an iris, or a face.

Face anti-spoofing technology may be used to improve the security of a user authentication system. Face anti-spoofing determines whether a face of a user input to a computing device is fake or genuine. For this purpose, features such as local binary patterns (LBP), histogram of oriented gradients (HOG), and difference of Gaussians (DoG) may be extracted from an input image, and whether an input face is fake may be determined based on the extracted features. Face spoofing is a form of attack using photographs, moving images or masks, and it is important to identify such attacks in a face verification process.

A liveness test method and apparatus are disclosed in <CIT>. The liveness test method comprises detecting a face region in an input image for a test target, implementing a first liveness test to determine a first liveness value based on a first image corresponding to the detected face region, implementing a second liveness test to determine a second liveness value based on a second image corresponding to a partial face region of the detected face region, implementing a third liveness test to determine a third liveness value based on an entirety of the input image or a full region of the input image that includes the detected face region and a region beyond the detected face region, and determining a result of the liveness test based on at least one of the first liveness value, the second liveness value, and the third liveness value.

In one general aspect, a method includes detecting a face region in an input image; generating, based on the detected face region, weight map data related to a face location in the input image; generating concatenated data by concatenating the weight map data with feature data generated from an intermediate layer of a liveness test model or image data of the input image; and generating a liveness test result based on a liveness score generated by the liveness test model provided with the concatenated data.

The weight map data may include a first region corresponding to the face region in the input image and a second region corresponding to a non-face region in the input image. A weight of the first region and a weight of the second region may be different from each other.

A weight of the weight map data may vary based on a distance from a center of a corresponding region of the weight map data corresponding to the face region.

The weight map data may include a reduced region of the corresponding region, the corresponding region, and an extended region of the corresponding region. The reduced region, the corresponding region, and the extended region may overlap to be disposed based on the center of the corresponding region.

A first weight of the reduced region may be greater than a second weight of a region between the corresponding region and the reduced region, and the second weight may be greater than a third weight of a region between the extended region and the corresponding region.

The liveness test model may include providing the concatenated data to another intermediate layer of the liveness test model, in response to the concatenated data being generated by concatenating the weight map data and the feature data generated from the intermediate layer. The other intermediate layer may be subsequent to the intermediate layer.

The liveness test model may include providing the concatenated data to an input layer of the liveness test model, in response to the concatenated data being generated by concatenating the weight map data and the image data of the input image.

The generating of the weight map data may include generating the weight map data using a neural network-based weight map generation model.

A weight of a first region of the weight map data corresponding to the face region may be different from a weight of a second region of the weight map data corresponding to an occlusion region in the face region.

The generating of the concatenated data may include adjusting a size of the weight map data to correspond to a size of the feature data, and generating the concatenated data by concatenating the feature data and the weight map data of which the size is adjusted.

In a general aspect, embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the any one or any two or more of any combination, or all operations described herein.

In another general aspect, an apparatus includes a processor configured to detect a face region in an input image; generate, based on the detected face region, weight map data related to a face location in the input image; generate concatenated data by concatenating the weight map data with feature data generated from an intermediate layer of a liveness test model or image data of the input image; provide the concatenated data to the liveness test model; and determine a liveness test result based on a liveness score determined by the liveness test model.

The processor may be further configured to provide the concatenated data to another intermediate layer of the liveness test model, in response to the concatenated data being generated by concatenating the weight map data and the feature data generated from the intermediate layer, and the other intermediate layer may be subsequent to the intermediate layer.

The processor may be further configured to provide the concatenated data to an input layer of the liveness test model, in response to the concatenated data being generated by concatenating the weight map data and the image data of the input image.

The processor may be further configured to generate the weight map data using a neural network-based weight map generation model. A weight of a first region of the weight map data corresponding to the face region may be different from a weight of a second region of the weight map data corresponding to an occlusion region in the face region.

The apparatus may further include a memory storing instructions. The processor may be further configured to execute the instructions, which configures the processor to perform the detection of the face region, generation of, based on the detected face region, the weight map data, generation of the concatenated data, and determination of the liveness test result.

In another general aspect, an electronic device includes a camera configured to obtain an input image and a processor. The processor is configured to detect a face region in the input image; generate, based on the detected face region, weight map data related to a face location in the input image; generate concatenated data by concatenating the weight map data with feature data generated from an intermediate layer of a liveness test model to which the input image is input or image data of the input image; provide the concatenated data to the liveness test model; and determine a liveness test result based on a liveness score determined by a liveness test model.

The weight map data may include a first region corresponding to the face region in the input image and a second region corresponding to a non-face region in the input image, and a weight of the first region and a weight of the second region may be different from each other.

In another general aspect, an apparatus includes a processor configured to detect a face region in an input image; generate, based on the detected face region, weight map data related to a face location in the input image, the weight map data including a plurality of regions each having different weights; generate concatenated data by concatenating one of the regions of the weight map data with feature data generated from a layer of a liveness test model; provide the concatenated data to the liveness test model; and determine a liveness test result based on a liveness score determined by the liveness test model.

A first region of the plurality of regions may correspond to the face region and a second region of the plurality of regions may correspond to a non-face region in the input image.

A first region of the plurality of regions may correspond to a reduced region of the face region, a second region of the plurality of regions may correspond to the face region, and a third region of the plurality of regions may correspond to an extended region of the face region.

The liveness test model may be a machine learning model or neural network model.

However, various changes, modifications of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. Also, descriptions of features that are known after understanding of the disclosure of this application may be omitted for increased clarity and conciseness.

Spatially relative terms such as "above," "upper," "below," and "lower" may be used herein for ease of description to describe one element's relationship to another element as shown in the figures. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, an element described as being "above" or "upper" relative to another element will then be "below" or "lower" relative to the other element. Thus, the term "above" encompasses both the above and below orientations depending on the spatial orientation of the device. The device may also be oriented in other ways (for example, rotated <NUM> degrees or at other orientations), and the spatially relative terms used herein are to be interpreted accordingly.

Due to manufacturing techniques and/or tolerances, variations of the shapes shown in the drawings may occur. Thus, the examples described herein are not limited to the specific shapes shown in the drawings, but include changes in shape that occur during manufacturing.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and after an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like elements and a repeated description related thereto will be omitted.

<FIG> and <FIG> illustrate examples of biometric authentication and a liveness test, according to one or more embodiments.

Biometric authentication is an authentication technology that uses personal biometric information such as fingerprint, iris, face, vein, and skin, among authentication technologies, for user verification. In biometric authentication, face verification is an authentication technology that determines whether a user attempting an authentication is a valid user based on the face information of the user. For example, face verification may authenticate a valid user for user login, payment service, and access control.

Referring to <FIG>, an electronic device <NUM> (e.g., an electronic device <NUM> of <FIG>) may authenticate an object <NUM> (e.g., a user) attempting to access the electronic device <NUM> through biometric authentication. The electronic device <NUM> may obtain image data related to the object <NUM> using a camera <NUM> included in the electronic device <NUM> and determine an authentication result by analyzing the obtained image data. A biometric authentication process may include extracting a feature from the image data, comparing the extracted feature with an enrolled feature for a valid object, and determining whether authentication is successful based on the comparison result. For example, assuming the electronic device <NUM> is locked, when it is determined that the authentication of the object <NUM> is successful, the electronic device <NUM> may be unlocked, and conversely, when it is determined that the authentication of the object <NUM> is not successful, the electronic device <NUM> may remain in a locked mode or block access to the object <NUM>. Herein, it is noted that use of the term 'may' with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

A valid user of the electronic device <NUM> may enroll biometric features of the user in advance to the electronic device <NUM> through an enrollment process. The electronic device <NUM> may store information to be used to identify the valid user in a storage device or cloud storage. For example, a face image or a facial feature of the valid user extracted from the face image may be stored as the enrolled biometric feature of the valid user.

A liveness test may be performed in the biometric authentication process described above. The liveness test may examine whether the object <NUM> is animate and determine whether a biometric authentication means is genuine. For example, the liveness test may examine whether a face appearing in an image captured by the camera <NUM> is a genuine or fake face of a person. The liveness test may distinguish between an inanimate object (e.g., a photograph, paper, a moving image, a model, a mask, etc.) and an animate object (e.g., a person's real face). Herein, the term "liveness test" may be replaced with "liveness detection. " Depending on the example, the electronic device <NUM> may perform either one or both of the liveness test and the biometric authentication.

<FIG> illustrates examples of a fake face <NUM> and a genuine face <NUM>, according to one or more embodiments. The electronic device <NUM> may identify whether a face shown in an image is the genuine face <NUM> through a liveness test. In an example, the electronic device <NUM> may distinguish between a fake face <NUM> shown in any one of a screen, a photograph, paper, a model, and the like and the genuine face <NUM> through the liveness test.

Referring to <FIG>, an invalid user may attempt to induce false acceptance of a biometric authentication system using a spoofing technique. For example, in the face verification process, the invalid user may present a substitute, such as a photograph, a moving image, or a model in which a face of a valid user is shown, to the camera <NUM> to induce a false acceptance. The liveness test may filter out an authentication attempt (or a spoofing attack) made with these substitutes to prevent false acceptance. As a result of the liveness test, when an object to be authenticated is determined to be an inanimate object, a final determination may be made that the authentication is not successful, without performing an authentication determination of whether the object is valid. But even in an example where the authentication process is performed, a final determination may be made that the authentication is not successful, regardless of the authentication result.

The electronic device <NUM> may perform the liveness test using face location information in a liveness test process. The electronic device <NUM> may detect a face region in the image data, and use location information of the detected face region to generate weight map data (or face location map data) related to a location of the face region. The electronic device <NUM> may use the weight map data to determine liveness by concatenating the weight map data with the image data or feature data output from a layer of a liveness test model (e.g., a liveness test model <NUM> of <FIG>). The electronic device <NUM> may determine a liveness test result considering features of the face and features around the face including contextual information by performing a liveness test using the image data and the weight map data. The liveness test may be more accurate when the weight map data is used than when the weight map data is not used.

<FIG> is a flowchart illustrating an example of a liveness test method, according to one or more embodiments. The liveness test method may be performed by a liveness test apparatus (e.g., a liveness test apparatus <NUM> of <FIG>).

Referring to <FIG>, in operation <NUM>, the liveness test apparatus may detect a face region in an input image (or a face image). The face region may include the main parts of a face (e.g., an eye, a nose, a mouth, and an eyebrow) shown in the input image, or only a partial region of the face and not necessarily the entire region of the face. The liveness test apparatus may receive the input image to determine liveness and use at least one of various face region detection techniques to detect the face region in the input image. For example, the liveness test apparatus may use a Haar-based cascade AdaBoost classifier, a neural network trained to detect a face region, or a Viola-Jones detector to detect the face region in the input data. However, the scope of examples is not limited thereto, and the liveness test apparatus may detect the face region using various other face region detection techniques. For example, the liveness test apparatus may detect facial landmarks in the input image and detect a bounding region, including the detected landmarks as a face region.

In an example, the liveness test apparatus may detect reference coordinates for defining a reference location of the face region in the input image, a height from the reference location, and a width from the reference location to define the face region. The face region may be detected, for example, as being a square-shaped region, and in this example, the reference coordinates may be two-dimensional coordinates of a location of an upper left vertex of the detected square-shaped region. However, the scope of examples is not limited thereto, and the face region may be detected, for example, as a circle, an ellipse, or a polygon, and the reference coordinates may be defined as a central location or another vertex location of the face region, for example.

In operation <NUM>, the liveness test apparatus may generate weight map data related to a face location in the input image based on the detected face region. The weight map data may represent a weight distribution that depends on the location of the face region in the input image. According to an example, the weight map data generated by the liveness test apparatus may be defined in various forms. For example, the weight map data may include a first region corresponding to the face region in the input image and a second region corresponding to a non-face region in the input image, and a weight allocated to the first region and a weight allocated to the second region may be different from each other. For example, the weight allocated to the first region may be greater than the weight allocated to the second region.

In an example, the weight map data may have a weight that varies according to a distance from a center of a corresponding region of the weight map data corresponding to the face region of the input image. The weight map data may include a reduced region of the corresponding region, the total corresponding region, and an extended region of the corresponding region. The reduced region, the corresponding region, and the extended region may overlap to be disposed based on the center of the corresponding region. Here, a first weight allocated to the reduced region may be greater than a second weight allocated to a region between the corresponding region and the reduced region, and the second weight may be greater than a third weight allocated to a region between the extended region and the corresponding region.

In an example, the liveness test apparatus may generate the weight map data using a neural network-based weight map generation model. The input image may be input to the weight map generation model, and the weight map generation model may output the weight map data related to the location of the face region in the input image. The weight map generation model may be a model trained based on training data (e.g., a training image) and desired weight map data corresponding to the training data. In a training process, the weight map generation model may update parameters (e.g., connection weights) thereof so as to output weight map data most similar to the desired weight map data corresponding to the input training data. When the weight map generation model is used, a weight for an occlusion region generated by obstacles or accessories (e.g., a hat, a mask, sunglasses, and glasses) that may exist in the face region may be determined more accurately. In the weight map data generated by the weight map generation model the weight allocated to the first region of the weight map data corresponding to the face region may be different from the weight allocated to the second region of the weight map data corresponding to the occlusion region in the face region. The weight allocated to the first region may be greater than the weight allocated to the second region.

In operation <NUM>, the liveness test apparatus may generate concatenated data by concatenating the weight map data with feature data output from a first intermediate layer of a liveness test model and image data of the input image. The liveness test model (e.g., the liveness test model <NUM> of <FIG> and a liveness test model <NUM> of <FIG>) may be a model that outputs a liveness score based on the input image and may be based on a neural network (e.g., a convolutional neural network (CNN)). When concatenating the weight map data and the feature data output from the first intermediate layer, the liveness test apparatus may adjust the size (or a resolution) of the weight map data to correspond to the size (or a resolution) of the feature data and generate the concatenated data by concatenating the feature data and the weight map data of which the size is adjusted.

In operation <NUM>, the liveness test apparatus may input the concatenated data to the liveness test model. When the concatenated data is generated by concatenating the weight map data and the feature data output from the first intermediate layer of the liveness test model, the liveness test apparatus may input the concatenated data to a second intermediate layer of the liveness test model. The second intermediate layer may be an upper layer to the first intermediate layer. When the concatenated data is generated by concatenating the weight map data and the image data of the input image, the liveness test apparatus may input the concatenated data to an input layer of the liveness test model.

In operation <NUM>, the liveness test apparatus may determine a liveness test result based on the liveness score determined by the liveness test model. The liveness test model may be a neural network trained to output the liveness score (or a determination of whether an object is live) based on the input data (e.g., the image data of the input image or the concatenated data of the input image and the weight map data). The liveness test model may output a value calculated by intrinsic parameters as the liveness score in response to the input data. The liveness test model may be, for example, a deep neural network (DNN), a CNN, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recurrent DNN (BRDNN), a deep Q-network, or a combination of two or more thereof, but is not limited to the foregoing examples. The liveness test model may be implemented by hardware, including a neural processor, or a combination of hardware and instructions implemented the hardware, e.g., as a processor configured to execute the instructions, which configures the processor to implement the liveness test model.

The liveness score output from the liveness test model may be a reference value for determining whether a test subject is live and represent values, such as a numerical value, a probability value, or a feature value indicating that an object, that is the test subject, corresponds to a genuine object or a fake object based on the input data. In an example, the liveness score may represent a value indicating whether a face object corresponds to a genuine face or a fake face. The liveness test apparatus may determine whether the object is live based on whether the liveness score satisfies a preset condition. For example, the liveness test apparatus may determine the object to be a genuine animate object when the liveness score is greater than a threshold value and an inanimate fake object when the liveness score is less than or equal to the threshold value. In an example, when the liveness score is a value indicating whether the object corresponds to a genuine object or a fake object, the liveness test apparatus may determine whether the object is live based on the liveness score.

<FIG> illustrates an example of generating weight map data, according to one or more embodiments.

Referring to <FIG>, when an input image <NUM> in which an object <NUM> is shown is given a liveness test, a liveness test apparatus may detect a face region <NUM> of the object <NUM> in the input image <NUM>. A location and size of the face region <NUM> may be defined, for example, by coordinates of an upper left vertex of a square that defines the face region <NUM> and the width and height of the square.

The liveness test apparatus may generate weight map data <NUM> by converting location information of the face region <NUM> into a form of a weight map based on the location of the face region <NUM>. The weight map data <NUM> may be the location information of the face region <NUM> converted into a form of a two-dimensional (2D) map and may include weight information that depends on the location of the face region <NUM>.

In the weight map data <NUM>, different weights may be allocated to a corresponding region of the weight map data <NUM> corresponding to the face region <NUM> and a region corresponding to a non-face region. In addition, the weight map data <NUM> may be implemented to have a weight that varies according to a distance from a center of the corresponding region of the weight map data <NUM> corresponding to the face region <NUM>. In the example of <FIG>, the weight map data <NUM> may include a corresponding region (an 1T region) <NUM> corresponding to the face region <NUM>, a reduced region (a <NUM>. 5T region) <NUM> of the corresponding region <NUM>, extended regions (a <NUM>. 5T region and a 2T region) <NUM> and <NUM> of the corresponding region <NUM>, and an outermost region <NUM>. Regions <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may each have a different aspect ratio and overlap to be disposed based on a center of the corresponding region <NUM>.

In the weight map data <NUM>, a weight allocated to each of the regions <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be different from each other. For example, a weight of <NUM> may be allocated to the reduced region <NUM> corresponding to an inner central region of the face region <NUM>, and a weight of <NUM> may be allocated to a region between the corresponding region <NUM> corresponding to the face region <NUM> and the reduced region <NUM>. For example, a weight of <NUM> may be allocated to a region between an extended region <NUM> and the corresponding region <NUM>, and a weight of <NUM> may be allocated to a region between an extended region <NUM> and the extended region <NUM>. For example, a weight of <NUM> may be allocated to a region between the outermost region <NUM> and the extended region <NUM>. In this way, weight may be distributed and the weight map data <NUM> may show the weight gradually decreasing according to a distance from a center of the corresponding region <NUM> corresponding to the face region <NUM>. However, this is merely an example, and the distribution of weight according to the weight map data <NUM> may vary. For example, the distribution of the weight may be continuously changing in proportion to a distance from the corresponding region <NUM> instead of changing in a stepwise manner as in the weight map data <NUM>. When a region has a greater weight in the weight map data <NUM>, region information of the input image <NUM> corresponding to the region may have a relatively higher impact on a liveness test result than that of the input image <NUM> corresponding to another region.

Referring to <FIG>, weight map data <NUM> may be generated by a neural network-based weight generation model <NUM>. An input image <NUM> in which an object <NUM> is shown may be input to the weight map generation model <NUM>, and the weight map data <NUM> corresponding to the input image <NUM> may be output from the weight map generation model <NUM>. The weight map generation model <NUM> may dynamically allocate a weight of the weight map data according to contextual information, such as face occlusion in the object <NUM>, in the input image <NUM>. As shown by the example of <FIG>, when it is assumed that the input image <NUM> shows an occlusion region of a face region created by an obstacle, the weight map data <NUM> generated by the weight map data generation model <NUM> may include a first region <NUM> corresponding to the face region of the object and a second region <NUM> corresponding to the occlusion region in the face region, and a weight allocated to the first region <NUM> and a weight allocated to the second region <NUM> may be different from each other. An identical weight or a weight gradually decreasing as a distance from a center of the first region <NUM> increases may be allocated to the first region <NUM>. Meanwhile, a weight of <NUM> may be allocated to the second region <NUM> corresponding to the occlusion region, and thus, the impact that the occlusion region shown in the input image has on a liveness test result may decrease.

<FIG> illustrates an example of inputting concatenated data to a liveness test model, according to one or more embodiments.

Referring to <FIG>, to perform a liveness test, an input image <NUM> may be input to a liveness test model <NUM>. The liveness test model <NUM> may be, for example, a CNN. When weight map data (e.g., the weight map data <NUM> of <FIG> or the weight map data <NUM> of <FIG>) is generated, a liveness test apparatus may input the weight map data to the liveness test model <NUM>. The liveness test apparatus may generate concatenated data by concatenating feature data <NUM> (e.g., an activation map or vector data) output from a first intermediate layer <NUM> (e.g., a convolutional layer) of the liveness test model <NUM> and input the concatenated data to a second intermediate layer <NUM> of the liveness test model <NUM>. To generate the concatenated data, feature data of another intermediate layer other than the first intermediate layer <NUM> may be used. For example, the second intermediate layer <NUM> may be an upper layer right above the first intermediate layer <NUM>.

In generating the concatenated data, when the feature data <NUM> and the weight map data <NUM> each have a different size (for example, when the feature data <NUM> and the weight map data <NUM> each have different horizontal and vertical lengths), the liveness test apparatus may adjust a size of the weight map data <NUM> to correspond to a size of the feature data <NUM> to concatenate the weight map data <NUM> and the feature data <NUM>. The liveness test apparatus may generate the concatenated data by concatenating the feature <NUM> and the weight map data of which the size is adjusted.

The liveness test model <NUM> may output a liveness score based on the input image <NUM> and the concatenated data input to the second intermediate layer, and the liveness test apparatus may determine a liveness test result based on the liveness score. Using the weight map data <NUM> may enable the face region of the input image <NUM> to have a larger impact on the liveness test result than another region, and accordingly, the accuracy of the liveness test result may be improved.

Referring to <FIG>, after weight map data <NUM> (e.g., the weight map data <NUM> of <FIG> or the weight map data <NUM> of <FIG>) is generated, the liveness test apparatus may generate concatenated data by concatenating the weight map data <NUM> and the image data of the input image <NUM> and input the concatenated data to an input layer of a liveness test model <NUM>. The liveness test model <NUM> may output a liveness score based on the input concatenated data, and the liveness test apparatus may determine a liveness test result based on the liveness score.

<FIG> is a block diagram illustrating an example of a configuration of a liveness test apparatus, according to one or more embodiments.

Referring to <FIG>, a liveness test apparatus <NUM> may perform a liveness test on an object shown in an input image. The liveness test apparatus <NUM> may include a processor <NUM> and a memory <NUM>.

The memory <NUM> may store a variety of data used by a component (e.g., the processor <NUM>). The variety of data may include, for example, instructions and input data or output data. The memory <NUM> may include either one or both of a volatile memory and a non-volatile memory. Execution of such instructions by the processor <NUM> may configure the processor to implement any one, any combination, or all operations and/or methods described herein.

The processor <NUM> may execute instructions to perform operations of the liveness test apparatus <NUM>. The processor <NUM> may execute, for example, instructions to configure the processor to control at least one other component (e.g., a hardware or hardware-implemented software component) of the liveness test apparatus <NUM> connected to the processor <NUM> and perform various types of data processing or operations.

As at least part of the data processing or operations, the processor <NUM> may store instructions or data in the memory <NUM>, process the instructions or data stored in the memory <NUM>, and store result data in the memory <NUM> or a storage device <NUM>. The processor <NUM> may include a main processor (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor (e.g., a graphics processing unit (GPU) and a neural processing unit (NPU)) that is operable independently of, or in conjunction with the main processor.

The processor <NUM> may perform one or more operations described or illustrated herein in connection with a liveness test. For example, the processor <NUM> may detect a face region in an input image, that is a liveness test subject, and generate weight map data related to a face location in the input image based on the detected face region. The processor <NUM> may generate concatenated data by concatenating the weight map data with feature data output from a first intermediate layer of a liveness test model or image data of the input image and input the concatenated data to the liveness test model. When the concatenated data is generated by concatenating the weight map data and the feature data output from the first intermediate layer, the processor <NUM> may input the concatenated data to a second intermediate layer, which is an upper layer to the first intermediate layer of the liveness test model. When the concatenated data is generated by concatenating the weight map data and the image data of the input image, the processor <NUM> may input the concatenated data to an input layer of the liveness test model. Then, the processor <NUM> may determine a liveness test result based on a liveness score determined by the liveness test model.

<FIG> is a block diagram illustrating an example of a configuration of an electronic device, according to one or more embodiments.

Referring to <FIG>, an electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may be an electronic device in any of various forms. For example, the electronic device <NUM> may be a smartphone, a tablet computer, a personal digital assistant (PDA), a netbook, a laptop, a product inspection device, a personal computer, a wearable device (e.g., augmented reality (AR) glasses, a head mounted display (HMD), a smart car, a smart home appliance, a security device, an automated teller machine, or a server device, but is not limited thereto. The electronic device <NUM> may perform a liveness test on an object.

The electronic device <NUM> may include a processor <NUM>, a memory <NUM>, a camera <NUM>, a sensor <NUM>, an input device <NUM>, an output device <NUM>, and a communication device <NUM>. At least some of the components of the electronic device <NUM> may be coupled mutually and communicate signals (e.g., instructions or data) therebetween via an inter-peripheral communication interface <NUM> (e.g., a bus, general purpose input and output (GPIO), a serial peripheral interface (SPI), a mobile industry processor interface (MIPI)).

The processor <NUM> may control overall operations of the electronic device <NUM> and execute functions and instructions to be executed within the electronic device <NUM>. The processor <NUM> may perform operations of the liveness test apparatus described herein (e.g., the liveness test apparatus <NUM> of <FIG>).

The memory <NUM> may store the instructions executable by the processor <NUM> and input/output data. The memory <NUM> may include a volatile memory such as a random-access memory (RAM), a dynamic random-access memory (DRAM), and a static random-access memory (SRAM) and/or a non-volatile memory known in the art such as a read-only memory (ROM) and a flash memory.

The camera <NUM> may capture an image. The camera <NUM> may obtain, for example, a color image, a black and white image, a gray image, an infrared image or a depth image. The camera <NUM> may obtain an input image in which an object is shown, and the processor <NUM> may perform a liveness test based on the obtained input image.

The sensor <NUM> may detect an operational state (e.g., power or temperature) of the electronic device <NUM> or an external environmental state (e.g., a state of a user), and generate an electrical signal or data value corresponding to the detected state. The sensor <NUM> may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The input device <NUM> may receive a user input from a user through a video, audio, or touch input. The input device <NUM> may include, for example, a keyboard, a mouse, a touch screen, a microphone, or any other device that transmits a user input to the electronic device <NUM>.

The output device <NUM> may provide an output of the electronic device <NUM> to the user through a visual, auditory, or haptic channel. The output device <NUM> may include, for example, a liquid crystal display or a light emitting diode (LED)/organic light emitting diode (OLED) display, a micro LED, a touch screen, a speaker, a vibration generating device, or any other device capable of providing the output to the user.

The communication device <NUM> may support the establishment of a direct (or wired) communication channel or a wireless communication channel between the electronic device <NUM> and an external electronic device, and support the communication through the established communication channel. According to an example, a communication module may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). The wireless communication module may communicate with the external device via a short-range communication network (e.g., Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or a long-range communication network (e.g., a legacy cellular network, a <NUM> network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)).

<FIG> illustrates an example of training of a liveness test model, according to one or more embodiments.

A liveness test model described herein may have parameters (e.g., connection weights) determined through a training process. Referring to <FIG>, in a training process, a large quantity of training data <NUM> (e.g., a training image) and a desired value corresponding to each piece of the training data <NUM> is provided. In the training process, a liveness test model <NUM> may receive the training data <NUM> and output a result value corresponding to the training data <NUM> through a calculation process based on the parameters. Here, a weight map generator <NUM> may generate weight map data corresponding to the training data <NUM> based on the training data <NUM>. The weight map generator <NUM> may generate the weight map data as described in <FIG> or <FIG>, and thus, detailed descriptions thereof are not provided here.

Depending on the example, the weight map data generated by the weight map generator <NUM> may be concatenated with feature data output from a first intermediate layer of a liveness test model <NUM> or the current training data <NUM>. When concatenated data is generated by concatenating the feature data of the first intermediate layer and the weight map data, the concatenated data may be input to a second intermediate layer right above the first intermediate layer. When concatenated data is generated by concatenating the training data <NUM> and the weight map data, the concatenated data may be input to an input layer of the liveness test model <NUM>.

A training apparatus <NUM> may update parameters of the liveness test model <NUM> based on a result value output from the liveness test model <NUM>. The training apparatus <NUM> may calculate a loss based on a difference between the result value output from the liveness test model <NUM> and a desired value corresponding to the training data <NUM> and adjust the parameters of the liveness test model <NUM> to reduce the loss. Various loss functions may be used to calculate a loss and adjustment of parameters may be performed by, for example, a back propagation algorithm. The training apparatus <NUM> may iteratively perform this process for each piece of the large quantity of the training data <NUM>, and accordingly, the parameters of the liveness test model <NUM> may be desirably adjusted in a gradual manner. The training apparatus <NUM> may train the liveness test model <NUM> using various machine learning algorithms in addition to the training method described herein.

When the weight map generator <NUM> is a neural network-based weight map generation model, the training apparatus <NUM> may also train the weight map generator <NUM>. Here, the training data <NUM> and desired weight map data corresponding to the training data <NUM> are provided, and the training apparatus <NUM> may update parameters of the weight map generator <NUM> for the weight map generator <NUM> to output weight map data most similar to the desired weight map data corresponding to the training data <NUM> input to the weight map generator <NUM>.

The liveness test apparatus, processor, sensor, electronic device, liveness test apparatus <NUM>, processor <NUM>, <NUM>, electronic device <NUM>, <NUM>, sensor <NUM>, communicative device <NUM>, weight map generator <NUM>, and training apparatus <NUM>, in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A processor-implemented method, comprising:
detecting (<NUM>) a face region (<NUM>) in an input image (<NUM>);
generating (<NUM>), based on the detected face region, weight map data (<NUM>) related to a face location in the input image;
generating (<NUM>) concatenated data by concatenating the weight map data with feature data generated from inputting the input image to an intermediate layer of a liveness test model or image data of the input image, the liveness test model comprising a convolutional neural network;
providing (<NUM>) the generated concatenated data to a second intermediate layer of the liveness test model that is above the first intermediate layer or to an input layer of the liveness test model, respectively; andgenerating (<NUM>) a liveness test result based on a liveness score generated by the liveness test model provided with the concatenated data,
characterised in that
the weight map data comprises a first region corresponding to the face region in the input image and a second region corresponding to a non-face region in the input image; and
a weight of the first region and a weight of the second region are different from each other, and a weight of the weight map data varies based on a distance from a center of the face region.