Patent Description:
Face recognition is a biometric recognition technology for performing identity recognition based on facial feature information of a person, and is important application in the field of artificial intelligence (artificial intelligence, AI). Face recognition uses analysis and comparison of visual feature information of a face to perform identity identification. With rapid development of computer and network technologies, a face recognition technology has been widely applied to many industries and fields, such as intelligent access control, intelligent locks, mobile terminals, public security, entertainment, and military.

Face recognition includes: collecting a face image, detecting a face in the image, and performing face image preprocessing on the detected face, then, extracting an image feature, and performing recognition and matching. The image feature extraction refers to extracting the image feature including a structure and texture from the image, the subsequent recognition and matching are completed based on the image feature, and completeness of the image feature is a key factor that affects success or failure of face recognition.

In the conventional technology, the image feature is extracted based on structure and texture information of an entire face image, and a face recognition effect depends on the completeness of the image feature. When the face image is interfered by an external environment, for example, has uneven light or is blocked by a hat, a scarf, or a mask, a part of the image feature is lost, incompleteness of the image feature results in a low success rate of face recognition.

<CIT> discloses a method for face recognition, to determine whether the face correlates with an enrolled face, including: generating a personalized three-dimensional (3D) face model based on a two-dimensional (2D) input image of the face, acquiring 3D shape information and a normalized 2D input image of the face based on the personalized 3D face model, generating feature information based on the 3D shape information and pixel color values of the normalized 2D input image, and comparing the feature information with feature information associated with the enrolled face.

<CIT> discloses a face feature acquisition method and device, a terminal and a storage medium.

<CIT> discloses a system and a method of face recognition comprising multiple phases implemented in a parallel architecture.

This application provides a face recognition method, a face recognition apparatus and a computer program product, as defined in the appended set of claims, used to recognize a face image, especially a face image that is blocked, to improve a success rate of face recognition.

The invention is carried out according to the appended claims.

Embodiments of this application provide a face recognition method, used to recognize a face image, especially a face image that is blocked, to improve a success rate of face recognition.

Brief introduction to terms in this application is as follows:.

The following describes embodiments of this application with reference to the accompanying drawings. It is clear that the described embodiments are merely some rather than all of embodiments of this application. A person of ordinary skill in the art may learn that, with technology development and emergence of a new scenario, the technical solutions provided in embodiments of this application are also applicable to a similar technical problem.

In the specification, claims, and accompanying drawings of this application, terms "first", "second", and the like are intended to distinguish between similar objects but do not necessarily indicate a specific order or sequence. It should be understood that data termed in such a way are interchangeable in appropriate circumstances, so that embodiments described herein can be implemented in other orders than the order illustrated or described herein. In addition, terms "include", "have", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device including a series of steps or modules is not necessarily limited to those clearly listed steps or modules, but may include other steps or modules that are not clearly listed or are inherent to such a process, method, product, or device. Naming or numbering of steps in this application does not mean that the steps in a method procedure need to be performed in a time/logical order indicated by the naming or numbering. An execution order of the steps in the procedure that have been named or numbered can be changed based on a technical objective to be achieved, provided that same or similar technical effects can be achieved.

<FIG> is a schematic diagram of a main framework of artificial intelligence. The main framework describes an overall working procedure of an artificial intelligence system, and is applicable to a general requirement of the field of artificial intelligence.

The following describes the foregoing main framework of artificial intelligence from two dimensions: an "intelligent information chain" (a horizontal axis) and an "IT value chain" (a vertical axis).

The "intelligent information chain" reflects a series of processes from data obtaining to data processing. For example, the process may be a general process of intelligent information perception, intelligent information representation and formation, intelligent inference, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

The "IT value chain" reflects a value brought by artificial intelligence to the information technology industry from an underlying infrastructure and information (providing and processing technology implementation) of human intelligence to an industrial ecological process of a system.

The infrastructure provides computing capability support for the artificial intelligence system, implements communication with the external world, and implements support by using a base platform. A sensor is configured to communicate with the outside. A computing capability is provided by an intelligent chip (a hardware acceleration chip, for example, a CPU, an NPU, a GPU, an ASIC, or an FPGA). The base platform includes related platform assurance and support such as a distributed computing framework and a network, and may include cloud storage and computing, an interconnection and interworking network, and the like. For example, the sensor communicates with the outside to obtain data, and the data is provided to an intelligent chip in a distributed computing system for computation, where the distributed computing system is provided by the base platform.

Data from a higher layer of the infrastructure indicates a data source in the field of artificial intelligence. The data relates to a graph, an image, a voice, and text, further relates to internet of things data of a conventional device, and includes service data of an existing system and perception data such as force, displacement, a liquid level, a temperature, and humidity.

Data processing usually includes manners such as data training, machine learning, deep learning, searching, inference, and decision-making.

Machine learning and deep learning may perform symbolized and formalized intelligent information modeling, extracting, preprocessing, training, and the like on data.

Inference refers to a process in which a human intelligent inferring manner is simulated in a computer or an intelligent system, and machine thinking and problem solving are performed by using formal information according to an inferring control policy. A typical function is searching and matching.

Decision-making refers to a process in which a decision is made after intelligent information is inferred, and usually provides functions such as classification, ranking, and prediction.

After data processing mentioned above is performed on data, some general capabilities may be further formed based on a data processing result, for example, an algorithm or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image recognition.

Intelligent products and industry applications refer to products and applications of the artificial intelligence system in various fields, and are package of an overall solution of artificial intelligence. Decision making for intelligent information is productized and an application is implemented. Application fields mainly include intelligent manufacturing, intelligent transportation, smart home, smart health care, intelligent security protection, autonomous driving, a safe city, an intelligent terminal, and the like.

Refer to <FIG>. An embodiment of this application provides a system architecture <NUM>. A data collection device <NUM> is configured to collect face image data and store the face image data in a database <NUM>. A training device <NUM> generates a target model/rule <NUM> based on the face image data maintained in the database <NUM>. The following describes in more detail how the training device <NUM> obtains the target model/rule <NUM> based on the face image data. The target model/rule <NUM> can be used in application scenarios such as face recognition, image classification, and virtual reality.

In this embodiment of this application, training may be performed based on the face image data. Specifically, the data collection device <NUM> may be configured to collect various face images, including a face image that is blocked, and store the face images in the database <NUM>. In addition, the face image data may be alternatively directly obtained from a common database, for example, an LFW database, a YaleB database, a CMU PIE database, a CFW database or a Celeba database.

The target model/rule <NUM> may be obtained based on a deep neural network. The following describes the deep neural network.

Work of each layer in the deep neural network may be described by using a mathematical expression: y = a(W · x + b). The work of each layer in the deep neural network may be physically understood as completing transformation from input space to output space (namely, row space to column space of a matrix) by performing five operations on the input space (a set of input vectors). The five operations include: <NUM>. dimension raising/dimension reduction, <NUM>. scaling out/scaling in, <NUM>. rotation, <NUM>. translation, and <NUM>. The operations <NUM>, <NUM>, and <NUM> are performed by W · x, the operation <NUM> is performed by +b , and the operation <NUM> is performed by a(). The word "space" is used herein for expression because classified objects are not single things, but a type of things. Space refers to a collection of all individuals of such type of things. W is a weight vector, and each value in the vector represents a weight value of one neuron in the neural network at this layer. The vector W determines space transformation from the input space to the output space described above. In other words, a weight W of each layer controls how to transform space. A purpose of training the deep neural network is to finally obtain a weight matrix (a weight matrix formed by vectors W of a plurality of layers) of all layers of a trained neural network. Therefore, a training process for the neural network is essentially a manner of learning control of space transformation, and more specifically, learning a weight matrix.

Because an output of the deep neural network is expected to be as much as possible close to a value that actually wants to be predicted, a current predicted value of the network and an expected target value are compared, so that a weight vector of each layer of the neural network can be updated based on a difference between the predicted value and the target value (certainly, there is usually an initialization process before first updating, to be specific, a parameter is preconfigured for each layer of the deep neural network). For example, if the predicted value of the network is large, the weight vector is adjusted to obtain a smaller predicted value, and is continuously adjusted until the neural network can predict the expected target value. Therefore, "how to obtain, through comparison, a difference between the predicted value and the target value" needs to be predefined. This is a loss function (loss function) or an objective function (objective function). The loss function and the objective function are important equations that measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training of the deep neural network becomes a process of reducing the loss as much as possible.

The target model/rule obtained by the training device <NUM> may be applied to different systems or devices. In <FIG>, an I/O interface <NUM> is configured for an execution device <NUM>, to exchange data with an external device. A "user" may input data into the I/O interface <NUM> by using a client device <NUM>.

The execution device <NUM> may invoke data, code, and the like in a data storage system <NUM>, or may store data, instructions, and the like in the data storage system <NUM>.

A computing module <NUM> uses the target model/rule <NUM> to process the input data. Face image recognition is used as an example. The computing module <NUM> may parse the input face image to obtain an image feature, for example, texture information, in the face image.

An associated function module <NUM> may preprocess the image data in the computing module <NUM>, for example, perform face image preprocessing, including face alignment.

Finally, the I/O interface <NUM> returns a processing result to the client device <NUM>, and provides the processing result for the user.

More deeply, the training device <NUM> may generate, for different targets, corresponding target models/rules <NUM> based on different data, to provide a better result for the user.

In a case shown in <FIG>, the user may manually specify data to be input into the execution device <NUM>, for example, may operate on an interface provided by the I/O interface <NUM>. In another case, the client device <NUM> may automatically input data into the I/O interface <NUM> and obtain a result. If the client device <NUM> needs to obtain permission of the user for automatically inputting the data, the user may set corresponding permission on the client device <NUM>. The user may view, on the client device <NUM>, a result output by the execution device <NUM>, and a specific presentation form may be a specific manner, for example, display, a sound, or an action. The client device <NUM> may be also used as a data collection end to store the collected training data in the database <NUM>.

It should be noted that <FIG> is merely a schematic diagram of a system architecture according to an embodiment of this application. A location relationship between devices, components, modules, and the like shown in the figure does not constitute any limitation. For example, in <FIG>, the data storage system <NUM> is an external storage device relative to the execution device <NUM>, and in another case, the data storage system <NUM> may be alternatively disposed in the execution device <NUM>.

A convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure, and is a deep learning (deep learning) architecture. In the deep learning architecture, multilayer learning is performed on different abstract levels by using a machine learning algorithm. As the deep learning architecture, the CNN is a feed-forward (feed-forward) artificial neural network. Image processing is used as an example. Neurons in the feed-forward artificial neural network respond to an overlapping area in an image input into the CNN.

As shown in <FIG>, a convolutional neural network (CNN) <NUM> may include an input layer <NUM>, a convolutional layer/pooling layer <NUM>, and a neural network layer <NUM>. The pooling layer is optional.

As shown in <FIG>, for example, the convolutional layer/pooling layer <NUM> may include layers <NUM> to <NUM>. In an implementation, the layer <NUM> is a convolutional layer, the layer <NUM> is a pooling layer, the layer <NUM> is a convolutional layer, the layer <NUM> is a pooling layer, the layer <NUM> is a convolutional layer, and the layer <NUM> is a pooling layer. In another implementation, the layer <NUM> and the layer <NUM> are convolutional layers, the layer <NUM> is a pooling layer, the layer <NUM> and the layer <NUM> are convolutional layers, and the layer <NUM> is a pooling layer. In other words, an output of a convolutional layer may be used as an input for a subsequent pooling layer, or may be used as an input for another convolutional layer, to continue to perform a convolution operation.

The convolutional layer <NUM> is used as an example. The convolutional layer <NUM> may include a plurality of convolution operators. A convolution operator is also referred to as a kernel. In image processing, the convolution operator functions as a filter that extracts specific information from an input image matrix. The convolution operator may be essentially a weight matrix, and the weight matrix is usually predefined. In a process of performing a convolution operation on an image, the weight matrix is usually used to process pixels at a granularity level of one pixel (or two pixels, which depends on a value of a stride (stride)) in a horizontal direction on the input image, to extract a specific feature from the image. A size of the weight matrix should be related to a size of the image. It should be noted that a depth dimension (depth dimension) of the weight matrix is the same as a depth dimension of the input image. During a convolution operation, the weight matrix extends to an entire depth of the input image. Therefore, a convolution output of a single depth dimension is generated by performing convolution with a single weight matrix. However, in most cases, a plurality of weight matrices of a same dimension rather than the single weight matrix are used. Outputs of the weight matrices are stacked to form a depth dimension of a convolutional image. Different weight matrices may be used to extract different features of the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract a specific color of the image, still another weight matrix is used to blur an unnecessary noise in the image, and so on. Because the plurality of weight matrices have the same dimension, feature maps extracted by using the plurality of weight matrices with the same dimension also have a same dimension. Then, the plurality of extracted feature maps with the same dimension are combined to form an output of the convolution operation.

Weight values in these weight matrices need to be obtained in an actual application through massive training. The weight matrices that are formed based on the weight values obtained through training may be used to extract information from the input image, to help the convolutional neural network <NUM> perform correct prediction.

When the convolutional neural network <NUM> includes a plurality of convolutional layers, a larger quantity of general features are usually extracted at an initial convolutional layer (for example, the convolutional layer <NUM>). The general features may be also referred to as low-level features. As a depth of the convolutional neural network <NUM> increases, a feature extracted at a more subsequent convolutional layer (for example, the convolutional layer <NUM>) is more complex, for example, a high-level semantic feature. A feature with higher semantics is more applicable to a to-be-resolved problem.

Because a quantity of training parameters usually needs to be reduced, the pooling layer usually needs to be periodically introduced after a convolutional layer. To be specific, for the layers <NUM> to <NUM> in the convolutional layer/pooling layer <NUM> shown in <FIG>, one convolutional layer may be followed by one pooling layer, or a plurality of convolutional layers may be followed by one or more pooling layers. During image processing, the pooling layer is only used for reducing a space size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator, to perform sampling on an input image to obtain an image with a small size. The average pooling operator may compute a pixel value in the image within a specific range, to generate an average value. The maximum pooling operator may be used for selecting a pixel with a largest value within a specific range as a maximum pooling result. In addition, similar to a case in which a size of a weight matrix in the convolutional layer should be related to a size of the image, an operator in the pooling layer should be also related to the size of the image. A size of a processed image output from the pooling layer may be less than a size of an image input into the pooling layer. Each pixel in the image output from the pooling layer represents an average value or a maximum value of a corresponding sub-area of the image input into the pooling layer.

After processing is performed at the convolutional layer/pooling layer <NUM>, the convolutional neural network <NUM> still cannot output required output information. As described above, at the convolutional layer/pooling layer <NUM>, only a feature is extracted, and parameters resulting from an input image are reduced. However, to generate final output information (required type information or other related information), the convolutional neural network <NUM> needs to use the neural network layer <NUM> to generate an output of a quantity of one required type or one group of required types. Therefore, the neural network layer <NUM> may include a plurality of hidden layers (<NUM>, <NUM>,. , and 13n shown in <FIG>) and an output layer <NUM>. Parameters included in the plurality of hidden layers may be obtained through pre-training based on related training data of a specific task type. For example, the task type may include image recognition, image classification, super-resolution image reconstruction, and the like.

The plurality of hidden layers included in the neural network layer <NUM> are followed by the output layer <NUM>, namely, the last layer of the entire convolutional neural network <NUM>. The output layer <NUM> has a loss function similar to a categorical cross entropy, and the loss function is specifically used to compute a prediction error. Once forward propagation (for example, propagation from the layers <NUM> to <NUM> in <FIG> is forward propagation) of the entire convolutional neural network <NUM> is completed, reverse propagation (for example, propagation from the layers <NUM> to <NUM> in <FIG> is reverse propagation) is started to update weight values and deviations of the layers mentioned above, to reduce a loss of the convolutional neural network <NUM> and an error between a result output by the convolutional neural network <NUM> by using the output layer and an ideal result.

It should be noted that the convolutional neural network <NUM> shown in <FIG> is merely used as an example of a convolutional neural network. During specific application, the convolutional neural network may alternatively exist in a form of another network model, for example, a plurality of parallel convolutional layers/pooling layers shown in <FIG>, and extracted features are all input into the entire neural network layer <NUM> for processing.

The following describes a diagram of a system architecture for a face recognition method according to an embodiment of this application with reference to <FIG>.

The face recognition method provided in this embodiment of this application is applicable to face recognition in various scenarios such as home and security protection, including use scenarios such as identity verification, personalized customization, and expression simulation of products such as a robot, a smartphone, a desktop computer, a tablet computer, a television, a household or public security surveillance camera, a camera, an access control system, a door lock, an attendance machine, and smart glasses. A face recognition process may be completed by the foregoing entities, or may be completed by a dedicated server by connecting the foregoing entities to the dedicated server through a network. This is not specifically limited herein.

In a scenario in which the server is used to perform face recognition, the network includes one or more of a plurality of types of wireless or partially wireless communication networks, for example, a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a wide area network (WAN), an intranet, the Internet, a peer-to-peer network, a point-to-point network, or a mesh-bonding network. This is not specifically limited herein.

The following describes in detail a face recognition method according to an embodiment of this application with reference to <FIG>.

A face recognition apparatus obtains the face image. Optionally, the face recognition apparatus shoots an image by using a built-in or peripheral camera, and then detects the face image including face information from the shot image.

For example, a robot may obtain the face image by collecting an image by using a camera and detecting that the image includes a face.

The face recognition apparatus preprocesses the face image. An original face image cannot be directly used due to restrictions and random interference of various conditions. Image preprocessing such as grayscale correction and noise filtering need to be performed on the original face image at an early stage of image processing to obtain a preprocessed image for subsequent feature extraction.

The face image preprocessing may include face alignment, light compensation, grayscale transformation, histogram equalization, normalization processing, georectification, median filtering, sharpening, and the like. A specific processing procedure is not limited herein. Normalization processing is used to obtain standardized face images with a same size and a same grayscale value range, and median filtering may be used to perform a smoothing operation on the image to eliminate noise.

Optionally, face alignment processing is performed on the face image, and faces of different scales and in different directions are normalized to a unified scale based on locations of face feature points to obtain an image with a positive face location. The face alignment includes:.

Face frames are normalized to a unified scale. A range of a face in an image may be identified by a face frame area. A size of the face frame is related to a distance between a human body and a camera during image shooting. If the distance is short, a face is large. If the distance is long, a face is small. The face frames can be normalized to the unified scale through face alignment to facilitate subsequent feature extraction.

Face rotation: Because a person in a face image may have different postures, for example, a detected face may be a front face, or may be a side face, face rotation is performed through face alignment, and faces with different postures are rotated to a same angle as much as possible for ease of recognition. Optionally, the faces are rotated to a front direction of a nose.

Face alignment can reduce impact of the distance or the posture on subsequent feature extraction, to recognize the faces on a unified scale.

The face recognition apparatus detects facial geometric feature points in a face image to be recognized. The facial geometric feature points include feature points used to represent face parts, for example, eyes, a nose, a mouth, and a facial contour.

The face recognition apparatus may detect the facial geometric feature points by using a preset algorithm. A specific type of the preset algorithm is not limited herein. In addition, a quantity of extracted facial geometric feature points is not limited, and <NUM> feature points or <NUM> feature points may be extracted. For example, <FIG> is a schematic diagram of facial geometric feature points according to an embodiment of this application. The figure shows <NUM> facial geometric feature points.

A plurality of face part feature point sets may be determined based on target geometric feature points. For example, in <FIG>, a face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a left eyebrow part <NUM>. A face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a right eyebrow part <NUM>. A face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a left eye part <NUM>. A face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a right eye part <NUM>. A face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a nose part <NUM>. A face part feature point set including facial geometric feature points <NUM> to <NUM> is used to indicate a mouth part <NUM>.

A relative location relationship of various parts of a face is fixed, and the parts are divergently distributed by taking the nose as a center, so that a connection relationship between the face part feature point sets is computed by using prior information and the detected facial geometric feature points, to construct a face topology structure and obtain the face topology structure feature.

There are a plurality of methods for obtaining the face topology structure feature.

Optionally, the face topology structure feature is represented as a feature vector set, and a feature vector in the feature vector set is used to indicate a relative location relationship between any two of the plurality of face part feature point sets. <FIG> is a schematic diagram of a face topology structure according to an embodiment of this application.

P represents a face part feature point set, a vector ci,j represent a connection relationship between face part feature point sets i and j.

The face topology structure is represented as [c0,<NUM>, c0,<NUM>, c0,<NUM>, c1,<NUM>, c1,<NUM>, c3,<NUM>, c2,<NUM>, c2,<NUM>, c1,<NUM>]. For example, the vector c0,<NUM> may be obtained based on location coordinates of P0 and location coordinates of P1. A vector representation method only lists a feature relationship having a connection relationship. It may be understood that a connection relationship between two face parts that do not directly have a connection relationship may be also indirectly obtained based on topology structure information.

Normalization is performed on face structure features. Normalization refers to using a unified measurement unit and measurement standard for a feature. For example, the nose is used as an origin, and a distance from an eye to the nose is C0,<NUM>. The distance may use an image coordinate system as a computing system, or the distance may be further normalized to a value between <NUM> and <NUM>.

Optionally, the face topology structure feature is represented as a feature matrix. An element in the feature matrix is used to indicate a relative location relationship between any two of the plurality of face part feature point sets. As shown below, cij that has no connection relationship is <NUM>. For example, there is no connection relationship between p1 and p4 in the foregoing figure, and C1,<NUM> may be represented as <NUM>.

It should be noted that, in addition to a connection relationship shown in <FIG>, another connection relationship may be defined, for example, a Euclidean distance, curvature, or angle. This is not specifically limited herein.

Optionally, a mapping relationship between the plurality of face part feature point sets and face part feature point sets of a standard face is constructed, and the mapping relationship is input into a pre-trained face topology structure feature extraction network to obtain the face topology structure feature. A mapping relationship between each face part in the face image and each corresponding face part in a standard face may be expressed by using a measurement relationship, for example, a distance and/or an angle, and is used to indicate a relative location relationship between the plurality of face part feature point sets.

The standard face is a face image used as a reference standard. Refer to <FIG>, on the middle standard face, at least one feature point is marked on parts such as the eyes, the nose, the eyebrows, and the mouth. For example, the nose may be marked by using one feature point, or may be marked by using a plurality of feature points. A quantity of feature points marked on each part is not specifically limited, and may be specified by a developer. In a case in which a topology structure of feature points of the standard face is known, a location relationship between all feature points in the face image to be recognized and the relative location relationship between the plurality of face part feature point sets may be determined based on a mapping relationship between face part feature point sets in the face image to be recognized and feature point sets of various parts on the standard face.

The mapping relationship may be expressed by using a measurement relationship, for example, a distance and/or an angle. For example, in the facial geometric feature points detected in the face image to be recognized, a measurement relationship between one feature point in a nose part feature point set and one feature point in a nose part feature point set of the standard face is a.

For example, it is assumed that a quantity of facial geometric feature points in the standard face is <NUM>, and a quantity of facial geometric feature points detected in the face image to be recognized is <NUM>. In a planar coordinate system, one feature point may be identified by using two-dimensional data. A data volume of a location relationship between one feature point in the face image to be recognized and the <NUM> points in a standard face image is <NUM> *<NUM>*<NUM>, and a data volume of a location relationship between the <NUM> feature points in the face image to be recognized and the <NUM> points in the standard face image is a data block of <NUM>*<NUM>*<NUM>.

The face topology structure feature may be obtained by inputting data that represents the mapping relationship into the pre-trained face topology structure feature extraction network.

The face recognition apparatus may determine each face part in the face image.

There are a plurality of methods for determining each face part in the face image. A pre-trained face part discrimination model is configured to extract the face part.

Optionally, the face parts extracted from the image include a mouth, a nose, a left eye, a left eyebrow, a right eye, and a right eyebrow. The face parts to be discriminated may be preset. This is not limited herein. <FIG> is a schematic diagram of a face part image and blocking flipping processing according to an embodiment of this application.

Optionally, whether each face part is blocked is determined based on a preset part discrimination model for each face part. If a blocked part is one of paired parts, and the other of the paired parts is not blocked, for example, as shown in <FIG>, one eyebrow or eye is blocked, the other part that is not blocked of the paired parts is horizontally flipped, and is used as a restored image of the current blocked part.

It should be noted that an execution sequence of step <NUM> and step <NUM> is not limited, and step <NUM> may be performed before step <NUM>, or step <NUM> may be performed before step <NUM>.

A complete face image preprocessed in step <NUM> is input into a global face feature extraction network obtained by training a first network for feature extraction, and a global face feature is output.

The face part image obtained in step <NUM> is input into a pre-trained part feature extraction network for feature extraction, and a target part feature is output. It should be noted that a corresponding feature extraction network may be preset for each face part. For a plurality of parts of a face, different part feature extraction networks may be respectively preset for feature extraction, and then the target part feature is obtained by comprehensively using a single part feature extracted for each part.

Optionally, the target part feature is a weighted average value of a plurality of part features, and weight values of the plurality of part features are preset values. Optionally, the face recognition apparatus detects whether a face part in a plurality of face part images are blocked. For a blocked part, a weight value may be reduced based on an initial preset weight value of the part. In this way, impact of part blocking on face recognition can be reduced.

For training processes of the global face feature extraction network, the face topology structure feature extraction network, and the part feature extraction network, refer to an embodiment corresponding to <FIG>.

<NUM>: Perform feature comparison to obtain a face recognition result.

A feature matching module matches a feature of a current face with a face feature in a face database, and obtains the face recognition result based on similarity measurement between the features.

In the face recognition method provided in this embodiment of this application, face recognition is divided into integral face recognition, part recognition, and feature recognition based on structured information. Task dissection can reduce a network volume to some extent, reduce a network depth, and increase a width, so that a feature extraction process is more efficient and effective.

A feature is extracted for a part, to reduce impact of a low-response area, enhance a receptive field of a high-response area, learn more texture information, and enhance judgment and robustness of the feature. A policy with a weight distribution is used during part identification, to reduce impact of a low-response part on a final discrimination result, and improve impact of the high-response area, and is beneficial to face recognition.

When a part is blocked and information of the part is invalid, a face part to be matched is flipped, a feature is extracted, and a blocking discriminator is added, to reduce impact of the blocked part on face recognition.

A feature extraction unit dedicated to learning the face topology structure is proposed to learn a difference between a face and the standard face and a link between topology nodes. By linking information between layers, more structured information is obtained, and impact of blocking on face recognition can be reduced without affecting normal face recognition.

The following describes a feature extraction network training method in the face recognition method provided in this application. Refer to <FIG>.

<NUM>: Obtain a face image from a face database.

A face recognition apparatus obtains the face image from the face database. The face database may be downloaded from the Internet. Alternatively, the face recognition apparatus can establish a part of the database. This is not specifically limited herein. A common face database includes an LFW dataset, a Yale series, and the like.

Optionally, the face database includes a blocked face image, or blocking processing is performed on the face image in the face database to generate a blocked face image. This is not specifically limited herein.

Step <NUM> and step <NUM> are similar to the methods in step <NUM> and step <NUM> in the embodiment corresponding to <FIG>.

<NUM>: Construct a face topology structure.

Refer to step <NUM> in the embodiment corresponding to <FIG>, the mapping relationship between the plurality of face part feature point sets and the face part feature point sets of the standard face is constructed, and the data that represents the mapping relationship is obtained to represent the face topology structure.

<NUM>: Input the face image, the part image, and the face topology structure into different networks respectively for training.

The face image, the part image, and face topology data are respectively input into a network <NUM>, a network <NUM>, and a network <NUM> for training. <FIG> is a schematic diagram of an architecture of a feature extraction network according to an embodiment of this application.

The first network may be any existing global face feature extraction network. This is not specifically limited herein.

For example, a face image with image data of H*W*<NUM> is input, where H is a height of the image, W is a width of the image, and <NUM> represents RGB (red, green, blue) three channels. Optionally, images of a plurality of scales are used as an input for training, so that the images are robust in the plurality of scales.

The input image passes through nine convolutional layers, each block in <FIG> includes three convolutional layers, and a vector [T*<NUM>] is output.

A fully connected layer is followed by a softmax layer (not shown in the figure). An input of this layer is to input the vector [T*<NUM>] into the softmax layer, and an output is also the vector [T*<NUM>]. Each output vector is normalized to [<NUM>, <NUM>]. The vector output by softmax is a probability that a sample belongs to each class.

A loss function L1 of the first network is as follows: <MAT>.

Ls is an interclass difference softmax loss, Lc is an intraclass individual difference center loss, m is a batch size (batch_size), Xi is an ith value of the vector [T*<NUM>], Xj is a jth value, wi is a weight value of convolution, and is learned by reverse propagation, and bi is a corresponding offset.

The input image passes through the network <NUM> to obtain a feature matrix (<NUM>*W dimension). A score corresponding to each image block is a real number in the range of [<NUM>, <NUM>].

A training process of the part feature extraction network obtained by training the network <NUM> corresponds to parts <NUM> and <NUM> identified in <FIG>.

Different face parts are of different importance to face recognition. Generally, a weight of eyes is highest, weights of a nose and a mouth is second highest, and a weight of eyebrows is lowest.

In addition, whether a part is blocked also affects a face recognition effect. If a part is blocked, a weight of the part should be reduced.

In this step, an input of each part network is a part image of H*W*<NUM>, and a part discriminator processes an image of each part to obtain a result about whether each face part is blocked.

The preceding figure shows an example of a part network. A small part network can be used as the part network to extract a feature for a part, to reduce the impact of a low-response area, enhance the receptive field of a high-response area, learn more high-resolution texture information, and enhance judgment and robustness of the feature.

Based on response features of different face parts, an initial weight value is set for each part, and can be defined by a product developer. For example, the weight of the eyes is <NUM>, the weights of the nose and the mouth are <NUM>, and the weight of the eyebrows is <NUM>.

An N*<NUM>-dimension feature vector and a final part weight value wi of each part are obtained through the part network based on image data of a current part and a result about whether the part is blocked that are processed by the part discriminator.

If a part is blocked, a weight is adjusted for decrease based on an initial weight value. For example, if an eye is blocked, a weight value of the eye part is adjusted to <NUM> after passing through a part network.

wi is a weight value of convolution, k represents a quantity of extracted face parts, and Li is a loss computing function of each part. A computing formula is as follows: <MAT>.

θj,i is an angle between vectors Wj,xi , θi,j ∈ (<NUM>, π), and a cosine represents an angle cosine value of a feature vector and a weight vector. A parameter m is used to make an interclass distance large enough and an intraclass distance small enough.

The face topology structure feature extraction network obtained by training the network <NUM> corresponds to a part <NUM> identified in <FIG>.

When a face topology structure is constructed in a standard face manner, data representing a mapping relationship is obtained, and the data is input into the network <NUM> for training to obtain the face topology structure feature extraction network that can be used to obtain a face topology structure feature.

The face topology structure feature extraction network can be obtained by training the network <NUM>. A measurement relationship of <NUM> face feature points in two dimensions is used as an example. A network input is a data block of <NUM>*<NUM>*<NUM>. A differentiated feature between a face and the standard face is extracted and constructed by using a convolutional layer and fully connected layer of a network, and an M*<NUM>-dimension feature vector is output.

A loss function L3 of the network <NUM> is as follows: <MAT>.

m is a batch size (batch_size), n is XX, Xi is an ith value of a vector [T*<NUM>], Xj is a jth value, wi is a weight value of convolution, wj represents XX, and is learned by reverse propagation, bi is an offset corresponding to the ith value, and bj is an offset corresponding to the jth value.

An output Si of the network <NUM>, an output Sj of the network <NUM>, and an output Sk of the network <NUM> are integrated into one output.

Optionally, in a training process, the part feature (network <NUM>) may not be trained. In a recognition process, the face recognition is completed by using the global face feature and the face topology structure feature.

Optionally, in a training process, the global face feature (network <NUM>) may not be trained. In a recognition process, the face recognition is completed by using the face part feature and the face topology structure feature.

The following describes a face recognition apparatus for implementing face recognition. <FIG> is a schematic diagram of an embodiment of a face recognition apparatus according to an embodiment of this application.

The face recognition apparatus includes:.

Optionally, the face topology structure feature includes:.

Optionally, the determining module <NUM> is further configured to:
construct a mapping relationship between the plurality of feature point sets and a plurality of feature point sets of a standard face, where the mapping relationship is used to determine the relative location relationship between the plurality of feature point sets.

The obtaining module <NUM> is specifically configured to input the mapping relationship into a pre-trained face topology structure feature extraction network to obtain the face topology structure feature.

Optionally, the mapping relationship includes a distance and/or an angle between the plurality of feature point sets and the plurality of feature point sets of the standard face.

Optionally, the face topology structure feature extraction network is obtained after a first network is trained.

The extraction module <NUM> is further configured to extract a plurality of facial geometric feature points from a face image training sample to determine a plurality of sample feature point sets. Each of the plurality of sample feature point sets corresponds to one face part of the training sample, and the sample feature point set includes at least one facial geometric feature point.

The obtaining module <NUM> is further configured to obtain a mapping relationship between the sample feature point sets and the feature point sets of the standard face, and input the mapping relationship into the first network for training to obtain a first loss value.

The obtaining module <NUM> is further configured to update a weight parameter in the first network based on the first loss value, to obtain the face topology structure feature extraction network.

Optionally, the extraction module <NUM> is specifically configured to:
input the face image into a pre-trained global face feature extraction network to extract a global face feature.

Optionally, the extraction module <NUM> is specifically configured to:.

Optionally, the first part feature extraction network is obtained after a second network is trained. The obtaining module <NUM> is further configured to:
input a face part image extracted from the face image training sample into the second network for training to obtain a second loss value.

The obtaining module <NUM> is further configured to update a weight parameter in the second network based on the second loss value, to obtain the first part feature extraction network.

The extraction module <NUM> is further configured to:
extract a plurality of face part images from the face image.

The extraction module <NUM> is specifically configured to respectively input the plurality of face part images into a plurality of pre-trained part feature extraction networks to extract a plurality of part features.

The determining module <NUM> is further configured to determine a target part feature of the face image based on the plurality of part features.

Optionally, the target part feature is determined based on a weighted average value of the plurality of part features, and weight values of the plurality of part features are preset values.

The face recognition apparatus further includes:
a detection module <NUM>, configured to detect whether face parts in the plurality of face part images are blocked.

The determining module <NUM> is further configured to determine, if a first face part in a first face part image is blocked, a second face part in a second face part image is not blocked, and the second face part is a symmetric part of the first face part, a horizontally flipped image of the second face part image as a restored image of the first face part. The restored image is used to be input into the part feature extraction network to extract the part feature.

Optionally, the face recognition apparatus further includes:
an updating module <NUM>, configured to update a weight value of a part feature of the first face part based on that the first face part is blocked, where an updated first weight value is less than a preset first weight value of the first face part.

Optionally, the obtaining module <NUM> is further configured to:
preprocess the face image to obtain a preprocessed face image, where preprocessing includes face alignment, and the preprocessed face image is used in extracting of the face image feature and extracting of the plurality of facial geometric feature points.

<FIG> is a diagram of a hardware structure of a chip according to an embodiment of this application.

The convolutional neural network-based algorithms shown in <FIG> and <FIG> may be implemented in an NPU chip shown in <FIG>.

A neural-network processing unit NPU <NUM> NPU, as a coprocessor, is mounted to a host CPU (Host CPU), and the host CPU allocates a task. A core part of the NPU is an operation circuit <NUM>. The operation circuit <NUM> is controlled by a controller <NUM> to extract matrix data in a memory and perform a multiplication operation.

In some implementations, the operation circuit <NUM> internally includes a plurality of process elements (process engine, PEs). In some implementations, the operation circuit <NUM> is a two-dimensional systolic array. Alternatively, the operation circuit <NUM> may be a one-dimensional systolic array or another electronic circuit that can perform mathematical operations such as multiplication and addition. In some implementations, the operation circuit <NUM> is a general-purpose matrix processor.

For example, it is assumed that there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit extracts corresponding data of the matrix B from a weight memory <NUM>, and buffers the corresponding data into each PE in the operation circuit. The operation circuit extracts data of the matrix A from an input memory <NUM>, to perform a matrix operation on the matrix B, and a partial result or a final result of an obtained matrix is stored in an accumulator (accumulator) <NUM>.

A uniform memory <NUM> is configured to store input data and output data. Weight data is transferred to the weight memory <NUM> by using a direct memory access controller (direct memory access controller, DMAC) <NUM>. The input data is also transferred to the uniform memory <NUM> by using the DMAC.

A BIU is a bus interface unit, in other words, a bus interface unit <NUM>, and is configured to perform interaction between an AXI bus, and the DMAC and an instruction fetch buffer <NUM> (Instruction Fetch Buffer).

The bus interface unit (bus interface unit, BIU for short) <NUM> is used by the instruction fetch buffer <NUM> to obtain instructions from an external memory, and is further used by the direct memory access controller <NUM> to obtain original data of the input matrix A or the weight matrix B from the external memory.

The DMAC is mainly configured to transfer input data in the external memory DDR to the uniform memory <NUM>, or transfer the weight data to the weight memory <NUM>, or transfer the input data to an input memory <NUM>.

A vector computing unit <NUM> may include a plurality of operation processing units. If required, further processing is performed on an output of the operation circuit, for example, vector multiplication, vector addition, an exponential operation, a logarithmic operation, or size comparison. The vector computing unit <NUM> is mainly configured to perform network computing at a non-convolution/FC layer in a neural network, for example, pooling (pooling), batch normalization (batch normalization), or local response normalization (local response normalization).

In some implementations, the vector computing unit <NUM> can store a processed output vector in the uniform memory <NUM>. For example, the vector computing unit <NUM> may apply a non-linear function to an output, for example, a vector of an accumulated value, of the operation circuit <NUM> to generate an activation value. In some implementations, the vector computing unit <NUM> generates a normalized value or a combined value or both. In some implementations, the processed output vector can be used as an activation input for the operation circuit <NUM>, for example, used in a subsequent layer in the neural network.

The instruction fetch buffer (instruction fetch buffer) <NUM> connected to the controller <NUM> is configured to store instructions used by the controller <NUM>.

The uniform memory <NUM>, the input memory <NUM>, the weight memory <NUM>, and the instruction fetch buffer <NUM> are all on-chip memories. The external memory is private for the NPU hardware architecture.

An operation at each layer in the convolutional neural network shown in <FIG> and the convolutional neural network <FIG> may be performed by a matrix computing unit <NUM> or the vector computing unit <NUM>.

<FIG> is a schematic diagram of another embodiment of a face recognition apparatus according to an embodiment of this application. The face recognition apparatus provided in this embodiment may include a terminal or a server. A specific device form of the face recognition apparatus is not limited in this embodiment of this application.

The face recognition apparatus <NUM> may differ greatly due to different configurations or performance, and may include one or more processors <NUM> and a memory <NUM>. The memory <NUM> stores a program or data.

The memory <NUM> may be a volatile memory or a non-volatile memory. Optionally, the processor <NUM> is one or more central processing units (Central Processing Units, CPUs). The CPU may be a single-core CPU, or may be a multi-core CPU. The processor <NUM> may communicate with the memory <NUM>, and execute, on the face recognition apparatus <NUM>, a series of instructions in the memory <NUM>.

The face recognition apparatus <NUM> further includes one or more wired or wireless network interfaces <NUM>, for example, an Ethernet interface.

Optionally, although not shown in <FIG>, the face recognition apparatus <NUM> may further include one or more power supplies and one or more input/output interfaces. The input/output interface may be configured to connect to a display, a mouse, a keyboard, a touchscreen device, an image collection module, or another sensing device. The input/output interface is an optional component, and may or may not exist. This is not limited herein.

For a procedure executed by the processor <NUM> in the face recognition apparatus <NUM> in this embodiment, refer to the method procedures described in the foregoing method embodiments.

For example, the foregoing described apparatus embodiments are merely examples. For example, division of the units is merely a logical function division and may be another division during actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or may not be performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in an electrical form, a mechanical form, or another form.

The units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units. To be specific, the components may be located at one location, or may be distributed on a plurality of network units.

When being implemented in the form of a software function unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of this application essentially, or the part contributing to the conventional technology, or all or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in embodiments of this application. The foregoing storage medium includes any medium that can store program code, for example, a USB flash drive, a removable hard disk, a read-only memory (read-only memory, ROM), a random access memory (random access memory, RAM), a floppy disk, or a compact disc.

Claim 1:
A face recognition method, applied to an apparatus, comprising:
obtaining (<NUM>) a face image to be recognized;
extracting (<NUM>) a face image feature based on the face image by using a pre-trained feature extraction network;
extracting (<NUM>) a plurality of facial geometric feature points from the face image to determine a plurality of feature point sets, wherein each of the plurality of feature point sets corresponds to one face part, and the feature point set comprises at least one facial geometric feature point;
obtaining (<NUM>) a face topology structure feature based on the plurality of feature point sets, wherein the face topology structure feature is used to determine a relative location relationship between the plurality of feature point sets; and
performing (<NUM>) matching in a preset face database based on the face topology structure feature and the face image feature, to obtain a face recognition result,
wherein the method further comprises:
extracting a plurality of face part images from the face image; and
the extracting a face image feature based on the face image by using a pre-trained feature extraction network comprises:
respectively inputting the plurality of face part images into a plurality of pre-trained part feature extraction networks to extract a plurality of part features; and
determining a target part feature of the face image based on the plurality of part features.
wherein the target part feature is determined based on a weighted average value of the plurality of part features, and weight values of the plurality of part features are preset values, and
wherein the method further comprises:
detecting whether face parts in the plurality of face part images are blocked; and
determining, if a first face part in a first face part image is blocked, a second face part in a second face part image is not blocked, and the second face part is a symmetric part of the first face part, a horizontally flipped image of the second face part image as a restored image of the first face part, wherein the restored image is used to be input into the part feature extraction network to extract the part feature.