Patent ID: 12230058

DETAILED DESCRIPTION

Disclosed implementations learn an optimal face tree topology to be used in subsequent components of an image recognition network. However, the potential for differences, such as different poses present in facial images, or different angles of view in images of other objects, can cause erroneous variations in the tree topology, even when the same information is contained in the image. To remedy this, a Deep Attention Generative Adversarial Network (DA-GAN) can be used for face frontalization or other normalization techniques, thus making the disclosed implementations more robust against variations, such as pose variations. Image landmarks, such as facial landmarks in the example of a facial image, are determined from the frontalized images using a deep regression architecture and an integer number of landmarks from each input image.

FIG.1shows a face image along with 4 random trees that were built on the extracted facial landmarks. Traversing these trees results in 4 different sequence-based embeddings that can be fed, for example, to 4 independent LSTM networks for a 7-class FER classification task. The resulting recognition rates are shown below each image inFIG.1. An obtained performance range of 3.7% confirms the importance of finding the optimal tree for further image recognition processing.

FIG.2illustrates a pipeline architecture200in accordance with disclosed implementations, which can be implemented by a programmed computing system. Pipeline architecture200can be divided into 3 functional components: tree topology component210; structure stream component220; and texture stream component230, as shown inFIG.2. In tree topology component210, the tree learning is performed by extraction of facial landmarks by Pose-Weighted Generative Adversarial Network (PW-GAN)212and optimization component214to obtain the optimal tree. The optimized tree can be processed by structure stream component220and/or texture stream component230to learn separate structure and texture representations. The optimized tree is traversed to form a sequence, whose resultant embedding is then fed to structure stream component220. Specifically, Long Short Term Memory network (LSTM)222receives the embedding as an input and the output of LSTM220can be fed to soft attention module224to focus on the salient cells of the LSTM network. LSTMs are a special kind of neural network, capable of learning long-term dependencies. “Soft attention” is when a context vector is calculated as a weighted sum of the encoder hidden states. In texture stream component230, ResNet50 features are extracted from the local patches around extracted image landmarks determined by bounding box process module232, prior to training separate LSTM234and fed to softs attention module236, similar to the structure stream. The resulting embedding is fused with that of the structure stream to yield an expression class prediction. The operation of pipeline architecture200is described in more detail below.

In tree topology component210, a weighted fully connected graph (kn) is constructed from the extracted facial landmarks. The initial weights of the graph can be chosen at random. The optimum minimum-cost spanning tree with regards to these weights is determined, whose traversal generates a sequence specifying an embedding that can be input into an LSTM network for recurrently learning.

The minimum-cost spanning tree can be defined as:

aij⁢∑i=1n∑j=1naij∋G⁡(A)⁢is⁢tree,(1)
where A is an adjacency matrix of graph G(A) and aijis an element of matrix A. The resulting G(A) is a minimum-cost spanning tree. The Prim's algorithm can be used to solve Eq. 1. The Prim's algorithm is a known algorithm that only visits a vertex once in each step. During this visit, an edge with a minimum weight, among all edges that are connected to that vertex, is considered as the selected edge for the minimum-cost spanning tree. By visiting all vertices, Prim's algorithm efficiently builds a “minimum-cost”, e.g. most efficient, spanning tree. The built tree is then traversed using a traversal algorithm such as a preorder depth-first order algorithm, with the starting point of the most centered landmark, e.g., the nose tip. In the traversal, when reaching a leaf node, the traversal backtracks in the tree, and returns to the starting point. The traversal forms a sequence, which in turn is used to form an embedding that is input to structure stream component220and texture stream component235.

FIG.3depicts the tree traversal process and building of the embedding. InFIG.3, the process is illustrated for a tree example with only 10 nodes for clarity. The weighted fully connected graph310can be traversed using a preorder depth-first order traversal technique and the traversal is backtracked when it reaches a leaf node, as noted above. The resulting tree optimization320is a data structure in which the most efficient traversals from each leaf to the root node are identified. The optimized tree in the example ofFIG.3yields a node sequence330of A-B-C-B-A-D-E-F-E-D-A-G-H-I-H-J-H-G-A that is used as embeddings to be input into LSTM network222(see alsoFIG.2).

Tree topology learning module210(FIG.20updates the learnable weights associated with kn. To this end, a combinatorial optimization algorithm as used. For the combinatorial optimizer's objective function, the loss function of the end-to-end pipeline is expressed as follows:

J⁡(w)=1m⁢∑i=1mL⁡(y^(i),yi),(2)
where w is the weights of the complete graph kn, m is the size of the training set, and L(x) is the loss function selected to maximize performance for the specific problem.

At each training epoch, a combinatorial algorithm generates a set of weights associated with kn, from which the minimum-cost spanning tree, whose traversal results in a particular sequence, is generated. Next, this sequence is frozen, and the subsequent structure and texture streams are fully trained. Once training of the two streams is complete, the final loss is fed to the combinatorial optimizer as the objective of an optimization, which then generates a new set of weights for knand this process can continue repeatedly until convergence. It should be noted that the obtained graph is converted to a tree for determining the optimal sequence because the acyclic nature of trees is better suited for traversal, and thus better for sequence generation.

The coordinates of the facial landmarks extracted in tree topology learning component210, along with the obtained sequence, are the inputs to structure stream component, 220, as shown inFIG.2. By using this sequence, an embedding is generated and can be fed to an LSTM network with peephole connections. The embedding can be expressed as:
O(X,Wgraph)=xαxβ. . . xγ,  (3)
where, X is the set of the facial landmarks' coordinates, Wgraphis the weights of the complete graph, and each of xαxβ. . . xγbelongs to the set X. Note that, since the tree traversal algorithm may visit a node more than once, as shown inFIG.3, the output chain may include repeated nodes. Soft attention modules224and236are included to better focus on more important parts of the embeddings by applying learned weights. To this end, the hidden state vectors hiof the LSTM cells are multiplied by a set of coefficients αiwhich determines the level of the attention that it receives and can be calculated using:

αi=eui∑k=1neuk,(4)
where, n is the number of LSTM units, and uiis calculated as:
ui=tanh(Whhi+bh).  (5)
Where Whand bh are the trainable weights and biases. The attentive output can be calculated using:

H=∑iαi⁢hi,(6)
where H is the final embedding of the structure stream.

The inputs of texture stream component230are the facial landmarks and the sequence generated in the face tree topology learning step. However, unlike structure stream component220, where the coordinates of facial landmarks are used to form an embedding for subsequent use in an LSTM network, in texture stream component230, the focus is on the texture of the facial images. To this end, first n×n patches centered around each facial landmark are formed. The patches are then cropped from the images and fed to a ResNet50 pretrained on, for example, the VGG-Face2 dataset. Through empirical experiments, it was found that a patch size of 17×17 pixels yields good results. These embeddings are stacked in accordance with the sequence obtained from the tree topology learning component210. The resulting embedding is then passed onto LSTM network234. This is followed by soft attention process component236in a manner similar to the structure stream (see Equations 4-6 above).

In order to fuse the outputs of the structure and texture streams, a two-stream fusion strategy, such as that disclosed in Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic,Hybrid attention based multimodal network for spoken language classification, Proceedings of the conference. Association for Computational Linguistics, Meeting, volume 2018, page 2379 can be used.FIG.4illustrates a fusion methodology400in accordance with disclosed implementations. This methodology has been designed to preserve the original characteristics of its input embeddings and has yielded state-of-the-art results in a number of areas where preserving and incorporating these characteristics are of particular importance.

Two dense, i.e. non-linear, layers of classifiers are used as encoders410and420, respectively, for each of the input streams (structure and texture). Each of encoders410and420generate stream-specific features. Then, a soft attention component430is applied to learn the weight by using the following equation:
α=softmax(tanh(Wf[T*,S*]+bf)),  (7)
where Wfand bfare trainable weights and biases respectively. T* and S* are the stream-specific features of texture and structure streams, respectively. Next, a dense layer classifier440is utilized to learn the optimal association across the embedding-specific features as:
y=tanh(Wy[(1+αT)T,(1+αS)S]+by),  (8)
where y is the final embedding, Wyand byare trainable weights and biases, and T and S are the output embeddings of texture and structure streams, respectively. The final decision is made by using a softmax classifier with y as its input.

FIG.5illustrates a high level method500for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition in accordance with disclosed implementations. At502image data representing the image of an object is received. At504, N landmarks are detected on the image using a deep regression algorithm, wherein N is an integer. At506, a weighted, fully connected, graph is constructed from the landmarks by assigning initial weights for the landmarks randomly and determining the optimized tree structure based on the initial weights. At508, a sequence is generated by traversing the tree structure. At510, a series of embeddings representing the object image is formed based on the sequence. At512, the embeddings can be inputted into a neural network to thereby generate an image recognition signal based on the embeddings.

Details of a specific example of the disclosed implementation applied to FRE will be discussed below. The entire architecture can be been implemented using TensorFlow and can be trained using a pair of Nvidia RTX 2080 Ti GPUs. For optimizing the weights of the two streams, an ADAM optimizer was used, with the learning rate, first-momentum decay, and second-momentum decay of 0.001, 0.9, 0.99 respectively. All of the reported results were obtained on the validation sets of the respective datasets.

Two popular in-the-wild facial expression datasets were used to evaluate the disclosed implementations. The first dataset, AffectNet includes more than 1 million images, all of which have been obtained from the internet by leveraging 3 search engines. About half of the dataset has been manually annotated in both categorical model for the presence of 7 facial expressions (anger, disgust, fear, happy, sad, surprise, and contempt) plus neutral, and in dimensional model for valence and arousal intensities. The validation set of this dataset includes 500 images for each expression class. The FER2013 dataset includes 28709, 3589, and 3589 images for training, validation, and testing. All the images have been resized to 48×48 pixels. Face images in this dataset are labeled as having one of the 6 basic expressions (anger, disgust, fear, happy, sad, and surprise) along with neutral.

FIG.6illustrates the probability mass function (pmf) of AffectNet and FER2013 datasets, showing that these distributions are highly skewed towards the ‘happiness’ expression. This matter can cause the FER methods to perform poorly on minority classes. To tackle this problem, focal loss was utilized as the loss function L(x) in Eq. 2. Focal
focL=−αb(1−pb)γlog(pb),  (9)
where γ is the focusing parameter and pbis defined as:

pb={py=11-potherwise,(10)
where p is the probability of y=1, which is estimated by
the model. Focal loss causes the learning algorithm to be less focused on the classes with higher number of samples.

The obtained recognition rates by the example (Face Tree Topology) and other state-of-the-art benchmarking methods are presented in Table 1 below for both AffectNet and FER2013 datasets. With respect to the AffectNet dataset, the disclosed implementations show an improvement of 1.52% and 3.28% over the BreG-NeXt50 and BreG-NeXt 32, which are currently the state-of-the-art FER methods. Note that BreG-NeXt 50 and BreG-Next 32 have also utilized focal loss as their loss function. Given the FER2013 dataset, the disclosed implementations outperform the current state-of-the-art FER method, i.e., BreG-Next 50, showing a performance improvement of 1.13%.

DatasetYearMethodRec. RateAffectNet2018CNN for inconsistent label handling57.312018Light CNN582019Ensemble deep learning62.112019Facial mask learning61.502020Expression synthesis from neutral602020BreG-NeXt 3266.742020BreG-NeXt 5068.502021Face tree topology70.02FER20132013CNN + SVM69.32016CNN66.42017New CNN architecture71.22020BreG-NeXt 3269.112020BreG-NeXt 5071.532021Face tree topology72.66

For a side-by-side comparison between the disclosed implementations and BreG-NeXt 50, F1 scores of each expression were outlined by using the one-vs-all method. The results are shown inFIG.7. With respect to the AffectNet dataset, the disclosed implementations (FTNet) have higher F1 scores than BreG-NeXt 50 in all the 8 expression classes except happiness and neutral. As mentioned before, AffectNet is skewed toward happiness and neutral expressions. However, the disclosed implementations have shown better results in the minority expressions which are more difficult to be recognized. With regards to FER2013, FTNet has achieved better F1 scores in 4 out of 7 expressions. The exceptions are surprise, happiness, and disgust expressions, in which BreG-NeXt 50 perform slightly better.

FIG.8shows some of the well-classified and misclassified samples, along with the corresponding input images from the validation sets of AffectNet and FER2013 datasets along with the corresponding predicted and ground truth expressions. It can be seen fromFIG.8that, in the pmf of miss-classified cases, ground truth expression has the 2nd highest probability. This indicates that even in miss-classified cases, the disclosed implementation have a probabilistic sense to the ground truths, and do not generate pmfs which are far away from the ground truth.

FIG.9illustrates an example of the minimum-cost spanning trees, obtained by disclosed implementations for AffectNet and FER2013 datasets. Traversal of these trees provides the optimal sequence to feed the sequence learner, e.g., an LSTM. It is clear fromFIG.9that larger variations between the two trees occurred near the mouth and eyes regions. However, the trees are almost identical around the jaw line. This is not surprising, because the regions around the eyes and mouth are very representative in FER, so facial expressions are more reflected in these regions. As a result, these regions are more sensitive, because they are designed based on the properties of each dataset.

The disclosed implementations can be implemented by various computing devices programmed with software and/or firmware to provide the disclosed functions and modules of executable code implemented by hardware. The software and/or firmware can be stored as executable code on one or more non-transient computer-readable media. The computing devices may be operatively linked via one or more electronic communication links. For example, such electronic communication links may be established, at least in part, via a network such as the Internet and/or other networks.

A given computing device may include one or more processors configured to execute computer program modules. The computer program modules may be configured to enable an expert or user associated with the given computing platform to interface with the system and/or external resources. By way of non-limiting example, the given computing platform may include one or more of a desktop computer, a laptop computer, a handheld computer, a tablet computing platform, a Smartphone, a gaming console, and/or other computing platforms.

The various data and code can be stored in electronic storage devices which may comprise non-transitory storage media that electronically stores information. The electronic storage media of the electronic storage may include one or both of system storage that is provided integrally (i.e., substantially non-removable) with the computing devices and/or removable storage that is removably connectable to the computing devices via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storage may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media.

Processor(s) of the computing devices may be configured to provide information processing capabilities and may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. As used herein, the term “module” may refer to any component or set of components that perform the functionality attributed to the module. This may include one or more physical processors during execution of processor readable instructions, the processor readable instructions, circuitry, hardware, storage media, or any other components.

Although the present technology has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the technology is not limited to the disclosed implementations, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present technology contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.