---
abstract: 'State-of-the-art methods of attribute detection from faces almost always assume the presence of a full, unoccluded face.  Hence, their performance degrades for partially visible and occluded faces.  In this paper, we introduce SPLITFACE, a deep convolutional neural network-based method that is explicitly designed to perform attribute detection in partially occluded faces.  Taking several facial segments and the full face as input, the proposed method takes a data driven approach to determine which attributes are localized in which facial segments.  The unique architecture of the network allows each attribute to be predicted by multiple segments, which permits the implementation of committee machine techniques for combining local and global decisions to boost performance.  With access to segment-based predictions, SPLITFACE can predict well those attributes which are localized in the visible parts of the face, without having to rely on the presence of the whole face.  We use the CelebA and LFWA facial attribute datasets for standard evaluations.  We also modify both datasets, to occlude the faces, so that we can evaluate the performance of attribute detection algorithms on partial faces.  Our evaluation shows that SPLITFACE significantly outperforms other recent methods especially for partial faces.' 
author:
- |
    Upal Mahbub$^{*}$[^1] Sayantan Sarkar$^{*}$ Rama Chellappa\
    Department of Electrical and Computer Engineering and the Center for Automation Research,\
    UMIACS, University of Maryland, College Park, MD 20742\
bibliography:
- 'FSAD.bib'
title: 'Segment-based Methods for Facial Attribute Detection from Partial Faces'
---

attribute detection, facial segment, committee machines, score fusion, local to global decision propagation;

Introduction and Motivation
===========================

The problem of attribute detection from face images has received much attention from the computer vision community in recent years [@JointFaceAndAttribDetect_He_2017] [@MOON_Rudd2016] [@EmilyNet_AAI_Multitask] [@HuHan_DeepMultiTaskHeteroAttrib].  Successful detection of facial attributes has numerous practical applications, such as user-verification [@KumarBelhumer_FaceVerificationFromAttribute2009] and image search [@KumarBelhumeurImagesearch2011], video surveillance [@videoSurveillance2009], age and gender estimation to assist salutation for HCI [@EmilyNet_AAI_Multitask], and facial expression estimation for mood analysis [@EmilyNet_AAI_Multitask].  Most attribute detection algorithms assume the availability of a full, near frontal and aligned face, and we find that their performance degrades significantly in domains where partially visible faces are frequent.  One such domain is front-camera images of smartphones, which are used for continuous active authentication of users [@VMP_SPM_AA_2016] [@SegFaceDeepSegFace_FG2017]. 

To develop a method that detects attributes from full as well as partial faces, we consider the following key observations:

-   Some attributes can be inferred correctly even if the face is partially occluded.  For example, it is possible for humans to infer the gender from only the left half or upper half of the face. 

-   Some attributes are strongly localized in certain part of the face, such as beard or mustache can only be inferred from the lower half of the face. 

Given this observations, it is desirable that a technique for attribute detection be designed, whose performance degrades gracefully with increasing occlusion, rather than suffer catastrophic failures.  In this paper, we present a two-step deep convolutional neural network-based method for facial attribute detection that takes into account the relative strength of different facial segments in detecting different facial attributes.  We analyze the detection results obtained in the first step where all facial segments were tasked to decide the attributes.  We then present a method to automatically assign selective sets of attributes to different facial regions, resulting in a performance boost in the second step.  We also determine the appropriate thresholds for deciding on each attribute at each segment based on the detection results obtained from the validation set.  Finally, we combine the predictions from different facial segments and produce the final result.  Some special features of the proposed algorithm are:

-   We have implemented a local to global attribute detection approach that harnesses the strength of different facial segments into determining different attributes.  For example, bottom-half of a face has information about the beard, while the upper-half has information about the hair.  Our divide and conquer approach extracts intermediate results from each segment and combines them in the end to boost the overall performance. 

-   Not all facial segments have to be present for the proposed method to work.  The method relies on the whole face and one or more facial segments to estimate all the attributes.  The individual facial segments are self-sufficient for estimating the attributes they are assigned to.  Hence, the method demonstrates superior performance when the full face is not visible due to partial occlusion or pose variation. 

-   We analyze the local aspects of facial attributes by associating them with facial segments and develop an automated method to utilized the local information. 

-   It is a well known fact that an ensemble of networks generally outperform a single network.  However training an ensemble is very time-consuming.  Our proposed architecture provides us with $16$ predictors with only one round of training.  We show that a significant increase in the final result is achieved by combining scores from these predictors. 

In section \[RelatedWorks\], a summary of related works done on facial attribute detection is given.  In sections \[ProposedMethod\], the proposed Segmentwise, Partial, Localized Inference in Training Facial Attribute Classification Ensembles network is described in detail.  All the analyses and experimental results for the proposed methods and comparisons with state-of-the-art methods are provided in section \[ExperimentalResults\].  Finally, a brief summary of this work as well as future directions of research are included in section \[Conclusion\]. 

Related Works {#RelatedWorks}
=============

There has been significant amount of research on attribute extraction starting from learning separate models for each attribute [@Huo2016DeepAD] [@AgeAndGender_Eidinger] to jointly learning multiple attributes in a multi-task learning fashion [@JoingAgeGenderEthnicity_Guo_FGNonCNN] [@MOON_Rudd2016] [@EmilyNet_AAI_Multitask] [@JointFaceAndAttribDetect_He_2017] [@HuHan_DeepMultiTaskHeteroAttrib].  Multi-task optimization is found to improve performance in comparison to training independent models for each attribute detection task[@EmilyNet_AAI_Multitask] [@MOON_Rudd2016]. 

In recent times, the research on attribute detection mostly revolves around two challenging, publicly available datasets namely, CelebA and LFWA [@CelebA_liu2015faceattributes].  Both datasets have annotations for forty different attributes along with identity information.  The CelebA dataset contains $162,770$ images for training, $19,867$ image for validation and $19,962$ more for testing.  It is a very challenging dataset with wide variations in pose, illumination and image quality.  The LFWA dataset is a much smaller dataset with $6263$ training and $6880$ test images.  The datasets are introduced in [@CelebA_liu2015faceattributes] where the authors proposed a cascaded system of two DCNNs to jointly perform face localization and attribute detection.  In [@MOON_Rudd2016] the authors addressed the multi-label imbalance problem of the CelebA dataset and proposed a mixed objective optimization network (MOON) that utilizes a unique loss function comprised of a mixed multitask objective with domain adaptive re-weighting. 

Some authors, such as in [@EmilyNet_AAI_Multitask] [@HuHan_DeepMultiTaskHeteroAttrib], categorized the attributes into different groups to take advantage of their mutual relationships.  The authors in [@EmilyNet_AAI_Multitask] suggested an auxiliary network on top of the multi-task DCNN to further exploit the relationships among the attributes.  On the other hand, the authors in [@HuHan_DeepMultiTaskHeteroAttrib] defined a modified AlexNet with both shared and category-specific feature learning to assist attribute extraction. 

Some researchers also implemented the attribute detection task as an auxiliary task of another task.  For example, in [@JointFaceAndAttribDetect_He_2017], the authors proposed a DCNN architecture similar to Faster RCNN [@FasterRCNN_NIPS2015] with additional losses for joint detection of face and associated facial attributes without requiring explicit face alignment.  However, the method does not address partial face detection, which is a challenging problem in itself [@DRUID_Umahbub].  Other notable attribute detectors for unaligned face are proposed in [@UnalignedFA_HuiDing] and [@AFFECT_AlignmentFreeAttrib_Boult].  In [@UnalignedFA_HuiDing], the authors proposed a cascade network to concurrently localize face regions to different attributes and perform attribute classification.  While this method might be suitable for attribute extraction from partially visible faces if trained properly, the authors presented no such extension or analysis.  Also, the original network is huge, consisting of separate DCNN branches for each of the $40$ attributes and therefore not easily scalable.  In [@AFFECT_AlignmentFreeAttrib_Boult], the authors introduced a data augmentation technique to assist attribute detection from unaligned faces.  They improved the detection performance by augmenting the test data and combining the results.  Even though their reported accuracy using an ensemble network of three ResNets is very good on unaligned faces, the architecture does not incorporate any mechanism for partially visible faces and also require combining scores from $162$ transformations of the test image to achieve the best performance. 

Proposed Method {#ProposedMethod}
===============

The basis of our approach is dividing the task of detecting attributes among different segments of the face.  By segments, we mean portions of the face such as left half, right half, upper half, bottom half, nose segment etc.  We divide a face into $14$ such facial segments (adopted from [@FSFD_Mahbub]) using 21 fiducial keypoints (as shown in Fig.  \[Fiducials21\]) which are upper-left-half (UL12), upper-half (U12), upper-right-half (UR12), upper-left-three-fourth (UL34), upper-three-forth (U34), upper-right-three-fourth (UR34), left-half (L12), left-three-fourth (L34), eye-pair (EP), nose region (NS), right-half (R12), right-three-fourth (R34), bottom-three-fourth (B34), and bottom-half (B12).  Let us denote the points shown in Fig.  \[Fiducials21\] with $(x_k, y_k)$ where $x_k$ and $y_k$ are the horizontal and vertical pixel distances from the $(0,0)$ pixel coordinate (top-left corner) of an image image of width $W$ and height $H$, and $k\in\{\mbox{TL, BR},1, 2, \hdots, 21\}$. TL and BR corresponds to Top-Left and Bottom-Right coordinate of the full face bounding box.  The fiducials and full face bounding boxes are obtained from All-in-One Face [@UltraFace_RRanjan] along with visibility scores $v_k$ where $k\in\{\mbox{TL, BR},1, 2, \hdots, 21\}$. Now, $\forall v_j\vert_{j=1}^{21}\geq\tau$ where $\tau$ is a visibility threshold, the bounding boxes of segments $\boxdot_L$, were $L\in\{\mbox{UL12, UR12}, \hdots, \mbox{B12}\}$are defined as $$\begin{aligned}
\boxdot_{\mbox{EP}}&=&[max(x_{\mbox{TL}}, min(x_i\vert _{i=1}^{12})), \nonumber\\ && max(y_{\mbox{TL}}, min(y_i\vert _{i=1}^{12})-\Delta_{\mbox{EP}}),\nonumber\\&& min(x_{\mbox{BR}}, max(x_i\vert _{i=1}^{12})), \nonumber\\&& min(y_{\mbox{BR}}, max(y_i\vert _{i=1}^{12})+\Delta_{\mbox{EP}})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{NS}}&=&[max(x_{\mbox{TL}}, min(x_i\vert_{i\in\{8, 14, 15, 16, 18\}})),\nonumber\\ &&max(y_{\mbox{TL}}, max(0, \frac{1}{3}\sum_{i=14}^{16}y_i - 2\Delta_{\mbox{NS}})),\nonumber\\ &&min(x_{\mbox{BR}}, max(x_i\vert_{i\in\{11, 14, 15, 16, 20\}})) \nonumber\\&& min(y_{\mbox{BR}}, max(H, \frac{1}{3}\sum_{i=14}^{16}y_i + 2\Delta_{\mbox{NS}}))]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{UL12}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, max(x_i\vert_{i\in\{3, 9, 14, 15, 19\}}), \nonumber\\&&max(y_i\vert_{i=14}^{16})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{U12}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, x_{\mbox{BR}}, max(y_i\vert_{i=14}^{16})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{UR12}}&=&[min(x_i\vert_{i\in\{4, 10, 15, 16, 19\}}), y_{\mbox{TL}}, \nonumber\\&& x_{\mbox{BR}}, max(y_i\vert_{i=14}^{16})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{UL34}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, max(x_i\vert_{i\in\{5, 11, 16, 20\}}), \nonumber\\&&max(y_i\vert_{i=18}^{20})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{U34}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, x_{\mbox{BR}}, max(y_i\vert_{i=18}^{20})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{UR34}}&=&[min(x_i\vert_{i\in\{2, 8, 14, 18\}}), y_{\mbox{TL}}, \nonumber\\&& x_{\mbox{BR}}, max(y_i\vert_{i=18}^{20})]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{L12}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, max(x_i\vert_{i\in\{3, 15, 19\}}), y_{\mbox{BR}}]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{L34}}&=&[x_{\mbox{TL}}, y_{\mbox{TL}}, max(x_i\vert_{i\in\{5, 11, 16, 20\}}), y_{\mbox{BR}}]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{R34}}&=&[min(x_i\vert_{i\in\{2, 8, 14, 18\}}), y_{\mbox{TL}}, x_{\mbox{BR}}, y_{\mbox{BR}}]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{R12}}&=&[min(x_i\vert_{i\in\{4, 10, 15, 16, 19\}}), y_{\mbox{TL}}, x_{\mbox{BR}}, y_{\mbox{BR}}]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{B12}}&=&[x_{\mbox{TL}}, min(y_i\vert_{i=14}^{16}), x_{\mbox{BR}}, y_{\mbox{BR}}]
\nonumber\end{aligned}$$ $$\begin{aligned}
\boxdot_{\mbox{B34}}&=&[x_{\mbox{TL}}, min(y_i\vert_{i=7}^{12}), x_{\mbox{BR}}, y_{\mbox{BR}}].\end{aligned}$$ Here, $\Delta_{\mbox{EP}}=max(\vert y_i - y_{i-6}\vert_{i=7}^{12})$ and $\Delta_{\mbox{NS}}=0.5*(max(y_i|_{i=18}^{20}) -min(y_i|_{i=14}^{16}))$, $\forall v_j\geq\tau$, where $j\in\{1, 2, \hdots 21\}$.

Intuitively, certain segments are more effective at predicting a subset of attributes than others.  For example, we can expect that segments related to the upper part of the face (e.g. $U12$, $U34$ etc.)  would contain information about the person being bald or having certain types and color of hair.  Therefore, even if the some other part of the face is occluded (e.g. $B12$, $EP$, or $NS$ being not visible), by looking at the upper portion of the face, one can still predict attributes related to hair.  Thus detecting attributes from parts as opposed to the whole face, has the advantage of allowing graceful degradation of performance rather than catastrophic failures with increasing occlusion. 

While some attributes can be easily predicted from facial segments, some attributes reflect more global characteristics.  For example, one can get hints if a person is young or not from multiple parts of the face, but youth is a global attribute.  Therefore it is important to combine the segment predictions into a global prediction, so that multiple parts can contribute to the final prediction.  Naturally, the following questions arise:

-   *Global vs local attributes:* How does one decide if an attribute is better predicted by facial segments or by a global predictor? 

-   *Optimal segment selection:* How does one decide which facial segment is more suitable for predicting a particular attribute? 

-   *Combining results multiple networks:* Given that each attribute is predicted by multiple segments of the proposed network, how does one combine the results optimally? 

-   *Handling occlusion:* If a certain facial segment responsible for predicting a certain attribute is not visible, how does one get a reasonable prediction? 

We can summarize the solutions to these problems as follows. 

-   *Network architecture:* The first problem is solved by choosing an architecture of a DCNN that is not only able to predict attributes from facial segments, but also performs feature-level fusion of intermediate features through a global prediction network to produce accurate global predictions.  Also, the sub-modules of the network have a Global Average Pooling which endows the networks with localization ability [@Zhou_2016_CVPR]. 

-   *Output pruning:* The second question is answered by the two-stage training approach that is adopted in this work, where the first stage primarily is used to prune the outputs of the segment networks by deciding which segments are good at predicting which attributes. 

-   *Committee Machines:* For the third problem, we use two-committee machines to perform score level fusion of the multiple predictors to significantly improve the performance of any single constituent network. 

-   *Hierarchy of best predictors and Segment Dropout:* Finally, to address the fourth problem, we keep track of a hierarchy of segments, which are good at predicting that attribute.  Therefore, even if the best segment for an attribute is not present, one can fall back on other segments that are known to do somewhat well for that attribute.  We also train our network with ‘Segment Dropout’ [@7961801] to make it more robust to partial faces. 

These ideas are core to our proposed method: **S**egmentwise, **P**artial, **L**ocalized **I**nference in **T**raining **F**acial **A**ttribute **C**lassification **E**nsembles (SPLITFACE).  The algorithm looks at the facial segments and learns to infer local attributes, to better handle partial faces.  The next four subsections expand on these ideas. 

Local to Global Network Architecture
------------------------------------

Here the three constituents of the proposed network namely, the Full Face Network, the Facial Segment Networks and the Global Predictor Network, and their training losses are described:

*Facial Segment Networks*: Let $I_1, I_2, \hdots, I_M$ denote face regions for the aforementioned $M=14$ facial segments.  Each segment has some predictive power which is unknown initially.  In the next section, we describe our data-driven approach to find which attributes are predicted more accurately by each segment.  For now, let us say that segment $i$ predicts a set of attributes $N_i$, where the number of attributes predicted by each segment $|N_i| \le K$, where $i\in\{1, 2, \hdots, M\}$. Initially, all segments predict all $K=40$ attributes, but later each segment is allowed to specialize, as described in the next section.  We denote these Segment Networks $S_i$, where $i\in\{1, 2, \hdots, M\}$. When the facial segment $I_i$ is passed through its corresponding segment network $S_i$, it yields attribute scores $s_i$ for each attribute in $N_i$ and a feature for that segment $C_i$, i.e. $$s_i, C_i = S_i(I_i)
\label{segeq}$$ where $C_i$ is tapped from the last convolutional layer of $S_i$. The architectures of all the segment networks $S_i$ are same, as described in table \[NetArchTable\], and each of these segment networks is independent of one another. 

*Full Face Network*: Let $I_0$ represent the full-face region, which is passed through a DCNN $S_0$. We have adopted a seven layer deep convolutional network as $S_0$. Details on the network architecture are provided in table \[NetArchTable\].  The full face region is expected to always predict all $K$ attributes ($|N_0|=K$).  Hence, it outputs a vector $s_0$ of length $K$, and also a compact feature representation $F_0$ after global pooling the last convolutional feature, i.e. $$s_0, F_0 = S_0(I_0)
\label{fulleq}$$

*Global Prediction Network for local feature fusion*: In the Global Prediction Network, we combine the results from the local segment networks and the full face network to produce predictions for all the $K$ attributes.  To do so, we first concatenate the convolutional features $C_i$ from the $M$ segment networks, convolve them, then apply global pooling to get a flattened feature from the segments, $F_s$. This is concatenated with $F_0$, the flattened features from the Full Face Network, and passed through a few fully connected layers, to finally yield predictions for all the $K=40$ attributes.  The Global Prediction Network can be thought of as a feature level fusion of the different segments, as opposed to the score level fusion of committee machines described in the next section. 

$$s_{M+1} = S_{M+1}(F_0, C_1, \hdots, C_M)
\label{globeq}$$

The color-coded network architecture for the SPLITFACE network is shown in Fig.  \[Task1Diagram\].  It shows the above mentioned architectural choices, namely predictions from segments, predictions from the full face and fusion of segment and full face features to provide a global prediction. 

For further discussions, we shall use the word ‘predictor’ to mean any of the $M+2$ sub-networks $S_i, i \in \{0,1,\hdots,M+1\}$ described above, that is, any of the $M$ Facial Segment Networks, the Full Face Network or the Global Prediction Network. 

*Localization using Global Average Pooling*: It has been shown in [@Zhou_2016_CVPR] that Global Average Pooling (GAP) introduced in [@NIN] has remarkable localization properties.  Since we are aiming to predict localized attributes well from partial segments, we use a GAP layer in the architecture to transition from convolutional to fully connected layers.  Using Class Activation Maps (CAM) in section \[camheatmap\] we observe that this provides the network with the desired property of being able to focus on regions of interest, thus making the process more interpretable. 

*Loss*: We use binary crossentropy loss for all the predictor outputs $s_i, i \in \{0,1,\hdots,M+1\}$ described in \[segeq\], \[fulleq\], \[globeq\], weighted by the inverse of priors.  Then loss $L$ incurred on image $I$ is: $$L(I) = \sum_{i=0}^{M+1} \sum_{j=0}^{K-1} w_{I,j} \log(s_i(j))
\label{losseq}$$ In \[losseq\], $w_{I,j}$ is a weight based on the ground truth $I_j$ and the prior probability $p_j$ of attribute $j$ being present, which is precomputed on the training set.  The weight $w_{I,j}$ defined in \[priorweight\] helps to mitigate the challenges due to unbalanced class distributions which are prevalent in datasets like CelebA [@MOON_Rudd2016].  $$w_{I,j}=\begin{cases}
  p_j,& \text{if } I_j=0 \\
  1-p_j & \text{if } I_j=1
\end{cases}
\label{priorweight}$$

            $S_i$, $i\in\{1,2, \hdots, M\}$                                  $S_0$                                             $S_{M+1}$
  --------------------------------------------------- --------------------------------------------------- ---------------------------------------------------
   conv$3$-$3$2 $\rightarrow$ BN $\rightarrow$ ReLU    conv$3$-$32$ $\rightarrow$ BN $\rightarrow$ ReLU    conv$3$-$512$ $\rightarrow$ BN $\rightarrow$ ReLU
          2D MaxPool $3\times 3$, Stride $2$                  2D MaxPool $3\times 3$, Stride $2$                        global average pooling
   conv$3$-$64$ $\rightarrow$ BN $\rightarrow$ ReLU    conv$3$-$64$ $\rightarrow$ BN $\rightarrow$ ReLU                    merge$(F_0, F_s)$
   conv$3$-$64$ $\rightarrow$ BN $\rightarrow$ ReLU           2D MaxPool $3\times 3$, Stride $2$                    dense-$256$D$\rightarrow$ ReLU
          2D MaxPool $3\times 3$, Stride $2$           conv$3$-$64$ $\rightarrow$ BN $\rightarrow$ ReLU                     dropout$(0.2)$
   conv$3$-$128$ $\rightarrow$ BN $\rightarrow$ ReLU          2D MaxPool $3\times 3$, Stride $2$          
          2D MaxPool $3\times 3$, Stride $2$           conv$3$-$128$ $\rightarrow$ BN $\rightarrow$ ReLU  
   conv$3$-$128$ $\rightarrow$ BN $\rightarrow$ ReLU          2D MaxPool $3\times 3$, Stride $2$          
   conv$3$-$256$ $\rightarrow$ BN $\rightarrow$ ReLU   conv$3$-$128$ $\rightarrow$ BN $\rightarrow$ ReLU  
                                                       conv$3$-$256$ $\rightarrow$ BN $\rightarrow$ ReLU  
                                                              2D MaxPool $3\times 3$, Stride $2$          
                                                       conv$3$-$256$ $\rightarrow$ BN $\rightarrow$ ReLU  

\[NetArchTable\] -5pt

Optimal segment selection for output pruning
--------------------------------------------

Intuitively, not all segments predict all attributes well.  Therefore it is counterproductive to train the network to produce all $K$ predictions from all $M$ segment networks.  Instead, we follow a data-driven approach to prune the segment networks. 

*Stage 1*: Initially, we predict all $K=40$ attributes from all $M$ segment networks, the full face network and the global prediction network.  Hence each attribute is predicted by $M+2$ networks.  After training for several epochs, we evaluate the detection accuracy of each of the $M$ segment networks, $S_i$. For each attribute, we sort the $M+2$ networks according to accuracy on validation set, and pick the top $d$ networks for each attribute.  The global predictor (GP) and the full face network can be expected to be among the best predictors most of the time, since they have a top view of the sum of parts.  The rest of the $d-2$ predictors of each attribute are segment networks and therefore the most associated $N_i$ attributes for segment $i$ where $i\in\{1,2, \hdots, M\}$ and $N_i\leq K$ is determined this way. 

In Fig.\[Top7SegHeatMap\] a table for $d=7$ and $M=14$ is shown where for the segment networks (row three and below) the non-zero numbers for different attribute columns denote the attributes assigned to the segment after pruning.  The total number of attributes assigned to the segment networks after pruning are shown in the last column. 

*Stage 2:* After the association of attributes with segments as described above, a second round of training is performed.  The pruning process in stage 1 allows the segment networks to focus on attributes that they perform best on, without having to worry about attributes they are just not capable of predicting.  Also, we have intentionally assigned all the attributes to GP and FULL networks (as shown in Fig.  \[Top7SegHeatMap\]), since the receptive field for these two networks encompasses the entire face.  So, we make the inherent assumption that these two networks are capable of successfully predicting all the attributes. 

Committee Machines for Score-Level Fusion
-----------------------------------------

While the global predictor or the full face networks has good predictive power, using only those two predictors does not harness the full potential of the proposed network architecture.  To utilize all the segment networks along with the global and full face predictors, we describe two committee machines here namely, the Highest Ranked Predictor (HRP) and the Normalized Score Aggregation (NSA) methods, that perform score-level fusion.  For both methods, using the validation set $V$, we first compute the optimal thresholds, $t_{a,i_a}$ for each attribute $a\in \{1,2,\hdots,K\}$ and for each predictor $i_a\subset \{0, 1, \hdots, M+1\}$ which are responsible for predicting $a$. For CelebA, the table in Fig.  \[Top7SegHeatMap\] shows information about $i_a$, which is an ordered set or tuple, ordered in descending order of validation accuracy.  For example, if we consider the attribute ‘goatee’, then $a=16$ and $i_{a=16}=(15, 1, 14, 13, 0, 12, 10)$, which correspond to B12, FULL, B34, R12, GP, R34 and L34.  Denoting $I_a$ as the ground truth of attribute $a$ for sample $I$ and $\mathds{1}$ as the indicator function, the optimal thresholds that maximize validation accuracy are computed as: $$t_{a,i} = \operatorname*{arg\,min}_{t \in [0,1]} \sum^{}_{I\in V} \mathds{1}_{I_a = \mathds{1}_{S_i(I)<t}}
\label{optth}$$

We denote visibility of a segment $j$ for image $I$ by $V_{I,j}\in\{0,1\}$. Clearly $V_{I,0}=V_{I,15}=1$, since the both the Full Face Predictor and Global Predictor Network can predict attributes no matter what the occlusion is.  Finally, we define an ordered set $i_a^v$ which contains the top usable predictors for attribute $a$ (ones that have visible segments) as $$i_a^v = (x : x \in i_a, V_{I,x}=1).$$

### Highest Ranked Predictor (HRP) Committee Machine

After the completion of the two training stages, we evaluate the performance of each of the predictors (segment, full face and global networks) on the validation set to find a hierarchy of best performing predictors for each attribute.  The results are shown in the table in Fig.  \[Top7SegHeatMap\].  For example, we can see that the best predictors for ‘goatee’ are B12, FULL, B34, R12, GP, R34 and L34, in descending order of validation accuracy. 

When making a prediction for an attribute, we find the topmost usable predictor $j=i_a^v(1)$ that is usable/visible for that image.  This score from predictor $S_j(I_j)$ is then thresholded with the optimal threshold of that segment for that attribute, which was precomputed from the validation set following \[optth\].  The prediction outcome based on the optimal threshold would be $$P_a(I) = \mathds{1}_{S_j(I_j)<t_{a,j}}. 
\label{v2eqn}$$ Continuing our example, to predict ‘goatee’, we use the prediction of B12 ($=i_a(0)$), and if that segment is not visible, we use FULL ($=i_a(1)$). 

### Normalized Score Aggregation (NSA) Committee Machine

In general, different predictors trying to predict the same attribute might have different optimal thresholds.  Once they are aggregated (say, by taking their mean or product or median), one needs to calculate the optimal threshold for the aggregate score.  Instead, we could normalize the scores of the predictors so that after aggregation, the optimal threshold for the aggregate score is $0.5$. [@Cappelli_ThresholdNorm] suggests a double sigmoid score normalization function for fusing scores from multiple predictors.  However, it involves $2$ hyper-parameters, which need to be found by cross-validation.  Instead we propose a simpler normalization function below, which does not require any hyper-parameters. 

*Linear Threshold Normalization:* Consider a binary classification problem, where we have to decide the class $C\in \{0,1\}$, given a score $X\in [0,1]$. We assume that the optimal threshold $t$ that maximizes separation between the 2 classes is known.  Therefore, $P(C=1|X=t)=0.5$. Consider a transformation $Y=T(X)$. We wish to identify the function $T$, such that, $P(C=1|Y=0.5)=0.5$.

$$\begin{aligned}
P(C=1|Y=0.5)&=&P(C=1|T(X)=0.5)\nonumber\\
&=& P(C=1|X=T^{-1}(0.5))\nonumber\\\end{aligned}$$

If we choose an invertible function $T$, such that $T(t)=0.5$, then above equation yields $P(C=1|Y=0.5)=0.5$. Thus, given multiple scores $X_i$, and their optimal thresholds $t_i$, we can transform the scores to $Y_i=T(X_i)$, so that after aggregating $Y_i$, say by averaging, the optimal threshold is $0.5$.

For our algorithm, we use a piecewise linear transformation \[piecewiselin\], that satisfies the $T(t)=0.5$ criterion discussed above.  $$T_t(x) = 
\begin{cases}
  (0.5/t) \times x,& \text{if } 0 \leq x \leq t \\
  (0.5x + 0.5 -t)/(1-t) & \text{if } t <x\leq 1
\end{cases}
\label{piecewiselin}$$

We transform the scores of at least $p=5$ top predictors out of the visible ones, $i_a^v$, using \[piecewiselin\] to yield $Z$ (\[transformedscores\]).  $$Z=\{T_{t_{a,i_a^v(k)}}(S_{i_a^v(k)}(I_{i_a^v(k)})) : k\in\{1,2,\hdots,min(|i_a^v|,p)\}\}
\label{transformedscores}$$

Finally we use an aggregation function $A$ on $Z$ for the prediction.  The decision rule using the aggregator function is $$P_a(I) = \mathds{1}_{A(Z) \geq A(\{1-z:z\in Z\})}.$$

Possible choices of aggregator functions are:

-   *Bayes’ Rule or Product Rule*: As discussed in the score fusion literature [@Jain_CombinationApproach] we can use a product rule to combine the decisions of $N$ binary classifiers according to the following decision rule $$\prod^N P(C=1|x)\lessgtr \prod^N (1-P(C=1|x)). 
    \label{bayesrule}$$ Thus the product aggregator function is $A(Z)=\prod_{z\in Z}z$ for all $z\in Z$.

-   *Median Rule*: As proposed in [@DBLP:journals/corr/cs-NE-9905012], the median aggregator function is $A(Z)=med(Z)$.

Segment Dropout and Hierarchy of Best Predictors for Handling Occlusion
-----------------------------------------------------------------------

*Segment Dropout:* When training the network with image $I$, only a subset of the $M=14$ segments might be present.  The visible segments are randomly dropped with probability $30\%$ when training.  This is called Segment Dropout, which was introduced in [@7961801] to augment the dataset for handling occlusion.  When a certain segment is not present in a face the input to corresponding segment branch is zero.  In order to make SPLITFACE robust against such cases and generalize better to detect attributes from the available segments, random segment dropout is performed. 

*Hierarchy of Best Predictors:* As described in the earlier section, we compute a hierarchy of predictors $i_a^v$ that are visible.  Thus even if a face is partially occluded and the best segment is not available, the other segments provide reasonable predictive power to the committee machine. 

The unique architecture of SPLITFACE allows the use of predictor hierarchy and thereby improving the detection accuracy.  In addition, it ensures that even if some part of the input face is not visible due to occlusion or failure of the face detector, the attribute detection network would rely on the visible segments to still make a good prediction.  Note that the input to GP are the features from all the segment networks and the full face network, and our partial face augmentation approach during training enables it to handle missing segments while predicting attributes. 

Experimental Setup and Evaluation {#ExperimentalResults}
=================================

Datasets
--------

We use the CelebA [@CelebA_liu2015faceattributes] and LFWA [@CelebA_liu2015faceattributes] datasets for both training and evaluation.  Also, to evaluate the SPLITFACE’s capability for handling partially visible faces when estimating facial attributes, we created several variations of these two datasets and evaluate the performance of SPLITFACE on those variations.  We follow the data augmentation scheme described in [@DRUID_Umahbub] for generating partially visible faces by cropping the images keeping only L12 or L34 or R12 or R34 or U12 or U34 portion.  We replace the rest of the pixels with white pixels.  Hence, we create six variations of both datasets and named them C$-{P}$ and L$-{P}$, respectively for CelebA and LFWA, where $P\in\{\mbox{L12, L34, R12, R34, U12, U34}\}$ .  Some sample images for C$-{P}$ dataset are shown in Fig.  \[FSDatasetSamples\]. 

Implementation Details
----------------------

The proposed network has $26,090,334$ trainable parameters, which are tuned using the adaptive moment estimation (ADAM) optimizer [@ADAMOptimizer].  The initial learning rate was set to $0.001$. For CelebA, we train the network for $10$ epochs for both stages, while for LFWA, we train it for $180$ epochs in the first stage and $270$ epochs in the second stage.  The full face region is resized to $196\times 196 \times 3$ and given as input to the full face branch, which the inputs to the facial segment branches are all resized to $64 \times 64 \times 3$. The experiments were performed on NVIDIA Quadro P6000 GPUs, with training batch size of $200$, and the code was written using the KERAS python library [@KERAS_chollet2015keras] with tensorflow [@Tensorflow] backend.  Apart from segment dropout, horizontal flipping was applied for data augmentation.  Among state-of-the-art methods, the authors of AFFACT [@AFFECT_AlignmentFreeAttrib_Boult] have provided the source code in their paper, which we used for performance comparison on partial face datasets.  However, the accuracy obtained from this implementation is slightly less than the accuracy reported in [@AFFECT_AlignmentFreeAttrib_Boult], perhaps because we have not applied test time data augmentation.  For all the other methods, we directly report the results in corresponding publications. 

Visualizing Network Response using Class Activation Maps {#camheatmap}
--------------------------------------------------------

The class activation map was proposed in [@Zhou_2016_CVPR] to visualize the localization properties of the network.  Given a network which terminates in a Global Average Pooling (GAP) layer followed by a dense layer, we can compute the CAM $C_i$ of a particular class $i$ as a weighted average of the activation maps of the layer just before the GAP layer as $$C_i = \sum^N_j w_{i,j}M_j,
\label{CAMeq}$$ where $M_j$ is the $j^{\text{th}}$ feature map in a feature tensor of depth $N$ just before the GAP layer and $w_{i,j}$ is the corresponding weight of the dense layer after the GAP layer.  In Fig.  \[CAMheatmap\], we show the CAM superimposed on some facial segments from CelebA. Clearly, the activation maps are localized in interpretably meaningful regions.  It can be seen from Fig.  \[CAMheatmap\] that for all the three attributes shown for UR12 (‘bald’, ‘receding hairline’ and ‘wearing hat’) and B12 (‘5 o clock shadow’, ‘goatee’ and ‘sideburns’), the network focuses on the same region: the top corner of the head for UR12 and the chin and cheeks for B12.  On the other hand, for segments EP and NS, the attention shifted to different regions for different attributes.  For example for segment EP, the attribute ‘bags under eyes’ is predicted when the network has high response near the eyes, ‘bangs’ are predicted when the response is high near the forehead and ‘eyeglasses’ are predicted by looking at the bridge of the nose.  Similarly the NS segment network shifted its attention to the nose, eyes or lips to predict ‘big nose’, ‘narrow eyes’ and ‘wearing lipstick’, respectively. 

Performance Comparison on Original CelebA and LFWA datasets
-----------------------------------------------------------

The performance of the proposed method is compared with state-of-the-art methods in tables \[tab:PerformanceCompareOnCelebA\] and \[tab:CompareResultsLFWA\] on the original CelebA and LFWA datasets, respectively.  Among state-of-the-art methods, the result of AFFACT is directly reported from [@AFFECT_AlignmentFreeAttrib_Boult], while the column AFFACT Unaligned contains results that we found by evaluating the full faces we use in our experiment.  Since the source codes were not available for any of the other state-of-the-art methods, we reported the results directly from corresponding publications.  The column titled ‘Prior’ shows the accuracies obtainable by only applying the knowledge from the prior probabilities of the presence or absence of an attribute in the datasets.  It can be seen that a staggering mean accuracy of $80.57\%$ in CelebA and $71.27\%$ in LFWA is achievable by only using the prior probabilities in decision making.  Even though the state-of-the-art methods and the proposed method increases this number by more than $10\%$, for certain attributes such as Big Lips and Narrow Eyes in table \[tab:PerformanceCompareOnCelebA\], the prior is higher than the trained methods for most of the approaches.  We put the prior column in the table as a baseline for evaluation. 

The last five columns in tables \[tab:PerformanceCompareOnCelebA\] and \[tab:CompareResultsLFWA\] show the attribute-wise accuracy and the mean accuracy for Full, GP, HRP, NSA Product rule and NSA Median rule, respectively, in both tables.  It can be seen from these tables that the committee machine approaches boost the results obtained from Full and GP for most of the attributes.  The mean accuracy of $90.42\%$ for CelebA and $85.85\%$ for LFWA obtained from NSA Product Rule closely matches the state-of-the-art results presented in the table.  Note that we adopted a very simple six layer convolutional network for the Full face branch that achieves $90.72\%$ accuracy on CelebA and $84.02\%$ accuracy on LFWA.  The result for CelebA is boosted for HRP but degrades slightly for NSA methods.  On the other hand, for LFWA, the committee machine approaches improves the overall performance.  Since LFWA is a much smaller dataset and hence the trained network over-fitted greatly on the training set, this boost in result shows that the proposed committee machine approaches, especially NSA, is effective for generalization due to their ensemble aggregation mechanism.  In later sections, we will present results for partially visible faces, where the committee machine approaches consistently improves over Full and GP branches and hence the adaptation of such methods is justified for practical purposes even with a slight loss in accuracy for the original dataset. 

Cross-Dataset Testing Accuracies
--------------------------------

In table \[tab:CrossDataSetResults\], we presented the cross-dataset testing performances of AFFACT, DMTL and SPLITFACE (NSA product rule).  For AFFACT and the proposed method, we presented two accuracies separated by $/$, the first one is for using the optimal threshold obtained from the validation set to find detection results (for AFFACT) or to normalize scores before applying product rule (for proposed).  And, the second accuracy is obtained by using the mid value of the score range ($0$ for AFFACT which gives scores between $-1$ and $+1$, and $0.5$ for proposed method for which score is between $0$ and $1$) as the threshold.  Higher accuracies are obtained by using optimal thresholds for the proposed method, while for AFFACT the accuracies dropped slightly.  It can be seen from this table that, for cross-dataset testing (trained on CelebA and tested on LFWA or vice versa), the proposed method outperforms both AFFACT and DMTL with a relatively large margin.  This again proves the generalization capability of SPLITFACE, which is achieved by the combination of its unique architecture with the committee machine. 

\[tab:CrossDataSetResults\]

Next, we evaluated the performances SPLITFACE on the modified CelebA and LFWA partial face datasets.  The results for evaluation on same and cross-dataset are presented in tables \[tab:CrossTableCelebAtoAll\] and \[tab:CrossTableLFWAToAll\].  In table \[tab:CrossTableCelebAtoAll\], results are presented form AFFACT and SPLITFACE network, both trained on the original CelebA training set and tested on the original and modified CelebA and LFWA datasets.  The results for both using and not using optimal threshold (using $0$ for AFFACT and $0.5$ for SPLITFACE as threshold instead) are shown in the table.  It can be seen that SPLITFACE, especially NSA with product rule and optimal threshold outperforms AFFACT in terms of accuracy for full face dataset, and both cross-domain and partial datasets.  The differences are more prominent when using the optimal thresholds, which show that threshold normalization step with a piece-wise linear function can boost the overall performance.  Similar scenario is found in Table \[tab:CrossTableLFWAToAll\], where SPLITFACE is trained on the original LFWA training set and tested on both original and partial CelebA and LFWA datasets.  Since the no pre-trained version of AFFACT on LFWA is publicly available, the results for AFFACT could not be provided in this table.  Note that in both tables \[tab:CrossTableCelebAtoAll\] and \[tab:CrossTableLFWAToAll\], the committee machine approaches improves over the full face branches, especially for partial face datasets.  This improvement can be attributed to the unique architecture of SPLITFACE that harnesses local information from unoccluded facial segments and to the ensemble aggregation approach by using committee machine. 

Analysis of Performance Degradation with Occlusion
--------------------------------------------------

One obvious observation from tables \[tab:CrossTableCelebAtoAll\] and \[tab:CrossTableLFWAToAll\] is that the attribute detection accuracies decrease with increasing occlusion.  For example, all the methods achieve higher accuracies for upper three-fourth faces present in C-U34 and L-U34 in comparison to C-U12 and L-U12, respectively.  In this section, we explore the effect of occlusion on the accuracy of each attribute using Fig \[PartialCompare\] which plots the decrease in accuracy of SPLITFACE (after stage 1 before output pruning) for the partial CelebA datasets, C$-P, P \in \{\mbox{U12, L12, L34}\}$ described in section \[datasets\] with respect to full face accuracy.  The differences are denoted as delta-U12, delta-L12 and delta-L34, respectively.  We observe that SPLITFACE fails in C-L12 and C-L34 for the same attributes such as wavy hair, high cheekbones and wearing lipstick, since part of the right side of the faces containing vital information in these regard are occluded in both cases.  On the other hand, C-U12 has reduced performance for attributes like ‘mouth slightly open’, ‘no beard’ and ‘smiling’, which are attributes localized in the lower part of the face, which is not visible in C-U12.  So, SPLITFACE avoids catastrophic failures during occlusion, since the prediction accuracy of other attributes remain near constant and only invisible localized attributes’ performance degrade.  The output pruning step of SPLITFACE removes the attributes for which a segment performs badly in the first stage.  When trained, SPLITFACE utilizes information from different segments to bolster its decision about an attribute as well as fill up the gaps in attributes in one segment by using information from other segments which predict those missing attributes. 

Performance for Partial Face Augmentation
-----------------------------------------

We also trained SPLITFACE with training samples from the modified partial face datasets.  When training, samples from the modified datasets were picked with a $0.3$ probability in each batch while the rest of the samples came from the original datasets.  The performances of the networks trained in this manner are presented in tables \[tab:CelebAWithFS\] and \[tab:LFWAWithFS\].  In comparison to tables \[tab:CrossTableCelebAtoAll\] and \[tab:CrossTableLFWAToAll\], we can see that for both partially modified CelebA and LFWA datasets, the performances improved greatly when the partial faces are augment the training samples in addition to segment dropout. 

      **Methods**       **CelebA**  **C-U12**   **C-U34**   **C-L12**   **C-L34**   **C-R12**   **C-R34**
  ------------------- ------------ ----------- ----------- ----------- ----------- ----------- -----------
       **Full**                       88.01       89.7        89.56       90.02       89.54       90.07
        **HRP**                       87.94       89.55       89.23       89.77       89.21       89.71
   **NSA Prod rule**                  88.43       89.77       89.76       90.02     **89.86**     90.11
   **NSA Med rule**                 **88.46**   **90.05**   **89.85**   **90.16**     89.85     **90.21**

\[tab:CelebAWithFS\]

    **Methods**    **LFWA**    **L-U12**   **L-U34**   **L-L12**   **L-L34**   **L-R12**   **L-R34**
  --------------- ----------- ----------- ----------- ----------- ----------- ----------- -----------
       Full          83.93       80.31       83.01       81.6         83         81.93       83.37
        HRP          85.46     **82.39**     84.45     **83.36**     84.59     **83.56**     84.79
   NSA Prod rule   **86.04**     82.06       84.87       83.07     **84.97**     83.43       85.14
   NSA Med rule      85.97       82.17     **84.88**     83.06       84.87       83.4      **85.23**

\[tab:LFWAWithFS\]

Conclusion and Future Work {#Conclusion}
==========================

In this paper, we introduced SPLITFACE, an algorithm for facial attribute extraction utilizing multiple facial segments, a unique deep convolutional network, and a committee machine approach for ensemble aggregation.  Through extensive experimentation, we have shown that the proposed method outperforms state-of-the-art facial attribute extraction methods when the faces are partially visible.  Also, utilizing a committee machine approach, SPLITFACE achieved better generalization and therefore superior performance across domains.  Moreover, when trained with both segment dropout and partial face data, the network achieved even higher attribute detection accuracy for partially visible faces.  The overall accuracies might be boosted by replacing the full face and segment branches with more advanced deep neural network architecture such as ResNet.  On the other hand, since the segments are heavily overlapping and therefore can assist each other greatly, similar performance might be achievable with smaller input images.  Finally, it would be interesting to see if a cross-stitch network [@crossstitchNet] can improve performance when connected to the segment network branches at certain intervals, by allowing the segment networks to share data. 

Acknowledgments {#acknowledgments .unnumbered}
===============

Acknowledgment {#acknowledgment .unnumbered}
==============

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R$\&$D Contract No.  2014-14071600012.  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government.  The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. 

\[[![image](UpalPhoto2-001.JPG){width="1in" height="1.25in"}]{}\][Upal Mahbub]{} received the M.Sc.  degree in Electrical and Computer Engineering from the University of Maryland College Park in 2017 and the B.Sc.  and M.Sc.  degree in Electrical and Electronic Engineering from the Department of EEE, BUET, Dhaka, Bangladesh, in 2009 and 2011, respectively.  He currently a doctoral candidate in the department of ECE at the University of Maryland, College Park.  His doctoral research is primarily focused on multi-modal active user authentication on smartphones using computer vision and machine learning techniques.  Upal is the recipient of the best paper award at IEEE UEMCON 2016, best poster award at BTAS 2016, best paper award at ICCIT 2011 and distinguished graduate fellowship from the A. James Clark School of Engineering at the University of Maryland. 

[Sayantan Sarkar]{} received his B.Tech.  degree in Instrumentation Engineering from Indian Institute of Technology Kharagpur, India, in 2012.  After working in mobile computing at Nvidia for 2 years, he is currently a Research Assistant at University of Maryland College Park.  His research interests include face analysis and active authentication using deep learning.  He received Best Poster Award at IEEE BTAS 2016. 

[Rama Chellappa]{} is a Distinguished University Professor, a Minta Martin Professor in Engineering and Chair of the Department of Electrical and Computer Engineering at the University of Maryland, College Park, MD.  He received the B.E. (Hons.)  degree in Electronics and Communication Engineering from the University of Madras, India and the M.E. (with Distinction) degree from the Indian Institute of Science, Bangalore, India.  He received the M.S.E.E. and Ph.D. Degrees in Electrical Engineering from Purdue University, West Lafayette, IN.  At UMD, he is an affiliate Professor of Computer Science Department, Applied Mathematics and Scientific Computing Program, member of the Center for Automation Research and a Permanent Member of the Institute for Advanced Computer Studies.  His current research interests span many areas in image processing, computer vision and machine learning.  Prof. Chellappa is a recipient of an NSF Presidential Young Investigator Award and four IBM Faculty Development Awards.  He received two paper awards and the K.S. Fu Prize from the International Association of Pattern Recognition (IAPR).  He is a recipient of the Society, Technical Achievement and Meritorious Service Awards from the IEEE Signal Processing Society.  He also received the Technical Achievement and Meritorious Service Awards from the IEEE Computer Society.  Recently, he received the inaugural Leadership Award from the IEEE Biometrics Council.  At UMD, he has received numerous college and university level recognitions for research, teaching, innovation and mentoring of undergraduate students.  In 2010, he was recognized as an Outstanding ECE by Purdue University.  He received the Distinguished Alumni Award from the Indian Institute of Science in 2016.  Prof. Chellappa served as the EIC of IEEE Transactions on Pattern Analysis and Machine Intelligence, as the Co-EIC of Graphical Models and Image Processing, as the Associate Editor of four IEEE Transactions, as a Co-Guest Editor of many special issues, and is currently on the Editorial Board of SIAM Jl.  of Imaging Science and Image and Vision Computing.  He has also served as the General and Technical Program Chair/Co-Chair for several IEEE international and national conferences and workshops.  He is a Golden Core Member of the IEEE Computer Society, served as a Distinguished Lecturer of the IEEE Signal Processing Society and as the President of IEEE Biometrics Council.  He is a Fellow of IEEE, IAPR, OSA, AAAS, ACM, and AAAI and holds six patents. 

[^1]: \* First two authors contributed equally
