# Face Recognition and Tracking Framework for Human-Robot Interaction
## 4 Methodology and Proposed Framework
### Face Detection and Alignment  
For face detection tasks, we use a deep CNN-based face detector by employing a single-shot, multi-level face localization method, called RetinaFace [3]. RetinaFace unifies three different face localization tasks together under one single shot framework: face box prediction, 2D facial landmark localization, and 3D vertices regression. Additionally, all points for these three tasks are regressed on the image plane. RetinaFace proposes a single-shot, multi-level face localization model, which consists of three components: the feature pyramid network, the context head module, and the cascade multi-task loss. First, the feature pyramid network generates five feature maps of different scales. Then, the feature map of a particular scale is fed to the context head module to compute the multi-task loss, i.e., the first context head module predicts the bounding box from the regular anchor. Afterward, the second context head module predicts a more accurate bounding box using the regressed anchor generated by the first context head module. Finally, the anchors are matched to ground-truth boxes if the Intersection over Union (IoU) is greater than 0.7 and 0.5 for the first and second context head respectively, and are matched to the background if IoU is less than 0.3 and 0.4 for the first and second context head, respectively. Furthermore, the unmatched anchors are ignored during training. For any training anchor \(i\), RetinaFace minimizes the following multi-task loss [3]:  
\[\mathcal{L}=\mathcal{L}_{cls}(p_{i},p_{i}^{*})+\lambda_{1}p_{i}^{*}\mathcal{L }_{\text{box}}(t_{i},t_{i}^{*})+\lambda_{2}p_{i}^{*}\mathcal{L}_{pts}(l_{i},l _{i}^{*})+\lambda_{3}p_{i}^{*}\mathcal{L}_{mesh}(v_{i},v_{i}^{*}), \tag{1}\]  
where \(t_{i},l_{i},v_{i}\) are box, five landmarks and 1k vertices predictions, \(t_{i}^{*},l_{i}^{*},v_{i}^{*}\) is the corresponding ground-truth, \(p_{i}\) is the predicted probability of anchor \(i\) being a face, and \(p_{i}^{*}\) is 1 for the positive anchor and 0 for the negative anchor. The classification loss \(\mathcal{L}_{cls}\) is the softmax loss for binary classes (face/not face). The loss-balancing parameters \(\lambda_{1}\) and \(\lambda_{2}\) are set to 0.25 and 0.1, respectively.  
For the face landmarks and alignment task, we use a deep CNN-based network by utilizing a practical facial landmark detector (PFLD) by [PERSON] et al. [29]. PFLD employs a branch of the network to estimate the geometric information for each face in order to regularize the landmark localization. Moreover, it adds a multi-scale fully connected (MS-FC) layer to enlarge the receptive field, catch the global structure, and precisely localize  
Figure 4: An overview of the proposed face recognition and tracking framework. The predicting face locations and identities are published to the ROS network for broadcasting to RoSA workstations and modules.  
landmarks on faces. For predicting landmark coordinates, it utilizes the MobileNet network as a backbone to enhance the processing speed and model size. As a result, it achieved a size of 2.1 Mb and over 140 fps per face on a mobile phone with high accuracy against complex faces, including unconstrained poses, expressions, lighting, and occlusions.  
In the face detection and alignment module, all the faces in the images or the video frames are detected with RetinaFace. RetinaFace outputs bounding boxes and five landmarks (2 eyes, nose, and mouth) with a confidence score. For real-time constraints, we select MobileNet-0.25 [49] as a lightweight backbone network, which achieves the real-time speed of 40 fps at GPU for 4K images (\(4096\times 2160\)) with outstanding performance.  
Next, the filtered faces, i.e., the detection boxes with high confidence scores are sent to face alignment for calibrating to the canonical view and for cropping it to a size of \(112\times 112\) to be suitable for the subsequent task of face feature extraction. For the face landmarks and alignment task, we used the compact model of the PFLD.