# Face Recognition and Tracking Framework for Human-Robot Interaction
## 7 Conclusions  
We propose a face recognition system for human-robot interaction (HRI) boosted by face tracking based on deep convolutional neural networks (CNNs). To ensure that our framework can work in real-time HRI systems, we developed our framework based on lightweight CNNs for all face recognition stages, including face detection, alignment, tracking, and feature extraction. Furthermore, we implemented our approach as a modular ROS package that makes it straightforward for integration in different HRI systems. Our results suggest that the use of face tracking alongside face recognition increases the recognition rate.  
We utilize the state-of-the-art loss function _ArcFace_ for the face recognition task and the _RetinaFace_ method for face detection combined with a simple online and real-time face tracker. Furthermore, we propose a face tracker to tackle the challenges faced by the existing face recognition methods including various illumination conditions, continuous changes in head posture, and occlusion.  
The face tracker is designed to fuse the tracking information with the recognized identity and associate it with the faces once they are detected for the first time. For the updated trackers, the last recognized identity will be kept alongside the tracker. Despite what preceded, a new identity prediction is required for the new trackers. This method improved the overall processing time and face recognition accuracy and precision for the unconstrained face.  
The proposed framework is tested in real-time experiments applied in our real HRI system \"RoSA\" with 11 participants interacting with the robot to accomplish different tasks. Furthermore, to confirm the obtained results, we tested it on the recorded videos from the Wizard-of-Oz study, which contains videos of 36 subjects doing the same tasks on \"RoSA\" with the same results. The results showed that the framework can improve the robustness of face recognition effectively and boost the overall accuracy by an average of 25% in real-time. It achieves an average of 99%, 95%, and 97% precision, recall, and F-score respectively.  
Conceptualization, A.K. and A.-H.; methodology, A.K., A.A.A., D.S., J.H. and T.H.; software, A.K., D.S., J.H. and T.H.; validation, A.K., D.S., J.H. and A.A.A.; investigation, A.K., D.S., J.H. and A.A.A.; resources, A.A.-H.; writing--original draft preparation, A.K., A.A.A., D.S., J.H. and T.H.; writing--review and editing, A.K., A.A.A., D.S., J.H., T.H. and A.A.-H.; visualization, A.K., A.A., D.S., J.H. and T.H.; supervision, A.A.-H.; project administration, A.K. and A.A.-H.; funding acquisition, A.A.-H. All authors have read and agreed to the published version of the manuscript.  
This work is funded by the Federal Ministry of Education and Research of Germany (BMBF) RoboAssist No. 03 ZZ0448L and Robo-Lab No. 03 ZZ04X02B within the Zwanzig20 Alliance 3 DSensation.  
The study was conducted according to the guidelines of the Declaration of Helsinki. Ethical approval was done by Ethik Komnision der Otto-von-Guericke Universita (IRB00006099, Office for Human Research) 157/20 on 23 October 2020.  
Informed Consent Statement:Informed consent was obtained from all subjects involved in the study.  
Data Availability Statement:Not applicable.