# Face Recognition and Tracking Framework for Human-Robot Interaction
## 4 Methodology and Proposed Framework
### Improved Face Recognition Using Face Tracking  
For the face tracking task, we build the face tracking algorithm based on a simple online and real-time tracking algorithm (SORT) [46]. SORT uses a Kalman filter for estimating the location of the face in the current frame given the location in the previous frame. It starts with detecting the target face in the initial frame \(i\). After that, predict the future location \(i+1\) of the target face from the initial frame using the Kalman filter. Noting that the Kalman filter just approximates the face's new location, which needs to be optimized. Finally, the Hungarian algorithm is used for face location optimization and association.  
The main problem we are targeting is the speed/accuracy trade-offs. Continuous face detection and face recognition processing are time-consuming. Moreover, the quality of the face features depends on the face pose, where the frontal face pose generates the best facial features and degrades in a departure from the frontal pose. Therefore, instead of detecting all faces around all input video frames, we assign only each newly detected face a tracker and start the tracking instead of detection. Furthermore, for each new tracker only, the face embedding will be inferred and compared against the stored embeddings by calculating the cosine similarity to generate the user identity (ID), then add the ID to the tracker metadata for fast recognition, i.e., retrieve the ID from the tracker in the successive frames without the need for recognition. These will improve the processing time, recognition rate, and reduce the recognition errors caused by variations from frontal face poses.  
In the proposed tracking Algorithm 1, for each input frame, we are detecting faces using face detection and alignment in Section 4.1. Initially, a new tracker for each detector box will be created by applying SORT [46]. SORT analyzes previous and current frames and predicts face locations on the fly by utilizing the Kalman filter and Hungarian algorithm. Then, the user ID will be obtained using face recognition in Section 4.2 and assigned with the face tracker for use in fast recognition in the further frames. Finally, the tracker will be associated with the detected faces and maintained throughout tracking, and the user ID is assigned for each face tracker. We update the tracker in each frame to validate if a face is there inside the box to improve the tracking quality. If not, we are deleting the tracker to prevent unbounded growth in the number of trackers. Moreover, the actual user identity is attached to the face tracker instead of a unique face tracker ID to improve the face recognition speed.  
```
Inputs : Video, Detections, KalmanFilter, Headjoints, SubjectIDs Output : Recognized Tracked Faces  Initialize \(KalmanFilterTracker\); foreach frame \(f_{i}\in Video\)do  Trackers \(\leftarrow\) Predict();  Trackers \(\leftarrow\) Assign(Detections, Trackers);  TrackersID \(\leftarrow\) Attach(Trackers, SubjectIDs);  TrackersID \(\leftarrow\) Assign(TrackersID, Headjoints);  Update \(KalmanFilterTracker\); foreach tracker \(t_{i}\in TrackersID\)do \(ROS\leftarrow\) Publish(\(t_{i}\));  end if  end foreach
```  
**Algorithm 1**The Proposed Face Tracking Algorithm.  
To improve the proposed face tracking algorithm and minimize the tracking error, we obtain the head joint from the tracked skeleton provided by the WS1 Kinect V2 camera and try to assign it with the face center. If the assignment is successful, we update the face tracker with the fine location. Otherwise, the tracker will be deleted. Further, this reduces the number of identity switches through longer periods of occlusions.