## 3 Measurement of State Data Based on Non-Contact Image Analysis
### Face Feature Point Detection and Facial Expression Recognition Implementation  
Facial feature points were detected using CE-CLM [33], a deep learning-based algorithm. A total of 68 major facial feature points to be detected were used as facial expression recognition and behavior analysis data. Figure 19 is a facial feature detection and facial expression recognizer using CE-CLM that can detect facial feature points at FHD resolution  
Figure 16: Optical flow tracking result for pixels detected by ROI. in real time and analyze facial behavior such as facial pose tracking and gaze tracking based on the detected facial feature points. Since the location and change of facial feature points have different size and direction distributions for each person due to differences in appearance, a normalization function was implemented that can measure changes in facial feature points based on their neutral expressions in order to normalize individual differences. In Figure 20, facial rotation and movement were corrected and individual differences were normalized by measuring the  
Figure 19: Facial feature point detection and facial expression recognizer. Figure 17: Changes in blood flow due to respiration observed in PPG (Photoplethysmogram). Figure 18: The result of removing the skin color change and heart rate component measured from the actual skin of the face. movement of each facial element after aligning the neutral expression and the expression to be measured using rigid body transformation for the feature points of the joy feature and the neutral feature obtained through Figure 19.  
In addition, facial asymmetry has been studied as an index of facial behavior that can grasp the psychological state, and since artificial and spontaneous expressions are expressed in different motor cortex, there is a difference in the degree of facial lateral asymmetry. The asymmetry measure in Figure 21 measures the degree of asymmetry of a pair of feature points in a lateral symmetry relationship by a geometric operation using the dot product between the face center vector and the feature point vector. For real-time state data analysis, a facial expression recognition model with a fast and small amount of computation is required, and the input data dimension of the model must be reduced. Accordingly, an expression recognition model based on facial feature points (Figure 22) was designed. In the case of images, data is stored in the form of three dimensions (image height, image width, image channels), which requires a lot of computation when using input data. Dimensional reduction was performed using facial feature points with geometry features according to facial expressions as input data of the facial expression recognition model. The facial feature point data used as input enables facial expression recognition in consideration of the movement and rotation of the face through the facial feature point normalization method described above. In addition, features using HOG [34] (Histogram of Oriented Gradients) are used as input data of the model, and even texture features are used as input data. For deep learning-based real-time facial feature point extraction using CE-CLM model, parallel processing using GPU is essential, and real-time performance of facial feature point extraction using CE-CLM model cannot be guaranteed in an environment without GPU. Therefore, we implemented a real-time facial feature extraction function suitable  
Figure 21: Face side asymmetry meter using facial feature points. Figure 20: Changes in the position of facial feature points by movement (**a**), before rigid body transformation (**b**), after rigid body transformation (**c**). for a GPU-free environment using face alignment provided by the dlib library. dlib's face alignment outputs two-dimensional facial feature points, and enables the extraction of facial feature points with a speed of 40 fps or more with only an operation using only the CPU (i7-6700).