# Viewpoint Robustness of Automated Facial Action Unit Detection Systems
## 4 Discussion  
We evaluated the robustness of three automated facial action detection systems to changes in facial angle by calculating the accuracy of predicted AUs. OpenFace had the best performance, followed by FaceReader and Py-Feat. The performance was significantly lower at 45\({}^{\circ}\), and there was no significant performance difference among 0\({}^{\circ}\), 15\({}^{\circ}\) and 30\({}^{\circ}\). Importantly, the robustness to angle changes differed among the systems. FaceReader had markedly poorer performance at 45\({}^{\circ}\). For OpenFace, performance was similar between 0\({}^{\circ}\) and 15\({}^{\circ}\), and between 30\({}^{\circ}\) and 45\({}^{\circ}\). However, the performance gradually decreased as the angle moved toward the profile view. Although the performance of Py-feat was poorer than the other two systems, it did not vary by angle. FaceReader provided inaccurate results at 45\({}^{\circ}\). The poor performance of FaceReader for recognizing dynamic facial expressions reported in a previous study [19] may be due to its vulnerability to angle changes. In this study, the AUC values were significantly lower at 45\({}^{\circ}\), especially for AUs 1 (inner brow raised), 2 (outer brow raised), 5 (upper lid raised), 25 (lips parted), and 26 (jaw dropped). These AUs correspond to the prototypical surprised facial expression [8]. Therefore, FaceReader may not be suitable for evaluating the prototypical surprised expression, suggesting that future models should learn to recognize that expression from various angles. OpenFace had the best AU detection performance among the systems evaluated in this study, consistent with the previous findings pertaining to dynamic facial expressions [19]. Among the pre-trained models, OpenFace had the best AU detection performance based on facial expression images. As shown in Figure 3, the AUC values for AUs 14 (dimpled), 15 (lip corner depressed), 17 (chin raised), and 23 (lip tightened) were lower at 45\({}^{\circ}\) compared to the frontal view. When using OpenFace, recognition of the lower part of the facial expression was likely to be affected by angle, whereas for static facial images, the AUC values were high for AUs 2 (outer brow raised), 9 (nose wrinkled), 12 (lip corner pulled), 17 (chin raised), and 25 (lips parted), even at 45\({}^{\circ}\). Although the overall performance of Py-Feat was not superior to that of FaceReader and OpenFace, it provided robust results at 0-45\({}^{\circ}\). Therefore, the use of a large amount of \"wild\" data (i.e., Aff-Wild2 [38; 39; 40; 41; 42; 43]), including various angles, for model pre-training may have been effective. The lower performance for the frontal view compared to the other angles for AU 23 may have been related to the properties of the training data. Importantly, despite the lower AU detection performance in the frontal view of Py-Feat compared to the other two systems, the detection of AU 5 (upper lid raised) was superior for Py-Feat (Figure 3). Therefore, Py-Feat may be particularly useful for evaluating eyelid opening in a static facial image. Novice users should use a pre-trained automated facial action detection system; OpenFace may be the best choice. The performance of OpenFace is affected by facial angle, but it was still more robust to angle changes than FaceReader (Figure 2). Despite the inadequate performance of Py-Feat in the current study, it is still very promising. Py-Feat uses multiple models for face detection (e.g., MTCNN [48] and FaceBoxes [49]), AUs (e.g., JAANET [10] and DRML [50]), and facial landmarks (e.g., PFLD [51] and MobileFaceNet [52]). In this study, we used the default settings, which are the best-performing combination [18], although other combinations of pre-trained models may be used depending on the properties of the target facial expressions. Moreover, the settings of Py-Feat and OpenFace can be fine-tuned. The overall performance of Py-Feat can be improved by using a database of frontal view facial expressions as training data.