# Racial Identity-Aware Facial Expression Recognition Using Deep Convolutional Neural Networks
## 1 Introduction  
The aim of a facial expression recognition system is to recognize the discrete categories of facial expressions such as happy, sad, surprise, angry, neutral, fear, contempt and disgust from still images or video sequences. Though several recent studies focus on image sequence based facial expression recognition tasks [1], still image based facial expression recognition remains a difficult problem due to the following three reasons. First, the facial structure variations among the subjects of different cultures thus make the classification task difficult in some cases [2]. Second, inter-expression resemblance between some expressions might be significant and thus challenging to recognize them accurately [3]. Third, different subjects express their emotions in different ways due to facial appearance variation and facial biometric shapes [4]. These three challenging issues can be visualized in Figure 1. It contains five representative faces, labeled as \"Moroccan\", \"Caucasian\", \"Taiwanese\", \"American\", \"Japanese\" cultural, ethnic and geographic region, respectively. However, different subjects have great variations in facial expression representations due to inter-expression resemblance, facial structure and facial appearance. Generally, the cross-culture variability of facial structure could possibly lead to incorrect recognition of facial expression because the facial image of an emotion from one culture could be very dissimilar to facial images of the same emotion from other cultures (e.g., the facial images of \"angry\" emotion in Figure 1). Furthermore, psychological studies show that facial expression representation varies from culture to culture, and subjects from different cultures express different arousal levels to present facial expressions [5]. Therefore, we believe that, with the inclusion of a compact description of racial identity as an auxiliary input to the model, the facial expression recognition system becomes more reliable and robust against cross-culture variations just as the role of culture identity in speech recognition systems [6]. Convolutional neural networks outperformed the traditional approaches and even human beings in other fields such as plant diseases identification [7], IoT based smart parking system [8], Brain Tumor Segmentation [9], and so on facial expression recognition [10]. Moreover, in cross-culture facial expression recognition process, there is still a need to enhance the performance of a cross-culture facial expression recognition system. Intra-culture and inter-culture expressions' variability, unreliable expression labels, and lack of a large-scale multi-culture facial expression dataset restrain the performance of cross-culture facial expression recognition techniques. Therefore, in this study, we developed a multi-culture facial expression dataset to jointly learn the cultural identity and facial expression for cross-culture facial expression recognition. In this paper, a novel joint deep learning technique is developed to recognize the racial identity aware facial expressions. The proposed racial identity-aware network (RIA-Net) learns the facial expressions from facial images and extracts the racial identity features from a pre-trained racial identity network (RI-Net). The RI-Net is trained using the multi-culture dataset with culture annotations such as Japanese, Taiwanese, American, Caucasian and Moroccan. The fusion of expression features and racial identity features makes the model robust to the racial identity variations. Therefore, this study makes a major contribution to research on multi-culture facial expression recognition by learning the cultural variations in facial expression representation. To the best of our knowledge, there is no literature work using a racial identity feature with expression features merged to jointly learn the multi-culture facial expression representation. In the following sections, we first provide an overview of related works in Section 2.