# AU-Guided Unsupervised Domain-Adaptive Facial Expression Recognition
## 1 Introduction  
Facial expression is one of the most important modalities in human emotional communication. Accurately recognizing facial expressions helps individuals understand various human emotions and intents, which is applied in a wide range of applications, such as human-computer interaction [1], service robots [2], and medicinal treatments [3]. Both in industrial and academic areas, in past decades, many well-labeled datasets [4, 5, 6, 7, 8, 9, 10, 11, 12] and high-performance algorithms [13, 14, 15] have been proposed to automatically recognize facial expressions.  
In general, deep learning-based facial expression recognition (FER) methods [13, 14, 10, 11, 15, 9, 11] achieve high performance only when the training domain is identical or similar to the testing domain. However, due to the diversities of inconsistent annotation and different image collection conditions, there inevitably exists annotation biases (domain gaps) among different datasets, which poses an evident challenge for adapting FER models trained on one dataset to another one. As shown in Figure 1, a naive cross-domain method that deploys a trained model on a different target domain often fails. To this end, many methods have been presented methods for mitigating the domain shift in FER [16, 17, 18], though almost all the methods follow general domain adaption algorithms, ignoring the sibling facial action unit (AU) detection task. Action units (AUs) represent the movement of facial muscles, which have lower bias than subjective facial expression annotations. Intuitively, AUs can be regarded as auxiliary cues to alleviate the annotation and data biases (domain gaps) among different FER datasets.  
In this paper, considering the remarkable progress and stable performance of AU detection [19; 20], we propose an AU-guided unsupervised domain-adaptive FER (AdaFER). The AdaFER consists of two crucial modules: AU-guided annotating (AGA) and AU-guided triplet training (AGT). Given two groups of images from a source domain and a target domain, we first utilize an advanced pretrained AU detection model to extract the AUs' coding from both domains. Then, we perform AU-guided annotating as follows: for each image on the target domain, we use its AU's coding to query the source domain. All the images on a source domain with the same AU coding as the query image will be used for annotating. By default, the query image is assigned with a soft label, which is the statistic label distribution of the retrieval images. Further, to achieve structure-reliable and compact facial expression features, we perform the AU-guided triplet training. Each triplet is generated by comparing the AU coding of source and target images. For example, when a source image is used as an anchor, we randomly sample a positive (negative) sample that has same (different) AUs as the anchor from all target images. With AdaFER, we are able to fine-tune a source-pretrained model on a target domain with both pseudo-soft labels and triplet loss, which effectively prevents FER performance degradation on the target domain.  
Overall, our contributions can be summarized as follows:  
* We heuristically utilize the relationship between action units and facial expressions for cross-domain facial expression recognition, and propose an AU-guided unsupervised domain-adaptive FER (AdaFER) framework;
* We elaborately design an AU-guided annotation module to assign soft labels for a target domain and an AU-guided triplet training module to learn structure-reliable and compact facial expression features;
* We conduct extensive experiments on several popular benchmarks and significantly outperform the state-of-the-art methods.