Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-13 17:02:13

Abstract: Facial action units (AUs), as defined in the Facial Action Coding System (FACS), have received significant research interest owing to their diverse range of applications in facial state analysis. Current mainstream FAU recognition models have a notable limitation, i.e., focusing only on the accuracy of AU recognition and overlooking explanations of corresponding AU states. In this paper, we propose an end-to-end Vision-Language joint learning network for explainable FAU recognition (termed VL-FAU), which aims to reinforce AU representation capability and language interpretability through the integration of joint multimodal tasks. Specifically, VL-FAU brings together language models to generate fine-grained local muscle descriptions and distinguishable global face description when optimising FAU recognition. Through this, the global facial representation and its local AU representations will achieve higher distinguishability among different AUs and different subjects. In addition, multi-level AU representation learning is utilised to improve AU individual attention-aware representation capabilities based on the multi-scale combined facial stem feature. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance over the state-of-the-art methods on most of the metrics. In addition, compared with mainstream FAU recognition methods, VL-FAU can provide local- and global-level interpretability language descriptions with the AUs' predictions.

Main paper: 
 related work: 
 Facial AU Recognition. The facial AU recognition task is popularly treated as a multi-label classification problem, utilizing multi-branch independent AU classifiers to predict activation states. From this perspective, most existing methods can be roughly categorized into two groups: region-localization based AU recognition [1, 7, 8, 34, 35, 49] and global-refinement based AU recognition [12, 16, 25, 38, 39] . The former methods represent precise AU representations by local patch feature learning and maintain certain differences between different AU features. For instance, in some early works [41, 48] , there is a need to predefine the patch location first. Recent JAA-Net [34] joined AU recognition and face alignment in a unified framework, employing detected landmarks to locate specific AU regions. However, all the above methods focused only on independent regions, without considering non-defined face areas and AU correlations. The latter global-refinement based methods extract AU branch features by directly refining the global face representation to ensure that useful information is not lost. For example, [12] proposed a transformer-based multi-branch AU recognition network, where each branch leverages a shared global face representation and refines it by independent convolution layers. Similarly, the correlations between AUs are also ignored. Recent works [7, 8, 16, 26, 38] pay attention to capturing the relations among AUs for modeling the inherent muscle linkages, supplying information transfers between AU branches. For instance, [16, 23] utilized multiple GCNs to model AU relations, requiring additional pre-statistical AU correlation references from the training set. However, this inter-branch information The jaw is lowered about as much as it can drop from relaxing of the muscles.The interaction somewhat breaks the original categorical independence between branches, which in turn weakens the discriminability of AU individuals. Moreover, all the above methods are flawed regarding explainable FAU recognition.Multi-task Joint Learning. Recently, multi-task joint learning [8, 11, 31, 50] has been an emerging topic in deep learning research, which enables the same target to efficiently obtain multiple representation capabilities in a unified model. For example, [35] joined face alignment into a facial AU recognition framework to assist in locating the muscles corresponding to AUs. [31] combined four face tasks into a unified framework and assigned an attention module for each face analysis subnet. However, these methods only focus on computer vision tasks and ignore the modality complementary advantages of multiple modality-task joint learning, i.e. interpretability complementarity and semantic complementarity. For example, [13] proposed a joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection, which can provide image sentiment predictions with explicit language explanations. However, most visual-language joint learning models [11, 13, 50] rely on the joint input of both modalities during the inference process.In this paper, we argue that explicit language can facilitate the modeling of inter-AU relevance and diversity. Each AU description can include muscle details and relationship descriptions identified by AU, which not only replace the visual relational reasoning among AUs, but also ensure the independence of AUs to improve distinguishability. The most relevant research to ours is SEV-Net [46] , which introduced AU language descriptions as a prior embedding into AU representations. However, significant drawbacks are that SEV-Net relied on the human pre-given language descriptions of activated AUs for inference and followed the mainstream AU recognition paradigm in Figure 1 for only AU prediction. In contrast to [46] , our VL-FAU introduces language generation auxiliary models for local AU prediction and global face refinement. It focuses on end-to-end language-explainable facial-image AU recognition without any pre-given reference for inference. 
approach: 
 As shown in Fig. 2 , the proposed approach -VL-FAU-consists of two key components, i.e., multi-level AU representation learning, and local and global auxiliary language generation. The former contains dual-level AU individual refinement for multi-branch FAU recognition based on multi-scale feature combinations from Swin-Transformer. The latter consists of local AU language generation, which facilitates semantic supervision of each AU and global facial language generation for whole-face semantic supervision. Both of these can help AU representations become more robust and distinguishable, thus improving the performance. Thus, we create an end-to-end framework with a multi-label classifier for explainable FAU recognition, supervised with local and global AU language generation auxiliaries.3.1.1 Global Facial Representation Extraction. Given a face image 𝐼 , we adapt the widely-used Swin-Transformer as the stem network [24] to extract the global feature 𝑉 by combining the multiscale representations from different stages. Multi-scale feature learning and combination [16, [29] [30] [31] is a popular image representation approach to leverage different levels of semantics from the backbone, where the low-level features from shallow blocks contain more [21, 40] . While existing research from both perspectives is extensive [7, 25, 31, 44] , this study represents the first adaptation within independent FAU branches for enhancing the independent fine-grained attention-aware AU representations rather than coarse-grained global representation. Specifically, following [44] , we extract the max-pooled and average-pooled channel-level vectors (𝐹 𝑐 𝑚𝑎𝑥 and 𝐹 𝑐 𝑎𝑣𝑔 ) along the spatial axis for channel-wise attention mining. After that, two shared learnable multilayer perceptrons (MLP) are used to obtain the mapping of channelwise vectors and then a sigmoid function is applied to get the 𝑖-th individual AU-channel attention. Similarly, we later extract the maxpooled and average-pooled spatial-level vectors (𝐹 𝑠 𝑚𝑎𝑥 and 𝐹 𝑠 𝑎𝑣𝑔 ) along the channel axis for spatial-wise attention mining. We utilise a 3×3 convolution to generate a spatial attention map, which focuses on highly responsive muscle areas associated with the current AU. Finally, the 𝑖-th individual AU attention-aware representation V𝑖 can be obtained as below:EQUATIONEQUATION)where 𝜎 is the sigmoid function.In addition to improving AU individual representations in multiple AU branches, our main innovation is to provide a new approach and inspiration for explainable FAU recognition, i.e., joint auxiliary language generation for FAU recognition, rather than a new complex language model. Language generation provides explicit semantic supervision while giving linguistic interpretability of the corresponding identified AUs, rather than just simply recognising the AUs. It contains two aspects: (i) global face language generation for explicit semantic auxiliary supervision of the whole face image representation, and (ii) local AU language generation for individual AU semantic auxiliary supervision.Different from the mainstream facial AU recognition [16, 34] , we introduce a new global language auxiliary model to generate the language description for activated AUs of the whole face. It brings benefits for subsequent multi-branch FAU recognition, i.e., it maintains the accuracy and variability of intra-and inter-subject stem features by explicitly focusing on multiple different AU muscles. Thus, it somewhat overcomes the data imbalance, i.e., inactive AUs are far more numerous than activated AUs in benchmarks. Specifically, as shown in Figure 2 , we treat the multi-scale stemfeature 𝑉 as the encoded image feature and input it into an attentionaware language decoder similar to image captioning [45] , which contains a soft-attention module to mine the attention-aware visual representation based on the past generated word 𝑠 𝑔 0:𝑖 -1 for new word 𝑠 𝑔 𝑖 . For example, when generating current word 𝑠 𝑔 𝑖 , we first calculate the soft-attention 𝛼 𝑖 between the image feature 𝑉 and the last hidden state ℎ 𝑖 -1 of generated word 𝑠 𝑔 𝑖 -1 by linear-based mapping operations with SoftMax function [15] . And then the 𝑖-th attention-aware visual feature 𝛼 𝑖 𝑉 is combined with the last hidden state ℎ 𝑖 -1 , which is fed into the 𝑖-th LSTM cell [10] with the previous cell state 𝑐 𝑖 -1 to generate the new hidden state ℎ 𝑖 and cell state 𝑐 𝑖 of new word. The above process can be formulated as:EQUATIONEQUATIONwhere 𝑊 𝑣 ∈ R 𝑑 ×𝑑 , 𝑊 ℎ ∈ R ℎ×𝑑 , and 𝑊 𝑎 ∈ R 1×𝑑 are the parameters of mapping function. During the training process, we use a shared learnable parameter 𝑊 𝑠 ∈ R ℎ×𝑣𝑜𝑐 to obtain vocabulary-length predicted vector and obtain the max-score index as the predicted word, as follows:EQUATIONDuring inference process, the beam search [14] can be used to obtain the most optimal global description 𝑆 𝑔 = [𝑠 𝑔 1 , ..., 𝑠 𝑔 𝑇 ]. 3.2.2 Local Language Generation. We introduce an individual local language generation model for each AU branch as an auxiliary semantic supervision, which can generate the corresponding finegrained language description for each AU determination. Compared with the global facial description, the local AU language description contains more facial muscle details described for corresponding AUs. Besides, different AUs have different local language descriptions, which are more fine-grained and diverse. The main motivation of the joint language model in each AU branch is not only to improve the distinguishability between AUs using fine-grained semantic auxiliary supervision, but also to provide specific language interpretation for each AU prediction.In particular, different from global language generation, each local language model utilizes the proposed DAIR to further refine the encoded face feature 𝑉 , obtaining the distinguishable attentionaware AU representation V𝑖 for the subsequent language generation. The decoder architecture of each local language generation model is the same as the global language model. To save space, we use ∮ EQUATIONNote that, the local language models among the multiple AU branches are shared to maintain efficiency.VL-FAU joints facial AU recognition (vision) and description generation (language) into an end-to-end multi-branch network. Among them, FAU recognition is predominant due to the explicit AU annotations, while the language models are used for auxiliary semantic supervision and to improve feature diversity and distinguishability of visual stem and sub-branches. For facial AU recognition, one fully connected layer with SoftMax function is used as a multi-label binary classifier to classify the AU activation state, which adopts a weighted multi-label cross-entropy loss function as follows,EQUATIONEQUATIONwhere 𝑁 is AU number, 𝑦 𝑖 and 𝑝 (𝑦 𝑖 ) denote the ground-truth and predicted probability for the 𝑖-th AU occurrence, respectively. 𝛾 𝑖 is a balancing weight of the 𝑖-th AU calculated by the 𝑖-th AU occurrence rate 𝜖 𝑖 in the training set.Local AU language generation auxiliary training provides detailed semantic supervision for AU predictions and is optimised by commonly used negative log-likelihood loss, as follows:EQUATION)where N is the number of AU branches and T is the length of each description. The 𝑡-th word 𝑠 𝑙 𝑡 ;𝑖 of AU 𝑖 description is generated based on the previous words 𝑠 𝑙 [0:𝑡 -1];𝑖 and the AU 𝑖 refined visual feature V𝑖 .Moreover, the introduced global language generation is also optimized by a similar objective function in Eq. (10) EQUATIONDifferent from the local language generation for specific AU branches, the global language generation faces the challenge of linguistic semantic diversity due to facial state changes. To this end, we add a global AU classification loss L 𝐺𝑎𝑢 together with L 𝐺𝑔𝑒𝑛 as a constraint, where AU states are predicted by a shared linear classifier based on the average attention-aware visual feature from language model. Finally, the joint loss of our VL-FAU model can be optimized by maximizing the following lower bound:EQUATION 
experiments 4.1 dataset and implementation details: 
 Dataset. We provide evaluations on the widely used datasets, i.e. BP4D [47] and DISFA [27] . BP4D is a spontaneous FAU database containing 328 facial videos from 41 subjects (23 females and 18 males) who were involved in 8 sessions. Similar to [35, 36] , the most expressive frames are manually labeled with AU occurrence from each session (around 400-500 frames). Overall, BP4D contains 12 AUs and 140K valid frames with labels. DISFA consists of 27 participants (12 females and 15 males). Each participant has a video of 4, 845 frames. 8 AU annotations are selected following the popular studies [17, 35] . Compared to BP4D, the experimental protocol and lighting conditions deliver DISFA to be a more challenging dataset. Data Processing1 . During training, language descriptions of different AU states are hand-crafted, containing descriptions of both activated and inactivated AU states. Each local AU description contains multiple muscle details with potential associations according to FACS [5] . The global face description is generated from the AU annotations and FACS [5] , which only focus on the activated AU muscles. The examples are shown in Figure 5 . We evaluated the model using the common 3-fold subject-exclusive cross-validation protocol [8, 19, 34, 35] .Training strategy. The whole end-to-end network is implemented with PyTorch on a single NVIDIA RTX 3090Ti GPU using AdamW solver with 𝛽 1 = 0.9, 𝛽 2 = 0.999, and a weight decay of 0.0005. [39] 53.1 46.1 56.0 76.5 76.9 82.1 86.4 64.8 51.5 63.0 49.9 54.5 63.4 FAU-Trans (CVPR2021) [12] 51. 7 [18] 55. 3 horizontally flipped. We randomly employ the cutout augmentation [4] to overcome overfitting and improve the robustness during training. The stem extractor (Swin-Transformer-base) is pre-trained on ImageNet. The dimensionality of the global feature is mapped from 1024 to 512 (d=512), matching the hidden state dimension (h=512).We perform extensive experiments to compare our VL-FAU with mainstream FAU recognition studies and the latest state-of-the-art methods on two widely used FAU benchmarks in Table 1 and Table 2 . The best and second-best results are bold and underlined. Evaluation metrics. For all methods, the frame-based F1 score (F1frame, %) is reported, which is the harmonic mean of the Precision P and Recall R and calculated by F1 = 2PR/(P + R). To conduct a more comprehensive comparison with other methods, we also evaluate the accuracy performance (%). In addition, the average results over all AUs (denoted as Avg.) are computed with "%" omitted.Quantitative comparison on DISFA: We compare our proposed VL-FAU with its counterpart in Table 1 and Table 3 . Our VL-FAU outperforms mainstream studies with impressive margins. In particular, compared with the state-of-the-art AAR [37] , which joint surprised local attention maps to obtain the multi-branch attentionaware AU representation, our VL-FAU increases the average F1frame by 2.3% and shows clear improvements for most annotated AU categories. Compared with the best model KDSRL [1] on DISFA, the average F1-frame score of our VL-FAU is also improved from 64.5% to 66.5%. Furthermore, we achieve the best performance in terms of average accuracy in Table 3 , compared with all methods. Quantitative comparison on BP4D: FAU recognition results by different methods on BP4D are shown in Table 2 and Table 3 , where the proposed VL-FAU model achieves a new state of the art compared with all methods in terms of average F1-frame score. Our VL-FAU outperforms the baseline JAA-Net [34] , which integrates AU detection and face alignment, in terms of average F1-frame and accuracy by 5.8% and 1.4%, respectively. This is mainly because JAA-Net focuses on the local AU regions based on the detected landmarks, resulting in poor distinguishability between different AUs, especially for the same individual with subtly different face states, which can be improved by our method. Moreover, VL-FAU achieves the best In addition, compared with SEV-Net [46] , which prior encodes the pre-provided linguistic descriptions into image features, our VL-FAU achieves 7.7% and 1.9% higher average F1-frame scores on DISFA and BP4D, respectively. Experimental results demonstrate the effectiveness of VL-FAU in improving AU recognition accuracy on DISFA and BP4D by our proposed joint learning with language generation. Besides, as shown in Figure 3 , we also provide the multi-label performance balancing analysis on DISFA and BP4D. Although the performance varies greatly between different AUs due to the inherent category imbalance in datasets, our VL-FAU still achieves a better performance-balance than existing methods and maintains the best overall results.We perform extensive ablation studies on BP4D to investigate how each component affects the overall performance of the proposed VL-FAU. Due to space limitations, we do not show the ablation results for DISFA, but it is consistent with BP4D. Table 4 presents component ablation studies focusing on the various modules within VL-FAU, including (1) multi-level AU representation learning (MARL), (2) local language generation auxiliary (LLGA), and (3) global language generation auxiliary (GLGA). In addition, Figure 4 gives a qualitative analysis of local and global language generations.(1) Multi-level AU Representation Learning (MARL). Compared with the baseline model ①, the result has an improvement of average F1-frame from 64.3% to 64.7% when considering the proposed multi-level AU representation learning (indicated variant ② with MARL). It indicates that the dual-level individual AU refinement based on the multi-scale stem feature combination could get richer and more fine-grained AU features and hence improve recognition performance.(2) Local Language Generation Auxiliary (LLGA). In Table 4 , we test the contributions of local language generation auxiliary (LLGA) of our VL-FAU model. Compared with variant ② and variant ③, after we provide local language generation auxiliary for each AU branch, the average F1-frame score has been improved from 64.7% to 65.3%. In addition, most of the 12 AUs annotated in BP4D achieve significant improvements. These observations demonstrate that by providing each AU branch with local language generation as an explicit auxiliary semantic supervision, the discriminative ability between AUs becomes stronger due to the gain of language expressiveness rather than single visual appearance features. (3) Global Language Generation Auxiliary (GLGA). Similarly, we conduct a comparison between variant ② and variant ④ to verify the effectiveness of the proposed global language generation auxiliary (GLGA) for whole-face representation learning. Results in Table 4 show that the proposed VL-FAU facilitates the target FAU recognition task when using GLGA (indicated variant ④). In particular, the averaged F1-frame score increases from 64.7 % to 65.2 % and most AUs achieve significant improvements. These validate the effectiveness of the proposed GLGA which provides better discriminability of global representations by focusing on the language semantics of activated facial AUs, especially for the same subject with subtly different AU states. Finally, when both local and global language generation auxiliaries (indicated model ⑤) are considered, the proposed VL-FAU achieves the best recognition performance in terms of average F1, significantly better than the single variants (variant ③ and ④) and variant ② in Table 4 . In addition, the local and global language generation performances also have certain improvements, achieving 86.6% and 64.7% on top-5 accuracy of word generation in LLGA and GLGA. These quantitative comparisons experimentally demonstrate that exploring explicit semantic-auxiliary supervisions for facial AU recognition is a beneficial way for discriminating different AU states under intra-and inter-subject.(4) Qualitative Analysis of Local and Global Generation Auxiliary. Besides the above quantitative comparisons, we further proved the detailed qualitative analysis of our main innovations -local and global generation auxiliaries for facial AU recognition. As shown in Figure 4 , we use t-SNE [43] to visualize the AU features learned by the proposed VL-FAU and the corresponding baseline without local and global generation auxiliaries (baseline ②). Note that, both models are used the same multi-branch networks with MARL. Specifically, we extract the AU features of 8 random gender-balanced subjects for clearer visualization from multiple AU branches before the final classification and then visualize them by t-SNE. We provide different coloring schemes in Figure 4 to analyze the impacts of VL-FAU on different aspects, including AU category, subject ID, and gender. (1) Comparisons of clustered AU features in the first row, our VL-FAU can better divide different AU features into different clusters, while the baseline is not sufficiently discriminative on some AU features (marked with red circles). Besides, we notice that AU features optimized from baseline are mapped in a narrow space compared with our VL-FAU. These observations indicate that by joining language generation auxiliary, our VL-FAU can maintain higher discriminability between multiple AU representations. (2) The second row shows the subject ID coloring results. Our VL-FAU distinguishes different subject AU features more widely within the same AU cluster, indicating that our VL-FAU provides higher discriminability between different subjects for facial AU recognition.(3) Due to the explicit language supervision of gender information in GLGA, VL-FAU can achieve clearer gender colorization results, as shown in the last row of Figure 4 .To further understand the quality of the proposed vision-language joint learning for FAU recognition and description generation, we visualize the predicted AU states and their corresponding local and global-level descriptions, as shown in Figure 5 . Two positive examples from BP4D contain visualizations of different genders with different AU states. Compared with the mainstream paradigm, our VL-FAU provides explainable FAU recognition with language generations. In detail, local descriptions contain multiple detailed muscle changes with natural connections, improving intra-AU semantics and inter-AU distinguishability. In addition, the global descriptions contain diverse activated AU states with gender information, which can improve the inter-face distinguishability within and between subjects. Besides, we provide a bad case, which makes wrong predictions in two AUs. However, the global description ignores the misrecognition. Although AU6 is incorrectly predicted, the detailed AU1 inactivated : inner brows are not raised , the skin between the eyebrows and above the forehead is flat and unchanged , there are no forehead lines , and there are no wrinkles in in the center of the forehead . AU17 activated : the chin boss shows severe to extreme wrinkling as it is pushed up severely , and the lower lip is pushed up and out markedly .Global Description: man raises upper lip , raises chin .Global Description: woman raises inner brow, lower brow, raises chin.  
conclusion and future work: 
 In this paper, we proposed a novel end-to-end vision-language joint learning (VL-FAU) for explainable FAU recognition along with language generations as explanations. As auxiliary supervisions, local and global language generations are joined into a multi-branch AU recognition network with multi-level AU representation learning.Local AU language generation provides explicit fine-grained semantic supervision for each AU classification with detailed language descriptions, improving the discrimination of inter-AU representations. Global language generation, employing multi-scale combined stem features, offers diverse semantic supervision for the whole facial feature to maintain diversity and distinction across intra-and inter-subject representation changes. Our VL-FAU finally provides predictions of AU states as well as interpretable language descriptions for individual AUs and global faces. Extensive experimental evaluations on DISFA and BP4D show that our VL-FAU outperforms state-of-the-art AU recognition methods with impressive margins. VL-FAU introduces a traditional language model for explainable FAU recognition considering computational power and efficiency limitations. We believe our attempts provide new inspiration for multimodal multi-task joint training for explainable FAU recognition. In the future, we would like to investigate the further combination of FAU recognition with popular LLMs for more diverse and finegrained explainable generations.  
: 
 states. However, hand-crafted shallow features prove inadequate to address the major challenges of FAU recognition, i.e. identifying subtle facial changes induced by different AUs, as well as variations stemming from individual physiology. Instead, deep learning based studies [22, 26, 28, 35] have recently been explored to improve the AU representations as a multi-label classification problem, where the AU individual characteristics are adequately learned to boost the FAU recognition. Therefore, improving the individual AU representation becomes the key to boosting the FAU recognition task. The main challenge is how to maintain discriminative inter-AU features while ensuring rich semantic representation capabilities within AUs.To enhance the individual representation of each AU, mainstream studies [8, 16, 26, [35] [36] [37] [38] [39] adopt a multi-branch AU recognition framework with independent classifiers to decouple different AUs. For instance, JAA-Net [34] proposes a face alignment network that predefines local muscle regions for various AUs using detected landmarks. It then refines all AU region representations through multiple independent convolutional networks dedicated to different AU classifications. The predetermined AU local regions and their feature representations establish explicit and distinct branches for conveying information essential for recognizing different AU states. However, they overlook the pertinent information originating from undefined facial areas. Therefore, ARR [37] is proposed to directly leverage a shared global face feature to explore the local AU representations by subsequent independent convolution blocks in different AU branches, covering the whole face information. However, recent studies [7, 16, 23] suggest that the above methods fail to capture intrinsic relationships between independent AU branches, which are actually undeniable [5] . To this end, some works [2, 7, 8, 16, 23] propose to build relationships between predefined AU branches by prior relationship statistics in datasets or implicit relationship reasoning in the modelling networks, such as Long Short-Term Memory network (LSTM) [9] , graph convolutional network (GCN), etc.Despite such progress, two prevalent defects remain unexplored: (i) improving the representation capability of individual AUs through inter-AU relational connections may compromise the distinctiveness of features among AUs, and (ii) directly obtaining classification results lacks an explainable basis for judgments. The main reason for the former is that inter-AU information based relationship reasoning networks, such as GCN [16, 38] or GNN [7] etc., are prone to the over-smoothing problem [33] among the different AU nodes, especially on small-scale face datasets. This makes differentiating the AU states difficult. The latter issue is caused by the current mainstream paradigm as shown in Figure 1 , which only focuses on the classification results and ignores the corresponding explanations.In this work, we bring fresh insights toward explainable facial AU recognition via the integration of language generation supervisors within an end-to-end FAU recognition network. We argue that detailed AU language descriptions, such as "The corners of the lips are markedly raised and angled up obliquely. The nasolabial furrow has deepened ..." for activated AU12 (Lip Corner Puller) in Figure 1 , can provide more linguistic semantics and relations to corresponding AU appearance features than mainstream relational reasoning methods [7, 16, 26] . Furthermore, different AUs correspond to different descriptions, keeping them better distinguishable. Different from encoding pre-provided activated AU language descriptions into AU classification in [46] , we introduce the language generation model as language semantic supervision for the classification of different AU states. It can provide AU prediction with the explainable language descriptions as Figure 1 while providing sufficient semantic to enrich AU representations and supervising the AU detection pertinently to distinguish AU. Motivated by the above insights, we propose a novel end-to-end vision-language joint learning model (termed VL-FAU) for facial AU detection with explainable language generation. Compared with the mainstream complex methods in Figure 1 , our main innovations lie in two aspects: (i) exploring the potential of joint language generation auxiliary training for intra-AU semantic enhancement and inter-AU semantic feature differentiation and (ii) providing interpretability for end-to-end FAU recognition. Specifically, we first introduce a new dual-level AU feature refinement based on multiscale combined representations from a vision backbone. This way, we provide each AU branch with a unique attention-aware representation ability. Secondly, to improve the semantics and guarantee the distinguishability of AU features as well as their interpretability, we integrate the end-to-end multi-branch framework with the local and global language generations for explicit semantic guidance. On one hand, each local branch of AU recognition is simultaneously supervised by a corresponding local language generator via finegrained semantic-supervised optimization. Such schema constrains the semantics and relationships of AU features by local language modeling and improves the inter-branch distinguishability for each face image. On the other hand, global facial features within and between subjects are difficult to distinguish because muscle changes are mostly subtle under different states. To this end, we introduce a global language model to generate a description focusing on all activated muscles as global semantic supervision. It provides better distinction between different whole-face representations within and between subjects via multiple facial state foci as shown in Figure 1 .Finally, vision AU recognition and language description generation are jointly optimized.The contributions of this work are as follows:• We propose a novel end-to-end vision-language joint learning scheme for explainable FAU recognition (VL-FAU), which leverages auxiliary training of language generators to improve discrimination and explanation for FAU recognition.  
 
 Appendix: The details of language descriptions will be released with paper acceptance. 


Decision: Reviewers 4rXg concerned about the data processing and training details, while wNWr and RifT questioned the structure and computational efficiency of the AU caption generator. Apart from the discussion of technical details, all reviewers support the novelty idea. And the rebuttal addressed most concerns 


Therefore, I recommend acceptance. Please include a comparison of computational efficiency and details on constructing AU captions in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper studied facial action unit recognition with intrinsic relationships between different AU and explainability problems. They proposed an end-to-end Vision-Language joint learning network for explainable FAU recognition(VL-FAU), which contains a dual-level AU representation learning framework and a joint supervision method with local and global language generations.
strength_weakness: strength_weakness: This paper provides a novel vision-language joint learning method, which combines facial action unit recognition with language explaniability to solve the lack of explanation in existing facial action unit recognition methods. By providing explainable language descriptions at local and global levels, the VL-FAU method not only improves the accuracy of AU recognition, but also enhances the explainability of the model, which is of great significance for the application of facial expression analysis in human-computer interaction, emotion computing and other fields. Experiments are carried out on DISFA and BP4D AU datasets to verify the effectiveness of the proposed method and shows that VL-FAU has better performance than the existing methods.
limitations: limitations: There are some problems, which must be solved before it is considered for publication. If the following problems are well-addressed, the reviewer believes that the essential contribution of this paper is important for facial action unit recognition.

1. **Line 498-500**: It is not explained where the calculation of $p(y_i)$ comes from, whether it uses FAU-related features, or takes into account features that describe FAU.

2. **Line 548-549**: As shown in Equation 12, when designing the loss function, the author directly chose to add different loss functions without considering their previous coefficient relationship, which we think is not rigorous and needs to be considered.

3. **Line 564-572**: It is said here that the details of language description are not developed. We do not know whether the author's method in the training process directly uses hand-crafted data or trains according to those data. If it is the latter, the relevant details of training are not developed in paper. It is important for reviewers to evaluate the effectiveness of the method. We suggest you to put them into the main body of the paper.

4. **In Table 3**, the author compare the accuracy with the SOTA methods. This seems to be a very convincing comparison experiment, but the authors did not uniformly compare the SOTA methods(such as KS, ARR, SMA-ViT and SEV-Net) in Tables 1 and 2. In this case, we consider the experiment to be inadequate.

5. Acoording to **Line 195-197**, the authors said they extended FAU datasets with new local and global language descriptions for different facial muscle states to facilitate language-interpretable FAU recognition. However, we only saw three related examples in Section 4.4, which seemed to us more like a case study demonstrating the effectiveness of the method than an effort to expand the data set. Because we do not see any more relevant extensive data description, and the author does not have an objective evaluation standard for these AU descriptions.
paper_summary: paper_summary: This paper proposes a new end-to-end vision-language joint learning model called VL-FAU for explainable facial action unit (AU) recognition. This paper would fit well under the theme "Multimedia Content Understanding" and "Vision and Language"
strength_weakness: strength_weakness: Integrating auxiliary language generation tasks with the AU recognition network to improve representation capability and provide linguistic interpretability is a unique approach. The use of local language generation modules for individual AU branches and a global language module for the whole face representation is an innovative idea to enhance discriminability and interpretability.

The paper provides a sound theoretical basis for the vision-language joint learning by leveraging language generation as an explicit semantic supervision for facial AU recognition.
The use of local descriptions to capture intra-AU semantics and inter-AU discriminability, and global descriptions to maintain diversity across facial states is well-motivated.

Established techniques like attention mechanisms, LSTM language models, and multi-scale feature fusion are adapted appropriately.

Quantitative results demonstrate the superiority of VL-FAU over previous state-of-the-art methods in terms of AU recognition performance.

The paper is well-organized and clearly describes the motivation, approach, and experiments. The use of informative figures aids in understanding the overall architecture and workflow.

Explainable facial AU recognition has diverse applications in fields like facial expression analysis, psychology, deception detection, and human-computer interaction.
limitations: limitations: The paper primarily focuses on the AU recognition performance and provides qualitative examples of the generated descriptions. However, a linguistic analysis evaluating the quality, diversity, and coherence of the generated text descriptions is lacking.

The inclusion of multiple language models along with the vision backbone could raise efficiency and scalability concerns, especially for deployment on resource-constrained devices. The paper does not discuss computational requirements and efficiency.

The paper uses traditional LSTM-based language models for generation. However, it does not explore or compare the performance with more recent transformer-based or large language models, which could potentially improve the quality and diversity of generated descriptions.
The experiments are limited to two facial AU datasets. The evaluation could be strengthened by testing the model's generalization capabilities on facial images from more diverse scenarios.
title: title: This paper utilizes language description to obtain more detailed AU detection, which is useful. But there are still some questions for speed and experimental designs.
paper_summary: paper_summary: This paper utilizes language description to obtain more detailed explainable AU detection, which is useful and novel. However, there are still some questions about inference speed and detailed experimental designs.
main_review: main_review: Pros:
1. This work achieves explainable AU detection by integrating language into the AU detection task. It changes a binary classification task into a caption generation task and classification task.
2. The performance seems great.

Cons:
1. The choice of the language model. How about using the pre-trained large language model to generate captions and help the classification task?
2. How about training time and inference speed?
3. Why does AU6 perform well in the BP4D dataset but badly in the DISFA dataset?
strength_weakness: strength_weakness: 1. This work achieves explainable AU detection by ingeniously integrating language into the AU detection task. By transforming a traditional binary classification task into a combined caption generation and classification task, it not only identifies AUs but also provides textual descriptions, enhancing the interpretability and transparency of the model's predictions. This novel approach opens avenues for deeper understanding and analysis of facial expressions.
2. The performance metrics indeed indicate remarkable results. The robustness and accuracy demonstrated in the experimental outcomes underscore the effectiveness of the proposed methodology. Additionally, the integration of language not only enhances performance but also enriches the output by providing descriptive captions alongside AU classifications, making the results more informative and actionable.
limitations: limitations: 1. The choice of the language model. How about using the pre-trained large language model to generate captions and help the classification task? 
2. How about training time and inference speed? Does the caption generation process cost too much?
3. Why does AU6 perform well in the BP4D dataset but badly in the DISFA dataset?