Domain Knowledge Enhanced Vision-Language Pretrained Model for Dynamic Facial Expression Recognition

Authors: liupeng  li
shupeng  liu
xiaoyin  xu
taihao 2024 li
yuhua  zheng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:14:16

Abstract: Dynamic facial expression recognition (DFER) is a rapidly developing field that focuses on recognizing facial expressions in video sequences. However, the complex temporal modeling caused by noisy frames, along with the limited training data significantly hinder the further development of DFER. Previous efforts in this domain have been limited as they tackled these issues separately. Inspired by recent advances of pretrained vision-language models (e.g., CLIP), we propose to leverage it to jointly address the two limitations in DFER. Since the raw CLIP model lacks the ability to model temporal relationships and determine the optimal task-related textual prompts, we utilize DFER-specific domain knowledge, including characteristics of temporal correlations and relationships between facial behavior descriptions at different levels, to guide the adaptation of CLIP to DFER. Specifically, we propose enhancements to CLIP's visual encoder through the design of a hierarchical video encoder that captures both short- and long-term temporal correlations in DFER. Meanwhile, we align facial expressions with action units through prior knowledge to construct semantically rich textual prompts, which are further enhanced with visual contents. Furthermore, we introduce a class-aware consistency regularization mechanism that adaptively filters out noisy frames, bolstering the model's robustness against interference. Extensive experiments on three in-the-wild dynamic facial expression datasets demonstrate that our method outperforms the state-of-the-art DFER approaches. The code is available at https://github.com/liliupeng28/DK-CLIP.

Main paper: 
 introduction: 
 Facial expression recognition (FER) has garnered significant attention in the computer vision community due to its crucial role in various applications, such as human-computer interaction (HCI) [5, 7, 26] , driver or student status monitoring [14, 22] , and medical diagnosis [2, 17] . According to the type of input data, FER can be categorized into static FER (SFER) and dynamic FER (DFER). SFER focuses on static facial images, while DFER deals with dynamic image sequences. The continuity of human emotional expression implies that dynamic image sequences offer richer information regarding the genuine emotional states, thereby facilitating more accurate emotion recognition. Consequently, DFER receives increasing attentions owing to its significance in practical applications.Previous works in DFER primarily focus on capturing temporal correlations within video frames using approaches based on 3DCNNs [4, 15] , RNNs [1, 11, 49] and Transformers [20, 25, 28, 50] . Although these methods have achieved notable performance, the issues of complex temporal correlations and scarce training data still hinder the progress of DFER. Firstly, the existence of non-target frames complicates the modeling of temporal correlations in DFER, distinguishing it from general video understanding tasks. As shown in Fig. 1 (a), dynamic facial sequences often exhibit a transition between target and non-target emotions. Meanwhile, current facial expression datasets typically offer only video-level labels, lacking snippet-level annotations with precise temporal locations. Models trained with this type of supervision are susceptible to producing inaccurate predictions owing to the disruptive influence of noisy frames. Recently, researchers have attempted to address the complex temporal correlations modeling in DFER. NR-DFERNet [21] mitigates the impact of noisy frames by introducing a dynamic class token and a snippet-based filter. M3DFEL [39] proposes mod- eling DFER as a multi-instance learning problem, where non-target frames are disregarded while target frames are emphasized. Secondly, the prohibitive cost of annotating facial expression data limits the availability of training data for previous DFER methods, thereby restricting their further advancement. MARLIN [3] and VideoMAE [34] leverage large-scale self-supervised pretraining on unlabeled facial videos to address the constraints posed by the limited training data in existing datasets. Despite their state-of-the-art performance, these methods exhibit notable limitations, including substantial pretraining costs, task-independent representations, and ignoring the impact of noisy frames. Most recently, the advancements in adapting vision-language pretraining models, such as CLIP [31] , to various downstream computer vision tasks [29, 52, 53] demonstrate their remarkable transfer and generalization abilities. CLIPER [19] and DFER-CLIP [51] finetune the pretrained CLIP model to promote the development of DFER, without pretraining on unlabeled facial videos. Although these CLIP-based methods have achieved excellent DFER performance, they still employ simple temporal average pooling or single-layer Transformers for feature aggregation across frames, ignoring the detrimental impact of noisy frames on learning robust temporal facial features. In addition, CLIPER necessitates a pretraining phase to learn multiple text descriptors for expressions, whereas DFER-CLIP entails carefully guiding large language models (LLMs) to initially produce descriptions of facial behaviors associated with expressions. The process of training or generating text descriptions is not only time demanding but also fails to ensure the model's generalization across diverse visual samples.To jointly address the issues of noisy frames and limited training data, we suggest utilizing the domain knowledge in DFER, including characteristics of temporal correlations and relationships between facial behavior descriptions at different levels, to effectively guide CLIP in adapting to DFER. In other words, we incorporate DFER-specific Domain Knowledge into the CLIP model, dubbed DK-CLIP, to further promote its performance for the DFER task. To tackle the issue of noisy frames, we present a hierarchical video encoder equipped with a cross-frame attention module to capture short-term temporal correlations in local snippets and a multi-snippets integration module to model long-term temporal correlations among these snippets. Subsequently, we introduce a class-aware consistency regularization mechanism aimed at mitigating the detrimental effects of noisy frames and bolstering the model's robustness through a noise reducing module and a consistency loss. As for the challenge of limited training data, our method aligns with existing works [19, 51] , reducing the need for labeled data by fine-tuning pretraining parameters. Furthermore, since the integration of expression class names with textual descriptions of facial action units (AUs) [9] offers semantic information that is highly pertinent to visual facial content (as illustrated in Fig. 1 (b )), we leverage the descriptions of AUs to enrich the semantic content of text prompt for basic expressions. Additionally, we condition the learning of textual prompts on visual features using a cross-modal attention module, enabling the model to adapt to diverse sample variations. To verify the effectiveness of DK-CLIP, we conduct experiments on three in-the-wild datasets (DFEW [15] , FERV39K [41] , MAFW [24] ). The results demonstrate that the proposed DK-CLIP outperforms the state-of-the-art methods, highlighting its ability to mitigate the effects of limited training data and noisy frames.In summary, our key contributions are as follows:• We introduce DK-CLIP, a novel CLIP-based approach that jointly addresses the limitations posed by insufficient training data and noisy frames in DFER. • DK-CLIP not only offers an effective hierarchical video encoder to modeling complex temporal correlations amidst noise interference, but also harnesses prior knowledge to steer task-related textual prompt generation. With the incorporation of the class-aware consistency regularization mechanism, DK-CLIP is also applicable for fine-grained expression analysis and expression intensity prediction in a weakly supervised setting. • We conduct extensive experiments on three DFER datasets, and our proposed DK-CLIP achieves state-of-the-art results compared with other methods. 
related works 2.1 dynamic facial expression recognition: 
 Deep learning has significantly promoted the advancement of DFER.Early works [11, 15, 18, 49] typically employ RNNs and 3DCNNs to model the temporal relationships among frames. Furthermore, several studies [28, 50] exploit the global dependency modeling ability of Transformer to extract spatial-temporal features from facial video sequences. However, these methods exhibit limited performance since they ignore the varying expression intensities and noisy frames within dynamic facial sequences. Recently, IAL [20] introduces an intensity-aware loss, enabling the network to distinguish frames based on their expression intensities and pay extra attention to low-intensity sequences. NR-DFERNet [21] employs a dynamic class token and a snippet-based filter to mitigate the impact of target irrelevant frames during temporal modeling and the decision-making stage, respectively. Additionally, M3DFEL [39] formulates DFER as a multi-instance learning problem, treating each video as a bag containing target and non-target instances, and devises a dynamic long-term instance aggregation module to learn the long-term temporal relationships and dynamically aggregate the instances. Although these methods have achieved good performance, they are limited by the insufficient training samples in the existing DFER datasets.To address the challenge of limited training samples, large-scale self-supervised pretrained methods have been proposed. For instance, MARLIN [3] and MAE-DFER [34] pretrain masked autoencoders on massive non-annotated facial videos, and then finetune them for DFER. The self-supervised pertained models exhibit notable limitations, including substantial pretraining costs and taskindependent representations. Furthermore, CLIPER [19] and DFER-CLIP [51] directly adapt the pretrained CLIP to DFER, enabling task-related representation learning and obviating the need for additional pretraining processes. However, these models primarily learn temporal dependencies through video masking autoencoders (VideoMAEs [36] ) or temporal average pooling, struggling to model complex temporal dependencies in the presence of noisy frames.Unlike existing CLIP-based methods for DFER, we propose leveraging unique domain knowledge in DFER to enable CLIP to handle complex temporal dependencies, thereby simultaneously addressing the issues of noise frames and limited training data.Vision-language pretraining [48] has demonstrated remarkable generalization capabilities to various downstream computer vision tasks. One of the most representative works is CLIP [31] , which leverages contrastive learning to jointly learn image-language representations from a large-scale dataset, comprising 400 million image-text pairs sourced from the web. CLIP's remarkable transferability has propelled advancements in various downstream tasks [16, 27, 40, 43] . A critical challenge in adapting the pretrained CLIP for downstream visual tasks is prompt engineering, which is highly time-consuming due to the substantial impact of minor wording variation on performance. CoOp [53] first pioneers the application of context optimization for automatic prompt engineering, introducing learnable prompts for textual inputs, rather than relying on hand-crafted templates. The follow-up CoCoOp [52] utilizes visual feature to generate input-conditional prompts, thereby enhancing generalization abilities. In addition, the unsuitability of the vision encoder devised for images in handling videos potentially restrict CLIP's application in video understanding tasks. X-CLIP [29] introduces a cross-frame attention mechanism to capture longrange frame dependencies along the temporal dimension, enabling CLIP to process videos as input. Moreover, CLIP-VIP [44] facilitates CLIP's ability to handle both images and videos via a proxy-guided video attention mechanism. In addition to the modifications suggested by existing methods, we further leverage domain knowledge to inform the design of video encoders and the learning of textual prompts, enhancing CLIP's adaptability to DFER.Facial action units (AUs) [10, 35] encode subtle facial behaviors commonly observed during the production of facial expressions. For instance, AU4, which describes the facial behavior of lowering eyebrows, is often associated with expressions of sad, fear, and angry. Actually, the categories of facial expressions provide holistic descriptions of facial behaviors, whereas AUs capture local variations on faces. The pioneering work [9] comprehensively summarizes the relationships between expressions and AUs, offering prototypical AUs for each basic expression. Given that AUs offer more detailed information about different expressions, several studies [6, 23, 30, 45] utilize them to enhance FER performance. Nonetheless, owing to the costly and labor-intensive nature of AU annotation, these studies often rely on prior relationships between AUs and expressions as supplementary supervision signals during model training. In our study, we utilize the textual descriptions of relevant AUs to enrich the semantic content associated with each expression class name. 
methodology 3.1 revisiting clip: 
 CLIP [31] comprises an image encoder and a text encoder to extract image and text features, respectively. After obtaining visual and textual features, CLIP learns a joint encoding space for both modalities using a contrastive loss based on the cosine similarity between the two features. Consequently, the prediction process of the image classification task based on the pretrained CLIP can be formulated as follows:EQUATIONwhere sim(•, •) denotes cosine similarity, 𝐾 represents the number of classes, and 𝜏 is a temperature parameter. The image feature x and the textual descriptions that specify the class c 𝑦 are extracted using the image encoder and text encoder of the pretrained CLIP. Our work is focused on effectively adapting CLIP for DFER through the design of an efficient video encoder and learnable textual prompts.As shown in Fig. 2 , our proposed method comprises visual feature extraction, textual feature extraction, and prediction. In the visual feature extraction part, facial expression sequences, consisting of 𝑇 frames, are first embedded into patch embeddings, which are subsequently divided into 𝑁 snippets. Following this, the crossframe attention module extracts local visual features, while the multi-snippet integration module extracts global visual features. In the textual feature extraction part, the class names of expressions are aligned with AUs based on prior knowledge summarized from previous work [9] . Subsequently, the cross-modal attention module takes textual features and noise-free visual features as inputs to learn textual prompts tailored for DFER. In the prediction part, the noise reducing module is employed to erase noisy local snippets. Then, all extracted visual features, excluding noisy ones, are utilized to calculate similarity with textual prompts for classification and engage in consistency regularization. We will elaborate on these components in the subsequent sections.The primary challenge in adapting CLIP to DFER lies in how to learn robust and discriminative visual features through effectively Specifically, given a video clip V ∈ R 𝑇 ×𝐻 ×𝑊 ×3 , consisting of 𝑇 sampled frames with 𝐻 and 𝑊 denoting the spatial resolution, we follow the Vision Transformer (ViT) [8] to divide the 𝑡-th frame into 𝑀 non-overlapping patches and subsequently embed them into patch embeddings, denoted as {x 𝑡,𝑖 } 𝑀 𝑖=1 , where 𝑡 ∈ {1, • • • ,𝑇 } denotes the absolute temporal index. Then, the patch embeddings of 𝑇 frames are evenly divided into 𝑁 snippets. To capture temporal correlations within each local snippet, we incorporate a cross-frame attention module (CFAM) composed of an L-layer Transformer. The input to the CFAM at frame t in the 𝑛-th snippet is denoted as:• • • •• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • * • • • • ℒ • ℒ • ℒ • • • • • • • • • • • • •EQUATIONwhere x 𝑛 t,𝑐 is the learnable class token, t is the relative temporal index in local snippets, and e 𝑠𝑝𝑎 represents the spatial position encoding. Subsequently, we utilize the message token m 𝑛 t , linearly transformed from x 𝑛 t,𝑐 , to abstract and exchange visual information across frames in each local snippet. Specifically, in the l-th layer of the CFAM, the message tokens of 𝑛-th snippet, denoted asm 𝑛 = [m 𝑛 1 , m 𝑛 2 , • • • , m 𝑛 T ], are utilized to capture spatio-temporal relationships among local frames in the local snippet, which is formulated as:EQUATIONwhere LN indicates layer normalization and MHSA represents multihead self-attention [38] . Subsequently, z 𝑛 t and m𝑛 t are concatenated to diffuse spatio-temporal dependencies across each frame, which defined as following:EQUATIONwhere [•, •] concatenates the frame tokens and message tokens.Ultimately, a snippet-level representation s 𝑛 is derived by:EQUATIONAs a result, the CFAM outputs a collection of snippet features, denoted ass = [s 1 , s 2 , • • • , s 𝑁 ].To derive a comprehensive representation for the entire video clip V, we introduce a multi-snippet integration module (MSIM) to synthesize local snippet features into a single video-level feature v, which is expressed as:EQUATIONwhere e 𝑡𝑒𝑚 is the relative temporal position embedding, AvgPool denotes the average pooling, and FFN indicates the feed-forward network.Existing works [19, 51] have demonstrated that constructing suitable textual prompts plays a key role in adapting pretrained visionlanguage models to DFER task. However, the ways they generate textual prompts involve complicated preprocessing steps that typically require a significant amount of time. Therefore, we propose a straightforward approach to enrich textual prompts by combining the class names of expressions and the textual descriptions of relevant AUs, guiding the model to focus on facial regions related to expressions. Furthermore, we leverage visual contents to enhance the constructed textual prompts, enabling the model to learn relevant information for each expression.Enriching Textual Prompts with AUs. Traditional ways of adapting CLIP to recognition tasks often rely on manually designed textual prompts, such as "A photo of a {class}" or simply "{class}", where "class" primarily refers to objects or scenes. However, due to the semantic gap between class names and visual contents, these traditional ways are not suitable for adapting CLIP to DFER. According to existing works [10, 35] , it is known that AUs describe facial behaviors closely related to basic expressions (excluding Neutral). As suggested in previous literature [9] that each expression is associated with several AUs, we consider aligning basic expressions with the primary relevant AUs to generate more semantically rich textual prompts, as shown in Tab. 1. Specifically, we concatenate the class name with the descriptions of related AUs to construct textual prompts. Meanwhile, we use "or" to indicate the semantics of AUs that are not always activated simultaneously. For instance, the textual prompt for "Happiness" is reformulated as "Happiness with cheek raiser, lip corner puller or lip part.", where "cheek raiser", "lip corner puller", and "lip part" are the textual descriptions of AU6, AU12 and AU25, respectively. Notably, the textual prompts for expressions without relevant AUs consist solely of the class name.Enhancing Textual Prompts with Visual Contents. Although textual prompts for basic expressions are enriched through AUs, the manually designed prompts remain inadequate for generalizing across varied visual situations. We follow the conditional prompt learning mechanism [29, 52] , leveraging visual content information to enhance textual prompts. Specifically, given a set of text embeddings c = [c 1 , c 2 , • • • , c 𝐾 ] derived from enriched textual prompts, we propose a cross-modal attention module (CMAM) to facilitate conditional prompt learning, which is formalized as following:EQUATIONEQUATIONwhere ĉ represents the enhanced prompts, 𝛾 denotes a learnable parameter initialized at 0.1, and z 𝑡 is the ultimate output of the 𝑡-th frame obtained from the CFAM, excluding the class tokens. In MHSA, we utilize the text representation c as the query and the visual content representation z as both the key and value.The previous work [21] has verified that noisy frames significantly impact the performance of DFER methods, while the current datasets lack supervised information for their identification. To mitigate the influence of noisy frames on the visual and textual features, Noise Reducing Module. Due to lacking supervision signals to identify noisy frames, we introduce a simple yet efficient approach to adaptively perceive noisy frames and mitigate their adverse effects on DFER, i.e., Noise Reducing Module (NRM). The NRM perceives and erases the noisy frames by leveraging visual similarities between local snippet features and the holistic feature as indicators. Specifically, we compute the similarity between each snippet feature s 𝑛 and the holistic feature v, resulting in a set of similarity scores denoted as a= [𝑎 1 , 𝑎 2 , • • • , 𝑎 𝑁 ].Then, we employ the normalized similarity scores a * to identify potential noisy snippet features and erase them, which can be defined as follows:EQUATIONwhere 𝜃 represents a pre-determined hyperparameter and ∅ denotes empty vector. The remained snippet features then constitute a new set s * , containing local features that are less impacted by noisy frames. Additionally, only 𝑇 * frames, excluding those identified as noisy, are utilized in computing z * . Meanwhile, z in Eq. 8 is replaced by z * to mitigate the impact of noisy frames on textual prompts. Consistency Regularization Loss. Due to noisy frames are identified based on local snippet features and holistic feature in an unsupervised manner, the precision of these identified noisy frames cannot be guaranteed in the early training phases. The inconsistency between the two features may result in erasing keyframes incorrectly. Meanwhile, the influence of noisy frames on the overall holistic visual features remain unchanged. Actually, the presence of noisy frames or erasing keyframes is equivalent to introduce perturbations to the sample, potentially resulting in inconsistency between local snippet features and holistic feature. Consequently, we introduce the consistency regularization employed in semisupervised learning [33, 46] to enforce similar predictions for perturbed versions of the same sample. Specifically, by utilizing the holistic feature v and the text features ĉ, we can compute the global prediction result p 𝑔 using Eq. 1. Additionally, we can derive a set of local prediction results based on the refined snippet features s * . Subsequently, we generate a unified local prediction p 𝑙 by fusing local predictions, which can be formulated as: EQUATIONEQUATIONwhere L 𝑔 and L 𝑙 represent the cross-entropy loss of p 𝑔 and p 𝑙 , respectively. L 𝑘 denotes the KL-loss [13] , which measures the KL-Divergence between p 𝑔 and p 𝑙 . 𝜆 is a scalar hyperparameter that controls the weight of the consistency regularization loss, which comprises L 𝑙 and L 𝑘 . We enforce consistent prediction outputs for the same sample across varying perturbations by aligning predicted categories and probability distributions. This not only enhances the NRM's capability to perceive noisy frames but also mitigate the influence of noisy features on global feature. 
experiments 4.1 experimental setup: 
 Databases. We conduct experiments on three in-the-wild datasets, including DFEW [15] , FERV39K [41] , and MAFW [24] . DFEW consists of 16,372 video clips collected from more than 1,500 movies, and each video clip is annotated with seven basic expressions, including Happiness, Sadness, Neutral, Anger, Surprise, Disgust, and Fear. FERV39K comprises 38,935 video clips collected from different scenarios, and each video clip is assigned to one of the seven primary expressions as in DFEW. MAFW is a multi-modal compound affective database with 10,045 video clips. Each video clip is annotated with 11 compound expressions (containing Contempt, Anxiety, Helplessness, Disappointment, and seven basic expressions) and a textual description of the subject's affective behavior. We only use the video modality to evaluate our proposed method. DFEW and MAFW provide data partitioning settings for 5-fold cross-validation, while FERV39K splits all data into training and test sets without overlapping. To ensure a fair comparison with other methods, we conduct experiments on the default setting provided by each dataset. Implementation Details. Our proposed method is based on the pretrained CLIP that uses ViT [8] as the image encoder and a transformer model [32] as the text encoder. Specifically, the patch embedding module and text encoder of our proposed method inherit the weights directly from the pretrained CLIP, while the CFAM partially inherits the weights from the pretrained ViT. The MSIM and CMAM are randomly initialized. All parameters of CFAM, MSIM, and CMAM are adjusted during training, while the parameters of the patch embedding and text encoder remain frozen. Our proposed model is trained for 50 epochs with five warm-up epochs using the AdamW optimizer. The learning rate, the weight decay, and the batch size are set to 2e-6, 0.001, and 8, respectively. We use data augmentation techniques consisting of Fmix [12] and Mixup [47] to increase the diversity of training data. We sample 16 frames for each video (𝑇 = 16), and divide them into 4 parts as inputs (𝑁 = 4). Meanwhile, we align only six basic expressions with AUs for all datasets. We implement the proposed method with the Pytorch toolbox and train it on 2 Tesla V100 GPUs. In all experiments, we take unweighted average recall (UAR, i.e., the average accuracy of each class) and weighted average recall (WAR, i.e., accuracy) as metrics.To demonstrate the superiority of our proposed DK-CLIP, we compare it with the state-of-the-art methods on three in-the-wild datasets. Since different methods employ distinct scales of ViT (B/16 and B/32) for feature extraction, we conduct experiments on both model scales.The average performances of 5-fold cross-validation results on DFEW are presented in Tab. 2. As can be seen, our proposed DK-CLIP achieves superior performance in both UAR and WAR. Specifically, in comparison to the previously best-performing MAE-DFER [34] , DK-CLIP achieves an improvement of 1.54% and 0.98% in terms of UAR and WAR, respectively. Additionally, it is noteworthy that DK-CLIP exhibits significant improvement across most expressions. The performance of DK-CLIP surpasses that of CLIPER [19] and DFER-CLIP [51] , thus highlighting the effectiveness of our proposed method in adapting CLIP for DFER. In addition, DK-CLIP significantly outperforms M3DFEL [39] , which is trained from scratch, by 8.85%/6.16% of UAR/WAR, demonstrating the effectiveness of leveraging pretrained knowledge for DFER. Tab. 3 presents the experimental results obtained on FERV39K and MAFW. Our proposed DK-CLIP achieves the best performance on both datasets. Specifically, DK-CLIP surpasses MAE-DFER [34] by margins of 1.39% UAR and 2.25% WAR on MAFW. Meanwhile, it outperforms DFER-CLIP with advantages of 1.28% UAR and 2.38% WAR. Owing to the diverse scenarios present in FERV39K compared to DFEW and MAFW, the dataset poses a greater challenge, leading to lower recognition accuracy. Concurrently, the improvement in performance resulting from the increased ViT scale is relatively minor compared to other datasets. In summary, the promising results on three in-the-wild datasets demonstrate the strong generalization ability of DK-CLIP in practical scenarios.In this section, we conduct ablation studies to investigate the impact of key factors in DK-CLIP. For simplicity, we only report the results of the ViT-B/16 based model on DFEW fold 3.Influence of Key Components. To evaluate the effectiveness of key components in DK-CLIP, we perform experiments with varying model architectures. The baseline model employs average pooling for temporal modeling and incorporates class names as textual prompts. As shown in Tab. 4, the results demonstrate the effectiveness of modeling temporal correlations via the hierarchical video encoder, along with the enhancement of textual prompts through including AUs and visual contents. Our proposed DK-CLIP outperforms the baseline model by 4.09%/3.17% of UAR/WAR. Furthermore, the integration of class-aware consistency regularization enhances performance by 0.74%/0.77% of UAR/WAR, highlighting its effectiveness in reducing the influence of noisy frames.Influence of Different Temporal Modeling. To evaluate the effectiveness of our proposed hierarchical video encoder, we conduct experiments utilizing diverse temporal modeling approaches. As shown in Tab. 5, our proposed hierarchical video encoder consistently outperforms other approaches, showcasing its prowess in modeling intricate temporal correlations within facial expression sequences.Influence of Different Textual Prompts. We conduct experiments to analyze the influence of different textual prompts on the performance of DK-CLIP. As shown in Tab. 6, we observe that the utilization of learnable prompts results in superior performance due to their capacity to learn task-relevant information throughout the [51] , our proposed learnable prompts enhanced by visual content and activated AUs exhibits an improvement of 1.54% and 0.52% in UAR and WAR, respectively. This improvement stems from the fact that the generated textual prompts exhibit significant semantic similarity, whereas our textual prompts, incorporating class names and AUs, exhibit greater discernibility. Furthermore, compared to using "and" in textual prompts to combine AUs that are not always activated simultaneously (Tab. 6 (d)), employing "or" to distinguish their semantics (Tab. 6 (e)) shows better performance.Influence of Hyper-parameters. We investigate the influence of the threshold 𝜃 and the number of local snippets 𝑁 . As shown in Figure 4 : High-level visual feature visualization on DFEW fold 3 using t-SNE. Fig. 3 , improper setting of noise frame threshold 𝜃 can significantly impact model performance. This is due to the fact that a threshold 𝜃 that is too low fails to effectively erase noise, whereas an excessively high threshold may lead to the erroneous removal of keyframes. Meanwhile, it is evident that setting the number of local snippets to 4 represents an optimal selection for a sampling rate of 16.• • * • • * • • * • • • • * • • * • • * • • * • • * • • * • • * • • * • • * • • *Feature Visualization. To demonstrate the superiority of DK-CLIP compared to state-of-the-art methods, we visualize the learned visual embedding using t-SNE [37] on DFEW fold 3. As can be seen in Fig. 4 , the embeddings learned by our method exhibit greater compactness and separability compared to those of other methods, which demonstrates the ability of DK-CLIP to learn highly discriminative visual representations for diverse dynamic facial expressions.Recognition Results Visualization. To further evaluate the effectiveness of the proposed method, we provide detailed local and global prediction results for four samples from DFEW fold 3, as shown in Fig. 5 . As we can see, our proposed method accurately identifies noisy frames and produces reliable predictions. Meanwhile, we observe that the similarity scores also serve as indicators for estimating the expression intensity, as exemplified in the third row of Fig. 5 . Furthermore, our proposed method demonstrates robustness against interference such as occlusion and low light conditions. 
conclusion: 
 In this paper, we propose a novel method to promote the development of DFER through effectively adapting CLIP to DFER. We present a hierarchical video encoder designed to jointly model shortand long-term temporal correlations in DFER. Meanwhile, we align facial expressions and AUs using prior relationships to construct semantically rich textual prompts, which are further enhanced with visual contents. Furthermore, we introduce a class-aware consistency regularization mechanism that comprises a noise reducing module and a consistency regularization loss, aiming to reduce the impact of noisy frames and enhance the robustness of the model to interference. The extensive experimental results demonstrate the superiority of our proposed method. 
 
 Appendix:  


Decision: Reviewer Ratings - 2xBR and 2xBA

The paper proposes a CLIP driven method for facial expression recognition. The results are convincing and a detailed analysis has been performed. The rebuttal answers most of the queries.

Reviews: review: paper_summary: paper_summary: This paper proposes a dynamic facial expression recognition framework that leverages DFER-specific domain knowledge, including temporal correlation characteristics and the relationships between different levels of facial behavior descriptions, to guide the adaptation of CLIP, which is pre-trained on general images, to DFER.
strength_weakness: strength_weakness: 1. The proposed hierarchical video encoder effectively captures both short- and long-term temporal correlations in videos. 
2. The introduced prior knowledge of the relationships between expressions and AUs enriches the textual prompts.
3. The proposed noise reducing module adaptively removes non-target noise frames, as visualized convincingly in Figure 5.
4. Extensive experimental comparisons and ablation analyses demonstrate the excellent performance of the proposed DK-CLIP on the DFER task.
limitations: limitations: 1. When enriching the textual prompts with AUs, the proposed method takes a relatively straightforward approach, such as "Happiness with cheek raiser, lip corner puller, or lip part." Despite using "or," a concern is that this sentence still includes textual descriptions of all related AUs. In reality, these AUs do not always activate simultaneously, leading to inaccuracies. An effective way to validate this is to replace "or" with "and" and check for changes in the model's performance.
2. The meanings of some elements in Figure 2 are unclear, such as the set of tokens after patch embedding and the corresponding indices. It is recommended to refine these elements to enhance readability.
3. The cross-frame attention module adopts the structure of the ViT from CLIP. Are the parameters of each cross-frame attention module shared? If not, wouldn't this result in an excessive number of parameters?
4. The paper mentions that "All parameters are adjusted during training." Are the parameters of the patch embedding and text encoder also trainable? This seems counterintuitive.
5. The paper mentions that 𝜆 is a scalar hyperparameter that controls the weight of the consistency regularization, but it also seems to control L_𝑙. Could this be a typo?
1. The conclusive analysis in lines 748 to 751 should be in the "Influence of Different Textual Prompts" section.
paper_summary: paper_summary: This research introduces a method to deal with data scarcity and noise issues in the DFER field by adapting CLIP to DFER. Including proposing a hierarchical video encoder to jointly model short-and long-term temporal correlations in DFER in the visual part, and aligning facial expressions and AUs using prior relationships to construct semantically rich textual prompts in the textual part. As well as proposing a class-aware consistency regularization mechanism to reduce the impact of noisy frames.
strength_weakness: strength_weakness: The paper is easy to understand and follow. The authors use clear language and provide sufficient background information for readers who may not be familiar with DFER, Vision-Language Pretraining or AUs. The experimental comparison is relatively sufficient.
limitations: limitations: 1. The article regarding the input of CMAM contains contradictory statements in several sections; please check for clarity or errors. Line 328 indicates that CMAM accepts global features as visual input, which seems inconsistent with the descriptions in Figure 2 and line 505.
2. The text overly relies on verbal descriptions, and it is suggested to include more formulas to aid reader comprehension. For example, the descriptions from line 396 to 404 could be entirely replaced with formulas, as well as further descriptions of MSIM in Equation 3 and CMAM in Equation 4.
3. The author has not made the models and algorithms discussed in this paper open source. It is hoped that the author will contribute their work to the community.
paper_summary: paper_summary: This paper utilizes domain-specific knowledge from DFER, including temporal features and the relationship between facial behavioral descriptions at different levels, to guide CLIP in adapting to DFER. This resolves the complex temporal modeling issue caused by noisy frames and addresses the limitation of training data.
strength_weakness: strength_weakness: The paper introduces the DK-CLIP method based on the CLIP model, addressing two limitations encountered by DFER. By capturing key facial region features and integrating textual cues, it achieves emotion recognition and classification tasks. The paper demonstrates clear logic, detailed content, complete structure, enriched experimental validation. It significantly enhances the accuracy of emotion recognition.
limitations: limitations: 1.If the number of non-target frames significantly exceeds the number of target frames in a video sequence, what strategies can be employed?
2.The visual cases chosen in the method all feature frontal or slightly off-center views of the face. If more than half of the face is occluded, it's uncertain whether the recognition accuracy of these video sequences will decrease. However, there is a lack of analysis on failure cases in such scenarios.
paper_summary: paper_summary: The paper proposes to utilize domain knowledge to optimize the application of the CLIP model by the proposed DK-CLIP method. DK-CLIP addresses two limitations in DFER: lack of temporal modeling ability and lack of optimizing the task-related textual prompts. Specifically, the authors design a hierarchical video encoder to capture the short- and long-term relationship, introduce the prior knowledge of facial action units and categorical emotion class into the feature generation of text, and devise a class-aware consistency regularization mechanism to aptly filter out noisy frames. Extensive experiments on datasets show superior performance over existing methods.
strength_weakness: strength_weakness: The writing is clear and easy to read.
limitations: limitations: The topic of upgrading existing CLIP-based methods for DFER is hot and crucial but the innovation has only some incremental.

1. In the introduction section, no paragraph is written to clarify that how the proposed work or previous CLIP-based works jointly alleviate the issues of limited training data and noisy frames.
2. Line 117: Does the sentence "restricts the availability of training data for DFER models" means the issue of a limited number of training samples? If it is, why are the following sentences  (L121~L124) mentioned and just only mention the methods for tackling noisy frames?
3. Line238: Why do the self-supervised pertained models lack the ability to handle complete temporal dependencies in DFER? There is no mention in related work 2.1. In addition, Transformer-based methods for DFER can also be divided into supervised and unsupervised techniques. So it is unsuitable to write "In addition to supervised learning methods, large-scale self-supervised pre-trained models..." in the first sentence of the second paragraph of related work 2.1.
4. Line 240: Line 240: "to enhance the pre-trained CLIP model" -> enable CLIP the ability of handling complex temporal dependencies? 
5. Line 241: It mentions "both issues simultaneously" but only the issue of lacking the ability to handle complex temporal dependencies in DFER involved.