FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-17 09:44:34

Abstract: Dynamic Facial Expression Recognition (DFER) is crucial for understanding human behavior. However, current methods exhibit limited performance mainly due to the scarcity of high-quality data, the insufficient utilization of facial dynamics, and the ambiguity of expression semantics, etc. To this end, we propose a novel framework, named Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs (FineCLIPER), incorporating the following novel designs:  1) To better distinguish between similar facial expressions, we extend the class labels to textual descriptions from both positive and negative aspects, and obtain supervision by calculating the cross-modal similarity based on the CLIP model; 2) Our FineCLIPER adopts a hierarchical manner to effectively mine useful cues from DFE videos. Specifically, besides directly embedding video frames as input (low semantic level), we propose to extract the face segmentation masks and landmarks based on each frame (middle semantic level) and utilize the Multi-modal Large Language Model (MLLM) to further generate detailed descriptions of facial changes across frames with designed prompts (high semantic level). Additionally, we also adopt Parameter-Efficient Fine-Tuning (PEFT) to enable efficient adaptation of large pre-trained models (i.e., CLIP) for this task. Our FineCLIPER achieves SOTA performance on the DFEW, FERV39k, and MAFW datasets in both supervised and zero-shot settings with few tunable parameters. Analysis and ablation studies further validate its effectiveness.

Main paper: 
 introduction: 
 Facial expressions are important signals to convey human emotions, thus accurately recognizing them has significant meaning for various tasks, including interpersonal communication, humancomputer interaction (HCI) [16, 22, 36] , mental health diagnosing [2, 21, 47] , driving safety monitoring [56, 62, 63] , etc. Traditional Facial Expression Recognition (FER) resorts to static images. However, since dynamic emotional changes could not be well-represented within a single image, research attention has been shifted to Dynamic Facial Expression Recognition (DFER), which distinguishes the temporally displayed facial expressions in videos.The study of DFER algorithms starts from highly-controlled environments [3, 37, 42] , where faces are frontal and non-blurry [30, 59] as shown in the upper right of Fig. 1 . However, such an ideal assumption makes the obtained models vulnerable to real-world situations. Therefore, researchers have turned to more open scenes and constructed several in-the-wild DFER datasets, e.g., DFEW [18] , FERV39k [60] , and MAFW [33] , to facilitate the development of corresponding methods [27, 28, 35, 39, 57, 68] . While category labels are treated without semantic meanings (e.g., Happiness may only be represented by a class id "0"), recent research on DFER [11, 26, 52, 70] has further delved into the exploration of vision-language multimodal learning beyond the traditional classification paradigm based on CNNs [10, 20, 24, 50] , RNNs [1, 61] , and transformers [25, 69] .Although huge efforts have been spent, the performance of DFER methods still suffers from noisy frames, small inter-class differences, and ambiguity between expressions, making it inappropriate to adopt video/action recognition techniques directly. Specifically, to distinctly improve the performance of DEFR algorithms, we have to face the following unique and tough challenges: 1) The ambiguity of semantic labels for dynamic facial expressions, and 2) The subtle and nuanced movements of local face parts (i.e., skeletons, muscles, etc.). The first challenge originates in the difficulty of accurate human labeling and complex expression ways adopted by different persons, while the latter demands additional focus on fine-grained details that happen in specific regions within a human face.To tackle the above challenges, we propose a novel framework called FineCLIPER, short for Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs. Specifically, we utilize the Contrastive Language-Image Pretraining (CLIP) model [46] , which is particularly suitable for providing a crossmodal latent space. To avoid the huge cost of fine-tuning such large pre-trained models, FineCLIPER adopts the Parameter-Efficient Fine-Tuning (PEFT) strategy by adding several adaption modules with small parameters for tuning (as shown in Fig. 2 ), achieving high efficiency while preserving the remarkable performance.Specifically, our FineCLIPER has the following characteristics that distinguish it from previous works:Firstly, by adopting the vision-text learning paradigm, we transform the ground truth label to form the textual supervision (e.g., "A person with an expression of {Label}"). But one noteworthy innovation is that we meanwhile generate and use the negative counterparts (e.g., "A person with an expression of No {Label}"). Such label augmentation via PN (Positive-Negative) descriptors is inspired by the negative prompting strategy [7, 43] , and found to be useful here for differentiating between ambiguous categories. A notable progress is observed in the "Disgust" category of the DFEW dataset, while most baselines [4, 25, 29, 34, 57] suffer from a nearby 0% accuracy, our FineCLIPER significantly promotes the performance by more than 25%, as shown in Tab. 2.Furthermore, we adopt a semantically hierarchical strategy to comprehensively mine useful information from the input video data. Specifically, features from directly embedding video frames stand at a relatively low semantic level. For middle semantic level, we utilize a well-trained face analysis model (i.e. FaceXFormer [44] ) to extract the face segmentation masks and landmarks from each frame. Intuitively, the former offers prior about face structures while the latter provides specific pivots for model attention. Additionally, we try to obtain descriptions at a high semantic level for describing dynamic facial changes across frames. This is realized by leveraging a well-trained MLLM, Video-LLaVA [31] to act as a facial expression analyst following given template-based prompts, and the generated descriptions will be carefully refined. All the above features at various semantic levels will be integrated to obtain the final representation of a given video.To summarize, our contributions are as follows:• We introduce FineCLIPER, a novel multi-modal framework that enhances Dynamic Facial Expression Recognition (DFER) through extensively mining useful information at different semantic levels from the video data, and all the obtained features (i.e., features embedded from visual frames, face segmentation, face landmarks, and the extra fine-grained descriptions obtained via MLLM) are integrated finally to serve as a more comprehensive overall representation; • To address the ambiguity between categories, we propose a label augmentation strategy, not only transforming the class label to textual supervision but also using a combination of both positive and negative descriptors; • Extensive experiments conducted on DFER datasets, i.e., DFEW, FERV39k, and MAFW, show that our FineCLIPER framework achieves new state-of-the-art performance on both supervised and zero-shot settings with only a small number of tunable parameters. Comprehensive ablations and analyses further validate the effectiveness of FineCLIPER. 
related work: 
 Dynamic Facial Expression Recognition. In early DFER research, the focus was on developing diverse local descriptors on lab-controlled datasets [3, 37, 38] . Then the rise of deep learning and accessible in-the-wild DFER datasets [18, 33, 60] leads to new trends towards DFER research. The first trend [10, 20, 24] involves the direct use of 3D CNNs [12, 54, 55] to extract joint spatiotemporal features from raw videos. The second trend [9, 19, 50, 61] combines 2D CNNs [5, 48] with RNNs [5, 13] for feature extraction and sequence modeling. The third emerging trend integrates transformer [8] , as demonstrated in works like Former-DFER [69] , STT [40] , and IAL [25] . These methods combine convolutional and attention-based approaches to enhance the understanding of visual data, especially in distinguishing samples based on varying visual dynamics. However, in prior efforts, the semantic meaning of class labels is neglected, and insufficient attention has been paid to the subtle and nuanced movements of the human face. Therefore, based on the well-trained large cross-modal models (i.e. CLIP), we propose to extend the class label to textual supervision both positively and negatively. Moreover, to fully exploit the visual information within videos, we also design a hierarchical information mining strategy to generate representative video features, which is a weighted fusion of various features involving different semantic levels, including video frame feature, the middle-level facial semantics from segmentation maps and detected landmarks, we well as the high-level semantics encoded from fine-grained descriptions provided by MLLM. CLIP in Classification. Vision-Language Models (VLMs), e.g., CLIP [46] , have recently demonstrated superior performance across various tasks, including video understanding [6, 45, 58, 66] , 3D generation or editing [15, 17, 32] , and region profiling [53, 64] , etc. CLIP leverages a vast corpus of image-text pairs to ground its framework in contrastive learning, resulting in robust pre-trained image and text encoders that demonstrate remarkable feature extraction capabilities. Recent studies [26, 52, 70] have also applied CLIP to the DFER task. Among them, A 3 lign-DFER [52] introduces a comprehensive alignment paradigm for DFER through a complicated design. CLIPER [26] adopts a two-stage training paradigm instead of end-to-end training; however, it is limited in capturing temporal information. Furthermore, DFER-CLIP [70] incorporates a transformer-based module to better capture temporal information in videos, but it requires fully fine-tune the image encoder and the proposed temporal module during training, leading to inefficiency. However, while these works have explored the semantic information of labels compared to traditional DFER, they often overlook the interrelations among facial expressions and the individual differences among humans as they directly extend labels into relevant action descriptions (e.g., Happiness→smiling mouth, raised cheeks, wrinkled eyes, ... [70] ). This oversight can lead to further ambiguity. In light of this, we propose PN (Positive-Negative) descriptors, extending the ground truth labels from contrastive views to better distinguish between ambiguous categories. 
methodology: 
 In this section, we first briefly go through the overall pipeline and basic notations of the framework in Sec. 3 on how to augment the original class labels to obtain positivenegative textual supervision in Sec. 3.2, followed by details about our hierarchical information mining strategy to obtain multi-modal features in Sec. 3.3. The integration of diverse features is introduced in Sec. 3.4. The overall pipeline is illustrated in Fig. 2 .Formally, given a video clip 𝑉 , the task of DFER aims to recognize the facial expression label 𝐶𝑙𝑠. Using text templates as "A person with an expression of {Cls}", the class label could be further transformed into textual supervision, which could better utilize the semantic meaning of the category name. Let V represents a set of videos and C denotes collections of augmented textual descriptions of labels, our framework could produce the embedded representations for both a given video and its corresponding textual supervision, resulting in v i and c i . Note that in our cases, v i is an integration of features from different semantic levels, namely low-level (video frames), middle-level (face parsing and landmarks), and high-level semantics (fine-grained captions of facial action changes obtained using MLLM). The similarity between v i and c i is calculated as 𝑠𝑖𝑚 𝑖 . To employ the cross-entropy loss, we calculate the prediction probability over class 𝑐𝑙𝑠 𝑖 as:EQUATION)where 𝑁 is the number of total classes and 𝜏 represents the temperature parameter of CLIP.Although in-the-wild DFER usually comprises limited categories (e.g., 7 in DFEW [18] and FERV39k [60] , or 11 in MAFW [33] ), the recognition difficulty does not reduce due to the high inter-class ambiguity (as shown in Tab. 2). Therefore, as stated in Sec. 3.1, class labels are transformed into textual supervision for utilizing their semantic meanings. While existing CLIP-based DFER models [26, 52, 70] mostly focus on enriching the textual descriptions for ground truth labels from a positive view, in this work, we devise a different label augmentation strategy by extending the original class labels from both positive and negative perspectives. Specifically, the Positive-Negative (PN) descriptors are derived as follows: i.e., P(ositive): "A person with an expression of {Cls}.", and N(egative): "A person with an expression EQUATIONwhere E denotes the positional encoding. } for positive and negative textual supervision, respectively. Then the encoded positive and negative textual features could be obtained via:z ( 𝑗 ) 𝑇 𝑃 = E ( 𝑗 ) 𝑇 𝑃 (A ( 𝑗 ) 𝑇 𝑃 (z 𝑇 𝑃 ( 𝑗 -1) )), z ( 𝑗 ) 𝑇 𝑁 = E ( 𝑗 ) 𝑇 𝑁 (A ( 𝑗 ) 𝑇 𝑁 (z ( 𝑗 -1) 𝑇 𝑁)).(3) We adopt the basic Adapter structure proposed in [14] for all adapters in our FineCLIPER framework. The structure of the adapter is illustrated in the middle of Fig. 2 . Then the final positive and negative text representations can be obtained by:EQUATIONwhere z(𝐿 𝑇 )𝑇 ,𝑙 is the last token of z(𝐿 𝑇 ) 𝑇and h 𝑇 is a projection layer.Our FineCLIPER adopts a hierarchical manner to mine useful information from: 1) low semantic level, where video frames are directly embedded; 2) middle semantic level, where face segmentation and landmarks are exploited, and 3) high semantic level, where fine-grained descriptions are obtained via MLLM to depict facial dynamics across frames. Details can be found as follows:Video Frames Embedding could provide semantically low-level features since the model operates at pixel-level. To effectively explore the spatial-temporal visual information, we resort to the strong spatial modeling abilities displayed by CLIP and utilize a temporal-expanded version inspired by [65] . Formally, given a video clip 𝑉 ∈ R 𝑇 ×𝐻 ×𝑊 ×3 , where 𝐻 × 𝑊 is the spatial size and 𝑇 is the temporal length. For 𝑡-th frame, we spatially divide it into non-overlapping patches {P 𝑡,𝑖 } 𝑀 𝑖=1 ∈ R 𝑃 2 ×3 , where 𝑀 = 𝐻𝑊 /𝑃 2 . These patches are then projected into patch embeddings X 𝑣,𝑡 ∈ R 𝑀 ×𝑑 , where 𝑑 represents the embedding dimension. Therefore, the representation for the given video 𝑉 could be z ∈ R 𝑇 ×𝑀 ×𝑑 . After the temporal information undergoes processing by the temporal adapter, the spatially adapted feature can be derived through the following procedure:EQUATIONEQUATIONwhere z( 𝑗 )𝑇 𝑒𝑚𝑉 and z( 𝑗 )𝑆𝑝𝑎𝑉 denotes the temporally and spatially adapted features, respectively.As a result, the adapter, operating in parallel with the MLP layer, aims to collectively refine the representation of spatiotemporal information. The final feature, scaled by a factor 𝑠 (set to 0.5 in our framework), can be expressed as follows:z ( 𝑗 ) 𝑉 = z ( 𝑗 ) 𝑆𝑝𝑎𝑉 + 𝑀𝐿𝑃 (𝐿𝑁 (z ( 𝑗 ) 𝑆𝑝𝑎𝑉 )) + 𝑠 • A ( 𝑗 ) 𝑉 (𝐿𝑁 (z ( 𝑗 ) 𝑆𝑝𝑎𝑉 )). (7)Thus, the ultimate video representation at a low semantic level is derived as v = h 𝑉 (z (𝐿 𝑉 ) 𝑉). Face Parsing and Landmarks Detection. Based on a given frame, we could further mine middle-level semantic information from it. In our task, as the main part of a frame is mostly human faces, we choose to utilize a powerful facial analysis model, FaceXFormer [44] , to obtain generalized and robust face representations. Specifically, we extract the facial segmentation map and perform landmark detection. Intuitively, the former implies the semantically grouped facial regions, while the latter could provide accurate locations indicating different face parts (e.g., eyes, nose, etc.) Specifically, given a specific video clip 𝑉 , the extracted parsing results and landmark maps are represented as 𝑃 and 𝐿, respectively. Following patch embedding, both 𝑃 and 𝐿 are fed into the corresponding segmentation encoder E 𝑃 and landmark encoder E 𝐿 , similar to the operation done for the frame data. The encoders E 𝑃 and E 𝐿 share weights to collaboratively capture middle-level face semantics. Finally, the parsing and landmark representations can be obtained as p = h 𝑃 (z(𝐿 𝑃 ) 𝑃 ) and l = h 𝐿 (z (𝐿 𝐿 ) 𝐿), and h 𝑃 and h 𝐿 are projection layers for 𝑃 and 𝐿, respectively. Additional Fine-grained Descriptions. In this part, we try to achieve fine-grained details describing the facial dynamics across video frames to serve as high-level semantics. Specifically, for each video clip 𝑉 , we adopt Video-LLaVA [31] , a MLLM, to generate detailed descriptions under the guidance of an elaborately designed prompt, where the model is asked to play a role as a facial expression analyst to provide details of facial changes, as illustrated in Fig. 3 . To elaborate, the provided text prompt raises requirements for the granularity of the descriptions, explicitly specifying movements involving various local facial regions. However, the generated description may include emotion-related words associated with the label or contain some redundant information. Hence, we thoroughly refined all generated descriptions to achieve a concise and highquality summary. The refinement works as follows. Initially, we employed a rule-based approach, utilizing pre-configured regular filters to eliminate redundant and irrelevant textual information. Popular text processing tools from the NLTK package were then utilized to remove noise. Subsequently, each data entry will go through manual inspection to filter out abnormal descriptions.The average number of tokens in our refined descriptions is approximately 35 tokens. However, research [67] demonstrates the actual effective length of CLIP's text encoder is even less than 20 tokens. Hence, to better explore the fine-grained description of facial changes, we adopt the text encoder of Long-CLIP [67] as our fine-grained text encoder E 𝐹 , which can support text inputs of up to 248 tokens. The refined fine-grained description, denoted as 𝐹 , is further tokenized and projected into embeddings X 𝐹 . Following a procedure similar to the text encoder described in Sec. 3.2, the input is further constructed as z (0) 𝐹 = X 𝐹 + E 𝐹 , where E 𝐹 is the positional encoding of 𝐹 . Subsequently, by feeding it into the projector h 𝐹 , we could contain the final feature vector of 𝐹 as: f = h 𝐹 (z (𝐿 𝐹 ) 𝐹,𝑙 ).Through the aforementioned semantically hierarchical information mining process, we obtain: 1) low-level video frame feature v, 2) middle-level face parsing features p and face landmark features l, and 3) high-level fine-grained description features f. The integration of these features is done using an adaptive fusion strategy.Specifically, given a specific video 𝑉 , the supervision for the 𝑖 𝑡ℎ class is represented by both the positive c 𝑖 𝑃 and negative c 𝑖 𝑁 . Suppose any representation m ∈ {v, p, l, f }, the similarity between m and c 𝑃 , as well as m and c 𝑁 is defined by calculating the cosine similarity:EQUATIONand the final similarity is obtained by: 𝑠𝑖𝑚 𝑖,m = 𝑠𝑖𝑚 𝑝𝑜𝑠 𝑖,m -𝑠𝑖𝑚 𝑛𝑒𝑔 𝑖,m , which further distinguishes similarity among similar categories Then, by finding the max similarity across all the categories, we obtain 𝑠𝑖𝑚 v = 𝑚𝑎𝑥 𝑁 𝑖=0 (𝑠𝑖𝑚 𝑖,v ). Similarly, we could get 𝑠𝑖𝑚 f , 𝑠𝑖𝑚 p , and 𝑠𝑖𝑚 l following corresponding max-similarity category. Normalizing these similarities, we obtain the weights corresponding to that representation as:EQUATION)Such weights could be calculated for p, l, f similarly, resulting in the corresponding weights 𝑤 v , 𝑤 f , 𝑤 p , and 𝑤 l . Then the overall multi-modal representation v 𝑚𝑚 of Multi-Modal Encoders can be obtained as follows:EQUATION)where the weights also correspond to the weights of the crossentropy loss for each modality. Then the overall loss function can ...The woman's eyes widen, her eyebrows raise, and her mouth opens. Her gaze is directed upward, and her breathing appears to be shallow and quick.The woman sits with her eyes slowly widening, then suddenly shivers back and asks in a confused voice. The wide eyes.... thus be expressed as:EQUATION 
experiment 4.1 setup: 
 Datasets and Evaluation. Following previous works, we adopt both supervised and zero-shot learning paradigms, evaluating our proposed FineCLIPER together with the baselines on the various in-the-wild DFER datasets, including DFEW [18] , FERV39k [60] , and MAFW [33] . We utilize UAR (Unweighted Average Recall) and WAR (Weighted Average Recall) as evaluation metrics for our assessments. Both DFEW and FERV39k have 7 dynamic facial expression categories to recognize, while MAFW has 11 categories. It is noteworthy that MAFW dataset comes with video captions for each video, making it a choice for pretraining in zero-shot setting. Implementation Details. All the experiments of our FineCLIPER are built on a CLIP model with the backbone of ViT-B/16 using a single NVIDIA RTX 4090 GPU for fairness and consistency. We process the input by resizing and cropping 16 video frames to a uniform size of 224×224 pixels. The SGD optimizer is employed with an initial learning rate of 3 × 10 -4 . FineCLIPER is trained in an end-to-end manner over 30 epochs with the temperature hyper-parameter 𝜏 = 0.01.Supervised Setting. The quantitative results in the supervised setting on three standard DFER datasets are depicted in Tab. 1. It can be observed that our proposed FineCLIPER achieves state-of-the-art performance compared with other DFER approaches. In addition, our method outperforms all CLIP-based DFER methods with the most lightweight architecture and also the least tunable parameters. Furthermore, we investigate three variants of our FineCLIPER, incorporating face parsing and landmark modalities, along with fine-grained text descriptions of facial changes, which justify the combination of these strategies. The superiority of our FineCLIPER is also supported by the substantial improvement in the most challenging category for previous methods, i.e., "Disgust (Dis.)", as Zero-shot Setting. To assess the generalization ability of FineCLIPER, we perform zero-shot DFER using captions extracted directly from each video. Our main baseline is EmoCLIP [11] , which is the first CLIP-based zero-shot DFER model, utilizes the MAFW [33] dataset for pertaining. The comparison between captions in MAFW and our generated fine-grained descriptions is shown in Fig. 4 . Tab. 3 reports the recognition performance of our FineCLIPER compared with other approaches in the zero-shot DFER setting. Not only did we surpass the previous methods when the pretraining data was consistent, but employing our generated fine-grained captions also led to a significant performance improvement. This further demonstrates the effectiveness of the fine-grained description obtained and used by our FineCLIPER, which focuses more on facial changes instead of video scenes (as in MAFW). In other words, fine-grained descriptions play a pivotal role in guiding the model's attention toward detailed aspects of specific facial regions in the zero-shot setting.Performance w.r.t. middle-level facial features. We investigate the effectiveness of using the middle-level face semantics obtained by face parsing and landmark detection, and the results are shown in Tab. 4. We have the following observations: 1) By comparing When we control other conditions, using our Pos-Neg augmentation achieves the best results across all metrics. Next, to further understand why the Pos-Neg descriptors perform well, we visualize class-wise cosine similarity between video representation and the positive text supervision (colored in green) as well as the negative supervision(colored in red), as shown in Fig. 5 . It can be observed that since the positive supervision could sometimes fail to work (as we can see that for some categories the pos-similarity is really low), the existence of its negative counterpart could address such a problem to a certain extent. Performance w.r.t. the usage of trainable adapters. We adopted several lightweight trainable adapters in our FineCLIPER to efficiently adapt the ability of large pre-trained models. The corresponding ablation studies are demonstrated in Tab. 5. We can see that given the same supervision settings (e.g. pose+neg for FineCLIPER), adding small adaptive modules could effectively boost the performance with only limited trainable parameters (e.g. 20M for all adapters in FineCLIPER).Effect of each components. To validate the effectiveness of each component module in our FineCLIPER, we visualize the attention map of the last transformer block, as shown in Fig. 6 . Specifically, we sequentially add components from top to bottom, including adding the Adapterers, using the parsing results and landmarks of faces, as well as using the high-level semantics from the finegrained descriptions generated by MLLM. We can see that the model's attention is shrinking to more crucial and concentrated face parts w.r.t. to certain categories. For example, it focuses on the mouth, eyes, and eyebrows when identifying Happiness, which aligns well with expression recognition using human vision. Such visualization results provide a vivid interpretation to explain the superior recognition performance of FineCLIPER. 
conclusion: 
 Dynamic Facial Expression Recognition (DFER) is vital for understanding human behavior. However, current methods face challenges due to noisy data, neglect of facial dynamics, and confusing categories. To this end, We propose FineCLIPER, a novel framework with two key innovations: 1) augmenting class labels with textual PN (Positive-Negative) descriptors to differentiate semantic ambiguity based on the CLIP model's cross-modal latent space; 2) employing a hierarchical information mining strategy to mine cues from DFE videos at different semantic levels: low (video frame embedding), middle (face segmentation masks and landmarks), and high (MLLM for detailed descriptions). Additionally, we use Parameter-Efficient Fine-Tuning (PEFT) to adapt all the pre-trained models efficiently. FineCLIPER achieves SOTA performance on various datasets with minimal tunable parameters. Detailed ablations and analysis further verify the effectiveness of each design.  
 
 Appendix:  


Decision: This paper proposes multi-modal fine-grained CLIP with adapters for facial expression recognition. 

Most reviewers tend to accept this paper. The authors should polish this paper in the final version by considering the comments of Reviewer VmG3. It is suggested to make the paper clearer including discussing the motivation why the designed method can address the stated issues.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs, i.e., FineCLIPER. FineCLIPER adopts a hierarchical manner to enhance visual features, and utilizes the Multi-modal Large Language Model (MLLM) to generate the detailed descriptions of facial changes across frames. Adapters are employed to fine-tune the Encoders. Experiments are conducted on DFEW, FERV39K and MAFEW datasets, showing state-of-the-art performance.
strength_weakness: strength_weakness: This paper employs many popular and novel frameworks and models to complete Dynamic Facial Expression Recognition task, such as Video-LLaVa, FaceXFormer, and Long-CLIP. This work achieves state-of-the-art performance and greatly upgrade the recognizing performance at the category of disgust. This paper is easy to follow. The experiments are enough to validate the efficacy.
limitations: limitations: Though the work keeps up with times, many limitations hinders the understanding of this paper.

1. In introduction section, the authors mentioned current Dynamic Facial Expression Recognition faces two challenges: 1) the ambiguity of semantic labels and 2) subtle movements of local face parts. However, The reason why the multi-modal framework can and how they do to alleviate the mentioned two challenges are not introduced. Sure the detailed operations are given in the following paragraphs, but the overall principle should be given before L113. Lacking the analysis of why the proposed methods can alleviate the corresponding issue makes this work like a simple compound of existing methods. I strongly recommend re-write your introduction section to introduce and issues and clarify the contribution.
2. For contribution 2): employing a hierarchical information mining strategy to mine cues from DFE videos at different semantic levels. How the proposed method solve one certain issue should be clarified, otherwise, this framework is considered as trivial and incremental design.
3. For contribution 3): Parameter-Efficient Fine-Tuning has widely used in existing works. It seems a basic trick for fine-tuning large model.
4. The paper has some confused symbols. L283: Does the meaning of i in \textbf{v_i} is the same as class i? Please clarify it. If it is, please remove the bold style for "i". L 288: What is the formula to calculate the similarity between the video feature and textual supervision? L551 formula(11): Missing the meaning of B, H in formula (11).
paper_summary: paper_summary: This paper proposed a multimodal fine-grained clip for dynamic facial expression recognition.
strength_weakness: strength_weakness: 1. The experiments show great performance.
2. The methodology is reasonable.
limitations: limitations: Instead of just using ViT-B/16 as the backbone, have you considered scaling up the model size?
paper_summary: paper_summary: The paper proposes multi-modal fine-grained CLIP with adapters for dynamic facial expression recognition. 

The novelty of the proposal includes extending positive and negative descriptions and introducing high-semantic level descriptions. 

The concerns mainly lie in the assumption of negative descriptors.
strength_weakness: strength_weakness: The proposal is novel. It extends the class label with negative descriptions. It utilizes the MLLM to generate high semantic-level information about facial changes.

The paper conducts extensive experiments. It shows advantages and includes ablation studies in three DFER datasets.

The writing and technical details are clear. The figures and expressions are in good quality. The paper poses the code for reproduction.
limitations: limitations: ### 1. The main concern is that the negative descriptors (e.g., a person with an expression of no {label}.) are too absolute. 

The facial expression class label is usually ambiguous, e.g., the neutral facial expression might contain fear emotion. Therefore, even if the apparent expression is neutral, we can't say there is no fear. Could you explain it?

### 2. There are some confusing or lack-verification expressions in the paper.
- #L10 ''scarcity of high-quality data". Why do the authors mention it?
- Figure 1. Why do the authors list sample numbers of SFER datasets in DFER?
- #L415 What is the meaning of "P" and "z"?
- #L419 "spatially adapted feature". What is it?
- Figure 2 and #L451. I can't see E_P and E_L share weights from the figure.
- #L469 "rule-based approach refinement". Can you give more details on how to reproduce it?
- #L697 "Zero-Shot DFER methods". I think it is a cross-domain setting since you use the DFER dataset.
- Figure 5. This picture is confusing. For the negative value, is lower better? In the 4th sample, I am confused about Happiness and Fear.
-  Table 5. What do you mean by an "X" of video?

### 3. Typos.
- #L505 "categories" needs a period.
- #L853 "supervision(colored" needs a blank.
paper_summary: paper_summary: In this work, it proposes a new framework named Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs (FineCLIPER), which combines the following novel designs:1)To better distinguish similar facial expressions, we extend the class labels from positive and negative aspects to textual descriptions and compute cross-modal similarity based on the CLIP model to obtain supervision; 2) FineCLIPER employs a hierarchical approach to efficiently mine useful cues from DFE videos. In addition, the paper uses Parameter Efficient Fine-Tuning (PEFT) to efficiently adapt to all pre-trained models, achieving high efficiency while maintaining excellent performance.
strength_weakness: strength_weakness: The motivation of integrating low level, middle level and high level sematics for expression recognitoin, is clear and the idea is interesting. The results also justify the effectiveness of adding sematics of different levels.
limitations: limitations: About clarity:
(1)	In line 381 of the article, each transformer layer is represented by the superscript i, but below it is represented by j. Is there any difference between the before and after?
(2)	The article mentions that FineCLIPER achieves high efficiency while maintaining excellent performance. However, the article uses multiple models, are they really achieving high efficiency? Can this be illustrated by metrics or graphs?
(3)	Why is a scaling factor s added to Eq. 7 and how is s=0.5 determined, please give appropriate experiments or references.

About experiments:
(1)	The EmoCLIP method in Table 1 is marked as published in FG, but the reference is shown as being from ArXiv.
(2)	What is the performance of the models in Table 4 without the use of the low semantic level, and with the use of the medium or high semantic level alone, respectively?
(3)	In Fig. 5, the video representations of some expressions do not obtain the highest similarity with positive texts (e.g. happiness and fear), please explain why. And elaborate on how the introduction of negative text supervision solves this problem.