Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting

Authors: yuanyuan  liu
yuxuan  huang
shuyang  liu
yibing  zhan
zijing  chen
zhe  chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:56:07

Abstract: In Video-based Facial Expression Recognition (V-FER), models are typically trained on closed-set datasets with a fixed number of known classes. However, these models struggle with unknown classes common in real-world scenarios. In this paper, we introduce a challenging Open-set Video-based Facial Expression Recognition (OV-FER) task, aiming to identify both known and new, unseen facial expressions. While existing approaches use large-scale vision-language models like CLIP to identify unseen classes, we argue that these methods may not adequately capture the subtle human expressions needed for OV-FER. To address this limitation, we propose a novel Human Expression-Sensitive Prompting (HESP) mechanism to significantly enhance CLIP's ability to model video-based facial expression details effectively. Our proposed HESP comprises three components: 1) a textual prompting module with learnable prompts to enhance CLIP's textual representation of both known and unknown emotions, 2) a visual prompting module that encodes temporal emotional information from video frames using expression-sensitive attention, equipping CLIP with a new visual modeling ability to extract emotion-rich information, and 3) an open-set multi-task learning scheme that promotes interaction between the textual and visual modules, improving the understanding of novel human emotions in video sequences. Extensive experiments conducted on four OV-FER task settings demonstrate that HESP can significantly boost CLIP's performance (a relative improvement of 17.93% on AUROC and 106.18% on OSCR) and outperform other state-of-the-art open-set video understanding methods by a large margin. Code is available at https://github.com/cosinehuang/HESP.

Main paper: 
 introduction: 
 Video-based Facial Expression Recognition (V-FER) targets the recognition of predetermined emotion categories from video data. Despite its advancements in enhancing human-centered visual comprehension, this method usually focuses on a limited set of predefined categories. This restriction does not capture the full complexity and diversity of human expressions, limiting its ability to recognize a broader range of emotional states. To address this limitation, we introduce the Open-set Video-based Facial Expression Recognition (OV-FER) task, which seeks to identify known and discover new human emotions from videos. Similar to other openset paradigms [4, 7, 20, 27, 28] , the OV-FER requires the ability to recognize and classify previously unseen or 'unknown' expressions. Rooted in human emotion understanding, an effective OV-FER model is supposed to provide a robust understanding of various facial expression patterns in diverse and unpredictable environments, thus benefiting diversified applications like smart healthcare [5] , human-computer interaction [6] , and driver assistance system [35] .Recently, several studies [21, 30, 34] have proposed leveraging large-scale pre-trained models like CLIP [29] to facilitate open-set recognition tasks. With CLIP's remarkable generalization capabilities, it becomes feasible to effectively identify new categories from open-set data. However, when applying CLIP to video analysis, a ...... small intra-video emotion changes learnable frozenKnown (Disgust)......Figure 1 : The motivation and intuitive results of our HESP for OV-FER. Affected by challenges in OV-FER, current methods struggle to capture effective expressive information due to subtle inter-video and intra emotion changes and differences.To address this limitation, HESP consists of a novel learnable textual prompting module, visual prompting module, and an open-set multi-task learning scheme, aiming to augment CLIP to obtain more expression-sensitive representations for both known and unknown emotions in OV-FER.critical issue arises: CLIP was originally designed for image-text associations, and its support for video analysis, particularly in the context of human emotion videos, is not addressed appropriately in the vanilla model. To tackle this, a few studies [37, 39] attempt to capture spatial-temporal dynamic information in videos to extend the capability of CLIP on open-set video understanding tasks. For instance, AutoLabel [39] introduces an additional self-attention temporal pooling for CLIP to aggregate frame-level features as video-level features. Open-VCLIP [37] proposes a temporal attention view of each self-attention layer in CLIP to facilitate the aggregation of global temporal information. Despite promising progress, these CLIP video-based methods still achieve unsatisfactory results on OV-FER, where changes in facial expressions both between and within videos can be very subtle [25] , as shown in Fig. 1 . To address the above problems when applying the CLIP on OV-FER, we propose a novel human expression-sensitive prompting (HESP) mechanism to significantly upgrade the CLIP by equipping it with the ability to understand facial details and model subtle human expressions effectively, therefore enabling the CLIP-based model to properly deal with unseen emotion categories in the OV-FER task. In general, our contributions are summarized as follows:• We are the first research to explore the OV-FER task. To tackle the task, we propose a novel Human Expressionsensitive Prompting (HESP) framework comprising textual and visual prompting modules, along with a multi-task learning component. HESP significantly augments CLIP's efficacy in achieving effective OV-FER. • Methodologically, first, we introduce learnable text prompts to better depict emotions in both known and unknown classes. We also create learnable visual prompts to guide CLIP in focusing on expression-sensitive areas in video frames, capturing temporal visual expression-sensitive information for both known and unknown data. Then, an open-set multitask learning scheme is proposed to facilitate the cross-modal prompt interaction, thus further enhancing the HESP's performance in capturing inter-video emotion differences for both known and unknown categories. • Experimentally, inspired by previous open-set tasks, we define experimental settings of OV-FER, and conduct extensive experiments on 4 open-set OV-FER settings, showing that our approach achieves new state-of-the-art results. Specifically, our approach demonstrates average relative gains of 20.85% in AUROC and 142.81% in OSCR, compared to the baseline method across all 4 OV-FER tasks, confirming the effectiveness and generality of our proposed model. 
related work: 
 Video-based Facial Expression Recognition (V-FER). Compared to static image FER, V-FER extracts dynamic changes of facial expressions and recognizes emotion categories. Currently, many studies use structures like LSTM [19, 22] , 3D-CNN [24, 26, 42] , Transformer [25, 43] to extract spatio-temporal expression information for V-FER. EST [25] proposes to model the subtle motion of facial expressions in videos by decomposing it into a series of expression snippets. Former-DFER [43] proposes a dynamic Transformer that learns expression features from both temporal and spatial perspectives. However, these methods are limited to closedset datasets with fixed emotion categories, struggling to be applied in real-world open-set environments with unknown emotions.Open-set Recognition. Current open-set recognition methods are classified as: discriminative models and generative models. Discriminative models adds rejection rules to identify unknown classes [4, 7, 16] . Generative models generate fake data that simulates novel classes during training [20, 27] . Although encouraging results have been achieved, both types of methods show limited generalization abilities to FER task due to the subtle inter-class differences in facial expressions. Recently, there have been studies [41, 46] addressing the issue of open-set image FER. For example, VBExT [46] designs a Bi-Attention Expression Transformer to extract more compact features. Open-set FER [41] treats it as a noisy label detection problem and utilizes attention map consistency to separate open-set samples. However, these methods are not designed to handle videos, which limits their applicability in OV-FER.Large-scale Vision-language Models and Prompting Learning.Pre-trained large-scale vision-language models have become significant in computer vision (CV). These models, for instance CLIP [29] and GPT-4 [1] , learning rich visual and linguistic features through pre-training on large-scale image and text data, achieving significant performance improvements in various downstream tasks, such as image classification [45] , object detection [33] , as well as openset recognition [21, 30, 34] . Prompt learning (PL) aids large-scale models in adapting to various downstream tasks [31, 38, 45] . CoOp [45] employs learnable text prompts, avoiding manual prompt tuning. VPT [17] uses trainable parameters as vector sequence visual prompts to improve the performance. Despite the progress, existing PL methods are mainly designed for generic image recognition, lacking effective PL for more challenging FER tasks.Open-set Multi-task Learning Scheme Then, an open-set multi-task learning scheme is devised to facilitate interactions between these modules, improving OV-FER performance by exploring both known and unknown emotion cues. 
method 3.1 preliminary: 
 Problem Definition. We first define the problem of the OV-FER task. In OV-FER, the training set is denoted as 𝐷 train = {(𝑉 𝑖 , 𝑦 𝑖 )}, where 𝑖 indexes over samples, 𝑉 𝑖 represents the 𝑖-th video sample, 𝑦 𝑖 ∈ 𝑌 𝑘𝑛𝑜𝑤𝑛 = {1, . . . , 𝐾 } is the label of 𝑉 𝑖 , and 𝐾 is the number of known classes. Following existing open-set settings [7, 46] , there are novel, unknown expression categories in the test set. We consider all novel, unknown categories as 𝑌 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 = {𝐾 + 1}. Therefore, the test set is defined as 𝐷 𝑡𝑒𝑠𝑡 = {(𝑉 𝑗 , 𝑦 𝑗 )}, where 𝑦 𝑗 ∈ 𝑌 𝑘𝑛𝑜𝑤𝑛 ∪ 𝑌 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 , and 𝑗 indexes over test samples. This task then aims to accurately recognize the category of video 𝑉 𝑗 from both 𝑌 𝑘𝑛𝑜𝑤𝑛 and 𝑌 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 .Negative Representations for Open-set Problems. Negative representations, opposite to known class representations, are particularly beneficial for discovering unknown patterns in open-set datasets [7, 8, 14] . In open-set data, negative representations help distinguish irrelevant patterns, like background features, enabling better recognition of meaningful, unknown patterns. Therefore, data with patterns can be collected from negative representations of all categories to discover unknown categories. For more details, we refer readers to [7] . In the OV-FER task, since the differences between videos could be very small, it is important to identify the patterns related to negative representations, thereby driving us to follow such a mechanism for implementing the CLIP-based OV-FER approach.CLIP Model. The vanilla CLIP [29] first uses a Transformerbased text encoder [32] , denoted as 𝜙 𝑇 , and uses a ViT [13] visual encoder, denoted as 𝜙 𝑉 , to extract high-level features from text and vision inputs, respectively. Suppose we have a textual input 𝑇 and a visual input 𝑉 𝑖 , the CLIP extract features:EQUATIONwhere 𝐹 𝑇 and 𝐹 𝑉 ,𝑖 are extracted textual and visual features, respectively. Then, CLIP correlates the text features with the visual features based on the relationships between the input texts and images. The correlation between 𝐹 𝑇 and 𝐹 𝑉 ,𝑖 is estimated by 𝑐𝑜𝑟𝑟 (𝐹 𝑇 , 𝐹 𝑉 ,𝑖 ) where 𝑐𝑜𝑟𝑟 is a correlation estimation process. Normally, 𝑐𝑜𝑟𝑟 is implemented by inner-product followed by a softmax, and the obtained scores can be used for various zero-shot purposes, like openset tasks. For example, if the correlation score between an image and the phrase "a photo of a happy expression" is very high, we can consider that this image contains a happy face, even without training or fine-tuning.Fig. 2 illustrates the overview of how our proposed HESP method augments CLIP for the OV-FER task. Specifically, our framework comprises three modules: a novel textual prompting module to enhance the CLIP's text encoder, a novel visual prompting module to improve the CLIP's visual encoder, and an open-set multi-task learning scheme to facilitate learning through the interaction between the textual and visual prompting modules. The textual and visual prompting modules enable CLIP to understand emotion-related descriptions and video data, and the learning scheme enables our approach to perform OV-FER effectively. In the CLIP process, we mainly recast the Eq. ( 1) by introducing inputs enhanced by textual and visual prompting:EQUATIONwhere 𝑇 𝐻 and 𝑉 𝐻,𝑖 are enhanced inputs based on the textual and visual prompting of our HESP, respectively, and 𝐹 ′ 𝑇 and 𝐹 ′ 𝑉 ,𝑖 are related refined high-level features for better OV-FER.In the textual prompting of HESP, rather than re-training the entire text encoder of the CLIP which would be extremely costly, we attempt to introduce learnable features more sensitive for both known and unknown categories in OV-FER.Formally, to obtain an enhanced text input 𝑇 𝐻 that could be sensitive to emotion phrases, we introduce learnable representations 𝛿 𝑇 to complement the 𝑇 . As a result, the 𝑇 𝐻 is defined as:EQUATIONwhere ⊕ represents a fusion operation. The detailed implementations are discussed as follows.Textual Prompting for Closed-set Data. For closed-set data, regarding the text description 𝑇 to be associated with visual input, we introduce specific emotional tokens denoted as [CLASS-k] for depicting the 𝑘-th facial expression. The [CLASS-k] is actually an emotional phrase, such as "happy". Normal CLIP usage applies a template of "a photo of X" where "X" here can be an emotional token [CLASS-k], but it cannot appropriately describe a video with human expressions. As a result, we instead introduce learnable templates to instantiate 𝛿 𝑇 .Using the emotional token 𝑇 for the closed-set data, we define that 𝛿 𝑇 is a sequence of learnable text representations, i.e.,EQUATIONwhere 𝑀 is a pre-defined length of 𝛿 𝑇 and 𝑘 refers to the same emotion category with [CLASS-k]. In this study, we set 𝑀 to 16 empirically. As a result, the Eq. (3) will have a form of:EQUATIONwhich means that the ⊕ actually implements a concatenation function that attaches together the 𝛿 𝑇 (𝑘) and the token [CLASS-k]. This design can be partially supported by CoOp [45] which demonstrates that learnable tokens can be integrated with normal tokens to obtain better text prompts regarding specific tasks. Now, by collecting 𝑇 𝐻 (𝑘) for all the closed-set categories, we have the final representation𝑇 𝐻 = {𝑇 𝐻 (1),𝑇 𝐻 (2), . . . ,𝑇 𝐻 (𝐾)}, where 𝐾 is the number of known categories for close-set data. The 𝑇 𝐻 is then fed into the 𝜙 𝑇 for text encoding with CLIP. It is worth mentioning that we calculate 𝑇 𝐻 (𝑘) independently, which means that𝐹 ′ 𝑇 = 𝜙 𝑇 (𝑇 𝐻 ) = {𝜙 𝑇 (𝑇 𝐻 (1)), 𝜙 𝑇 (𝑇 𝐻 (2)), . . . , 𝜙 𝑇 (𝑇 𝐻 (𝐾)))}.Negative Textual Prompting for Open-set Data. To discover unknown classes in open-set data, we follow [7] and introduce negative representations. Specifically, we introduce 𝐾 different tokens, each of which helps discover patterns that do NOT belong to a specific known emotion category. In practice, we apply 𝐾 fixed negative textual prompts as the tokens to help explore unknown emotion categories in open-set datasets. The negative textual prompts T (𝑘) are usually in the form of "This video is not [CLASS-k]", where the [CLASS-k] enumerates all the closed-set emotional categories. Then, we employ the the CLIP text encoder 𝜙 𝑇 to obtain the negative textual features F𝑇 ′ = {𝜙 𝑇 ( T (1)), . . . , 𝜙 𝑇 ( T (𝐾))}.For the OV-FER task, we augment CLIP by introducing learnable visual representations with attention to expression-sensitive areas. Besides, we introduce a temporal modeling module for this visual prompting, enabling the understanding of emotional motions in a video. Our visual prompting of HESP also does not require the re-training of the visual encoder of CLIP, thus we can still make the best use of the powerful visual feature extractors used in the vanilla CLIP like ResNet [15] or ViT [13] to encode visual features.Formally, to introduce a refined visual input 𝑉 𝐻,𝑖 that is sensitive to expression areas on frames of a video, we introduce learnable representations 𝛿 𝑉 , and we have:EQUATION)where ⊕ is a fusion operation. Although in a modality different from texts, we would like to mention that the benefits of adding learnable representations to visual input can also be partially validated by recent research [2, 9, 10, 36, 40] . In the following, we will discuss in detail how this process, for the first time, can be implemented in the OV-FER task.Visual Prompting for Closed-set Data. For closed-set data, we define that the 𝑉 is the video input consisting of a sequence of frames. We use the 𝑞 to index over each frame of a video, thus 𝑉 𝑖 (𝑞) refers to the 𝑞-th frame in 𝑖-th video. Since we aim to focus on expression-sensitive details on 𝑉 𝑖 , we need to first identify what area on a frame contains the expression-sensitive information, so that a learnable representation can complement the visual input 𝑉 𝑖 (𝑞) by emphasizing the expression-sensitive information for CLIP. To achieve this, we introduce an expression-sensitive mask M 𝑖 (𝑞) for the 𝑞-th frame.We define that the expression-sensitive mask M 𝑖 (𝑞) has the value of 1 for areas on expression-sensitive areas of a frame and the value of 0 for other areas. Using this mask, we implement the Eq. ( 6) as :EQUATIONwhich means that the fusion ⊕ is implemented by the weighted sum of 𝑉 𝑖 (𝑞) and 𝛿 𝑉 (𝑞) w.r.t. M 𝑖 (𝑞). For the learnable representation 𝛿 𝑉 (𝑞), we begin with pixel perturbations on the expressionsensitive areas indicated by M 𝑖 (𝑞) and then optimize the parameters within 𝛿 𝑉 (𝑞) during training. The M 𝑖 (𝑞) enables the learnable representation 𝛿 𝑉 (𝑞) to complement the raw visual input on expression-sensitive areas without disturbing other areas. To obtain a proper mask M 𝑖 (𝑞), we use the popular class activation mapping (CAM) [44] mechanism. More specifically, we use the visual encoder of CLIP to extract CAM and use the results to obtain expression-sensitive areas. Based on the CAM result, we find a rectangular area with a higher score and set the values of the mask inside this area to 1, keeping other values at 0. This rectangular area is defined to have a length of 𝑙, and 𝑙 is set to 56. In practice, we surprisingly found that keeping this rectangular area stable throughout the video is more beneficial, thus we only rely on the mask generated in the first frame. Please refer to the supplementary material for more details.For a video, we extract visual features from each of the frame and aggregate them together to obtain the final refined high-level visual features. That is, we have: 𝐹 ′ 𝑉 ,𝑖 = 𝑔(𝜙 𝑉 (𝑉 𝐻,𝑖 )) = 𝑔({𝜙 𝑉 (𝑉 𝐻,𝑖 (1)), 𝜙 𝑉 (𝑉 𝐻,𝑖 (2)), . . . , 𝜙 𝑉 (𝑉 𝐻,𝑖 (𝑁 ))}), where 𝑁 refers to the frame amount, and 𝑔(•) is a temporal global pooling to aggregate all the results from frame 1 to frame 𝑁 to obtain global temporal information.Negative Visual Prompting for Open-set Data. For open-set data, we still introduce negative representations. Instead of textual prompting where we have an explicit negation phrase "not", we attempt to apply implicit negative visual representations V𝑖,𝑘 that need to be optimized together with our HESP promoting modules. Specifically, given 𝐾 known categories, we start with randomly initialized 𝐾 tensors to represent the negative representations for each 𝐾 known category. To ensure consistency with closed-set data inputs for CLIP, we set the dimensions of each implicit negative visual representation as the same with a frame 𝑉 𝑖 (𝑞), namely 3 × 224 × 224. Then, using the encoder 𝜙 𝑉 in CLIP, we extract the negative visual prompting features as F ′ 𝑉 = {𝜙 𝑉 ( V𝑖,1 ), . . . , 𝜙 𝑉 ( V𝑖,𝐾 )}. After training, although the negative text representations already facilitate the identification of irrelevant visual patterns, the performance could be affected by visual noises. Therefore, further calculating the distances between open-set video input and learned implicit negative visual representations allows us to better identify irrelevant visual patterns and help discover more appropriate unknown patterns.To fine-tune and optimize HESP, we introduce an open-set multitask learning scheme consisting of three major objectives: loss for known category learning L 𝐾 𝑁 , loss for negative representation learning L 𝑁 𝐸 , and loss for final prediction L 𝐻 , thereby we have the overall learning objective L as follows:EQUATIONFor known category learning, with the visual features 𝐹 ′ 𝑉 ,𝑖 and textual features 𝐹 ′ 𝑇 augmented by HESP, we simply follow CLIP's procedure [29] to obtain prediction results as,EQUATIONwhere 𝑃 𝐾 𝑁 ,𝑖 is the known class prediction for the 𝑖-th video, 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 is the softmax operation and 𝑐𝑜𝑠 is cosine similarity. Then, we use the cross-entropy loss to make 𝑃 𝐾 𝑁 ,𝑖 predict known classes well. Besides, we also introduce a supervised contrastive loss to further enlarge inter-class feature distances and reduce intra-class feature distances, which is advantageous for feature learning in open-set data. As a result, the L 𝐾 𝑁 is written as:EQUATIONwhere 𝑦 𝑖 is the known class label for the 𝑖-th video, 𝐶𝐸 represents cross-entropy loss, and 𝐶𝐿 represents a contrastive learning loss. Note that the 𝐶𝐿 contrasts 𝐹 ′ 𝑉 ,𝑖 by making features of different known classes away from each other as much as possible.For negative representation learning, as mentioned previously, we follow [7] and use negative representations to de-emphasize potentially irrelevant patterns for better open-set recognition. Therefore, we will have another prediction, termed 𝑃 𝑁 𝐸,𝑖 , based on negative representations. We would like to clarify that this prediction relies on the double negation concept, that is, we attempt to find data that is not negative representations. We achieve this by calculating distances between inputs and negative representations:EQUATIONwhere 𝑑 is Euclidean distance, and F ′ 𝑉 is the collection of visual features extracted from learned negative visual representations. Note that 𝑃 𝑁 𝐸,𝑖 is also a 𝐾-classification problem. We also use crossentropy loss to optimize the prediction 𝑃 𝑁 𝐸,𝑖 , which encourages the negative representations to encode features of little interest to known class data as well as unknown class data. Additionally, we introduce a contrast CLIP loss [29] , represented as 𝐶𝐿𝐼𝑃, to further align the negative textual features F ′ 𝑇 and negative visual features F ′ 𝑉 as closely as possible. Then, we have:EQUATIONIt is worth mentioning that the supervised 𝐶𝐿 loss is not useful here since optimizing the above loss already makes negative representations away from known class features and there is no point in pulling irrelevant features together. Lastly, we have an overall class prediction 𝑃 𝐻,𝑖 :EQUATION)We optimize the 𝑃 𝐻,𝑖 using one another cross-entropy loss:EQUATIONFor inference, we followed existing open-set evaluation strategies [7, 28] . Given a novel video 𝑉 𝑗 as input, we first follow the HESP and CLIP to extract the visual features 𝐹 ′ 𝑉 , textual features 𝐹 ′ 𝑇 , negative visual features F ′ 𝑉 , and negative textual features F ′ 𝑇 . Then, we employ Eq. ( 9), Eq. ( 11), and Eq. ( 13) to calculate the overall class prediction 𝑃 𝐻,𝑗 . Now, still following negative-representation methods [7] for open-set problem, we analyze the probability distributions 𝑃 𝐻,𝑗 on both closed-set data and open-set data and use a dynamic thresholding method to distinguish the unknown class from known classes by introducing a calibration-free measure. We refer more details to [7, 28] .4 Experiments and analysis 4.1 Experiment Setup 4.1.1 Datasets. For evaluation, we used two challenging videobased FER datasets, i.e., AFEW [12] and MAFW [23] . AFEW [12] : The AFEW dataset contains videos from movies and TV series with spontaneous expressions under various conditions. With seven basic emotion labels, i.e., anger, disgust, fear, happiness, sadness, surprise, and neutral, AFEW comprises Train (738 videos), Val (352 videos), and Test (653 videos) splits. MAFW [23] : MAFW is the first large, multimodal, multi-label emotion database containing 11 single emotion categories, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment, 32 compound emotion categories and descriptive texts. MAFW collects 10,045 videos from movies, TV dramas and short videos in social medias. [28, 46] , we divide the expression categories in AFEW and MAFW into known and unknown classes and design four different OV-FER tasks. For better evaluation, we introduce the concept of openness [28, 46] as: 𝑂 (𝐾 : 𝑈 ) = 1 -√︁ 𝐾/(𝐾 + 𝑈 ). 𝐾 and 𝑈 are the number of divided known and unknown expression classes, respectively. A larger 𝑂 indicates more unknown emotions in openset data. The four OV-FER tasks are as follows: 1) OV-FER with 7 basic emotions. In AFEW, we divide the known classes and unknown class sets, according to four different openness values, i.e. , 𝑂 (5 : 2) = 0.15, 𝑂 (4 : 3) = 0.24, 𝑂 (3 : 4) = 0.35, 𝑂 (2 : 5) = 0.47.2) OV-FER with 11 emotions. In MAFW, known and unknown classes are segregated using four distinct openness values: 𝑂 (8 : 3) = 0.15, 𝑂 (6 : 5) = 0.26, 𝑂 (5 : 6) = 0.33, 𝑂 (3 : 8) = 0.48.3) OV-FER with fusion datasets. To simulate more challenging open-set scenario, we combined AFEW with MAFW, using 7 basic emotions as known classes and the other 4 emotions as unknown classes, with the openness 𝑂 (7 : 5) = 0.24. 4) OV-FER with compound emotions. For more challenging settings, we used the 7 basic emotions in MAFW as known classes and 9 compound emotions in MAFW as unknown classes, with the openness 𝑂 (7 : 9) = 0.34.It is worth noting that: 1) under each task, all unknown emotion classes are removed from the training set; 2) we randomly divide known and unknown classes 5 times according to each openness and report the average results for each openness. 
evaluation protocols.: 
 Following the existing work [7, 46] , we selected the Area Under the Receiver Operating Characteristic Curve (AUROC) and the Open-Set Classification Rate (OSCR) as our evaluation metrics.The model was implemented on Ubuntu 20.04 using the PyTorch framework on the NVIDIA GeForce RTX 4090. Facial videos underwent preprocessing with face detection and alignment [11] , followed by cropping to 224x224 size. Training utilized 200 epochs with stochastic gradient descent (SGD) optimizer with momentum. The learning rate initially set to 0.01, decreased by a factor of 0.1 every 30 epochs during training. For a fair comparison, we followed exsiting open-set inference testing method [7, 28] .To comprehensively evaluate the performance of the proposed HESP for OV-FER, we compared it with several open-set image recognition techniques including ARPL [7] , CSSR [16] , DIAS [27] , and Open-set FER [41] , as well as open-set video recognition methods including DEAR [3] and Open-VCLIP [37] . We also considered the integrated method CLIP alone and CLIP+ARPL as baselines for comparison. Extensive experimental results confirm the versatility AUROC OSCR ARPL [7] 56.10 15.31 CSSR [16] 51.29 17.56 DIAS [27] 56.13 15.56 DEAR [3] 55. 14 15.19 Open-set FER [41] 57.88 13.03 Open-VCLIP [37] 59.28 30.17 ARPL [7] 55. 37 15.85 CSSR [16] 60.00 17.64 DIAS [27] 51.63 15.99 DEAR [3] 40.44 13.18 Open-set FER [41] 53.70 14.81 Open-VCLIP [37] 60 random patches within videos. The results are shown in Fig. 3 (b), demonstrating that our learnable visual representations performs the best. We also conducted ablation experiments with various sizes of our learnable visual prompts to 14×14, 28×28, 56×56 and 112 ×112, respectively, as shown in Fig. 3(c ). Obviously, the best result was achieved when the size was 56×56. To further validate the effectiveness and universality of our method, we evaluated our HESP on a large dataset DFEW [18] containing 7 basic emotions, and the results are shown in Table 7 . Our method still performs best, relatively surpassing Open-VCLIP [37] 17.63% and 36.30% on AUROC and OSCR, respectively. Meanwhile, we compared the full model inference speed of various methods in Table 7 . Analysis of the table reveals that our method significantly improves model performance while reducing time consumption. 4 shows the probability scores for known and unknown emotion detection on open-set data through ARPL [7] , CLIP+ARPL, Open-VCLIP [37] , and our HESP. We normalized the probability scores to  
conclusion: 
 In this paper, we propose a Human Expression-Sensitive Prompting (HESP) mechanism to enhance the model's understanding of subtle differences and nuanced human expression patterns in video sequences. HESP comprises a textual prompting module and a visual prompting module, both designed to capture emotion-related patterns within videos and supplement CLIP's original input to better characterize known or unknown emotions. We also devise an open-set multi-task learning scheme to facilitate interactions between the textual and visual prompting modules, augmenting the understanding of novel human emotion-related patterns in video sequences. Extensive experiments on four OV-FER tasks demonstrate that HESP significantly outperforms other state-of-the-art open-set recognition methods. In future work, we will extend HESP to explore multimodal open-set recognition as well as more generalized category discovery in unknown multi-class recognition. 
 
 Appendix:  


Decision: I recommend accept based on the final ratings of all reviewers. Please carefully revise the paper  according to the reviewers' comments.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel Open-set Video-based Facial Expression Recognition (OV-FER) task, aimed at identifying both known and previously unseen facial expressions in videos. To address this challenge, the authors propose a Human Expression-Sensitive Prompting (HESP) mechanism, which enhances the CLIP model's ability to capture subtle facial expression details. HESP includes a textual prompting module, a visual prompting module, and an open-set multi-task learning scheme. Experimental results demonstrate that HESP significantly improves performance over existing methods across four different OV-FER tasks.
strength_weakness: strength_weakness: The study introduces a novel Human Expression-Sensitive Prompting (HESP) mechanism designed to enhance the model's ability to understand subtle differences and nuanced human expression patterns in video sequences. The study also proposes an open-set multi-task learning scheme that facilitates interactions between the textual and visual prompting modules.

The effectiveness of the proposed HESP mechanism and open-set multi-task learning scheme is demonstrated through extensive experiments on multiple OV-FER. The experimental results show that the proposed approach significantly outperforms other SOTA open-set recognition methods.
limitations: limitations: The proposed HESP mechanism involves a sophisticated combination of textual and visual prompting modules along with a multi-task learning scheme. This complexity can make the implementation and tuning of the model more challenging and time-consuming compared to simpler models.

While the paper demonstrates the effectiveness of HESP on several datasets, it focuses primarily on specific video-based facial expression recognition tasks. The generalizability of the approach to other types of open-set recognition tasks or different domains (e.g., non-facial expressions or different video contexts) remains to be fully explored and validated.
paper_summary: paper_summary: This paper proposes a novel Human Expression Sensitive Prompt (HESP) mechanism aimed at enhancing the model's understanding and recognition of facial expression details in videos. The main concept behind the HESP is to introduce new modules to enhance the open set recognition ability of CLIP models. Through intensive experiments, the authors show that their new modules and approach results in significant improvement over baseline methods.
strength_weakness: strength_weakness: 1). Novelty: The HESP mechanism proposed in the paper is conceptually novel, which is not complex yet effectively enhances the recognition capability for known and unknown emotion categories by combining text and visual prompt modules with an open-set multi-task learning scheme.

2). Significant performance improvement: According to the experimental results, HESP has achieved significant performance improvements on multiple standard datasets, especially the improvements in AUROC and OSCR metrics, which prove the impressive effects of the method.

3). Extensive experimental validation: The authors have conducted extensive experiments on different datasets and different openness settings, which helps to prove the robustness and universality of HESP. The paper also includes an ablation study, which helps readers understand the specific impact of different components and design choices on the final performance.
limitations: limitations: I think the overall idea and empirical studies in this paper have no severe limitations.  But I have some suggestions for paper writing:

1). Figure 2 which demonstrates the whole pipeline seems to be too complex. It is better that the author can simplify the pipeline in some way to make it easier to understand.
paper_summary: paper_summary: To deal with OV-FER, this paper proposes a novel Human Expression-Sensitive Prompting (HESP) mechanism, including a textual prompting module with learnable prompt representations, a visual prompting module that encodes temporal emotional information from video frames, an open-set multi-task learning scheme that boosts interactions between the textual and visual prompting modules.
strength_weakness: strength_weakness: - The paper is largely clearly written and makes an easy read.
- The paper addresses a significant limitation in the field of V-FER.
- The proposed approach is intuitive and solid.
limitations: limitations: - The authors need to evaluate how well the proposed approach scales with larger datasets and a more diverse set of expressions. Is there a point at which the model's performance increases or decreases?
- How well can the model deal with ambiguous or mixed expressions? In real-world scenarios, expressions may be ambiguous or mixed. It is recommended to evaluate how the model handles such cases and whether it can accurately classify these ambiguous expressions or recognize them as unknown.
- While the paper reports significant improvements in AUROC and OSCR, I’m wondering if these metrics are sufficient to capture all aspects of performance relevant to OV-FER tasks. Are there other metrics that should be considered?
- Based on my understanding, human expressions can vary widely based on cultural, individual, and contextual factors. Please analyze whether the proposed approach is sensitive to such variations and how it addresses potential biases in the training data. 
- Minor: What’s the latency of running inference with the model compared with existing works? The latency should be low enough for the model deployment for real-world applications.
paper_summary: paper_summary: This manuscript develops a human expression-sensitive prompting mechanism to model video-based facial expression details by using CLIP-based OV-FER method.  The experiments have been conducted to validate the effectiveness of the proposed method. However, there are some limitations as follows:
1. In Fig. 2, how to compute the loss between different outputs in detail? 
2. In Eq. 8, what are the definitions of different loss functions? How do the authors balance the trade-off between different loss? 
3. Please clarify the introduction and abstract clearly.
strength_weakness: strength_weakness: The task and idea are interesting.
limitations: limitations: The authors should polish the manuscript.
paper_summary: paper_summary: The author studied the open-set facial expression problem and proposed using the HESP mechanism, which utilizes CLIP to enhance performance in addressing the open-set issue.
strength_weakness: strength_weakness: The paper is written clearly and is easy to understand.

This work extends the original image-based open-set FER [40] to a video-based approach.

The use of CLIP significantly improves the performance of open-set facial expression recognition (FER).
limitations: limitations: The overall narrative of this paper is not very convincing to me, which prevents me from giving it a high ranking. The issue of small inter-video differences is crucial to open-set FER, but there is no deep discussion on how CLIP addresses this problem.

Figure 1 of this paper illustrates the small intra-video changes problem. However, this issue is neither mentioned nor addressed in the rest of the paper. If the intention of Figure 1 is similar to that of Figure 1 in [40], I believe there is no need to mention small intra-video changes.

The paper uses a visual prompting module to convert a video input into an image input. However, I am curious whether the introduction of CLIP will also enhance the performance of the original image-based open-set FER. It would be beneficial to include some discussion on this point.
paper_summary: paper_summary: The paper focuses on an open-set video-based facial expression recognition task. The categorical classes of facial expressions that are not included in the training set are considered the unknown classes. It aims to distinguish the unknown categorical facial expressions from the test set. The authors proposed a new method called Human Expression-Sensitive Prompting (HESP), which integrates textual and visual modules to improve the ability of the CLIP model to process and recognize both known and unknown facial expressions. Textual prompting and visual prompting are employed to upgrade the performance. A multi-task learning scheme is employed to achieve the training.
strength_weakness: strength_weakness: The authors introduce a new topic into video-based facial expression recognition (V-FER) tasks. The proposed method seems effective.
limitations: limitations: Though the performance upgrade seems obvious, the innovation to upgrade the existing CLIP method is incremental and trivial. The following are some suggestions to improve this paper. 

1. It will be better to first write down the problem that the video-based facial expression recognition faces to introduce the specific design of your proposed modules. 

2. Line 144 ~ Line 172: It is cumbersome to summary the novelty in such long paragraphs. Please simplify the proposed novelty. In addition, the difficulties and proposed methods can be clearly clarified in the first few paragraphs of the introduction section.

3. In line 201, the authors mentioned that the VBExT and Open-set FER can not handle videos. That is false. They can deal with video-based FER once they replace the backbone networks.

4. Line 273: The authors mentioned that the negative representations help distinguish irrelevant patterns from the relevant ones, enabling better recognition of meaningful, unknown patterns. It infers that the negative representations help to recognize the unknown classes. Therefore, it is unclear why the authors hold the view that the negative representation will be de-emphasized in the OV-FER task, even though the differences between the videos could be very small.

5. Line 317: ,including -> like 

6. About Experiments: It is worth providing the benchmark and testing the proposed method in a large-scale dataset with 7 categorical discrete emotions, such as the DFEW dataset. 

   Dfew: A large-scale database for recognizing dynamic facial expressions in the wild, in ACM MM 2020.

7. Related work: Please compare the prior works with your proposed method in the section of related work.

8. About the idea: The proposed prompting approach can be easily extended into image-based facial expression recognition. Why does the author emphasize this method is specifically designed for video-based facial expression recognition? 

9. In line 376, the authors said the designed method can be partially supported by CoOp. Please clarify this statement. In addition, the proposed method looks like a simple combination of some modules. Can the authors clarify how do they combine organically to present the upgraded performance?