Integrating Content-Semantics-World Knowledge to Detect Stress from Videos

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-27 14:08:05

Abstract: Stress has rapidly emerged as a significant public health concern in the contemporary society, necessitating prompt identification and effective intervention strategies. Video-based stress detection offers a non-invasive, low-cost, and mass-reaching approach for identifying stress. In this paper, we propose a three-level content-semantic-world knowledge framework, addressing three particular issues for video-based stress detection. (1) How to abstract and encode video semantics with frame contents into visual representation? (2) How to leverage general-purpose LMMs to augment task-specific visual representation? (3) To what extent could general-purpose LLMs contribute to video-based stress detection? We design a Slow-Emotion-Fast-Action scheme to encode fast temporal changes of body actions revealed from video frames, as well as subtle details of emotions per video segment, into visual representation. We augment task-specific visual representation with linguistic facial expression descriptions by prompting general-purpose Large Multimodal Models (LMMs). A knowledge retriever is built to evaluate and select the most proper deliverable of LMMs. Experimental results on two datasets show that 1) our proposed three-level framework can achieve 90.89% F1-score in UVSD dataset and 80.79% F1-score, outperforming state-of-the-art; 2) leveraging LMMs helps to improve the F1-score by 2.25% in UVSD and 3.55% in RSL, compared to using the traditional Facial Action Coding System; 3) purely relying on general-purpose LMMs.
is insufficient with 88.73% F1-score in UVSD dataset and 77.48% F1-score in RSL dataset, demonstrating the necessity to combine
task-specific dedicated solutions with world knowledge given by LMMs.

Main paper: 
 introduction: 
 In today's rapidly advancing society, people are experiencing unprecedented levels of stress, stemming from both traditional factors (such as further education, examination, marriage, etc.), as well as newly emerging stressors (such as the outbreak of the coronavirus epidemic and the displacement of jobs by artificial intelligence). Too much stress, if left unchecked, can have profound physical and psychological ramifications, exacerbating existing issues and ensnaring life in a harmful cycle [3, 15] . Recognizing the critical importance of early intervention, detection of stress emerges as a paramount concern [41] .Traditional ways rely on well-defined psychological questionnaires (e.g., Cohen's Perceived Stress Scale (PSS-14) [4] and Social Readjustment Rating Scale (SRRS) [17] ), various physiological signals (e.g., blood pressure, heart rate, electromyography, electroencephalogram, etc. [11, 32] ), and/or social media behaviors [8, 25, [40] [41] [42] for stress detection. Recently, with the wide deployment of surveillance videos, as well as the rapid development of machine learning techniques, detecting stress from videos based on handcrafted and/or deeply learned features has attracted research attention due to its non-invasiveness, low-cost, and mass-reaching advantages [13, 16, 19, 20, 22, 47, 48] .In this study, we are interested in examining the role of world knowledge delivered by Large Multimodal Models (LMMs) which made significant progress very recently in video-based stress detection. To this end, we build a three-level (content-semantic-world knowledge) framework, enclosing three particular questions to be addressed towards stress detection. According to the psychological studies, stress is a feeling of emotional strain and pressure [30] . Fostering comprehensive understanding of stress manifestations (emotion and emotion dynamics) could inevitably enhance the stress detection capability due to the high correlation of stressful states and emotions. Hereby, to address 𝑄 1 , we attempt to sense emotion (of seven possible categories -joy, love, surprised, angry, sorrow, anxiety, and hate) from each video frame to grasp the most prominent emotion per video segment. Specifically, as human emotions exhibit a slower pace of shift than their body actions [24, 31] , we draw inspirations from the powerful SlowFast mechanism [12] , and design a Slow-Emotion-Fast-Action scheme to encode segment-wise emotion with fast temporal changes of body actions as reflected from each frame in the video segment into visual representation.Furthermore, as emotions evoke specific patterns of facial muscle actions, but what is expressed by the muscle movements cannot be fully encapsulated by the emotions [2] , we augment the obtained visual representation with linguistic facial expression descriptions delivered by general-purpose pre-trained Large Multimodal Model (LMMs). Considering that LMMs tend to hallucinate, and may generate contents unfaithful to the specific requirement [49] , we develop a knowledge retriever to evaluate multiple generations of facial expression descriptions from LMMs based on original face images, and then select the best generation as the final facial expression description to address 𝑄 2 .In response to 𝑄 3 , we evaluate the performance of the proposed three-level stress detection framework on two stress detection datasets. The experimental results show that 1) our method can achieve strong performance with 90.89% F1-score in UVSD dataset and 80.79% F1-score in RSL dataset; 2) leveraging LMMs enables to improve the detection by 2.25% F1-score in UVSD and 3.55% F1-score in RSL, compared to using the traditional Facial Action Coding System (FACS) [10] which decomposes facial expressions into actions appearing on different facial parts, named Action Units (AUs) [23] .3) purely relying on general-purpose LMMs is insufficient with in 88.73% F1-score in 77.48% UVSD and F1-score in RSL, demonstrating the necessity to combine task-specific dedicate solutions with world knowledge given by LMMs.The contributions of the paper can be summarized as follows.• We present a three-level framework, unifying frame-wise video content, video segment-wise emotions, and world knowledge about facial expressions for video-based stress detection. • We design a Slow-Emotion-Fast-Action scheme to encode fast temporal changes of body actions revealed from video frames, as well as subtle details of emotions per video segment, into visual representation. • We explore the use of world knowledge by prompting generalpurpose Large Multimodal Models (LMMs) to augment taskspecific visual representation with linguistic facial expression descriptions. A knowledge retriever is particularly designed to evaluate and select the most proper deliverable of LMMs. 
related work 2.1 video-based stress detection by applying deep learning methods: 
 Deep learning methods have achieved impressive results in various video-related tasks. There have been early efforts in video-based stress detection tasks. Jeon et al. [20] proposed a method for stress detection combining spatial and temporal attention. Its core idea is that the two attention mechanisms enable the model to focus on frames highly correlated with stress and the regions on a single frame that are highly correlated with stress. Gao et al. [13] , Zhang et al. [48] correlated emotion with stress detection, and concluded that stress is sensitive to anger, fear and sadness. In this light, they proposed to classify a video as stressful when the percentage of expressions related to negative emotions in the video surpass a threshold. Furthermore, Zhang et al. [47] combined the dynamics of action with emotion for stress detection, introducing a two-leveled stress detection network named TSDNET. They utilized facial emotion recognition to obtain the most expressive and most expressionless facial images in the video to generate emotions representations, and obtained action representations by computing optical flow between the first and the last video frame. Despite the success in integrating action representations with emotion representations for stress detection, human action and stress-related emotion tend to exhibit different pace of shifts [24, 31] . In this light, we model action and emotion with lens of different scales. Specifically, we propose Slow-Emotion-Fast-Action with a temporally varying dual-stream structure, in which dynamics of actions and emotions are encoded with a different pace. Meanwhile, in prior psychological studies, facial expressions can be decomposed into a group of units, in which certain co-occurrence of units can reveal states of inner feeling [5] . We thus delve into the facial expressions alongside action and emotion representations.Large Multimodal Models (LMMs) represent the cutting-edge advances in artificial intelligence, designed to comprehend a diverse range of modalities, including text, images, audio, and video. LMMs are extensively pre-trained on large, diverse datasets that contain a variety of genres. Such pre-training process helps the model encode a wealth of concepts and cross-modal relationships, which lays the foundation for the downstream tasks. LMMs represented by GPT-4V have demonstrated stunning performance in image captioning and visual question answering [38] . Very recently, Wu et al. [43] , Yang et al. [46] have successfully employed GPT-4V to tackle visual understanding, language comprehension, and visual puzzle solving tasks. In this work, we delve into the capability of LMMs to recognize facial expressions incorporating psychological FACS knowledge, which enhances the representation of stress. 
overall framework: 
 Given a video 𝑉 of a subject, we aim to detect whether the subject is stressed or not. To capture the body actions of the subject, we follow Feichtenhofer et al. [12] to set a sampling rate 1/𝜏 that uniformly samples one frame out of 𝜏 frames from video 𝑉 , obtaining a frame sequence named "Action Stream" denoted as𝑆 𝑎 = {𝐴 1 , 𝐴 2 , • • • , 𝐴 𝑛 },in which 𝑛 is the number of frames sampled. As the shift of human emotions exhibits a slower pace than actions, we set a slower sampling rate 1/𝑚𝜏 to capture the details of emotions, in which 𝑚 is the sampling rate ratio. Specifically, for 𝑆 𝑎 of 𝑛 frames, we analyze the subject's emotion based on a segment of 𝑚 (𝑚 << 𝑛) consecutive frames at each time, thus obtaining a set of ⌊𝑛/𝑚⌋ emotion features of this subject. For each segment of 𝑚 frames, we design an emotion extraction module to obtain the representative emotion of the subject and the most emotional face image. After gathering emotion features from ⌊𝑛/𝑚⌋ segments, we acquire a stream of subjects' emotions (referred to as "Emotion Stream") and a stream of the most emotional facial images (referred to as "Face Stream"), denoted as𝑆 𝑒 = {𝐸 1 , 𝐸 2 , • • • , 𝐸 ⌊𝑛/𝑚⌋ } and 𝑆 𝑓 = {𝐹 1 , 𝐹 2 , • • • , 𝐹 ⌊𝑛/𝑚⌋ } respectively.To construct visual representations for each video, we introduce the Slowfast mechanism to model the "Face Stream" and "Action Stream" jointly. Meanwhile, to enrich facial features while dealing with subtle differences in facial expressions, we introduce textualized facial description features by employing a large multimodal model (i.e., GPT-4V [1] ) and pre-trained facial analysis model (i.e., SCN [39] ) based on the "Face Stream". A sequence-level encoder then generates linguistic representation of the video after acquiring segment-level linguistic facial description features. Finally, we concatenate the visual and linguistic representation and employ a fully-connected stress predictor to detect the stress of the subject. The overall framework is illustrated in Figure 1 .For each segment of 𝑚 consecutive frames from video 𝑉 , denoted as𝑆 𝑖,𝑖+𝑚 𝑎 = {𝐴 𝑖 , 𝐴 𝑖+1 , • • • , 𝐴 𝑖+𝑚 },we employ a selection process that extracts the most representative emotion 𝐸, and the most emotional face image 𝐹 , details shown below. Image-level Emotion Detection. We adopt a facial recognition algorithm MTCNN [44] to localize the facial regions of each frame to obtain the collection of face images, denoted as𝐹𝑎𝑐𝑒 𝑖,𝑖+𝑚 = {𝑓 𝑎𝑐𝑒 𝑖 , 𝑓 𝑎𝑐𝑒 𝑖+1 , • • • , 𝑓 𝑎𝑐𝑒 𝑖+𝑚 }.Then we employ a Self-Cure Network (SCN) [39] to recognize the emotion of each face image with the form of seven categorical emotions (i.e., "angry", "disgust", "fear", "happy", "sad","surprise", "neutral"), and obtain the sequence of emotions, denoted as𝐸𝑚𝑜𝑡𝑖𝑜𝑛 𝑖,𝑖+𝑚 = {𝑒𝑚𝑜 𝑖 (𝑝 𝑖 ), 𝑒𝑚𝑜 𝑖+1 (𝑝 𝑖+1 ), • • • , 𝑒𝑚𝑜 𝑖+𝑚 (𝑝 𝑖+𝑚 )}, where 𝑝 𝑘 (𝑘 = 𝑖, • • • , 𝑖 + 𝑚) ∈ [0, 1] is the confidence of the recog- nized emotion 𝑒𝑚𝑜 𝑖 .Most Representative Emotion Selection. In order to minimize the noise introduced by the emotion recognition model's misjudgment of individual frames, we select the emotion that appears most frequently in the 𝐸𝑚𝑜𝑡𝑖𝑜𝑛 𝑖,𝑖+𝑚 as the most representative emotion in this phase. It can be denoted as𝐸 = 𝑀𝑂𝐷𝐸 (𝐸𝑚𝑜𝑡𝑖𝑜𝑛 𝑖,𝑖+𝑚 ),where 𝑀𝑂𝐷𝐸 is the mode of the set. Most Emotional Face Image. After obtaining the most representative expression 𝐸, we query in the 𝐸𝑚𝑜𝑡𝑖𝑜𝑛 for the elements with 𝑒𝑚𝑜 value 𝐸, next we select the element with the highest 𝑝 value, and define the face image corresponding to this element in 𝐹𝑎𝑐𝑒 as the most emotional face image.Here is an example with a segment of 6 face images, the sequence of emotion and the sequence of confidence obtained by SCN are 𝐸𝑚𝑜𝑡𝑖𝑜𝑛 = { "happy"(0.8982),"happy" (0.8803), "neutral"(0.7655), "surprise"(0.8763), "happy"(0.9866), "happy"(0.9951)}. First, we obtain the most representative emotion 𝐸 ="happy" by calculating the mode of the 𝐸𝑚𝑜𝑡𝑖𝑜𝑛, and then we select the element from all "happy" elements with the highest confidence, so the last face image recognized as "happy" with confidence 0.9951 is selected as the most emotional face image.As the shift of actions and emotions exhibits different paces, we propose Slow-Emotion-Fast-Action framework inspired by [12] , which learns the visual representation of a video through a temporally varying dual-stream structure. Specifically, the slow pathway has larger channel sizes and lower frame rate to capture subtle details of emotions, while the fast pathway has high temporal resolution to capture fast body motions. Therefore, we adopt 𝑆 𝑓 as the input of the slow pathway, and the larger size of channels can obtain more semantic information for each face image. On the other hand, we take 𝑆 𝑎 as the input of fast pathway to capture the change of movement with higher temporal resolution. The visual representations of video 𝑉 are computed as follows:𝑅 𝑉 = 𝑆𝑙𝑜𝑤 𝑓 𝑎𝑠𝑡 (𝑆 𝑓 , 𝑆 𝑎 ).The details of the slow pathway and the fast pathway are shown in Table 2 .We further exploit knowledge from large multimodal models (LMMs) to enrich semantics conveyed in "Emotion Stream"𝑆 𝑒 = {𝐸 1 , 𝐸 2 , • • • , 𝐸 𝑛/𝑚 } and "Face Stream" 𝑆 𝑓 = {𝐹 1 , 𝐹 2 , • • • , 𝐹 𝑛/𝑚 }.For the 𝑖th segment, the most representative 𝐸 𝑖 is combined with the most emotional face image 𝐹 𝑖 and fed into the segment-level encoder to acquire linguistic representation. Given ⌊𝑛/𝑚⌋ segment-level linguistic representations, sequence-level linguistic representation of the whole video can be generated via the sequence-level encoder.Facial Description Generation and Selection. Large multimodal models (LMMs) have demonstrated the capacity to recognize finegrained visual features [26, 50] . For face image 𝐹 𝑖 , we employ a LMM (i.e., GPT-4V) to recognize facial expressions with an instruction based on Facial Action Coding System (FACS). FACS is a widely applied manual extraction method for facial feature extraction, which divides facial expressions into 46 primary units, focusing on 9 regions (i.e., eyes, nose, lid, lips, lip corner, neck, jaw, cheek and eyebrow). We thus instruct the LMM to focus on these 9 regions and describe corresponding movements, the instruction and result shown in Figure 2 .Unfortunately, the outputs of LMM can be diverse, including unfaithful content that does not meet the demand due to hallucination [28, 49] . Therefore, for each face image 𝐹 𝑖 we sample 𝑘 times to obtain 𝑘 different descriptions from the LMMs, denoted as {𝑑 1 , • • • , 𝑑 𝑘 }. We design a FACS knowledge retriever 𝑅 that selects the best output among these descriptions, which is a vision-language model consisting a linguisticencoder T and image encoder I. For each LMM description 𝑑 𝑗 it generates a score: 𝑅(𝑑 𝑗 , 𝐹 𝑖 ) = sim(T (𝑑 𝑗 ), I (𝐹 𝑖 )), where sim(•, •) denotes cosine similarity, T (𝑑 𝑗 ) and I (𝐹 𝑖 ) denotes linguistic representation of 𝑑 𝑗 and visual representation of 𝐹 𝑖 encoded by T and I. As a result, the final facial description for the 𝑖-th segment is selected as:𝐷 𝑖 = argmax 𝑑 𝑗 ∈ {𝑑 1 ,••• ,𝑑 𝑘 } 𝑅(𝑑 𝑗 , 𝐹 𝑖 ).To help the retriever 𝑅 effectively distinguish helpful LMM answer from unfaithful ones, we train 𝑅 before stress detection learning. We build a face image set from the training samples of stress detection, for each face image 𝐹 𝑖 , we employ expert to annotate facial action units based on FACS knowledge, and transform them into natural language descriptions as ground truth d. We compute linguisticsimilarity between d and 𝑑 1 , • • • , 𝑑 𝑘 , which are 𝑘 answers from LMMs to describe facial expressions of 𝐹 𝑖 . We label the answer with highest similarity with d as positive and 𝑘 ′ answers with lowest similarity as negative, denoted as 𝑑 + and 𝑑 - 1 , • • • , 𝑑 - 𝑘 ′ respectively. We can now define the typical contrastive learning objective and minimize for each face image the negative log likelihood of the positive example:L (𝐹 𝑖 , 𝑑 + , 𝑑 - 1 , . . . , 𝑑 - 𝑘 ′ )=log 𝑒 sim( I (𝐹 𝑖 ),T (𝑑 + ) )𝑒 sim( I (𝐹 𝑖 ),T (𝑑 + ) ) + 𝑘 ′ 𝑗=1 𝑒 sim( I (𝐹 𝑖 ),T (𝑑 - 𝑗 ) ).(EQUATIONSegment-level Encoding. After retrieving the best facial expression description with 𝑅, we construct a manual template to integrate the emotion 𝐸 𝑖 with facial expressions generated by LMMs in a natural language form, which can be fed into the BERT encoder. The template is shown in Table 1 . We denote the above text integrating 𝐷 𝑖 and 𝐸 𝑖 as 𝑇 𝑖 . We then feed 𝑇 𝑖 into a trainable BERT [7] encoder to obtain segment-level representations 𝑥 𝑖 . Video-level linguistic representation. Given the linguistic representation of segment-level facial description 𝑥 𝑖 , we feed the representation of all phases 𝑋 = {𝑥 1 , 𝑥 2 , • • • , 𝑥 ⌊𝑛/𝑚⌋ } into a Long Short-Term Memory network (LSTM) to acquire the hidden states in the sequence:ℎ 𝑡 = 𝐿𝑆𝑇 𝑀 (𝑥 𝑡 , ℎ 𝑡 -1 ),where ℎ 𝑡 and ℎ 𝑡 -1 denote the hidden state vector of LSTM at time 𝑡 and 𝑡 -1. In addition, we aggregate the hidden states from all phases via a phase attention layer to obtain the attention vector 𝛼 that assign different weights to each phase.𝛼 = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 ( h × 𝑤 𝑎 + 𝑏 𝑎 ),where h={ℎ 1 , ℎ 2 , • • • , ℎ 𝑛/𝑚 } is the aggregation of the hidden states from all phases, 𝑤 𝑎 and 𝑏 𝑎 are trainable parameters of the phase attention layer. Finally, we obtain the video-level linguistic representation 𝑅 𝑇 of the video:𝑅 𝑇 = (𝛼 𝑇 × h) × 𝑤 𝑡 + 𝑏 𝑡 ,where 𝑤 𝑡 and 𝑏 𝑡 are trainable parameters. After we acquire the visual and linguistic representations of the video, we fuse the two representations and classify the stress of the subjects in the video. We merge visual representation 𝑅 𝑉 and global linguistic representation 𝑅 𝑇 , and then apply a fully connected stress predictor 𝑝 to compute whether the subject is under stress:ŷ = Sigmoid(𝑝 (𝑅 𝑉 ⊕ 𝑅 𝑇 )).Given stress label 𝑦, we optimize the detection framework with the following cross entropy loss:EQUATION4 EXPERIMENT 4.1 Datasets. Subjects were asked to watch videos while being recorded by a video camera. Videos were labeled as "unstressed" when subjects were asked to watch videos of scenery, food production, and variety show episodes, and "stressed" when subjects were asked to watch knowledge-intensive videos and take a question and answer test after they finished watching. The dataset contains a total number of 2,092 video samples, including 920 "stressed" samples, and 1,172 "unstressed" samples. 
reality show about lies (rsl) dataset.: 
 Lying is a complex psychological behavior intricately tied to cognitive processes and mental activities [6] . For most people without special training, lying induces psychological stress which can be reflected in facial expressions, sounds and body movements [34] . Drawing inspiration from this, we curated a video stress dataset based on a reality TV program named "Odd Man Out". In an episode titled "6 Introverts vs 1 Secret Extrovert," for instance, an extrovert impersonates an Surviving until the end, the liar(s) stand to claim the prize money, motivating heightened engagement throughout the program. Encountering constant scrutiny, the liar(s) undergo persistent stress from beginning to the end. All video frames featuring the liar(s) were labeled as "stressed" except for the end-of-show statement. Conversely, the end-of-show statement by the liar(s) and all footage of other participants were labeled as "unstressed".To construct the dataset, we selected 30 truth-tellers and 30 liars from a total of 195 participants over 28 sessions, maintaining a oneto-one male-to-female ratio. All video frames were refined to depict only one participant per frame, with clear visibility of their face and upper body. This meticulous process yielded a dataset comprising 706 video clips, further detailed in Table 3 as the Reality Show about Lies (RSL) Dataset.For all experiments, we perform 10-fold cross validation. The original format of image frames from UVSD and RSL dataset is 640×480 (ℎ𝑒𝑖𝑔ℎ𝑡 × 𝑤𝑖𝑑𝑡ℎ). We resize each original frame into 224 × 224, and feed them into the model. The format of each face image obtained by the MTCNN is also 224 × 224. We leverage Resnet-18 as the backbone network structure in visual representations with Slow-Emotion-Fast Action. we set the sampling rate 1/𝜏 to 1/8 following Feichtenhofer et al. [12] , and the sampling ratio 𝑚 to 8. More details of the slow pathway and fast pathway are shown in Table 2 . We adopt gpt-4-vision-preview from the GPT-4 family for facial description, and we also limit the token number of model output within 150. We employ CLIP-B/16 (CLIP-B) [33] as the knowledge retriever 𝑅. Specifically, it consists of a ViT-B/16 Transformer as the image encoder and a masked self-attention Transformer [36] as the text encoder. To collect training samples for retriever 𝑅, we annotate AU labels with 300 face images, which are randomly selected from 150 stressed video samples and 150 unstressed video samples of UVSD dataset. The maximum sentence length of the information integration is set to 200, including the [CLS] token at the beginning of the sentence and the [SEP] token prior to the facial description. To optimize the knowledge retriever with loss term Eq. 1, we set the learning rate to 3e-5 and the batch size to 32. We set the number of negative samples 𝑘 ′ to 3. To optimize the detection model with loss term Eq. 2, we set the batch size to 16 and learning rate to 3e-3. Adam [21] is adopted as the optimizer. We categorize the currently competitive video-based stress detection methods into three groups, and then compare our method with representative methods from each category.(1) Action units-based stress detection Dependent Model. Viegas et al. [37] propose to extract 17 different action units from the subject's face image, and then employ different simple classifiers (i.e. Random Forest, LDA, Gaussian Naive Bayes and Decision Tree) to recognize stress based on the action units. We implement this baseline using the best performing Gaussian Naive Bayes classifier.FDASSNN. Gavrilescu and Vizireanu [14] employ an Active Appearance Model (AAM) [9] to detect intensities of different action units, and a multi-layer perceptron to transform action unit intensities into stress detection result.(2) Frame-wise Emotion Detection and Aggregation Gao et al. Gao et al. [13] extract 49 feature points of each face image from a video and apply SVM to classify each frame as positive or negative emotions. When the percentage of frames with negative emotions in a video exceeded a threshold, the video can be classified as stressed.Zhang et al. Zhang et al. [48] leverage a Convolutional Neural Network (CNN) to detect emotion in each video frame. If two-thirds of the frames show emotions of anger, sadness, or fear, the video can be detected as having stress. TSDNet. Zhang et al. [47] first obtain face-and action-level representations separately, and then fuse the results through a streamweighted integrator with local and global attention for video stress detection.GPT-4V. GPT-4-vision [1] (GPT-4V) is a state-of-the-art large multimodal model. GPT-4V demonstrates impressive vision understanding ability, yet it does not possess the ability to accept video data directly. To implement video-based stress detection with GPT-4V, we take 10 frames evenly from each sample and allow the GPT-4V to determine the stress labels of the videos.As shown in Table 4 , our framework outperforms others on the UVSD dataset, achieving the highest accuracy (91.25%) and F1score (90.89%). This represents a substantial improvement of over 4.83% and 5.36% compared to TSDNet [47] , the best video-wise stress detection baseline. Additionally, compared to the leading frame-wise emotion detection and aggregation method by Zhang et al. [48] , our method demonstrates improvements of over 9.77% and 18.8%. When contrasted with machine learning-based methods like Dependent Model [37] and FDASSNN [14] that rely on action units, the enhancements our method brings are even more pronounced. In the RSL dataset, our framework also achieves the best performance, with 86.50% accuracy and 80.79% precision, outperforming the best baselines such as TSDNet [47] by 4.74% and 5.80%, respectively. These all results confirm the effectiveness of our framework in incorporating the Slow-Emotion-Fast-Action scheme, FACS, and LMM for stress detection.4.5.1 Effectiveness of Visual Encoding and Linguistic Encoding. To evaluate the impact of different modalities, we conducted a modality ablation study with ablation variant "only vision" that only uses visual representations for stress detection, and "only linguistic" that only uses linguistic representations for stress detection. The results are presented in Table 5 . We observed a performance decline of 3.42% and 4.11%, when excluding the text modality. Similarly, excluding the vision modality led to a decline of 15.4% and 12.0%. These results highlight the greater contribution of the visual modality to our model compared to the linguistic modality.The effectiveness of leveraging LMMs to generate facial description. We validate the effectiveness of our facial description generation by prompting LMMs, compared with directly using Action Units (AUs) as facial description. We design ablation variant "AUs" which replaces our linguistic representation with AU-based representation by detecting AUs on each frame and transforming the AU signals into a video-level representation with a LSTM encoder [29] . As shown in Table 6 , using the rich linguistic information obtained from the LMMs for facial description provided stronger representation information than action units with only discrete AU signals in the stress detection task, this mechanism enhances performance by 2.25% and 3.55% on the respective datasets.To evaluate the rationality of the Slow-Emotion-Fast-Action scheme, we conducted experiments focusing on the simultaneous consideration of emotion and action at the same sampling frequency. We design ablation variants with pure slow or fast sampling rates. Specifically, 1. "SE-SA" represents temporal invariant dual-stream structure with slowpath, and 2. "FE-FA" represents temporal invariant dual-stream structure with fastpath. The results are presented in Table 7 . When both emotion and action use a slow sampling frequency, the performance levels are 80.53% and 68.32%, respectively. Conversely, using a fast sampling frequency for both yields performance levels of 86.56% and 76.68%. However, employing the Slow-Emotion-Fast-Action scheme leads to the highest performance of 90.89% and 80.79%, underscoring the rationale and effectiveness of this approach. Our proposed method is based on the phenomenon that shifts of human emotions have a slower pace than actions, thus we set different sampling frequencies for emotions and actions. A sampling rate for emotions that is too low can result in ignoring the emotions that actually occur, and a sampling frequency that is too high can suffer from noise interference due to problems with the accuracy of emotion classifiers. As is illustrated in Figure 3 , we enumerate the outcome of SCN, one of the most effective emotion recognition models, Deepface [35] , the most commonly used facial emotion recognition tool, and GPT-4V for emotion recognition on consecutive video frames. In addition to the significant lack of ability of GPT-4V in emotion recognition [27] , the other two dedicated models are unable to maintain consistent results in frame-by-frame emotion detection with variations and their subtleties. We empirically verify the impact of different sampling ratios on the experimental results. As shown in Figure 4 , the best result of 90.89% is obtained when the ratio reach to 2 3 on the UNVD dataset and 81.83% at 2 4 on the RSL dataset. Before obtaining the best result, the model's performance on both datasets increase considerably with the rise of the sampling ratio, with a comparison of the performances at a sampling ratio of 2 1 increasing by 9.72% and 10.55%, respectively. Thereafter the model's performance on both datasets start to decrease with the growth of the sampling ratio, decreasing by 4.33% and 5.15% when the sampling ratio reaches 2 6 , respectively.We provide a case study to analyze our prompting LMMs to generate facial descriptions and selecting the best generation based on FACS knowledge. As shown in Table 9 , the LMMs output with the FACS-based prompt and retriever (see "with FACS Knowl.") fully encompasses the knowledge that the Action Units can provide, and also provides more detailed description of the face region by region. with FACS Knowl. The character in the photo is showing a broad smile with their eyes closed. The smile leads to the cheeks being 𝑟𝑎𝑖𝑠𝑒𝑑 6 and a bunching of the skin around the eyes, which is often associated with a genuine, joyful smile. This facial expression typically indicates happiness or amusement.Eyes: The eyes are 𝑛𝑎𝑟𝑟𝑜𝑤𝑒𝑑 𝑎𝑛𝑑 𝑠𝑙𝑖𝑔ℎ𝑡𝑙𝑦 𝑐𝑟𝑒𝑎𝑠𝑒𝑑 42 at the corners, which is typical when a person is smiling or laughing. Check: The cheeks are 𝑟𝑎𝑖𝑠𝑒𝑑 6 , pushing up towards the eyes, which contributes to the squinting appearance of the eyes. Lid: The eyelids are partially 𝑐𝑙𝑜𝑠𝑒𝑑 7 , as they often are when a person smiles broadly. Lips: The lips are 𝑝𝑎𝑟𝑡𝑒𝑑 𝑎𝑛𝑑 𝑠𝑡𝑟𝑒𝑡𝑐ℎ𝑒𝑑 20 horizontally, revealing the upper teeth in a smiling gesture. Mouth: 𝑂𝑝𝑒𝑛 27 in a wide smile, showing teeth, which suggests happiness or laughter... 
conclusion: 
 This paper shows an effective way to integrate task-specific videobased stress detection model with general purpose Large Multimodal Models to achieve strong performance. Specifically, we introduce a three-level content-semantic-world knowledge framework for videobased stress detection. A Slow-Emotion-Fast-Action scheme is designed to encode temporal changes of body actions as well as subtle details of emotions. It explores the use of world knowledge by prompting general-purpose large multimodal models (LMMs) to generate linguistic facial expression descriptions. Furthermore, we capitalize on Facial Action Coding System (FACS) knowledge to design a prompt to instruct the LMMs and develop a knowledge retriever to select the most proper deliverable of LMMs. The experiments on two datasets show that our framework achieves state-of-art performance. 
limitations: 
 We identify one major limitation of this work is its input modality. Specifically, our method is limited to detecting stress with video inputs and ignores inputs in other modalities such as audio and electroencephalogram (EEG). Such modalities provide valuable information that can be used to enhance stress detection. Fortunately, through multi-modal pre-training models [18, 45] , we can obtain robust representations with more diverse modalities.  
 
 Appendix:  


Decision: The paper received mixed review. The paper is well-written, the motivation is clear and the approach is reasonably novel, which the reviewers agreed on. However, as reviewers pointed out, there are some limitations, and the design needs some more justification and more experimentation in the future.

Reviews: review: paper_summary: paper_summary: This paper proposes a content-semantic world knowledge framework for video-based stress detection. The authors design a Slow-Emotion-Fast-Action scheme to encode fast temporal changes of body actions revealed from video frames, as well as subtle details of emotions per video segment, into visual representation. They further augment task-specific visual representation with linguistic facial expression descriptions and introduce a knowledge retriever to evaluate and select the most proper deliverable of LMMs. 

Empirical results how the effectiveness of the proposed method.
strength_weakness: strength_weakness: I appreciate this paper's effort to develop an interpretable approach with world knowledge for emotion detection. The proposed approach  is reasonable and is a good example for further research to instill knowledge into model design, instead of only relying on black-box models for decision making. Therefore, I recommend "Accept" for this paper to advocate the endeavors on this direction.
limitations: limitations: Have the facility descriptions generated by LMMs been evaluated by human experts? It would be more convincing to if the outputs from LMMs can be validated by human experts.
paper_summary: paper_summary: This paper tackles the problem of stress detection from videos. Their system utilizes video frame features and language descriptions extracted via the video frames itself. To generate the natural language descriptions, authors extract representative emotion label $E_i$, one of seven discrete emotion labels, and a corresponding representative face image $F-i$. For $m$ segments within a video, $F_i$ serves as an input to LMM (GPT-4V) with additional context about the Facial Action Coding System (FACS). To mitigate hallucinations, authors sample $k$ different descriptions from LMMs and use an additional retriever $R$ (CLIP-B) to select the descriptions. The final facial description proposed by $R$ serves as input to the BERT-encoder, which yields the final segment-level representations. To obtain video-level linguistic representation, all the segment representations are fed into an LSTM model, and the final aggregated hidden state is extracted. Video features are processed via the Slow-Fast network. The representations extracted for both modalities are fused over which a fully connected predictor/network is applied for stress label classification. The authors conduct experiments on UVSD and the newly proposed RSL dataset.
strength_weakness: strength_weakness: 1. Covers important experiments to justify the modalities, motivation to use the Slow-Fast approach, and effectiveness of LMMs to generate factual descriptions.
2. Well written paper and easy to follow.
limitations: limitations: 1. The proposed method is computationally complex and performs stress detection in a very restrictive environment. The proposed approach cannot scale to in-the-wild settings with multiple people, narration, subject not facing the camera, etc. The restrictive setting does injustice to the potential of all the models necessary to the pipeline (GPT-4, CLIP-Base, BERT-encoder, MTCNN, SCN, Slow-Fast, LSTM, FN-Classifier).
2. For the proposed RSL dataset, claiming all the frames with liars as "stressed" is unfair. "Odd Man Out" TV show had many scenarios where actors laugh, act cool, and show no signs of stress.
3. Need additional details on GPT-4V baseline. Were the 10 frames provided to GPT-4V sampled from the subset used by the author's model? If different, how relevant were the frames for identifying stress from the video?
4. The justification for using a slow-fast network is not sufficient. The network expects the inputs to have different sampling rates, and therefore, modifying those is unfair. An appropriate experiment and comparison would be to replace the slow-fast backbone with any other backbone, such as ResNet50, etc. It would also be interesting to see a pre-trained emotion recognition backbone (ResNet50 pre-trained on FER13, SFEW, etc, datasets). Such pre-trained models will provide richer emotional representations. 
5. Line 572, CLIP is limited to 77 input tokens; it is unclear how authors set the input sentence length to 200, which includes the [CLS] and [SEP] tokens.
paper_summary: paper_summary: This paper studies the task of stress detection based on videos and proposes a three-tier Content-Semantic World Knowledge Framework. The innovation of this framework lies in leveraging the LMMs to enhance the capability of video frames in expressing emotional and linguistic facial features for the task. And it utilizes a knowledge retriever to evaluate and select the most suitable LMMs output representation. Moreover, through the Slow-Emotion-Fast-Action Scheme, it acquires visual representations capable of capturing both the speed of actions and subtle emotional details. Experimental results demonstrate that compared to other methods, this framework achieves superior performance in video-based stress recognition.
strength_weakness: strength_weakness: 1. The article is well-written, clearly presenting the proposed framework and experimental results. The description of the methods is detailed, making it easy to understand the article's structure.
2. The article effectively leverages large-scale language models for application in video pressure detection tasks, yielding favorable results. 
3. The experimental results of this approach are robust compared to other methods and include comprehensive elimination experiments.
limitations: limitations: 1. The method proposed in the article is based on the phenomenon where human emotional changes are slower than actions. Is there existing research that serves as a foundation for this setup? Some emotional changes can also occur rapidly.
2. The facial description selection method mentioned in the article involves additional data annotation and training before the experimental training. This could significantly increase the training cost of the model.
3. The baseline method for experimental comparison is quite outdated. It would be beneficial to incorporate recent research articles for comparison.
4. The workflow of this framework is rather complex. Considering the complexity of the framework, it is advisable to include some experimental analyses.
paper_summary: paper_summary: This paper proposes a video-based stress detection method. The authors design a Slow-Emotion-Fast-Action scheme to encode fast changes of body actions, as well as subtle details of emotions into visual representation. Then, they augment the derived visual representation with linguistic facial expression descriptions by prompting general-purpose Large Multimodal Models (LMMs). Considering the potentially unfaithful descriptions, a knowledge retriever is designed to evaluate and select the most proper deliverable of LMMs. Experiments show state-of-the-art performance on two benchmark datasets.
strength_weakness: strength_weakness: The paper is well-organized and easy to follow.

The idea is natural and technically sound. I like the interpretability of the intermediate results of the model.

Extensive experiments are conducted, verifying most statements in the introduction and method sections. The improvements look solid. I like the insightful discussions in the experiments.
limitations: limitations: I believe this paper is above the borderline, although the following concerns prevent me from considering this paper for direct acceptance.

[INTRODUCTION] Some motivations are not clear.

The authors claim they build a "three-level (content-semantic-world knowledge) framework". I am confused about what semantics and world knowledge correspond to after reading the paper. Do they refer to facial expression descriptions and the knowledge retriever? Could the authors reformulate lines 108-120 to clarify this?

I notice that the authors adopt several pre-trained models, e.g., the facial recognition algorithm MTCNN, Self-Cure Network (SCN) [39] to recognize the emotion of each face image in the form of seven categorical emotions, and GPT-4V to recognize facial expressions with an instruction based on the Facial Action Coding System (FACS). I wonder if the proposed method majorly benefits from different pre-trained models rather than the designed pipeline.

In this case, fine-tuning a pre-trained LMM on the datasets could achieve equivalent performance without the need for the pipeline, taking the predictions of smaller models as inputs in the prompt.

[INTRODUCTION] Some details are missing.

The complex process and lack of details may hinder the reproducibility of the method. I am curious about the network structure of the proposed method, such as the hidden dimensions in the sequence-level encoder. I also wonder if some details are omitted in the Slow-Emotion-Fast-Action Scheme and stress predictor, e.g., the settings of activation functions. I couldn't find the specification of some hyperparameters, e.g., the number of training epochs.

The authors claim they "employ experts to annotate facial action units based on FACS knowledge, and transform them into natural language descriptions as ground truth". More detailed statistics and examples of the annotated results should be provided.

[Experiments] Some experiments are omitted.

I like the design of "Most Representative Emotion Selection & Most Emotional Frame Selection", but the authors omit the ablation study. I wonder about the performance gain brought by the designed selection compared to sampling at fixed intervals or random sampling.

I wonder if the proposed method can only be used for full video recordings, which should contain more than 2^6 frames according to the setting of m in Fig. 4. It might be a rather impractical experimental setup that hinders real-time stress detection. I wonder how the performance varies with different numbers of frames, n.