MDDR:Multi-modal Dual-Attention aggregation for Depression Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:53:40

Abstract: Automated diagnosis of depression is crucial for early detection and timely intervention. Previous research has largely concentrated on visual indicators, often neglecting the value of leveraging a variety of data types. Although some studies have attempted to employ multiple modalities, they typically fall short in investigating the complex dynamics between features from various modalities over time. To address this challenge, we present an innovative Multi-modal Dual-Attention Aggregation Architecture for Depression Recognition (MDDR). This framework capitalizes on multi-modal pre-trained features and introduces two attention aggregation mechanisms: the Feature Alignment and Aggregation (FAA) module and the Sequence Encoding and Aggregation (SEA) module. The FAA module is designed to dynamically evaluate the relevance of multi-modal features for each instance, facilitating a dynamic integration of these features over time. Following this, the SEA module determines the importance of the amalgamated features for each frame, ensuring that aggregation is conducted based on their significance, to extract the most relevant features for accurately diagnosing depression. Moreover, we propose a unique loss calculation method specifically designed for depression assessment, named DRLoss. Our approach, evaluated on the AVEC2013 and AVEC2014 depression audiovisual datasets, achieves unparalleled performance.

Main paper: 
 related work: 
 Considering the challenges faced in existing research on automated depression diagnosis using patient facial videos, we can categorize them into the following types.Multi-modal learning. Recently, Multi-modal methods have made valuable progress in many depression recognition tasks [12, 13, 22] . For instance, Zhang et al. utilized state-of-the-art models to extract Multi-modal (audio and visual) features, then fused these features using a Transformer Encoder, achieving excellent results in multiple emotion analysis tasks [14] . Sun et al. used multilayer perceptrons (MLPs) instead of self-attention to mix across three dimensions (modality, sequence length, features) for tasks like sentiment analysis and depression estimation. [15] . Chao et al. concatenated audio features, facial appearance features, and facial shape features for early exploration of multimodal learning in depression recognition [18] . Chen et al. employed audio, textual, and visual features to design a feature fusion framework both within and across modalities, achieving advanced results in depression recognition [16] . However, the complex relationships between modalities over time have not been effectively explored. In this paper, we designed two Attention Aggregation modules to dynamically fuse features from various modalities over time, achieving a more refined multimodal integration.Spatio-temporal modeling. To fully utilize the spatio-temporal information in patients' videos, researchers employ Dual-stream approaches or recurrent neural networks (RNN, LSTM) to model the spatio-temporal representation of the entire video [18] [19] [20] . For instance, Azher et al. utilized Bi-directional Long Short-Term Memory networks (BiLSTM) to capture the global representation of entire videos for depression recognition [20] . Jazaery et al. employed RNNs to model the local and global spatio-temporal information of continuous facial expressions to identify patients' depressive states [7] . Zhu et al. and Melo et al. propose a depression assessment method based on a two-stream framework, where one stream learns static features in the video through convolutional neural networks, and another stream learns dynamic features, combining results from both streams to diagnose depression [19, 29] . However, these methods focusing solely on the variation of spatial features temporally, overlooking the interrelations between spatial features at different times, represent a significant oversight. In this paper, we employ a self-attention mechanism to capture the interrelations among spatial features at different times, obtaining an enhanced representation that incorporates global relationships and spatiotemporal information.Pre-trained models. Due to the scarcity of publicly available depression video datasets, researchers utilize pre-trained models to extract facial depression features from videos [19] [20] [21] . For instance, Azher et al. used the Inception-ResNet-v2 network, pretrained on the ImageNet dataset, to extract static features from videos for depression assessment [20] . Melo et al. employed a two-stream framework, with both streams utilizing the ResNet-50 network, pretrained on the VGG Face dataset, to capture appearance and motion information from videos for depression assessment [19] . Zhou et al. utilized a ResNet network trained on the CASIA largescale facial database, subsequently fine-tuning it to capture facial depression features in patients [17] . Jazaery et al. and Melo et al. pre-trained the C3D network on the Sports-1M and UCF101 video datasets to learn depression representations in videos [7, 21] . However, these methods use pret-rained data that significantly differ from the depression recognition task, limiting the improvement in model performance. Considering the strong correlation between facial emotional states and depression outcomes, in this paper, we first utilize a pre-trained model, trained on a large-scale facial emotion 
methodology 3.1 overview: 
 Our framework takes a face video containing audio as input. As illustrated in Figure 2 , the Multi-modal features pre-extraction module extracts three modal features from given video: visual features 𝑋 𝑣 , acoustic features 𝑋 𝑎 and Head movement features 𝑋 𝑝 . The Features Alignment and Aggregation (FAA) module aligns and stacks these features into 𝑋 ∈ R 𝐿×𝑀 ×𝐷 , where L represents the length of the sequence, M represents the number of modalities, and D represents the feature dimension. The 𝑋 is processed by the Intermodality Attention aggregation network to produce 𝐹 ∈ R 𝐿×𝐷 . Following this, the Sequence Encoding and Aggregation (SEA) module uses a self-attention-based Encoder to process 𝐹 and produce 𝑂 ∈ R 𝐿×𝐷 , followed by the Temporal Attention aggregation network that yields an aggregated representation 𝐸 ∈ R 𝐷 . Finally, 𝐸 is processed by a classification network based on a Multilayer Perceptrons (MLPs) to produce the final depression assessment results, which is a depression score ranging from 0 to 63.To derive more generalized features indicative of depression, we utilize pre-trained models to extract deep representations from each modality within raw audiovisual data. This paper focuses on extracting three types of features: visual, acoustic, and head movement features, each demonstrating unique characteristics associated with depression. For the extraction of head movement and acoustic features, we employ OpenFace [34] and HuBERT [35] , respectively. Specifically, considering the strong correlation between depression-related visual features and facial expressions, coupled with the challenge conventional image pre-trained models (such as those based on ImageNet [44] ) face in extracting effective emotional features, we have developed a facial feature extraction model. This model, which is pre-trained using ResNet-50 [47] on the extensive emotion recognition dataset AffectNet [43] , achieves a generalized deep representation of facial expressions.Due to variations in the methods used for pre-extracting features across different modalities, there can be discrepancies in sequence lengths and feature dimensions. These differences pose challenges for the subsequent fusion of information among modalities. To align the acoustic and head movement features with the visual features, we utilize two alignment networks that process these two types of features separately. Both alignment networks employ a 1D-CNN as their backbone, enabling them to effectively capture the local features within the input sequence. After processing, we obtained three modalities of features with aligned dimensions, including visual features 𝑋 𝑣 ∈ R 𝐿×𝐷 , acoustic features 𝑋 𝑎 ∈ R 𝐿×𝐷 and head movement features 𝑋 𝑝 ∈ R 𝐿×𝐷 .In diagnosing depression, clinicians dynamically focus on patients' depressive characteristics at different modalities over time and considering them comprehensively based on the context. To emulate this complex diagnostic approach, we utilize an attention mechanism to adaptively determine the importance of each modality at each timestep, leveraging these weights to compile a fused multimodal representation for each moment. Referred to as Intermodality Attention aggregation network, as illustrated in Figure 3 (a). First, we stack these aligned features from different modalities, denoted as 𝑋 ∈ R 𝐿×𝑀 ×𝐷 . This step is primarily for the ease of subsequent unified data processing, with the attention weights for each modality learned separately. Next, we utilize a multi-layer fully connected network with shared weights to learn the attention weights of each modality at different times, which are then normalized using a softmax function. Finally, we multiply the attention weights of each modality by their respective feature vectors and sum these products, obtaining a fused feature that incorporates the significance of each modality over all times, defined as 𝐹 ∈ R 𝐿×𝐷 . This method involves a linear combination of each feature based on attention weights, proves to be much faster and more space-efficient in practice. The Attention aggregation calculation formula is as shown in Equation 1, where 𝑊 𝑚 represents a trainable parameter,EQUATIONThe complex interrelations between depressive characteristics at different times may contain subtle clues of depression. To capture these clues, we utilize a self-attention-based Encoder module to encode the input sequence. The Encoder consists of multiple Blocks, each comprised of a self-attention network and a fully connected layer. Within a Block, the process begins with a Multi-Head Attention layer, followed by a residual connection and Layer Normalization, then fed into a Feed Forward network, with multiple Blocks repeating this process to finally output a sequence. Additionally, the input vectors are not fed directly into the Encoder but are first augmented with Positional Encoding, which is crucial for processing time-series data. The self-attention calculation formula is as shown in Equation 2, where 𝑄, 𝐾, and 𝑉 represent the query, key, and value vectors, respectively. 𝑂 ∈ R 𝐿×𝐷 denotes the output obtained from self-attention computations that encompasses global relationship information.EQUATIONWe implemented a method similar to the Inter-modality attention aggregation described in Section 3.3 to aggregate temporal information, referred to as Temporal attention Aggregation network, as illustrated in Figure 3 In clinical practice, depression scores obtained through interviews and scales are usually range values, and doctors categorize scores within the same range into one category [39] . Therefore, we designed a loss calculation method suitable for depression assessment (DR), which introduces a range discrimination hyperparameter 𝛿 aimed at reducing the model's sensitivity to outliers and ensuring convergence of prediction results within a specified range. Specifically, when the predicted value and the target value are in the same category and the prediction error is less than 𝛿 (𝑙𝑜𝑠𝑠 ≤ 𝛿 ), the model's predictions can be considered consistent with clinical diagnoses. In this case, we multiply the loss by a very small weight 𝑤 to minimize the gradient impact from the current sample. In all other cases, the Huber loss is applied to calculate the loss. Huber loss is a loss calculation method that combines the advantages of Mean Square Error (MSE) and Mean Absolute Error (MAE), offering robustness against outliers. [40] .EQUATIONThe DR loss calculation formula as shown in Equation 3, where 𝑦 represents the target value, 𝑓 (𝑥) represents the predicted value. 𝐶 denotes a specific depression category, 𝐿 𝐻 is the Huber loss function.𝑤 is a minimal weight value, set here as 0.1. The hyperparameter 𝛿 can be selected through cross-validation for optimal value.  
experimental setup 4.1 dataset: 
 We conducted extensive experiments on the AVEC2013 and AVEC2014 audiovisual depression datasets [30, 31] . These are the currently known two public datasets containing original video data of patients with depression, providing conditions for our research to capture depressive features from raw audiovisual data. The AVEC2013 depression dataset comprises 150 videos from 82 participants, with an average video length of about 25 minutes. Each video documents participants performing a series of tasks, including sustained vowel phonation, speaking loudly, counting from 1 to 10, among others. The AVEC 2014 depression dataset contains 300 recorded videos from 84 participants, with an average video length of about 2 minutes. It encompasses two types of recordings: one in which participants loudly read excerpts from the German fable "The North Wind and the Sun" and another where they answer routine questions.Each video is labeled with an assessment based on the Beck Depression Inventory (BDI-II), representing its level of depression. The BDI-II is the most widely used self-rating questionnaire related to depression, extensively employed for detecting depressive symptoms in the general population and assessing the severity of depression in psychiatric patients [38] . It consists of 21 questions measuring aspects of depression, including typical symptoms such as sadness, guilt, suicidal thoughts, and lack of interest. BDI-II scores range from minimal (0-13), to mild (14) (15) (16) (17) (18) (19) , moderate (20) (21) (22) (23) (24) (25) (26) (27) (28) , and severe (29-63).The model performance is evaluated using Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), both of which are used to measure the degree of deviation between predicted values and target values. RMSE represents the square root of the average squared differences between predicted values and target values. MAE represents the average absolute deviation between predicted values and target values. The definitions of MAE and RMSE are as follows:EQUATION)Where 𝑁 is the total number of samples, x𝑖 is the predicted value for the 𝑗-th sample, and 𝑥 𝑖 is the true value for the 𝑖-th sample.Considering the redundancy in video data, we extracted one frame out of every 100 for AVEC 2013 and four frames out of every 100 for AVEC 2014, cropping the images to a size of 224×224 pixels. We utilize facial emotion pre-trained model to extract visual features from facial images, resulting in 2048-dimensional deep representations. We use the audio processing library librosa to extract audio signals from videos with a sampling rate of 16000 Hz. We utilize speech pre-trained model to extract acoustic features, resulting in 1024-dimensional deep representation. Using the open-source tool OpenFace, we extract head pose and gaze features from the face images after frame extraction, obtaining a feature dimension of 12.For missing sequence features, we use zero-padding to ensure that each sample sequence length is consistent. We employed the PyTorch deep learning framework for all experiments, which were conducted on AMD Ryzen 9 5900HX with GeForce RTX 3080 hardware. In our method, we adopted the Stochastic Gradient Descent (SGD) algorithm with a momentum of 0.9 and a weight decay of 0.0005. The initial learning rate was set to 0.001, employing learning rate decay (the learning rate is multiplied by 0.1 every 50 training rounds). The batch size was set to 16. We utilized DRLoss as the loss function, with hyperparameters set to 2. Data normalization was performed using LayerNorm, with ReLu as the activation function and a dropout rate of 0.1. The AL-CNN uses three 1D convolutional layers for acoustic features with strides (5, 5, 2), kernels (10, 5, 3), and filters (128, 256, 512), and a single layer for head movement features with a stride of 1, a kernel size of 1, and 512 filters. The hidden dimension of the sequence Encoder module is 2048, the number of heads in the multi head attention model is 8, and the number of encoder layers is 6. The classification network employs two fully connected layers, with a hidden layer dimension of 128.We introduce the baselines compared in our experiments as follows, including common base classifiers and novel deep learning methods.LPQ [32, 33] : This approach employs artificial features such as Local Phase Quantization (LPQ) to extract features of patients' facial expressions. However, this method is susceptible to changes in the external environment, can only represent spatial information, and fails to capture dynamic information.Two-stream [19, 29] : This approach utilizes convolutional neural networks to extract static features from video frames and employs optical flow or temporal pooling methods to capture the dynamic features of the video. Finally, the results from both streams are aggregated to determine the ultimate depression outcome.C3D [7, 21, 23] : 3D convolutional neural network can simultaneously extract spatial and temporal features from 3-dimensional data, making them more suitable for video data modeling. However, the number of parameters in 3D CNN is much larger than in 2D CNN, and due to hardware, memory, and runtime limitations, they cannot model excessively long video sequences at once.RNN [18, 20] : This approach uses 2D CNN to extract static features from video frames or employs 3D CNN to extract short-term spatio-temporal features from video clips. Then, recurrent neural networks (RNN, LSTM) are used to further learn global spatiotemporal information from sequential features, facilitating depression assessment.Attention [17] : The authors utilize the ResNet-50 network to learn frame-level features from the video, outputting a set of feature vectors. Subsequently, two cascaded attention modules are employed to adaptively learn the weights of different facial images, generating an aggregated representation. Finally, a regression layer uses the aggregated features to produce a depression score.AUs [28] : The authors use Facial Action Units (AUs) as the underlying features for each frame and employ spectral heatmaps and spectral vectors to characterize multi-scale spatio-temporal information in videos. Subsequently, the constructed spectral representations are fed into Convolutional Neural Network (CNN) and Artificial Neural Network (ANN) for depression analysis. 
experimental results and analysis 5.1 comparative experimental results: 
 We compared our proposed method with the latest results on the AVEC2013 and AVEC2014 audiovisual depression datasets, as shown in Table 1 . Our method outperforms all schemes in the table, achieving state-of-the-art results. In AVEC2013, our method improved the MAE results by 3.8% compared to the previous stateof-the-art approaches. In AVEC2014, both RMSE and MAE results saw improvements of 3.5% and 0.4%, respectively, against prior state-of-the-art methods. Specifically, 1) Methods using deep neural networks surpassed those relying on manual features (LPQ) [30] [31] [32] [33] . 2) Our approach exceeded the performance of all convolutional neural network methods [8, 21] . Such methods focusing solely on the spatial features and overlooking the temporal features, which impacts model accuracy. 3) Our method also outperformed both Dual-stream and recurrent neural network approaches [7, 20, 29] . Such methods primarily focuses on the temporal changes in facial spatial features, and the interrelations between spatial features at different times have not been adequately addressed, resulting in limited model effectiveness. 4) Our method surpassed those using other pre-trained models [17, 19, 20] . Interestingly, although both [19] and [20] employed Dual-stream approaches and used pre-trained models to extract facial features, Melo et al. utilized a face recognition pre-trained model that is more closely related to the task of depression recognition, yielding better results. Notably, we found that attention-based methods are effective [17, 25] . For example, [25] utilized two attention modules to capture facial depression characteristics in patients, achieving commendable depression detection results even without the use of pre-trained models. Furthermore, Song et al. employed additional low-dimensional features related to emotions, such as Facial Action Units (AUs) [28] . This approach also yielded favorable results, providing new insights for our subsequent research.To further validate the effectiveness of each module we designed, we conducted extensive ablation experiments.Pre-trained models. To assess the effectiveness of facial emotion pre-trained models, we conducted experiments under three conditions: using models pre-trained on AffectNet [43] , using models pre-trained on ImageNet [44] , and without using any pre-trained models, as shown in To verify the effectiveness of our proposed DR loss, experiments were conducted to compare depression assessment results using different loss calculation methods. According to the classification method in Section 4.1 using the BDI-II, depression severity is categorized into four levels: minimal (white, circle), mild (green, triangle), moderate (yellow, square), and severe (blue, diamond) [38] . Red dots represent outliers, indicating data points that do not fall into the same category as their actual depression level. All true values and predicted values were plotted on one graph, as illustrated in Figure 4 . The results show that the number of outliers with the Huber loss method was 47, while with the DR loss method, it was 43. Our loss calculation method resulted in fewer outliers and depression scores within the same category being closer together, which also aligns more closely with clinical diagnostic outcomes in real-world scenarios. 
conclusion: 
 In this paper, We propose a novel Multi-modal Dual-Attention aggregation architecture for depression recognition (MDDR). It extracts multiple modalities features from patient video as inputs and employs two Attention aggregation modules to capture the complex dynamics between features from various modalities over time, facilitating accurate and efficient automatic depression assessment. Additionally, we propose a loss calculation method (DR) suitable for depression assessment. Our method was evaluated on two standard depression audiovisual datasets (AVEC2013 and AVEC2014), demonstrating its effectiveness and superiority.Due to the scarcity of data, the method of extracting depression features from video data using complex deep neural networks is limited. In future work, we will focus on mining more effective and generalized depression features from small sample data. Furthermore, we will collect more data from depression patients and establish a multimodal database of depression patients (including EEG, physiological indicators, etc.) to enhance our research. 
: 
 Figure 1 : Our approach employs a Multi-modal framework that effectively capture the complex dynamics between features from various modalities over time, aimed at achieving accurate and efficient automatic depression assessment.due to factors such as social stigma and a significant imbalance between the number of doctors and patients, over 50 percent of individuals with depression do not receive relevant mental health services, missing the optimal time for intervention [2, 3] . In fact, diagnosing depression is a highly complex and time-consuming task.Clinicians must comprehensively consider multiple diagnostic outcomes, including patient interviews and depression test scales, to assess the symptoms and severity of a patient's depression [4] . Nonetheless, such assessments, which heavily rely on the subjective analysis of clinicians, are prone to errors and inefficiency.In recent years, significant advancements have been made in the research on automatic depression diagnosis based on patient facial videos, thanks to the rapid development of computer vision and deep learning technologies. Initial research primarily focused on analyzing video data of patients, utilizing basic models such as convolutional neural networks (CNNs) to detect facial features indicative of depression [7, 8, 17, 21, 23] . However, subsequent findings have revealed that the indicators of depression are not confined to visual cues alone but also include the patient's speech patterns and head movements [5, 6, 45, 46] . These elements are vital as they provide significant insights into an individual's emotional and cognitive states, thereby presenting a more comprehensive set of indicators for depression.Recent efforts have sought to incorporate multiple modalities, analyzing features within individual modalities before integrating them [16, 18, 26, 27, 42] . This approach, however, tends to overlook the nuanced diagnostics practiced in clinical settings, where clinicians assess patients' conditions through a combination of verbal and non-verbal cues and dynamically adjust their focus based on the context. This situation highlights the need for methodologies that not only merge multimodal features but also adaptively prioritize their importance over time, mirroring the variable emphasis clinicians place on different aspects of patient behavior. To date, existing research has yet to fully embrace this approach, marking a significant gap in the development of depression recognition systems that effectively replicate clinical diagnostic processes.In this paper, we aim to bridge the identified gap by proposing a model that dynamically weights multimodal features, offering a more nuanced and clinically aligned method for depression recognition, as illustrated in Figure 1 . We introduce a novel Multi-modal Dual-Attention Aggregation framework for this purpose, referred to as MDDR. To enhance the generalizability of our model, we have employed pre-trained models to extract features related to facial expressions, acoustic properties, and head movements separately.To explore the complex dynamics between features from various modalities over time, we introduce two attention-based modules: the Feature Alignment and Aggregation (FAA) module and the Sequence Encoding and Aggregation (SEA) module. The FAA module utilizes an attention mechanism to adaptively determine the importance of each modality at each timestep, leveraging these weights to compile a fused multimodal representation for each moment. The SEA module employs a self-attention mechanism to identify interrelations among fused features at different moments, generating an aggregated representation that reflects the significance of these features throughout the entire sequence.Furthermore, we recognize that the actual diagnosis of depression more accurately resembles an interval prediction challenge. To address this, we propose a novel loss function specifically designed for depression recognition, termed the Depression Recognition (DR) loss. The DR loss introduces a range discrimination hyperparameter, aiming to diminish the model's sensitivity to outliers and ensure the convergence of prediction results within a specified range.Our contributions can be summarized as follows:• We propose a novel Multi-modal Dual-Attention Aggregation architecture for Depression Recognition (MDDR), offering a more nuanced and clinically aligned approach to diagnosing depression. • We propose two Attention Aggregation modules (FAA and SEA), designed for the dynamic fusion of features from various modalities over time. • We have introduced a novel loss function specifically designed for depression recognition, named Depression Recognition (DR) loss. • We evaluated our approach on two depression audiovisual datasets: AVEC2013 and AVEC2014, both achieving stateof-the-art results.The rest of this paper is organized as follows. In Section 2, we briefly summarizes the related works for depression. In Section 3, we detail our proposed method (MDDR). Section 4 and Section 5 illustrate the experimental setup, along with the evaluate and analyze results, respectively. Finally, in Section 6, we discuss the conclusions of this paper and future work. 
 
 Appendix:  


Decision: this paper investigates the problem of depression recognition and introduces a new approach called Multi-modal Dual-Attention Aggregation for Depression Recognition (MDDR). to consider the facial expressions, acoustic properties, and head movements jointly, this paper introduces two attention mechanisms, and achieves competitive performance. 

initially, the paper received ratings of WR, WA, and BA. the reviewers commended the paper for the dual attention module, the DR loss, and the overall writing. they also voiced their concerns over the missing comparison against SOTAs, ablation and variant study, clarity issues, and need for discussion on topics including visual features and the two types of attentions. during rebuttal, the authors mostly addressed these concerns and the final ratings improved to WA, WA, and WA.

since the AC was not able to finish the meta-review in time, the PC stepped in and went through all the reviews and rebuttal. after careful consideration, the PC recommends Accept.

Reviews: review: paper_summary: paper_summary: The paper presents a novel architecture called Multi-modal Dual-Attention Aggregation for Depression Recognition (MDDR), designed to automatically diagnose depression by analyzing multimodal data such as audio and visual cues from patients. The MDDR framework utilizes pre-trained models to extract features related to facial expressions, acoustic properties, and head movements. It introduces two attention aggregation mechanisms: (1) Feature Alignment and Aggregation (FAA) Module: This module dynamically evaluates the relevance of multi-modal features for each instance, facilitating the dynamic integration of these features over time. (2) Sequence Encoding and Aggregation (SEA) Module: This module determines the importance of the combined features for each frame, ensuring that the aggregation is conducted based on their significance to extract the most relevant features for depression diagnosis. Additionally, the paper proposes a unique loss calculation method called Depression Recognition Loss (DRLoss), which is designed specifically for depression assessment.
strength_weakness: strength_weakness: The MDDR framework introduces a novel approach to depression recognition by utilizing a dual-attention mechanism that dynamically evaluates and integrates multimodal features over time. This work describes a technically sound methodology, with clear explanations of the pre-trained models used for feature extraction, the alignment and aggregation of multimodal features, and the sequence encoding process. The introduction of the Depression Recognition Loss (DRLoss) is also a technically appropriate solution to address the specific challenges of depression assessment.
limitations: limitations: 1. In the experimental design, the baseline comparison object selected by the authors is missing some SOTA work. such as: \
(1) de Melo W C, Granger E, Lopez M B. MDN: A deep maximization-differentiation network for spatio-temporal depression detection [J]. IEEE Transactions on Affective Computing, 2021, 14(1): 578-590. \
(2) Niu M, Tao J, Liu B, et al. Multimodal spatiotemporal representation for automatic depression level detection [J] IEEE Transactions on Affective Computing, 2020, 14(1): 294-307.\
(3) De Melo W. C., Granger E., and Hadid A. A deep multiscale spatiotemporal network for assessing depression from facial dynamics [J]. IEEE Transactions on Affective Computing, 2020, 13(3): 1581-1592.\
Based on the results of these references, this work cannot be concluded to achieve the level of sota.
2. In the ablation experiments, the authors only set the comparison using single-modal features and three-modal features. The case of any two combinations should also be taken into account.
3. FAN and SAN in Table 4 appear for the first time and should be explained with abbreviations. Or does the author mean FAA and SEA?
4. The authors compared the effect of extracting facial expressions from models pre-trained on different datasets on the final results but did not conduct similar experiments for the other two modalities. I think authors should consider comparing the impact of different feature extraction networks on the final results of this framework.
paper_summary: paper_summary: Objective: In this paper, the authors propose an innovative multimodal dual aggregation network for depression detection. The authors make use of multimodal pretraining features and propose two attention networks: the feature alignment and aggregation (FAA) module and a Sequence Encoding and Aggregation (SEA) module. Both of these modules are designed to evaluate the relevance of multimodal features for each instance facilitating dynamic integration of these features and determining the significance of each of these features. The authors also propose DR Loss suitable for depression assessment and evaluate their proposed methodology on AVEC 2013 and AVEC 2014 datsets. 

The authors use three different kinds of features – visual, acoustic, and head movement features extracted from pre-trained convolutional models, OpenFace and HuBERT respectively.
strength_weakness: strength_weakness: The paper is well written and the architecture is neatly explained. 

Main contributions: dual attention aggregation network for alignment of various multimodal features, and determine weights for the appropriate fusion of modalities; propose the use of DR loss.
limitations: limitations: 1.	Over a decade, OpenFace was used to extract visual features. And they were found to be robust for the task. It is not clear why pre trained models based on conv2D were employed specifically to derive visual features. 
2.	Head movement features – head movement is also a visual feature. Why there is a separate stream for head movement? Other visual features such as gaze are not included. If separate attention networks are considered, architecture becomes very complicated. 
3.	Authors use a simple method to handle missing modality. Better strategies could have been employed. 
4.	Why textual features are not considered? In AVEC 2014, participants answer casual questions. According to [1], [2], [3] and many previous methods, textual features proved to be the most significant modality among others. 
5.	Two datasets considered here are very different. The average duration for AVEC 2013 is 23 min while for AVEC 2014 is 2min. Is there any difference in the performance because multimodal alignment is challenging in AVEC 2013. 
6.	Evaluation of the proposed architecture could have been performed on multiple feature sets. 
References:
1.	Multimodal depression detection using task-oriented transformer-based embedding 
2.	Multi-level attention network using text, audio and video for depression prediction
3.	The Verbal and Non-Verbal Signals of Depression–Combining Acoustics, Text and Visuals for Estimating Depression Level
paper_summary: paper_summary: This paper designed two attention aggregation modules to consider the multi-modal feature and the temporal dynamics for depression detection.  The study presents a clear and deep background literature and the methodology proposed is clearly stated.
strength_weakness: strength_weakness: 1. This paper depends on pre-trained models to extract features that overcome the overfitting problem.
2. An FAA module is designed to align the multi-modal feature and an SEA module is designed to capture the temporal dynamics.
3. The paper is well-written and presented clearly. The proposed method can be more accurate in two datasets.
limitations: limitations: 1. More details about the pre-trained model used for uni-modal feature extraction can be provided.
2. How to use 1-D CNN for feature alignment if there are discrepancies in sequence lengths?
3. Why the attention used in the FAA and in the SEA are two different types? Will it be better to use the same attention mechanism or worse?
4. There may be a mistake in Table 4 about the name of the methods.
5. The experimental results show that the proposed DR loss can only bring 4 fewer outliers. What about the MAE and RMSE? In addition, the effect of the hyper-parameter $\delta$ and $w$ in the DR loss should be verified.