Year: 2021

Abstract:
Graph neural networks (GNNs) are widely used in regression and classification problems applied to text, in areas such as sentiment analysis and medical decisionmaking processes. We propose a novel form for node attributes within a GNN based model that captures node-specific embeddings for every word in the vocabulary. This provides a global representation at each node, coupled with node-level updates according to associations among words in a transcript. We demonstrate the efficacy of the approach by augmenting the accuracy of measuring major depressive disorder (MDD). Prior research has sought to make a diagnostic prediction of depression levels from patient data using several modalities, including audio, video, and text. On the DAIC-WOZ benchmark, our method outperforms state-of-art methods by a substantial margin, including those using multiple modalities. Moreover, we also evaluate the performance of our novel model on a Twitter sentiment dataset. We show that our model outperforms a general GNN model by leveraging our novel 2-D node attributes. The performance of our work demonstrates the generality of the proposed method.

heading: 1 INTRODUCTION
text: Nowadays, deep learning techniques have been frequently used in sentiment analysis (Tan et al., 2013; Mukhtar & Khan, 2018; Liao et al., 2021), particularly in movie recommendations and product reviews and ratings. In the healthcare domain, sentiment analysis with deep learning provides many benefits such as utilizing medical information to achieve the best result to increase healthcare quality (Abirami & Askarunisa, 2017; Bi et al., 2020). We focus our work on applications related to mental health such as using deep learning to make a diagnostic prediction.
There is a pressing need to find a convenient and automated method to assess depression severity. Worldwide, more than 300 million people are suffering depression (Organization et al., 2017). Apart from the high prevalence of Major Depressive Disorder, overall 85% of depressed individuals are underdiagnosed (Falagas et al., 2007). Research has shown that about 30% of patients suffering from an episode of major depression do not seek treatment, with only 10% of them being adequately treated (Falagas et al., 2007). We validate our approach based on its application to the diagnosis of MDD. We demonstrate that our approach can result in more accurate predictions of the severity of depression, which can enhance automated health care decision making.
Medical research (Tsakalidis et al., 2018) shows that text sentiment analysis methods can be effective in making inferences about people’s mental states, such as MDD. For example, depressed people use more first person pronouns than non-depressed people because they are more focused on themselves and less connected with others. These may not be reflected in a person’s visual appearance, i.e., visual behaviors. Visual data alone will therefore be insufficient for capturing depressive symptoms that are only manifested in a person’s verbal utterances.
Mental disorders are difficult to detect. In practice, clinicians first measure the severity of depressive symptoms in patients and then identify depression in them. During in-person clinical interviews, clinicians widely use a structured Patient Health Questionnaire (PHQ) — a clinically validated tool determining the severity of depressive symptoms across several personal dimensions (Kroenke & Spitzer, 2002; Arseniev-Koehler et al., 2018).
Inspired by the observation of the provided data set of such interviews, we formulated the hypothesis that the context of words in a transcript can be used to generate PHQ scores (Kroenke et al., 2009). This PHQ metric for depressive disorder forms a context constructed with eight items, such as “sleep problem”, “anxiety problem”, “fatigue problem”, “depression problem”, and “no motivation or interest in things” (Arseniev-Koehler et al., 2018). We observed that each text (see figure 1) covers information relating to at least one of these eight topics. This motivates use of a deep learning model that represents these eight PHQ topics, thereby introducing an inductive bias into the learning process. Thus the contextual information about these eight PHQ topics should be encoded to determine depression states.
We motivate generating graph-level representations for transcripts (as inputs) to encode contextual information from these transcripts (we assume a transcript encodes facts representing depressive symptoms). To achieve this, we propose a novel form for node attributes within a GNN based model that captures node-specific embeddings for every word in the vocabulary. The representations of each word is shared globally and can be updated according to associations among words in a transcript. We summarize the representations of all the words in the transcript to predict depression states.
Intuitively, the generated node-level embeddings maintain records linking certain existing facts that indicate the known symptoms of a subject. When subsequent facts are discovered which indicate additional depressive symptoms, records will be updated by aggregating both old and new facts. In other words, those records will be updated via a message passing mechanism over the transcript context until ideally all major depressive features (characterizing depressive symptoms) have been discovered.
In this way, we can represent the most relevant contextual information which is universal across all transcripts – possibly involving the context of eight PHQ topics we discussed above. Using graph structures to capture context-level features is an innovative pathway which we hypothesise can be used to measure different levels of depression. In our experiments, our approach realized a good generalization on a limited, incomplete, and unevenly distributed dataset (see figure 3a).

heading: 1.1 RELATED WORK
text: Research in measuring the severity of depressive symptoms aims to train a regression model to predict depression scores (Valstar et al., 2016; Alhanai et al., 2018; Tsakalidis et al., 2018). Some prior work (Ringeval et al., 2017; Alhanai et al., 2018; Haque et al., 2018) applies a sequencelevel deep learning model to capture implicit depressive signals. Such models, in general, use a multi-modal sentence embedding, rather than a mapping of a whole interview, to predict a PHQ score. Other prior work (Valstar et al., 2016; Cummins et al., 2017) implements statistical functions (e.g., max, min) on short-term features over an entire interview, but this may fail to preserve useful temporal information across an entire interview, such as some short-term signs of regret, anxiety, etc.
Current machine learning methods (Valstar et al., 2016; Alhanai et al., 2018; Song et al., 2018; Du et al., 2019) concentrate on learning features from multi-modalities –– as determinants for estimating the severity of depressive disorders. The method (Lin et al., 2020; Solieman & Pustozerov, 2021)
applies a multi-modal fusion network that summarizes all depressive features from both text and audio. The work (Lam et al., 2019) leverages a 1D CNN model to extract text-level embeddings from the text. The methods (Ray et al., 2019; Mallol-Ragolta et al., 2019; Zhang et al., 2020; Solieman & Pustozerov, 2021) utilize deep learning models with pretrained global word embeddings to extract contextual features from text. They only apply their work to the depression detection task.
In principle, we should be able to improve performance further using all available modalities, however feature-fusion method may lead to learn a wrong representation encoding inter-subject variability unrelated to depression (Williamson et al., 2016). The feature-fusion approach to learning from small data may cause poor generalization: with a small sample size, the number of features should also be small to avoid the problems of dimensionality and overfitting. However, given that the dimensions of audio and video features are very large, sparse parametrization needs to be taken into account when training machine learning models for learning a joint multi-modal feature vector (Williamson et al., 2016).

heading: 2 METHOD
text: We generate a node-level embedding matrix for each word of a transcript. These node attributes represent an underlying relationship between the current word and every other word in the transcript. In this way, a graph neural network (GNN) schematizes underlying associations among words via their internal representations. We propose to aggregate their internal representations to encode information from the context of a transcript generalizing all major depressive symptoms1.
Building a Text Graph: For a given transcript, we build a text graph. We regard all unique words appearing in a transcript as the nodes of the graph. Each edge starts from a word in the transcript and ends with a word within a fixed window either side of the word.
Let G = (V,E) denote a text level graph. V is a set of nodes representing all the unique words in a given text, and E is a set of undirected edges between pairs of these nodes, each represented by a set of the two nodes at either end of the edge. We build edges by using a fixed size sliding window in a transcript to collect co-occurrence statistics. Each node has an attribute which is a matrix. To emphasise this representational structure of the attribute matrix, we refer to it as a ‘schema’ (Dozois & Beck, 2008; Hammen & Zupan, 1984; Rudolph et al., 1997; Soygüt & Savaşir, 2001) Ui ∈ Rn×d. The jth row of Ui is a vector of length d containing the representation that node vi has of vj ; and n denotes the total number of unique words (the vocabulary size) in a corpus.

heading: 2.1 SCHEMAS
text: We generate a schema for each word node which performs a role of recording a global context. This global context retains information from interactions between the current word and every other word. In this way, each word node maintains “a dynamic record” (in the form of a schema) of the context from the given transcript. The schemas are progressively updated by a GNN model. This resulting model produces final embeddings of the words in a transcript in relation to all words in the vocabulary, including the other words that make up the current training transcript.
Schemas preserve structures that represent relationships between the identifier word and every other word. Thus we can exploit these schemas to capture context-level features in an explicit way by learning a GNN model. Our innovation is to represent word proximity through the graph structure and co-occurrence within the same transcript within the schema at each node.

heading: 2.2 SCHEMA-BASED GRAPH NEURAL NETWORK (SGNN)
text: We use multiple passes (layers) of the message passing mechanism (MPM) (Gilmer et al., 2017; Xu et al., 2018) to update the schema at each node of the text graph.
1our code links to: https://github.com/Clio-dl/Using-SGNN-for-Depression-Estimate. git/
Initialization: We first initialise an n × d1 matrix (in our experiments, we set d1 = 32) as the schema U (1)i at each node vi ∈ V . The schema is all zeros apart from the row corresponding to the word associated with this node, which is a random d-dimensional vector using a linear transformation.
Message passing layer: In this work, the operation of message passing is split into two steps to update the schema at each node.
We first modify the schema at each node vi:
Ûi (k) = U (k) i W (k) 1 +
1 n 11TU (k)i W (k) 2 + 1 n 1i1TU (k) i W (k) 3 (1)
where 1 ∈ Rn is a vector of ones. The first term updates each row independently. The second term operates on the sum of the columns, replicated in each row. The third term operates on the row corresponding to the current word, replicated in each row. These terms are a subset of equivariant linear functions which are computed by (Maron et al., 2018).
Second we compute the message function, which is defined as:
M (k) ( Ûi (k) , Ûj (k) ) = Ûj (k) +RELU ([ Ûi (k) ||Ûj (k)] W (k) 4 ) W (k) 5 (2)
where || denotes concatenation along the second axis. ( W (k) m ) 1≤m≤5 are learnable parameters. This is essentially a two-layer neural network.
In the next step, each node’s schema is updated as:
Ui (k+1) = ∑ j∈N(i) M (k) ( Ûi (k) , Ûj (k) ) ∈ Rn×dk+1 (3)
We apply equation (3) as the sum aggregator over the k-th layer of the SGNN.
After all K message-passing layers have been applied, the schemas are pooled to get representations of all nodes in the graph. We apply an elementwise max-pooling operater to extract node features. We use the READOUT (Ying et al., 2018; Xu et al., 2018) function which aggregates node features by summing them together:
hG = ∑ vi∈V U (K) i (4)
We use hG to predict a PHQ score for each transcript. In our experiments, we apply a 2-layer multi-layer perceptron (MLP).

heading: 2.3 DATASETS
text: We performed experiments on two datasets, DAIC-WOZ and a twitter sentiment dataset. Whilst our principal interest is in depression prediction, we included the second dataset to show the generality of the method we present here.
• The DAIC-WOZ dataset contains video-based facial actions, audio and the conversation transcribed to text for each participant. Both the 6th and 7th International Audio/Video Emotion Challenge (AVEC) (Valstar et al., 2016) used this dataset. We utilized only the text transcripts of all 142 individuals within the dataset. 43 out of 142 subjects (30%) were labeled as depressed. The provided dataset has been split into a training set having 107 patients and a development set containing 35 patients. In line with prior work (Valstar et al., 2016; Alhanai et al., 2018; Cummins et al., 2017; Haque et al., 2018) and to ensure comparable results, we test on the “development set” from the original competitions (Valstar et al., 2016), since the actual test set is not in the public domain.
Privacy: This data2 does not contain protected health information(PHI). Personal names, specific dates and locations were removed from the audio recording and transcription by the dataset curators.
• Borderlands Sentiment Twitter3dataset includes 74682 tweets labeled as positive (28%), negative (30%) and neutral (42%) respectively. Tweets range in length from 2 to 126 words with an average length of 25.3 words per tweet. We extracted 1% of full data for training.

heading: 3 EXPERIMENTS
text: In this section, we show the results of our method on two different tasks of depression prediction and twitter sentiment classification.
2https://dcapswoz.ict.usc.edu/ last accessed on 18/09/20 3https://www.kaggle.com/cameronwatts/bag-of-words-sentiment-analysis-with-keras/
data?scriptVersionId=78350767last accessed on 11/11/21

heading: 3.1 THE SEVERITY OF DEPRESSIVE SYMPTOMS PREDICTION
text: In this experiment, we predict the PHQ score for each participant. The loss objective is mean absolute error (MAE) for regression. We evaluate our SGNN model on DAIC-WOZ benchmark and we then compare our method to the state-of-art works centering on feature fusion learning algorithms.
We compare our method to prior work on measuring depressive symptom severity. The performance of our method and eleven other methods, including the state of the art method, is set out in Table 1. It is noted that results of some models are directly taken from their original papers. Our method outperforms all other methods, despite using only the textual modality.
We found that our model performs better than the standard GNN (Gilmer et al., 2017). The GNN without our schemas utilizes node representations which are initialized by pre-trained 300- dimensional GloVe word embeddings (Mikolov et al., 2013). However, our method constructs schemas which are initialized with random vectors. Moreover the standard GNN propagates node features in terms of vectors while our model uses 2-D node attributes. This change increases the expressive power of the MPM, resulting in better results.
The machine learning algorithms of (Valstar et al., 2016; Williamson et al., 2016) perform modeling statistics using handcrafted features derived from audio, text and (or) visual inputs. However, our method uses just raw text.
We also note that our model performs better than other prior work (Alhanai et al., 2018; Song et al., 2018; Haque et al., 2018; Du et al., 2019; Lin et al., 2020; Ray et al., 2019). That is likely due to the difference of representation learning. Their work uses a multi-modal sentence-level embedding to predict a PHQ score while our model is trained to learn a graph-level embedding of each transcript. The method (Lin et al., 2020) based on the BiLSTM model reaches to a similar lower MAE. They apply pre-trained w2v embeddings. However, we train word embeddings with random vectors.
(a) A histogram showing the distribution of PHQ scores on the dataset (of 189 subjects). The X-axis represents a PHQ score ranging from 0 to 24. The Y-axis represents the number of subjects for each score. (b) Results on the development set for our graph-based PHQ prediction system, with the true PHQ plotted as a function of predicted PHQ. The red color represents the performance of applying a generic GNN (Gilmer et al., 2017) algorithm, whereas the blue color represents the performance of implementing our SGNN. The yellow vertical line, as a cut-off, emphasizes a better generalization performance of a small group having scores higher than 13 (on the right of the figure).

heading: 3.1.1 ABLATION STUDY
text: To further analyze our model, we perform three ablation studies (see Table 2). To give more reliable measures of performance, we perform 10-fold cross validation on the combined training and development dataset. We concatenate the training and development set as one set and then divide it into 10 folds in a stratified manner. Each time one fold is used for testing and the other 9 folds are used for training.
In (i), we give equations 1-4 (in section 2.2) for ‘point-wise’ multiplication. According to the results in Table 2, we can see that using convolution layers can better model the relations between words compared with point-wise multiplication layers.
In (ii), we remove the second and third (equivariant) terms in (1). There was a significant reduction in performance (Table 2). As a result, the representation power of our model can increase when parametrized with these layers, thereby improving the model’s performance on the data set.
In (iii), we replace the max-pooling operator by a mean-pooling operator. From Table 2, we can see that the result was not good when applying mean operator of the node. Max operator highlighting features can enhance discriminating depressive features, which helps to achieve a better result.
Qualitative Analysis: Our model has some incorrect predictions which may be caused by the unequal distribution of dataset (see figure 3a). In figure 3b, our SGNN model made more biased predictions on those subjects with higher PHQ depression scores (plotted in blue): For instance, the model predicted a PHQ score of around 16 to the patient with an actual PHQ score of 13, and the patient who has the PHQ score of 23 has been overpredicted with a score of around 26. To improve the learning capacity of the model, more labeled training samples with PHQ scores higher than 8 would be needed.

heading: 3.1.2 INTERPRETABILITY ANALYSIS OF WORD EMBEDDINGS
text: We give an illustrative visualization of the word embeddings learned by SGNN model. Figure 4 can tell us what the content the model mainly focuses on, therefore helping us to understand its learning process to some extent. We show two examples with predictive depression scores higher than 15. The depression symptoms (of a patient) with corresponding PHQ-scores higher than 14 are classified as moderate-severe or severe by a domain expert (Kroenke et al., 2009).
We observed that the output of the final message passing layer of the SGNN model can be interpretable. We set the dimension with the highest value as a word’s weight value. We then select words by a customized threshold. Compared with the content of the transcripts illustrated by word clouds (figure 4a, figure 4c) of the left-side column, we found that word clouds (figure 4b, figure 4d) on the right-side column represent very different content of the same transcripts. We note that the
model can learn to discriminate semantic features relating to depressive symptoms or non-depressive symptoms from the text.
Word Visualization: We also qualitatively visualize word embeddings, in the form of bi-grams, learned by SGNN model. Table 3 shows the most important word pairs that have been learned to capture context-level semantic features, such as “feel, tough”, “lose, energi” or “ani diagnos” related to PHQ topics. This demonstrates that the high depression scores are predicted on the basis of appropriate semantic.

heading: 3.2 TWITTER SENTIMENT CLASSIFICATION
text: We evaluate the performance of our SGNN model on classifying three different sentiments on Twitter data (Naseem et al., 2019; 2020; Yenduri et al., 2021; Janjua et al., 2021). We train our model with a total of 7800 tweets. 10-fold stratified cross-validation is also adopted for our experiments. We compare our SGNN model with a general GNN model. Table 4 reports accuracies on both training and testing set. The row one method is initialized with pretrained 300-dimensional GloVe embed-
dings. The row two method initialises all the nodes with random vectors (we set 32-dimensional random vectors for experiments). We observed that our low dimensional 2-D node embeddings can propagate contextual information to the whole text well, while high dimensional pretrained embeddings do not improve classification performances.
Table 4 also shows the performance gains of the proposed model on the Borderlands Sentiment Twitter data when 2-D node embeddings applied. This gain demonstrates that a GNN model with 2-D node attributes can capture more context-level features of sentiments from text, and thus can greatly improve the accuracy of this domain task.

heading: 3.3 ANALYSIS OF 2-DIMENSION NODE ATTRIBUTES
text: We compared results of performances between a standard GNN model and our SGNN model on both the depression prediction task and the twitter sentiment classification task using 2-dimensional (2-D) node attributes.
On the first experiment, our graph representation learning framework using schemas achieves better generalisation on long text. Figure 3b shows that our SGNN model (plotted in blue) generalized the development set much better, particularly for the small group of samples in the class of having scores higher than 13. On the second experiment, we find our SGNN model also performs better on the short text Twitter dataset using schemas, showing that it can also model consecutive and short-distance semantics well.
We suggest that using 2-D node attributes (i.e. schemas) can improve the expressive power of the MPM. By setting a global context (matrices) of each node (as the input), we can obtain some parameters, in an explicit way, that capture the context of a word using each node’s schema. On the contrary, using simple 1D node embedding such as GloVe may result in losing contextual information when they are processed by several layers of a learning model.
Settings: We set the learning rate as 10−3, L2 weight decay as 10−4, the dropout rate as 0.5, and the window size as 4 to gather word-word occurrence statistics for both experiments. The loss objective for the first experiment is mean absolute error (MAE); the loss objective for the second experiment is cross-entropy loss. We trained the SGNN for a maximum of 500 epochs using the SGD optimizer (Kingma & Ba, 2014) and stopped training if the validation loss does not decrease for 10 consecutive epochs.

heading: 4 CONCLUSIONS
text: Our work shows a novelty of improving the performance of predicting depression states by training a deep graph learning model to learn contextual features from the text. Our results have demonstrated that it is possible to apply deep learning methods to tackle more specific problems within the field, and not just a more general problem, even with limited data. Future work might address finding a way of explaining a GNN model, such as, visualizing the relationship between the underlying depression features and depression scores, which helps us better understand clinical context behind the data.

Review summaries:
This paper proposes a schema-based GNN method to measure the severity of depression. To gain a global representation of each word, the proposed method constructs word nodes and uses schema structure to capture the context-level information. The main contribution of this paper is the introduction of the schema encoder. The experimental results show the superiority of schema GNN over other SOTA models.