Using Graph Representation Learning with Schema Encoders to Measure the Severity of Depressive Symptoms
Authors: Simin Hong
Anthony Cohn
David Crossland Hogg
Venue: ICLR 2022 Poster (2022)
Keywords: Graph neural networks sentiment analysis node-embedding algorithm  diagnostic prediction task

Abstract: Graph neural networks (GNNs) are widely used in regression and classification problems applied to text, in areas such as sentiment analysis and medical decision-making processes. We propose a novel form for node attributes within a GNN based model that captures node-specific embeddings for every word in the vocabulary. This provides a global representation at each node, coupled with node-level updates according to associations among words in a transcript. We demonstrate the efficacy of the approach by augmenting the accuracy of measuring major depressive disorder (MDD). Prior research has sought to make a diagnostic prediction of depression levels from patient data using several modalities, including audio, video, and text. On the DAIC-WOZ benchmark, our method outperforms state-of-art methods by a substantial margin, including those using multiple modalities. Moreover, we also evaluate the performance of our novel model on a Twitter sentiment dataset. We show that our model outperforms a general GNN model by leveraging our novel 2-D node attributes. These results demonstrate the generality of the proposed method.

Published as a conference paper at ICLR 2022
USING GRAPH REPRESENTATION LEARNING WITH SCHEMA ENCODERS TO MEASURE THE SEVERITY OF DEPRESSIVE SYMPTOMS
Simin Hong
Anthony G. Cohn
David C. Hogg
School of Computing University of Leeds, UK scsho @ @leeds.ac.uu
School of Computing University of Leeds, UK g cohn @ @leeds.ac.uk
School of Computing
University of Leeds, UK d. hogg @ lccdssaccuu
ABSTRACT
Graph neural networks (GNNs) are widely used in regression and classification problems applied to text, areas such as sentiment analysis and medical decision- making processes. We propose novel form for node attributes within a GNN based model that captures node-specific embeddings for every word in the vocab- ulary. This provides a global representation each node, coupled with node-level updates according to associations among words transcript. We demonstrate the efficacy of the approach by augmenting the accuracy of measuring major de- pressive disorder (MDD). Prior research has sought to make a diagnostic pre- diction of depression levels from patient data using several modalities, includ- ing audio, video, and text. On the WOZ benchmark, our method outper- forms state-of-art methods by substantial margin, including those using multiple modalities. Moreover, we also evaluate the performance of our novel model on a Twitter sentiment dataset. We show that our model outperforms general GNN model by leveraging our novel 2-D node attributes. These results demonstrate the generality of the proposed method.
INTRODUCTION
Deep learning techniques have been frequently used in sentiment analysis (TTn et al. 2013 Mukhtar & Khan 2018 Liao et al. (021), particularly in movie recommendations and product reviews and ratings. the healthcare domain, sentiment analysis with deep learning provides many benefits such utilizing medical information to increase healthcare quality (Abirami & Askarunisa 2017 Bi et al. 2020). We focus our work on applications related to mental health such as using deep learning to make diagnostic prediction.
There is a pressing need to find a convenient and automated method to assess depression sever- ity. Worldwide, more than 300 million people are suffering depression (Organization et al., (017). Apart from the high prevalence of Major Depressive Disorder (MDD). overall 85% of depressed in- dividuals are underdiagnosed (Falagas et al. 2007). Research has shown that about 30% of patients suffering from an episode of major depression do not seek treatment, with only 10% of them being adequately treated (Falagas et al., (007). We validate our approach based on its application the diagnosis of MDD. We demonstrate that our approach can result in more accurate predictions of the severity of depression, which can enhance automated health care decision making.
Medical research (Tsakalidis et al. 2018) shows that text sentiment analysis methods can be effective in making inferences about people's mental states, such as MDD. For example, depressed people use more first person pronouns than non-depressed people because they more focused on themselves and less connected with others. These may not be reflected in a person's visual appearance, i.c., visual behaviors. Visual data alone will therefore be insufficient for capturing depressive symptoms that are only manifested a person's 's verbal utterances.
Mental disorders difficult detcet. In practice, clinicians first measure the severity of depressive symptoms in patients and then identify depression in them. During in-person clinical interviews, clinicians widely structured Patient Health Questionnaire (PHQ) clinically validated tool

---

Published as a conference paper at ICLR 2022
determining the severity of depressive symptoms across several personal dimensions Kroenke & Spitzer 2002
Arseniev-Koehlee et al.) 2018).
Inspired by the observation of provided data set of such interviews, formulated the hypothesis that the context words a transcript can be used to generate PHQ scores (Kroenke et al. 2009). This PHQ metric for depressive disorder forms context constructed with eight items, such "sleep problem", "anxiety problem". "fatigue problem". "depression problem". and "no motivation or in- terest in things" Jarsenie-Koohler information relating to at least one of these eight topics. This motivates use of a deep learning model that represents these eight PHQ topics. thereby introducing an inductive bias into the learning pro- cess. Thus the contextual information about these eight PHQ topics should be encoded to determine depression states.
al. 2018). We observed that each text (see figure I covers
always feel irritated. am lazy when do not sleep well. my mood was just not right, i was always feeling down and depressed and lack of energy. always want to sleep. am lack of interest. have gone therapy, it has been useful for me in the past. would love to talk to someone, i just feel like do not have anyone so do not depend on anyone. have always felt depressed in my life, my symptoms were lack of energy, wanting to sleep lot, lack of interest. my appetite uncontrollable either lack of or was just being gluttonous and eating the wrong things. have notices those changes in my behavior.....
Figure An extract from a raw transcript.
We motivate generating graph-level representations for transcripts (as inputs) to encode contextual information from these transcripts (we assume a transcript encodes facts representing depressive symptoms). To achieve this, we propose novel form for node attributes within GNN based model that captures node-specific embeddings for every word in the vocabulary. The representations of each word are shared globally and be updated according transcript. We summarize the representations of all the words in the transcript to predict depression states.
associations among words in
Intuitively, the generated node-level embeddings maintain records linking certain existing facts that indicate the known symptoms of a subject. When subsequent facts discovered which indicate additional depressive symptoms, records will be updated by aggregating both old and new facts. In other words, those records will be updated via a message passing mechanism over the transcript context until ideally all major depressive features have been discovered.
In this way, we can represent the most relevant contextual information which is universal across all transcripts possibly involving the context of eight PHQ topics we discussed above. Using graph structures to capture context-level features is an innovative pathway which we hypothesise can be used to measure different levels of depression. In our experiments, our approach realized good generalization on a limited, incomplete, and unevenly distributed dataset (see figure
1.1 RELATED WORK
Research in measuring the severity of depressive symptoms aims to train a regression model to predict depression scores (Valstar et al., [0066] Alhanai et al. 2018 Tsakalidis et al., 2018). Some prior work (Ringeval et al., 2017] Alhanai et al. 2018 Haque et al. 2018) applies sequence- level deep learning model to capture implicit depressive signals. Such models, general, use a multi-modal sentence embedding, rather than mapping of a whole interview, to predict a PHQ score. Other prior work (Valstar et al. 2016 Cummins et al. (017)) implements statistical functions (e.g., max, min) short-term features over entire interview, but this may fail to preserve useful temporal information across an entire interview, such as some short-term signs regret, anxicty, etc.
2

---

Published as a conference paper at ICLR 2022
Deep learning approaches Vallstar et al. 2016 Alhanai et al. 2018 Song al. 2018 Du et al. 2019) to fuse multi-modal features in depression detection appear to be particularly promising. Al- hanai et al. (2018) proposed a deep model which trained jointly with the acoustic and linguistic features. Lam et al. (019)) used multi-head attention modules to extract contextual information from clinical text to assess depression tendency. There existing studies utilizing deep learn- ing models with pre-trained word embeddings to extract global features from text (Ray et al.) 2019
Mallol-Ragolta al. 2019; Zhang et al.) 2020 (Solieman & Pustozerov 2021 The work (Lin et 2020) Solieman & Pustozerov (2021) applied multi-moda fusion network to summarize all depressive features from both text and audio. They used text features and audio features that were highly correlated with depression severity.
In principle, we should able to improve performance further using all available modalities, how- ever feature-fusion method may lead to learn a wrong representation encoding inter-subject vari- ability unrelated to depression (Williamson et al., 2016). The feature-fusion approach learning from small data may cause poor generalization: with small sample size, the number of features should also be small to avoid the problems of dimensionality and overfitting. However, given that the dimensions of audio and video features very large, sparse parametrization needs to be taken into account when training machine learning models for learning a joint multi-modal feature vector Williamson al. 2016).
2
METHOD
We generate a node-level embedding matrix for each word of a transcript. These node attributes rep- resent an underlying relationship between the current word and every other word in the transcript. In this way, graph neural network (GNN) schematizes underlying associations among words via their internal representations. We propose to aggregate their internal representations to encode in- formation from the context of a transcript generalizing all major depressive symptom?"
Building a Text Graph: For given transcript, we build a text graph. We regard all unique words appearing a transcript as the nodes of the graph. Each edge starts from word in the transcript and ends with a word within fixed window on either side of the word.
Let G = (V, F)) denote text level graph. is set of nodes representing all the unique words in given text, and E' is set of undirected edges between pairs of these nodes, each represented by set of the two nodes at either end of the edge. We build edges by using a fixed size sliding window in transcript
collect co-occurrence statistics. Each node has attribute which is matrix.
To emphasise this representational structure of the attribute matrix, we refer to it as a "schema' (aammen & Zupan 1984 Rudolph et al., 1997; Soygüt & Savaşir 2001 Dozois & Beck 2008) U. E 1R The jea row of Ug is a vector of length d containing the representation that node 0; has of by; and n denotes the total number of unique words (the vocabulary size) in corpus.
2.1 SCHEMAS
We generate schema for each word node which performs a role of recording a global context. This global context retains information from interactions between the current word and every other word. In this way, each word node maintains "a dynamic record" (in the form of schema) of the context from the given transcript. The schemas are progressively updated by GNN model. This resulting model produces final embeddings of the words a transcript in relation all words the vocabulary, including the other words that make up the current training transcript.
Schemas preserve structures that represent relationships between the identifier word and every other word. Thus we can exploit these schemas to capture context-level features in explicit way by learning a GNN model. Our innovation is to represent word proximity through the graph structure and co-occurrence within the same transcript within the schema at each node.
code be found here
3

---

Published as a conference paper at ICLR 2022
lack
life
is
friend
care
his
lack
friend
hopeless
am
is
his
hopeless
life
UK+1
hopeless
my
do
hopelees
life
10
KK
hopetess
For one of layers of SGNN model k, the upper figure showing an output is a schema generated for the word node "hopeless". This schema be treated node "hopeless", which is formed as Uh hopeless E n x de. where the generated d4-dimensional representation of a node presents as a row. The blank rows correspond to word nodes that have not been encountered with this identifier node "hopeless" in the current text context. The figure the bottom shows an example of a modified schema of the same word node "hopeless" after learning. Our model using schemas can encode information about associations between the node "hopeless" and its new neighbors, such as a node "his" (colored in green). As a result, the internal representation of this identifier node "hopeless" is updated by the model. Moreover, its original record containing its other already existing neighbors has been explicitly preserved while learning. Note: For the convenience of display, in this figure, we set the window size of 1 for displaying associated edges among the nodes; in our actual experiments we use larger window size.
Figure 2:
"inner record" of the
2.2 SCHEMA-BASED GRAPH NEURAL NETWORK (SGNN)
We use multiple passes (layers) of the message passing mechanism (MPM) (Gimmer et al. 2017 Xu al. 2018) to update the schema each node of the text graph.
Initialization: We first initialise an n dj matrix (in our experiments, set dj 32) as the schema (i!) at each node U; E V. The schema is all zeros apart from the row corresponding to the word associated with this node, which is random d-dimensional vector using linear transforma- tion.
Message passing layer:
In this work, the operation of message passing is split into two steps to update the schema at each node.
We first modify the schema at each node ty:
0, 0.
w(()
- w((k) +
(I)
n
where 1 R is vector of ones.
The first term updates each row independently. The second term operates on the sum of the columns, replicated in each row. The third tenn operates the row corresponding to the current word, replicated in each row. These terms are subset of equivariant linear functions which are computed using the methods of (Maron et al. 2018).
4

---

Published as a conference paper at ICLR 2022
Second compute the message function, which is defined
(K) _,, ; Uj (k) REEUU (h0 110; (4) (k) (Ú,
( w
(2)
where| || denotes concatenation along the second axis. (Wih)) 1smss are learnable parameters. This essentially a two-layer neural network.
In the step, each node's schema is updated as:
(8) U, (k+1)) = Σ M (1, (U, 0, (0) Ruxdx+
(3)
JEN()
We apply equation (3) as the aggregator over the k-th layer of the SGNN.
After message- message-passing layers have been applied, the schemas are pooled to get representations of all nodes in the graph. We apply elementwise max-pooling operater extract node features. We use the READOUT (Ying et al. 20018) Xu et al. 2018) function which aggregates node features by summing them together:
hg Σ U(!)
WEV
We use he to predict a PHQ score for each transcript. In our experiments, we apply a 2-layer multi-layer perceptron (MLP).
2.3 DATASETS
We performed experiments on two datasets, DAIC-WOZ and a twitter sentiment dataset. Whilst our principal interest is in depression prediction, we included the second dataset to show the generality of the method we present here.
The DAIC-WOZ dataset contains video-based facial actions, audio and the conversation transeribed
for each participant. Both the 6th and 7th International Audio/Video Emotion Challenge (AVEC) (Valstar et al.) (016) used this dataset. We utilized only the text transcripts of all 142 individuals within the dataset. 43 out 142 subjects (30%) were labeled depressed. The provided dataset has been split into training set having 107 pa- tients and a development set containing 35 patients. In line with prior work (Valstar et al.) 2016 Allanai al.) 22018] Cummins al. | parable results, we test on the "development set" from the original competitions (Valstar 2016), since the actual test set is not in the public domain.
et al. 2018) and to ensure com-
et
Privacy:
This dote not contain protected health information(PHD, Personal names, specific dates and locations were removed from the audio recording and transcription by the dataset curators.
Borderlands Sentiment Twitter Positteerlattsett lataset includes 74682 tweets labeled positive (28%), negative (30%) and neutral (42%) respectively. Tweets range in length from 2 to 126 words with average length of 25.3 words per tweet. We extracted 10% of the full data for training.
EXPERIMENTS
In this section, we show the results generated by our method on two distinct tasks of depression prediction and twitter sentiment classification respectively.
htttps://doaaasos..ttttt.ss https://ww..aggttt com came www.ttts///ooooooooo.oo.o..ts..t http:/attttatgaaeeeeeeaeeeeeabe accessed 11/11/21
last accessed on 18/09/20
sent iment: ana yaissw' keras/
5

---

Published as a conference paper at ICLR 2022
Table Comparison of machine learning approaches for measuring the severity of depressive symp- toms on the DAIC-WOZ development set using mean absolute error (MAE). The task evaluated is: PHQ score regression. Modalities: A: audio, visual, L: linguistic(text), A+V+L: combination. The result marked with has been computed by us; the others are taken from the cited papers.
Regression: PHQ score
Methods
Modalities
PHQ score MAE
Bascline Challenge (Valstar et al. 2016)
A+V
5.52
Gaussian Staircase Regression Williamson al. 2016
A+V+L
4.18
LSTM Haquu et al.
2018)
A+V+L
5.18
LSTM Alhanai et
2018
A+L
5.1
DCGAN Yang et al.
2020)
A
4.63
DepArt-Net (Du al.
2019
V
4.65
LSTM [Alhanar et al.
2018)
I.
5.2
C-CNN (Haque et al.
2018
6.14
BiL et al. 2020)
L
3.88
Multi-level Attentron network (Ray et al. 2019
4.37
*GNN (Gilmer et al.
2017
L
4.24
Our Proposed Approach
3.76
Table 2: Results of ablation studies. We apply 10-fold stratified cross-validation and give mean
results. MAE: mean absolute error. RMSE: root mean squared error.
Metric
MAE
RMSE
Setting
Train
Test
Train
Test
Original(schemm-GGNN
3.85
3.52
4.48
4.32
i) Fast(schema-GNN)
4.28
3.86
5.14
4.51
Without equivariant linear layers (Maron al. 2018)
4.14
3.95
5.60
4.49
iii Mean Reduction
4.32
5.16
4.45
5.12
3.1 THE SEVERITY OF DEPRESSIVE SYMPTOMS PREDICTION
In this experiment, we predict the PHQ score for each participant. The loss objective is mean absolute error (MAE) for regression. We evaluate SGNN model on DAIC-WOZ benchmark and we then compare our method to the state-of-art works centering on the feature fusion learning algorithms.
We compare our method to prior work on measuring depressive symptom severity. The performance of our method and eleven other methods, including the state of the method, out in Table The results of some models are directly taken from their original papers. Our method outperforms all other methods, despite using only the textual modality.
We found that our model performs better than the standard GNN Gilmer et al. 2017). The GNN without our schemas utilizes node representations which are initialized by pre-trained 300- dimensional GloVe word embeddings (Mikolov et al. 2013). However, our method constructs schemas which are initialized with random vectors. Moreover the standard GNN propagates node features in terms of vectors while our model uses 2-D node attributes. This change increases the expressive power of the MPM, yielding better results.
The machine learning algorithms of (Valstar et al. 2016 Williamson et al.) 2016) perform modeling statistics using handcrafted features derived from audio, text and (or) visual inputs. However, our method uses just raw text.
We also note that our model performs better than other prior work (Alhanai et al. 2018 Song et al., 2018 Haque et al. 2018 Du et al. 2019 Lin al. 2020 Ray et al. 2019). That is likely due to the difference of representation learning. Their work uses a multi-modal sentence-level embedding to predict PHQ score while model is trained to learn graph-level embedding of each transcript. The method (Lin et al., 2020) based on the BiLSTM model reaches similarly low MAE. They apply pre-trained w2v embeddings. However, we train word embeddings with random vectors.

---

Published as a conference paper at ICLR 2022
Figure 4: Results on the development set for our graph- based PHQ prediction system, with the true PHQ plotted as a function of predicted PHQ. The red color represents the performance of applying generic GNN (Gilmer ct al. 2017) algorithm, whereas the blue color represents the performance of implementing our SGNN. The green vertical line, as a cut-off, emphasizes better generaliza- tion performance of small group having scores higher than 13 (on the right of the figure).
Figure 3: A histogram showing the dis- tribution of PHQ scores on the dataset (of 189 subjects). The X-axis represents a PHQ score ranging from 24. The Y-axis represents the number of subjects for each score.
3.1.1 ABLATION STUDY
To further analyze our model, we perform three ablation studies (see Table 2). To give more reliable measures of performance, we perform 10-fold cross validation on the combined training and devel- opment dataset. We concatenate the training and development set as one set and then divide it into 10 folds in a stratified manner. Each time one fold used for testing and the other 9 folds are used for training.
In (i), we give equations 1-4 (in section 2.2) for 'point-wise' multiplication. According to the results in Table 2, we
that using convolution layers can better model the relations between words compared with point-wise multiplication layers.
In (ii), remove the second and third (equivariant) terms in (I), There significant reduction in performance (Table 2). As result, the representation power of our model can increase when parametrized with these layers, thereby improving the model's performance the data set.
In (iii), we replace the max-pooling operator by a mean- mean-pooling operator. From Table 2 we can see that the result not good when applying mean operator of the node. Max operator highlighting features can enhance discriminating depressive features, which helps to achieve a better result.
Qualitative Analysis: Our model has some incorrect predictions which may be caused by the
unequal distribution of dataset (see figure 3). In figure 4 our SGNN model made more biased predictions on those subjects with higher PHQ depression scores (plotted in blue): For instance, the model predicted PHQ score of around 16 to the patient with actual PHQ score of 13, and the patient who has the PHQ score of 23 has been overpredicted with a score of around 26. To improve the learning capacity of the model, more labeled training samples with PHQ scores higher than would be needed.
3.1.2 INTERPRETABILITY ANALYSIS OF WORD EMBEDDINGS
We give an illustrative visualization of the word embeddings learned by SGNN model. Figure can tell us what the content the model mainly focuses on, therefore helping us to understand its learning process to some extent. We show two examples with predictive depression scores higher than 15. The depression symptoms (of a patient) with corresponding PHQ-scores higher than 14 are classified moderate-severs or severe by domain expert (Kroenke al. 2009).
Compared with the content of the transcripts illustrated by word clouds (figure 5a figure 5c) of the left-side column, we found that word clouds (figure 5b figure 5d] the right-side column represent very different content the same transcripts. We note that the model can learn to discriminate semantic features relating depressive symptoms non-depressive symptoms from the text.

---

Published as a conference paper at ICLR 2022
Table 3: Performance of top-10 bi-gram word associates on the DAIC-WOZ development set. The word-word connections, in the context of PHQ-related topics, are generated by the output of the final message passing layer of SGNN model. gt:ground truth
Transcirpt
Transcript 2
gt score of 16
score of 19
predictive score 16.46
predictive score of 17.30 ('almost', 'thought')
('married', "upset')
('getting', 'upset')
('cheated', "marry")
('anyone', "argued')
('unconditional, 'trip')
('family", 'issue")
("depression', 'psychiatrist")
('feel', 'tough")
('certainly', 'argent")
('energy', 'explore') ('helping'. 'sleep')
('exhausted", 'never')
('therapy', 'asleep')
('lack", 'achieve')
('development', 'issue") ('married', 'upset') ('especially', 'breath')
('missing', 'every") ('slceping', 'depression')
teeltime
meant cantar
Diece èce
o
heeep.coo Pccoo
MOU
sifferent
and
nk
erar
upss mine
NC
you missing
reat
marieed
peop
yeah pretey
(a) transcript A with ground (b) the transcript A with predic-
truth PHQ score of before
PHQ score of 16.46 after
much makelifewantee
preety
y
AIN SAAN argent AESECALLION LXTLLE
time think
PUCEM
aammost
DEar
though
we I one
point
LOS
real
FOOR
di
ffeentt
know
ciMated
sur e
BIT
probable win BLAck at children
Farent
certain Vactua
(c) transcript B with ground the transcript B with predic- truth PHQ score 19 before tive PHQ score 17.30 after
Figure 5: A word cloud depicting words from a transcript on the development set before and after applying SGNN model. The two word clouds on the left-hand column depict the most salient words based the frequency of their occurrences over two different raw transcripts respectively. The word clouds on the right-hand column illustrate the most focused content selected by SGNN model. The output of the final message passing layer of word embeddings is used. The right and left-hand columns use the same transcript.
Word Visualization: We also qualitatively visualize word embeddings, in the form of bi- gramm,,, learned by SGNN model. Table shows the most important word pairs that have been learned to capture context-level semantic features, such as "feel, tough", "therapy, asleep", "sleeping, depres- sion" or "depression, psychiatrist" related PHQ topics. This demonstrates that the high depression scores are predicted the basis of appropriate semantie features.
3.2 TWITTER SENTIMENT CLASSIFICATION
We evaluate the performance of our SGNN model on classifying three different sentiments on Twit- ter data (Naseem et al. 2019 2020 Yenduri et al. 2021 Janjua et al.) (02I). We train our model with a total of 7800 tweets. 10-fold stratified cross- validaaiii is also adopted for our experiments. We compare our SGNN model with a general GNN model. Table reports accuracies on both training and testing set. The embeddings. The row two method initialize each node with a 2-dimensional (2-D) random node attribute. We observed that our low dimensional 2-D node embeddings can propagate contextual
method is initialized with pretrained 900-dinensional Glo Ve
8

---

Published as a conference paper at ICLR 2022
Table 4: Performance comparison between a general GNN model and the SGNN model with 2-D node attributes on the twitter dataset. We implement a standard GNN without schema encoders. We show the accuracy on sentiment classification task. We adopt 10-fold stratified aross-validation and report mean with standard deviation in the parentheses for both of these two models. The GNN model initialized by Glove word embeddings.
Model 1) GNN
Train 0.958 (0.1200)
Test 0.878 (0.0963) 0.941 (0.0013)
ii) SGNN with 2-D node embedding
0.951 (0.1146)
information to the whole text well, while high dimensional pretrained embeddings do not improve classification performances.
Table also shows the performance gains of the proposed model the Borderlands Sentiment Twitter data when 2-D node embeddings applied. This gain demonstrates that a GNN model with 2-D node attributes can capture more context-level features of sentiments from text, and thus can greatly improve the accuracy of this domain task.
3.3 ANALYSIS OF 2-DIMENSION NODE ATTRIBUTES
We compared results of performances between standard GNN model and our SGNN model on both the depression prediction task and the twitter sentiment classification task using 2-D node attributes.
On the first experiment, our graph representation learning framework using schemas achieves better generalisation on long text. Figure 4 shows that our SGNN model (plotted in blue) generalized the development set much better, particularly for the small group of samples in the class having scores higher than 13. On the second experiment, we find our SGNN model also performs better on the short text Twitter dataset using schemas, showing that it can also model consecutive and short-distance semanties well.
We suggest that using 2-D node attributes (i.e. schemas) can improve the expressive power of the MPM. By setting global context (matrices) of each node (as the input), rameters, in an explicit way, that capture the context of a word using each node's schema. On the contrary, using simple ID node embedding such as GloVe may result in losing contextual informa- tion when they are processed by several layers of a learning model.
obtain some pa-
Settings:
We set the learning rate as 10-3 3, L2 weight decay as 10-4, the dropout rate as 0.5, and the window size as 4 to gather word-word occurrence statistics for both experiments. The loss objective for the first experiment experiment is cross-entropy loss. We trained the SGNN for maximum of 500 epochs using the SGD optimizer (Kingma & Ba 2014) and stopped training if the validation loss does not decrease for 10 consecutive epochs.
mean absolute error (MAE); the loss objective for second
4 CONCLUSIONS
Our work shows novelty of improving the performance of predicting depression states by training deep graph learning model to Iearn contextual features from the text. Our results have demonstrated that it is possible apply deep learning methods to tackle more specific problems within the field, and not just a more general problem, even with limited data. Future work might address finding way of explaining GNN model, such as, visualizing the relationship between the underlying depression features and depression scores, which helps us better understand clinical context behind the data.
ACKNOWLEDGEMENTS
The second and third authors were partially supported by Fellowships from the Alan Turing Institute, UK. The second author was also partially supported by the EU Horizon 2020 Framework under grant agreement 825619 (AI4EU).

---

Published as a conference paper at ICLR 2022
REFERENCES
Ariyur Mahadevan Abirami and Abdulkhader Askarunisa. Sentiment analysis model to emphasize the impact of online reviews in healthcare industry. Online Information Review, 2017.
Tuka Alhanai, Mohammad Ghassemi, and James Glass. Detecting depression with audio/text se- quence modeling of interviews. In Interspeech, pp- 1716-1720, 2018.
Alina Arseniev- Koehler, Sharon Mozgai, and Stefan Scherer. What type of happiness are you look- ing for?-a closer look at detecting mental health from language. Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard Clinic, pp- 1-12, 2018.
Qiqing Bi, Lining Shen, Richard Evans, Zhiguo Zhang, Shimin Wang, Wei Dai, Cui Liu, et al. Determining the topic evolution and sentiment polarity for albinism a chinese online health community: machine learning and social network analysis. JMIR medical informatics, 8(5): e17813, 2020.
Nicholas Cummins, Vidhyasaharan Sethu, Julien Epps, James R Williamson, Thomas F Quatieri, and Jarek Krajewski. Generalized two-stage rank regression framework for depression score prediction from specch. IEEE Transactions on Affective Computing, 11(2):272-25 2017.
David JA Dozois and Aaron T Beck. Cognitive schemas, beliefs and assumptions. Risk Risk factors in depression, pp. 119-143, 2008.
Zhengyin Du, Weixin Li, Di Huang, and Yunhong Wang. Encoding visual behaviors with attentive temporal convolution for depression prediction. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-7. IEEE, 2019.
ME Falagas, KZ Vardakas, and PI Vergidis. Under-diagnosis of common chronic discases: preva- lence and impact human health. International journal of clinical practice, 6199):159-11579 2007.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Constance Hammen and Brian A Zupan. Self-schemas, depression, and the processing of personal information in children. Journal of Experimental Child Psychology, 37(3):598-66) -08, 1984.
Albert Haque, Michelle Guo, Adam S Miner, and Li Fei-Fei. Measuring depression symptom sever- ity from spoken language and 3d facial expressions. arXiv preprint arXiv: 2018.
Sadaf Hussain Janjua, Ghazanfar Farooq Siddiqui, Muddassar Azam Sindhu, and Umer Rashid. Multi-level aspect based sentiment classification of twitter data: using hybrid approach in deep learning. PeerJ Computer Science, 7:e433, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint 1rXiv:1412.6988 2014.
Kurt Kroenke and Robert L Spitzer. The phq-9: new depression diagnostic and severity measure. Psychiatric annals, 32(9):509-5555 2002.
Kurt Kroenke, Tara W Strine, Robert I. Spitzer, Janet BW Williams, Joyce Berry, and Ali H Mokdad. The phq-8 as measure of current depression in the general population. Journal of affective disorders, 1 114(1-3)163--13, 2009.
Genevieve Lam, Huang Dongyan, and Weisi Lin. Context-aware deep learning for multi-modal depression detection. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3946-3950. IEFE, 2019.
Wenxiong Liao, Bi Zeng, Jianqi Liu, Pengfei Wei, Xiaochun Cheng, and Weiwen Zhang. Multi- level graph neural network for text sentiment analysis. Computers & Electrical Engineering, 92: 107096, 2021.
10

---

Published as a conference paper at ICLR 2022
Lin Lin, Xuri Chen, Ying Shen, and Lin Zhang. Towards automatic depression detection: A BiL- STM/ID CNN- model. Applied Sciences, 10(23):8701, 2020.
Adria Mallol-Ragolta, Ziping Zhao, Lukas Stappen, Nicholas Cummins, and Bjöm Schuller. A hierarchical attention network-based approach for depression detection from transcribed clinical interviews. 2019.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint 1rXhv:1812.09902 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Neelam Mukhtar and Mohammad Abid Khan. Urdu sentiment analysis using supervised machine learning approach. International Journal of Pattern Recognition and Artificial Intelligence, 32 (02):1851001, 2018.
Usman Naseem, Shah Khalid Khan, Imran Razzak, and Ibrahim A Hameed. Hybrid words represen- tation for airlines sentiment analysis. In Australasian Joint Conference Artificial Intelligence, pp. 381-392. Springer, 2019.
Usman Naseem, Imran Razzak, Katarzyna Musial, and Muhammad Imran. Transformer based deep intelligent contextual embedding for twitter sentiment analysis. Future Generation Computer Systems, 113:58-69, 2020.
World Health Organization et al. Depression and other common mental disorders: global health estimates. Technical report, World Health Organization, 2017.
Anupama Ray, Siddharth Kumar, Rutvik Reddy, Prerana Mukherjee, and Ritu Garg. Multi-level attention network using text, audio and video for depression prediction. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, pp. 81-88, 2019.
Fabien Ringeval, Björn Schuller, Michel Valstar, Jonathan Gratch, Roddy Cowie, Stefan Scherer, Sharon Mozgai, Nicholas Cummins, Maximilian Schmitt, and Maja Pantic. Avec 2017: Real-life depression, and affect recognition workshop and challenge. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge, pp. 3-9, 2017.
Karen D Rudolph, Constance Hammen, and Dorli Burge. A cognitive-interpersoni approach to depressive symptoms in preadolescent children. Journal of abnormal child psychology, 25(1): 33 45, 1997.
Hanadi Solieman and Evgenii A Pustozerov. The detection of depression using multimodal models based text and voice quality features. In 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pp. 1843-1844 1848. IEEE, 2021.
Siyang Song, Linlin Shen, and Michel Valstar. Human behaviour-based automatic depression anal- ysis using hand-crafted statistics and deep learned spectral features. In 2018 13th IEEE Inter- national Conference Automatic Face & Gesture Recognition (FG 2018), pp. 158-165. IEEE, 2018.
Gonea Soygüt and 1şik Savaşir. The relationship between interpersonal schemas and depressive symptomatology. Journal of Counseling Psychology, 48(3):359, 2001.
Shulong Tan, Yang Li, Huan Sun, Ziyu Guan, Xifeng Yan, Jiajun Bu, Chun Chen, and Xiaofei He. Interpreting the public sentiment variations on twitter. IEEE transactions on knowledge and data engineering, 26(5): 158-118111 1170, 2013.
Adam Tsakalidis, Maria Liakata, Theo Damoulas, and Alexandra I Cristea. Can we assess mental health through social media and smart devices? addressing bias in methodology and evaluation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 407-423. Springer, 2018
11

---

Published as a conference paper at ICLR 2022
Michel Valstar, Jonathan Gratch, Bjöm Schuller, Fabien Ringeval, Denis Lalanne, Mercedes Tor- res Torres, Stefan Scherer, Giota Stratou, Roddy Cowie, and Maja Pantic. Avec 2016: Depression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th international workshop on audio/visual emotion challenge, pp. 3-10, 2016.
James R Williamson, Elizabeth Godoy, Miriam Cha, Adrianne Schwarzentruber, Pooya Khorrami, Youngjune Gwon, Shiaag-TTsung Kung, Charlic Dagli, and Thomas F Quatieri. Detecting depres- sion using vocal, facial and semantic communication cues. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 11-18, 2016.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2018.
Le Yang, Dongmei Jiang, and Hichem Sahli. Feature augmenting networks for improving depression severity estimation from speech signals. IEEE Access, 8:24033-24045, 2020.
Gokul Yenduri, BR Rajakumar, K Praghash, and D Binu. Heuristic-assisted bert for twitter sentiment analysis. International Journal of Computational Intelligence and Applications, 20(03))2150015, 2021.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 4805 481 15, 2018.
Yufeng Zhang, Yingxue Wang, Xueli Wang, Bochao Zou, and Haiyong Xie. Text-based decision fusion model for detecting depression. In 2020 2nd Symposium on Signal Processing Systems, pp. 101-106, 2020.
12


Reviews:
[{"review_id": "lkoJYLhhrdZ", "reviewer": "Reviewer_jCNH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Interesting application of GNN in a clinical field where there is a great need of such applications.Clear clinical benefit.\n-Shows superior performance compared to prior methods with limited text data only.\n\nWeakness:\n- Details of prediction are missing.Is the performance better in patients with severe depression vs mild.While average and aggregated data results are presented, details of underperformance and opportinities for improvement could be identified in subsequent work.\n-Unclear why the performance is better compared to multimodal data approaches.Details of the comparison might be worth investigating.", "summary_of_the_paper": "The authors describe a graph neural network model to predict severity of depression from available text.Application of this method to text data itself is not unique but the clinical context is relevant and important.Many a times in clinical scenarios we do not have access to audio-video data and that's where methods such as these are important utilizing transcribed text.Comparison with prior models is relevant and important and the improvement somewhat significant.", "summary_of_the_review": "This paper has a very practical clinical context and if generalizable has good opportunity to for application.It is unclear why the performance is superior to other multimodal methods which should be investigated more with details of performance variability.Overall good work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kRtmMlLhViu", "reviewer": "Reviewer_Bk2m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+Ves\n+ The idea of using schema GNN is interesting for understanding depressive symptoms.\n+ The paper is a nice effort with visualizations and nice structure, the ablation studies have been well motivated by pointing out the difference of using different operations. \n+ The results indicate the effectiveness of the proposed method.\n\n-Concerns\n\n-Considering the size of dataset, I deem that, the authors should do more experiments on multiple other datasets about depression detection to prove the good performance of the proposed method.\n-The related works are insufficient. There are many studies that worked on DAIC-dataset, such as: \n1. Mallol-Ragolta A, Zhao Z, Stappen L, et al. A hierarchical attention network-based approach for depression detection from transcribed clinical interviews[J]. 2019.\n2. Lam G, Dongyan H, Lin W. Context-aware deep learning for multi-modal depression detection[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019: 3946-3950.\n3. Ray A, Kumar S, Reddy R, et al. Multi-level attention network using text, audio and video for depression prediction[C]//Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop. 2019: 81-88.\n4. Zhang Y, Wang Y, Wang X, et al. Text-based Decision Fusion Model for Detecting Depression[C]//2020 2nd Symposium on Signal Processing Systems. 2020: 101-106.\nDinkel H, Wu M, Yu K. Text-based depression detection on sparse data[J]. arXiv preprint arXiv:1904.05154, 2019.\n5.Solieman H, Pustozerov E A. The Detection of Depression Using Multimodal Models Based on Text and Voice Quality Features[C]//2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). IEEE, 2021: 1843-1848.\n6. Lin L, Chen X, Shen Y, et al. Towards Automatic Depression Detection: A BiLSTM/1D CNN-Based Model[J]. Applied Sciences, 2020, 10(23): 8701.\n7. Arseniev-Koehler A, Mozgai S, Scherer S. What type of happiness are you looking for? - A closer look at detecting mental health from language[C]//Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic. 2018: 1-12.\n\n-Because the authors said the method could help us understand depression, the paper should contain some interpretability analysis using visual figures or examples of results, which can help us to know which words the model mainly focuses on and the learning process.\n\n-In Figure 2, Is there a threshold to check the neighbor word which is encountered?\n\n\nMinor comments: \n\n*The authors should add some references about schema structure.\n\n*The reference of the READOUT function should be added.\n\n*The figures are a bit blurry, please replace them with vector diagrams.\n", "summary_of_the_paper": "This paper proposes a schema-based GNN method to measure the severity of depression. To gain a global representation of each word, the proposed method constructs word nodes and uses schema structure to capture the context-level information. The main contribution of this paper is the introduction of the schema encoder. The experimental results show the superiority of schema GNN over other SOTA models.", "summary_of_the_review": "The idea of using schema-based GNN is interesting and have good performance on DAIC dataset. The paper is well written, but many related works are missing. In addition, the authors should add the interpretability analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4l06k-eoxDq", "reviewer": "Reviewer_ehqq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-The motivation of the paper is clear. And the overall framework is introduced well. The paper is easy to read.\n\nWeaknesses:\n\n-Graph neural networks have been widely leveraged to improve text analysis. The novelty of the method is limited.\n\n-The experimental analysis is insufficient. Comparisons with recent works on measuring depressive symptom severity are not involved, such as [1], [2], [3]. \n\n[1] Muzammel M, Salam H, Othmani A. End-to-End Multimodal Clinical Depression Recognition using Deep Neural Networks: A comparative Analysis[J]. Computer Methods and Programs in Biomedicine, 2021: 106433.\n\n[2] Yang L, Jiang D, Sahli H. Feature augmenting networks for improving depression severity estimation from speech signals[J]. IEEE Access, 2020, 8: 24033-24045.\n\n[3] Lin L, Chen X, Shen Y, et al. Towards Automatic Depression Detection: A BiLSTM/1D CNN-Based Model[J]. Applied Sciences, 2020, 10(23): 8701.\n", "summary_of_the_paper": "This paper tries to leverage graph neural networks to improve the performance of measuring the severity of depressive symptoms. Experiments are conducted on a benchmark dataset.", "summary_of_the_review": "The paper is presented clearly. But the method is not novel. The comparisons with recent works are also not involved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a graph neural network model to predict the severity of depression symptoms from text. It proposes to construct a graph with word nodes and use schema structure to capture the context information in the text. A schema encoder is introduced for modeling the constructed graphs.

Strength:
- Interesting application domain and clear motivation
- Experiment results demonstrate the effectiveness of the method
- Paper writing is clear and easy to follow

Weakness:
- Technical novelty of the method is limited
- Experiment comparison with some recent work is missing
- More in-depth analysis on the method are needed
- Some details of the method pipeline require further elaboration
Decision: Accept (Poster)