<figure><img src="image_1.png" /></figure>

HAL
open science

IRIT at e-Risk

Idriss Abdou Malam, Mohamed Arziki, Mohammed Nezar Bellazrak, Farah Benamara, Assafa El Kaidi, Bouchra Es-Saghir, Zhaolong He, Mouad Housni, Véronique Moriceau, Josiane Mothe, et al.

► To cite this version:

Idriss Abdou Malam, Mohamed Arziki, Mohammed Nezar Bellazrak, Farah Benamara, Assafa El Kaidi, et al. IRIT at e-Risk. 8th International Conference of the CLEF Association (CLEF 2017), Sep 2017, Dublin, Ireland. pp.1-7. hal-01912779

HAL Id: hal-01912779

https://hal.science/hal-01912779

Submitted on 5 Nov 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible

This is an author's version published in:
http://oatao.univ-toulouse.fr/n° de post 19082

**Official URL:** http://ceur-ws.org/Vol-1866/paper_135.pdf

**To cite this version:** Abdou Malam, Idriss and Arziki, Mohamed and Nezar Bellazrak, Mohammed and Benamara, Farah and El Kaidi, Assafa and Es-Saghir, Bouchra and He, Zhaolong and Housni, Mouad and Moriceau, Véronique and Mothe, Josiane and Ramiandrisoa, Faneva *IRIT at e-Risk*. (2017) In: International Conference of the CLEF Association, CLEF 2017 Labs Working Notes (CLEF 2017), 11 September 2017 - 14 September 2017 (Dublin, Ireland).
IRIT at e-Risk

Idriss Abdou Malam ¹ Mohamed Arziki ¹ Mohammed Nezar Bellazrak ¹

Farah Benamara ² Assafa El Kaidi ¹ Bouchra Es-Saghir ¹ Zhaolong He²

Mouad Housni ¹ Véronique Moriceau ³ Josiane Mothe ² Faneva Ramiandrisoa²

(1) IRIT, UMR5505, CNRS & ENSEEIHT, France,

(2) IRIT, UMR5505, CNRS & Univ. Toulouse, France,

(3) LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay, France

**Abstract.** In this paper, we present the method we developed when participating to the e-Risk pilot task. We use machine learning in order to solve the problem of early detection of depressive users in social media relying on various features that we detail in this paper. We submitted 4 models which differences are also detailed in this paper. Best results were obtained when using a combination of lexical and statistical features.

# 1 Introduction

The WHO (World Health Organization) reports that “the number of people suffering from depression and/or anxiety increased by almost 50% from 416 million to 615 million” from 1990 to 2013¹. Depression and Bipolar Support Alliance also estimates that “major depressive disorder affects approximately 14.8 million American adults” and “annual toll on U.S. businesses amounts to about $70 billion in medical expenditures, lost productivity and other costs” (http://www.dbsalliance.org).

Depression detection is crucial and many studies are devoted to this challenge [7]. While there are clinical factors that can help for early detection of patients at risk for depression [10], in this paper we present our approach to help early depression detection from social media analysis, as part of our participation to CLEF e-risk 2017 pilot task [6].

Recent related work focus on people communication and social media post analysis to detect depression. Rude's study shows that depressed people tend to use the personal pronoun (“I”) more intensively than others [9]. Other features have also been noticed. For example, De Choudhury et al. noticed that the depressive people show less activity during the day and more activity during the night [3]. Schwartz et al. reported that depressive people tend to use swear words and talk more about the past [11].

These previous studies show that some cues and features extracted from social media posts can be related to depression. In this paper, we report our investigations on using various features in order to answer the e-risk challenge

¹ http://www.who.int/mediacentre/news/releases/2016/depression-anxiety-treatment/fr/
as described in [6]. The e-risk pilot task aims to detect a depressive person as soon as possible by analysing her or his posts in Reddit<fnref n="2" /> that are provided as a simulated data flow.

In our participation runs, the features we used to characterize posts are of two types: lexicon-based (extracted using NLTK toolkit³) and numerical features. These features are used in a machine learning method using Weka.

The remaining of this paper is organized as follows: Section 2 provides an overview of the model we used. Section 3 details the different features we implemented to train different models. In Section 4 we detail the 4 runs we have submitted and the underlying models and present the results. In Section 5 we discuss the results and depict future work.

## 2 Model overview

Figure 1 presents an overview of the model we use.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of the model used by the e-risk IRIT system.</figcaption></figure>

<footnote n="2">² https://www.reddit.com/
³ NLTK is a platform for building Python programs for natural language processing that interfaces easily with text processing and machine learning libraries (www.nltk.org)</footnote>
The model is composed of three modules. In the first one, we pre-process the XML files that contain the users' posts. The second module aims at extracting the features. Notice that while some features capture information from any textual parts, others focus either on the Title part (corresponding to the initial post) or on the Text part which corresponds to comments on the initial post. The feature extraction module is extensible: while we developed some features, new features can easily be added. Then, in the formatting module, we select a subset of the features to be used in the model.

## 3 Features and models

We developed different types of features. Some have linguistic foundation while others are more statistically-based. We distinguish lexicon-based features from other numerical features.

For lexicon-based features, we rely either on previous observations on depressive subjects' behaviour [3, 11] or on hypothesis that we wanted to evaluate.

Table 1 presents the features that rely on a lexicon. Each feature is calculated as follows: (a) we extract the considered lexicon words from each user's post, (b) for a given user, each lexicon word is weighted by the normalized word frequency (division of the frequency by the total number of words in the user's posts), (c) we then create one feature by averaging the obtained weights over the lexicon words.

Features are calculated for each user as follows: we first calculate the feature value for each of his or her post or comment, then we average the value over his or her posts in the chunk ; when several chunks are used, we average the feature values obtained for each chunk for the considered user.

We also used some other numerical features that are described in Table 2. The details of the features are described in [8].

We submitted 4 runs corresponding to 4 models. The features that were used for each model are listed in Table 3. While we used model GPLA to start with, the other models were introduced later on. The second column of Table 3 indicates the chunk number when each model was introduced. The 4 runs corresponding to our 4 models were performed with the Random Forest learning algorithm under the Weka platform using the default parameters.

In order to decide whether to issue a decision for a subject or wait for more chunks, we used the prediction confidence rate that Weka generates for each prediction. We set a threshold (estimated using samples of depressive subjects) and we only issued decisions that had a prediction confidence that exceeds the selected threshold. The evolution of the threshold for each model through the runs and according to the chunks can be tracked using Table 4, a threshold of 0.5 basically means that all predictions are considered.
<table><caption>Table 1. Details of the features based on lexicons.</caption><thead><tr><th>Num</th><th>Name</th><th>Hypothesis or tool/resource used</th></tr></thead><tbody><tr><td>1</td><td>Self-Reference</td><td>High frequency of self-reference words.</td></tr><tr><td>2</td><td>Over generalization</td><td>Depressive users use words like: "everyone", "everywhere", "everything" a lot.</td></tr><tr><td>3</td><td>Sentiment</td><td>Use of Vader analyser [4] for assigning a polarity score to users' posts:<br/>- Negative &lt; -0.05 and Positive &gt; 0.05<br/>- Neutral otherwise</td></tr><tr><td>4</td><td>Emotion</td><td>High frequency of emotionaly negative words<br/>Used WordNet-Affect [12], to assign a label to each word: Negative, Positive or Ambiguous we then calculated the frequency of each category</td></tr><tr><td>5</td><td>Depression symptoms &amp; related drugs</td><td>From De Choudhury et al. [3] and Wikipedia list<sup>4</sup>.</td></tr><tr><td>6</td><td>Past words</td><td>High frequency of past words.</td></tr><tr><td>7</td><td>Specific verbs</td><td>High frequency of "were" and "was", "like" "have", "being"</td></tr><tr><td>8</td><td>Targeted "I"</td><td>Depressive people tend to target themselves more in subjective context expecially using adjectives</td></tr><tr><td>9</td><td>Negative words</td><td>High frequency of negative words<br/>Used SentiWordNet [1] to detect negative words in texts</td></tr><tr><td>10</td><td>Part-Of-Speech frequency</td><td>Higher usage of verbs and adverbs and lower usage of nouns</td></tr><tr><td>11</td><td>Relevant 3-grams</td><td>Higher frequency of 3-grams described by Gualtiero B. et al. [2] and suggested ones</td></tr><tr><td>12</td><td>Relevant 5-grams</td><td>Higher frequency of 5-grams described by Gualtiero B. et al. [2] and suggested ones</td></tr><tr><td>13</td><td>Relevant 1-grams</td><td>Higher frequency of 1-grams described by Gualtiero B. et al. [2] and suggested ones</td></tr></tbody></table>

## 4 Results

The evaluation takes into account not only the correctness of the output of the system (i.e. whether or not the user is depressed) but also the delay taken to emit its decision. To this aim, the ERDE (Early Risk Detection Error) metric proposed in [5] is used. This measure rewards early alerts and the delay taken by the system to make its decision is measured by counting the number of distinct textual items seen before giving the answer.

Our best results when considering ERDE measures are obtained using model GPLC which does not use POS results nor the most frequent n-grams. Including them in the model slightly improves F1 measure mainly because of higher recall
<table><caption>**Table 2.** Details of the other numerical features.</caption><thead><tr><th>Num</th><th>Name</th><th>Hypothesis or tool/resource used</th></tr></thead><tbody><tr><td>14</td><td>Variation of the number of posts</td><td>For depressive people, the variation of the number of posts is generally small.</td></tr><tr><td>15</td><td>Average number of posts</td><td>Depressive users have a much lower number of posts.</td></tr><tr><td>16</td><td>Average number of words per post</td><td>The two groups of users have different means.</td></tr><tr><td>17</td><td>Minimum number of posts</td><td>Depressive users have a lower value in general.</td></tr><tr><td>18</td><td>Variation of the number of comments</td><td>For depressive people, the variation of the number of comments is generally small.</td></tr><tr><td>19</td><td>Average number of comments</td><td>Depressive users have a much lower number of comments.</td></tr><tr><td>20</td><td>Average number of words per comment</td><td>The two groups of users have different variances.</td></tr></tbody></table>

<table><caption>**Table 3.** Features used in each run.</caption><thead><tr><th>Name</th><th>Details</th><th>Used for the first time in chunk</th></tr></thead><tbody><tr><td>GPLA</td><td>Features: 1, 7-9, 14-19</td><td>1</td></tr><tr><td>GLPB</td><td>Features: 1, 6 (verbs only), 7-9, 14-19</td><td>2</td></tr><tr><td>GLPC</td><td>Features: 1-9, 14-19</td><td>3</td></tr><tr><td>GLPD</td><td>All features</td><td>10</td></tr></tbody></table>

<table><caption>**Table 4.** Evolution of the decision threshold for the 4 models according to the considered chunk</caption><thead><tr><th>Model / Chunk</th><th>1</th><th>2</th><th>3</th><th>4 - 8</th><th>9</th><th>10</th></tr></thead><tbody><tr><td>GPLA</td><td>0.9</td><td>0.9</td><td>0.9</td><td>0.9</td><td>0.6</td><td>0.5</td></tr><tr><td>GPLB</td><td>-</td><td>0.8</td><td>0.8</td><td>0.8</td><td>0.6</td><td>0.5</td></tr><tr><td>GPLC</td><td>-</td><td>-</td><td>0.8</td><td>0.8</td><td>0.6</td><td>0.5</td></tr><tr><td>GPLD</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.5</td></tr></tbody></table>

(0.60 against 0.50) (see Table 5). Our run GPLB had the 2nd best Recall (0.83) across participants and GPLA the 5th.

## 5 Conclusion and Future Work

In the runs we submitted we consider 19 features. However, some additional features are worth studying. In future work, we aim at considering temporal features such as the date of the posts, part of the day, etc. Moreover, we would like to modify the way features are calculated : in the case of lexicon-based
<table><caption>**Table 5.** Results for our 4 runs.</caption><thead><tr><th>Name</th><th>ERDE<sub>5</sub></th><th>ERDE<sub>50</sub></th><th>F1</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>GPLA</td><td>17.33%</td><td>15.83%</td><td>0.35</td><td>0.22</td><td>0.75</td></tr><tr><td>GPLB</td><td>19.14%</td><td>17.15%</td><td>0.30</td><td>0.18</td><td>0.83</td></tr><tr><td>GPLC</td><td><strong>14.06%</strong></td><td><strong>12.14%</strong></td><td>0.46</td><td>0.42</td><td>0.50</td></tr><tr><td>GPLD</td><td>14.52%</td><td>12.78%</td><td><strong>0.47</strong></td><td>0.39</td><td>0.60</td></tr></tbody></table>

features, each lexicon item would be a distinct feature. By this way, we would obtain a richer representation of each user and potentially a better detection.

## References

1. S. Baccianella, A. Esuli, and F. Sebastiani. Sentimentnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. *Istituto di Scienza e Tecnologie dell'Informazione Consiglio Nazionale delle Ricerche Via Giuseppe Moruzzi 1, 56124 Pisa, Italy*, 2010.

2. G. B. Colombo, P. Burnap, A. Hodorog, and J. Scourfield. Analysing the connectivity and communication of suicidal users on twitter. *Computer Communications*, 2015.

3. M. De Choudhury, M. Gamon, S. Counts, and E. Horvitz. Predicting depression via social media. In *ICWSM*, page 2, 2013.

4. C. Hutto and E. Gilbert. Vader: A parsimonious rule-based model for sentiment analysis of social media text. *Eighth International Conference on Weblogs and Social Media (ICWSM-14)*. Ann Arbor, MI, June 2014, 2014.

5. D. E. Losada and F. Crestani. A test collection for research on depression and language use. In *Conference Labs of the Evaluation Forum*, page 12. Springer, 2016.

6. D. E. Losada, F. Crestani, and J. Parapar. eRISK 2017: CLEF Lab on Early Risk Prediction on the Internet: Experimental foundations. In *Proceedings Conference and Labs of the Evaluation Forum CLEF 2017*, Dublin, Ireland, 2017.

7. L.-S. A. Low, N. C. Maddage, M. Lech, L. B. Sheeber, and N. B. Allen. Detection of clinical depression in adolescents speech during family interactions. *IEEE Transactions on Biomedical Engineering*, 58(3):574-586, 2011.

8. I. A. Malam, M. Arziki, M. N. Bellazrak, A. El Kaidi, B. Es-Saghir, and M. Housni. Automatic detection of depression in social networks. Technical report, Universit de Toulouse, France, 07 2017.

9. S. Rude, E.-M. Gortner, and J. Pennebaker. Language use of depressed and depression-vulnerable college students. *Cognition & Emotion*, 18(8):1121-1133, 2004.

10. U. Sagen, A. Finset, T. Moum, T. Mørland, T. G. Vik, T. Nagy, and T. Dammen. Early detection of patients at risk for anxiety, depression and apathy after stroke. *General hospital psychiatry*, 32(1):80-85, 2010.

11. H. A. Schwartz, J. Eichstaedt, M. L. Kern, G. Park, M. Sap, D. Stillwell, M. Kosinski, and L. Ungar. Towards assessing changes in degree of depression through facebook. In *Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality*, pages 118-125, 2014.
12. C. Strapparava and A. Valitutti. Wordnet-affect: an affective extension of word-net. *Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC 2004)*, Lisbon, May 2004, pp. 1083-1086, 2004.