SlowGuess/NAACL_2022_OCR huggingface dataset:
Tulika Saha,\* Saichethan Reddy\* Anindya Das\* Sriparna Saha\* Pushpak Bhattacharyya
*Indian Institute of Technology Patna, India
$^{\dagger}$ Indian Institute of Technology Bombay, India
(sahatulika15,sriparna.saha,pushpakbh)@gmail.com
Mental Health Disorders continue plaguing humans worldwide. Aggravating this situation is the severe shortage of qualified and competent mental health professionals (MHPs), which underlines the need for developing Virtual Assistants (VAs) that can assist MHPs. The data+ML for automation can come from platforms that allow visiting and posting messages in peer-to-peer anonymous manner for sharing their experiences (frequently stigmatized) and seeking support. In this paper, we propose a VA that can act as the first point of contact and comfort for mental health patients. We curate a dataset, Motivational VA: MotiVAte comprising of 7k dyadic conversations collected from a peer-to-peer support platform. The system employs two mechanisms: (i) Mental Illness Classification: an attention based BERT classifier that outputs the mental disorder category out of the 4 categories, viz., Major Depressive Disorder (MDD), Anxiety, Obsessive Compulsive Disorder (OCD) and Post-traumatic Stress Disorder (PTSD), based on the input ongoing dialog between the support seeker and the VA; and (ii) Mental Illness Conditioned Motivational Dialogue Generation (MI-MDG): a sentiment driven Reinforcement Learning (RL) based motivational response generator. The empirical evaluation demonstrates the system capability by way of outperforming several baselines.
With an estimated 970 million individuals suffering from some sort of mental or neural diseases, mental health disorders are regarded one of the primary causes of disability globally<sup>1</sup>. Poor access, stigma, and prejudice, on the other hand, are likely to limit clinical care to only  $15\%$  of individuals who are affected. As a means of expressing their emotions and experiences (generally stigmatized), millions of people (also known as support seekers)
frequently turn to looking for emotional or mental health-related support (Eysenbach et al., 2004) on a variety of text-based peer-to-peer support platforms (De Choudhury and De, 2014), (talklife.co). While peer supports on these platforms are well-intentioned and willing to aid and help seekers, they are often untrained and unaware of best-practices in therapy, resulting in wasted opportunities to provide effective and mutually engaging solutions (Gage-Bouchard et al., 2018). As a result, developing human-computer interfaces in the form of Virtual Assistants (VAs) that can effectively reply and provide support to online support seekers becomes even more critical.
Empathy, or empathetic interactions (Elliott et al., 2018), has been studied extensively in recent years (Sharma et al., 2021, 2020) as one of the most important aspects in providing successful support and triggering beneficial results in support-based dialogues. In addition to empathy, imparting hope and motivation (the process of thinking about and the willingness to move towards one's goals) have been recognised as important affective elements (Dowling and Rickwood, 2016) in uplifting the spirits of support seekers in distress during supportive talks. This is critical because support seekers often engage in escapist or avoidant behavior in anticipation of negative consequences, making it difficult for them to cope with the crisis (Hecht, 2013). Quantitative research shows that instilling optimistic behaviour fueled by hope and motivation improves symptoms in terms of positive psychological transformation and a favourable alliance in mental health support (Jahanara, 2017).
In this paper, we propose a VA acting as the first point of contact for mentally distressed support seekers afflicted with some form of mental illness. The VA's efforts are aimed at reassuring and allowing support seekers to anonymously communicate and express their thoughts, emotions, challenges and seek support. The VA's response should be
<sup>1</sup>https://www(singlecare.com/blog/news/mental-health-statistics/
Human Language Technologies, pages 2436 - 2449
competent and proficient enough to provide support seeker with a natural human experience focused on imparting hope and motivation based on positive perspective. To mimic this human-like behavior of mental health supporters in a VA is quite challenging and the tasks employed are two folds. Firstly, for the VA to provide a conversational support in the absence of electronic health records or psychiatric notes, it is critical to recognise and differentiate various mental diseases because they are frequently communicated using similar language patterns and overall sentiment polarity. In the case of anxiety, for example, the supporter's purpose is to reduce avoidant behaviour and assist the patient in disconfirming a feared consequence. In depression, however, the goal is to assist the mental health seeker in experiencing positive feeling, a burst of energy, or another sort of pleasant contact with the world. Subsequently, the task of the VA is to generate response conditioned on the identified mental illness for modelling motivational conversations with positive outcome.
Due to the unavailability of conversational data for our proposed task, we introduce a dataset, MotiVAte comprising of 7k conversations between support seekers and VA collected from a peer-to-peer support platform. The key contributions of this paper are as follows: (i) To the best of our knowledge, this work is the first to propose a VA for providing motivational support and comfort to mental health patients; (ii) We curate a conversational dataset, MotiVAte, to advance research in mental health based support; (iii) Our end-to-end system employs two sub-modules, viz., Mental Illness Classification (MIC) framework, a dual attention based BERT classifier to identify the mental health disorder of the support seeker in the on-going conversation and Mental Illness Conditioned Motivational Dialogue Generation (MI-MDG) framework to generate mental illness conditioned sentiment driven Reinforcement Learning (RL) based motivational responses mimicking an ideal mental health supporter; (iv) Empirical results indicate that our proposed system outperforms several baseline models.
In this section, we explore mental health based analysis from social media posts and computational models for therapy (Pérez-Rosas et al., 2019).
Mental Health Identification. There are numerous studies over the years that use multi-modal
cues such as images and (Yazdavar et al., 2020) to diagnose diagnose from social media posts and activity (Gaur et al., 2018; Yazdavar et al., 2018; Qureshi et al., 2019; Yazdavar et al., 2017). Investigations have also been conducted on recognising mental illness in online users by their posts on social media (Syarif et al., 2019; Ji et al., 2020). The authors of (Patra et al., 2020) proposed a BiLSTM (Hochreiter and Schmidhuber, 1997) based classifier for classifying mental severity as crisis, red, amber, and green using data from a psychological forum. For detecting mental diseases from daily posts of an online user, (Rao et al., 2020) suggested a knowledge augmented ensemble learning classifier. Authors of (Ji et al., 2021) proposed a pre-trained transformer model named MentalBERT trained on a large corpora of data belonging to mental health care. (Saha et al., 2022) presented a hierarchical attention based classifier to detect mental illnesses from motivational conversations. (Martínez-Castano et al., 2021) proposed a BERT classifier for detecting severity of depression and likeliness towards self harm for social media users.
Mental Health in Conversations. (Althoff et al., 2016) presents an investigation on a large-scale counselling dialogue gathered from an SMS text-based counselling service. As a result of these, exploring empathic relationships in therapy has grown in popularity (Sharma et al., 2020; Morris et al., 2018). In order to help mental health supporters, (Sharma et al., 2021) investigated empathy rewriting as a text generation task. Authors of (Fitzpatrick et al., 2017) presented a conversational agent, named Woebot to deliver cognitive behavioral therapy by initiating daily conversations for mood tracking. Our work differs in the sense that our end-to-end system does not provide any clinical suggestions or therapy recommendations. The role of competence in responses to help-seeking posts on mental health was investigated in (Lahnala et al., 2021).
Sentiment/Emotion aware Dialogue Systems. To make the VA user-adaptive, the authors in (Saha et al., 2020c,d, 2018), proposed using a sentiment-based reward function for learning a dialogue policy in a task-oriented conversation. The authors of (Saha et al., 2020a) demonstrated how reinforcement learning may be used to generate meaningful responses while training generation frameworks. In (Saha et al., 2020b, 2021a,b,c), the authors show how subtleties in human communication, such as
Sample Conversation-1
User1 > I am really struggling to not take more sleeping pills and just pass out. I want to not wake up for a week. I feel worthless of myself not doing anything. I want to take the easy way out and just slip away...
VA > That's not you talking User1, it's the depression. Give yourself some TLC...do something you love like....You are here for a reason that you're not allowed to know about. Hugs, and keep venting...
$\text{User1} >$  I doubled my dose to sleep. But I'm still awake and typing fine.
VA > It's very easy to go from 2 to 3 to 4. The recommended dosage is there for a reason and unless you are amedical professional, you should follow what they say.
VA > You need to take responsibility for your thoughts, feelings, and behavior. It is not our fault we have this disease, we are not to blame. But we have to be responsible. Own it. Take care of yourself.
Sample Conversation-2
User2 > Is excessive guilt associated with OCD? I feel guilty about things that happened  $3+$  years ago....but I treat my mistakes like They are the most evil horrible things! Please help if you have any advice.
VA > Yes, people with OCD often feel an excessive guilt. Have you been diagnosed with OCD? one possible thing to do is to ignore our annoying thoughts, but if it doesn't work for you maybe you could try other methods...do you know a therapist who could suggest you something?
User2 > I have panic attacks and ocd. According to 2 therapists I've seen. They did recommend......I guess I just really want it to be gone. :
VA > You should be consistent with your efforts. It is going to take time, you have to be consistent. Be regular with therapy, try as much as you can to ignore the intrusive thoughts. Remember that they are just some random. Good luck!
Figure 1: Sample conversation from the MotiVAt e dataset (a) from the MDD thread, (b) from the OCD thread.
Criteria	Statistics
Total	MDD	OCD	Anxiety	PTSD
# of dialogues	7067	4046	1000	1000	1021
# of utterances	25947	16257	2461	2784	4445
# of utterance per dialogue (avg.)	3.67	4.01	2.46	2.79	4.36
# of utterance per dialogue (max.)	129	129	14	16	25
Maximum user utterance length (# of words)	3319	3319	1337	1028	2112
Maximum VA utterance length (# of words)	2869	2851	2869	1024	2116
# of unique users	2139	1060	349	323	407
# of unique words	56336	35666	14108	12135	16427
Table 1: MotiVAte dataset statistics for every mental disorder
sentiment and emotion, can help different information elicitation models in dialogues work better. Apart from these, several other work (Wei et al., 2019; Ide and Kawahara, 2021; Huo et al., 2020) that suggests using sentiment and/or emotion as an additional input in generation frameworks either during decoding or as reward to guide the models for generating responses aligned with the user's mood or feelings.
3 Motivational VA : MotiVAte Dataset
The MotiVAte dataset contains 7067 dyadic conversations with support seekers who have one of the four mental disorders: MDD, PTSD, anxiety, or OCD. Supplementary material contains descriptions of these illnesses as they appear in ICD-10. Table 1 displays the dataset statistics as well as the sample distribution amongst illnesses. Sample conversations from the dataset are shown in Figure 1. As evident from the conversations, we expect our VA to perform simple, ordinary and expected things in the form of providing comfort and assistance to the support seekers at the time of crisis and the curated dataset is full of such statements.
Data Collection. Existing mental health databases had limitations in the context of our proposed work. Some of the datasets, for example, (Choudhury et al., 2017; Yazdavar et al., 2017, 2018, 2020) were social media contents of anonymized users comprising of self-disclosures and self-expressive posts with no specific dyadic or multi-party discussions to draw on. Some of
the text-based counseling conversational datasets (Althoff et al., 2016; Dowling and Rickwood, 2014) were no longer open-sourced for research usage. Some of the open-sourced datasets such as DAIC-WOZ (Gratch et al., 2014) contained small-scale conversations. Recent support-based datasets (Sharma et al., 2020; Lahnala et al., 2021), on the other hand, featured pairs of seeker post and supporter response with no dialogic structure. Inspired by previous research, we create the MotiVAte dataset, which was acquired via a peer-to-peer support platform and is ideal for our objective. Pyschcentral<sup>2</sup> is a text based support forum where anonymous individuals can talk about their mental health problems and get help and advice from others who have had similar emotions, troubles, and grievances. It has various subforums about mental health, such as MDD, bipolar disorder, anxiety and panic attacks, schizophrenia, and so on. We gathered 10k multi-party interactions from four distinct subforums: OCD, Anxiety, PTSD and MDD. A manual assessment of the raw data confirmed that the chats were acceptable and can be utilised to develop a VA after some post-processing.
Data Preparation. The challenge next was to convert these multi-party dialogues into dyadic ones, so that they resembled a conversation between a support seeker (with a mental disability) and the VA providing mental health support. We presume that a source conversation starts with a post by a support seeker known as the poster (say). The commenters (say) are forum users who make comments on the poster's statement. The poster and the commentators engage in a multi-party conversation in order to assist the poster. We worked with one of the noted psychiatrists, who is currently working at a government-run hospital of national importance, to develop standards for changing multi-party dialogues and confirming the qual
$^{2}$ www.psychcentral.org
ity of the amended dataset. We hired three crowd-workers for the task of modifying the dialogues and trained them in an interactive session using the instructions that had been developed (Details of the training session conducted is drafted in the Supplementary material). Some of the important guidelines are as follows: (i) In the modified dialogue, the poster in the source conversation becomes the support seeker, and the comments of a specific commenter creating the longest conversational thread with the poster become the responses of the VA. From the responses of the poster and commenter, the crowd-workers were instructed to develop a turn-by-turn exchange of dialogue between the seeker and the VA, making the most of the responses from the source conversation; (ii) The VAs' response should be helpful and positive, with the goal of raising the user's morale. So, negative utterances from the commenter such as "I know, nothing can change, we have to struggle throughout" were changed to exhibit optimism and hope like "life indeed is a struggle for all, but one needs to always fight back and be strong in the face of adversities" etc; (iii) A VA cannot provide medical advise, even if the poster requests it (as evident in the source conversation), because we do not advocate that the VA can replace MHP. As a result, in such a circumstance, the utterances of the commentators providing medicinal advice were completely eliminated, while utterances such as "I suggest you to visit a doctor or a psychiatrist before resorting to such medicines" were incorporated as part of VA's response. Following these rules, a total of 7k dyadic conversations were created (The process of rejecting 3k remaining conversations along with the other guidelines and inter-annotator agreement are detailed in the Supplementary material).
Problem Definition. The problem statement involves two parts: Firstly, we aim to identify a textual on-going communication between a support seeker and the VA as the conversation progresses in order to detect and differentiate mental health conditions. For a conversation  $T$ , given a seeker utterance,  $X_{t} = (x_{t,1}, x_{t,2}, \ldots, x_{t,n})$ , a conversational context/history,  $C = (c_{1}, c_{2}, \ldots, c_{t-1})$ , the task is to assign the most appropriate mental illness tag (say  $y_{2}$ ) among a set of tags ( $Y = \{y_{1}, y_{2}, \ldots, y_{i}\}$ ), where  $i$  is the number of disorders considered). Thus, it is a multi-class classification problem. Formally, it can be represented as
:  $y = \arg\max_{y' \in Y} F(y'|X_t, C)$ , where  $F$  is the developed classifier. The subsequent or the second part involves to solve the task of generating the next textual response of the VA given the seeker utterance, its context of  $t - 1$  turns (say) and conditioned on the output  $y_k$  (say) of the mental illness identification classifier. Formally, given a seeker utterance,  $X_t = (x_{t,1}, x_{t,2}, \ldots, x_{t,n})$ , a conversational context/history,  $C = (c_1, c_2, \ldots, c_{t-1})$ , where  $c_i = (X_i, Z_i)$  and mental illness category,  $y_k$ , the task is to generate next textual response of the VA,  $Z_t = (z_{t,1}, z_{t,2}, \ldots, z_{t,n^{\prime\prime}})$ .
Summarization. While analyzing the dataset, we observed that the utterances in a dialogue have longer sequences implying longer context (also evident from Table 1). Intuitively, an effective encoding strategy needs to be employed to counter loss of information. In this regard, we first summarize each of the utterances of the individual speakers in every time-step of the dialogue to preserve the content and curate it to be concise for modeling long-term dependencies. In the absence of gold-standard summary of utterances, we obtain summaries from a state of the art summarization model named BART-large by Facebook AI (Lewis et al., 2019). For our setting, we use the BART-large model fine-tuned on the CNN/DM summarization dataset (Hermann et al., 2015) to obtain summaries of the individual utterance of the MotiVAte dataset. So, for a given utterance,  $X_{t} = (x_{t,1}, x_{t,2}, \dots, x_{t,n})$ , its corresponding summary is,  $M_{t} = (m_{t,1}, m_{t,2}, \dots, m_{t,k})$  (Evaluation of the summaries obtained is presented in the Supplementary material). Consequently, we utilize the summarised version of the dataset for developing the system (handling longer sequences as in the original dataset will be dealt as a sub-task in the future).
4.1 Mental Illness Classification (MIC) Framework
In this section, we discuss the details of the attention based classification framework.
Feature Extraction. The classification framework inputs two different kinds of features. (i) Embedding Features: To extract textual features of an utterance  $U$  having  $n_u$  number of words, the representation of each of the words,  $w_1, \ldots, w_u$ , where  $w_i \in \mathbb{R}^{d_u}$ ,  $w_i$ s are obtained from BERT (Devlin et al., 2019), where dimension,  $d_u = 768$ . (ii) Semantic Features: An examination of the dataset revealed that users who expressed their emotions
Figure 2: Architectural diagram of the proposed VA with the MIC and MI-MDG frameworks
and pains reflected their overall sentiment to some level. These details must also be recorded in order to create a more accurate sentence representation that takes into account the user's mental state. We achieved this by using the Vader Sentiment Intensity Analyzer (VSIA) to count the number of positive and negative utterances in each speaker's encoding and using them as features. From strongly negative (-1) to strongly positive (+1), this rating seeks to convey the overall affect of the entire text.
Network Architecture. Three key components make up the proposed classification network: (i) Utterance Encoders (UE): the features retrieved above for each of the speakers (here VA and support seeker) for a particular conversation are fed into UE which generate relevant speaker encodings, (ii) Dual Attention Subnetwork (DAS) that encompasses self and cross attention, (iii) Classification Layer (CL): the output channel for classification is contained in the CL.
Utterance Encoders. The embedding features produced for each of the speakers' utterances (described above) are then processed through two discrete Bi-LSTMs for a specific time-step of the conversation. For a user level view (say), the final hidden state matrix for the textual representation of the utterances is  $H_{u} \in \mathbb{R}^{n_{u} \times 2d_{l}}$ .  $d_{l}$  represents the number of hidden units in each LSTM and  $n_{u}$  is the number of utterances of the respective speaker.
Dual Attention Subnetwork. We employ a similar notion proposed by the authors of (Vaswani et al., 2017), in which attention is computed by mapping a query and a set of key-value pairs to an output. The speaker level context encodings are passed through three fully-connected layers,
each termed as queries,  $Q$  and keys,  $K$  of dimension  $d_{k} = d_{f}$  and values,  $V$  of dimension  $d_v = d_f$ . Thus, we obtain two triplets of  $(Q,K,V)$  as:  $(Q_u,K_u,V_u),(Q_v,K_v,V_v)$ . These triplets are then combined in various ways to compute attention scores for specific reasons.
Self Attention. We compute self attention (SA) for each of these speaker encoders to learn the interdependence between the current and the previous part of the same speaker's conversation. In a sense, we want to connect distinct positions of utterances in order to estimate a final representation for each speaker (Vaswani et al., 2017). Thus, the SA score for individual speaker level is calculated as:
$$
S A = \operatorname {s o f t m a x} \left(Q _ {i} K _ {i} ^ {T}\right) V _ {i} \tag {1}
$$
where  $SA\in \mathbb{R}^{n_u\times d_f}$  for  $SA_{u}$ $SA\in \mathbb{R}^{n_v\times d_f}$  for  $SA_{v}$
Cross Attention. Similarly, we compute cross attention (CA) amongst triplets of the speaker level encodings to learn interdependence between speaker queries as:
$$
C A = \operatorname {s o f t m a x} \left(Q _ {i} K _ {j} ^ {T}\right) V _ {i} \tag {2}
$$
This is done to relate different positions of the utterances of the different (cross) speakers and to identify significant contributions amongst different speakers for a particular time-step to learn optimal features for the task. Thus, we obtain two  $CA$  scores as  $CA_{uv} \in \mathbb{R}^{n_u \times d_f}$  and  $CA_{vu} \in \mathbb{R}^{n_v \times d_f}$ .
Attention Fusion. Next, we concatenate each of these computed  $SA$  and  $CA$  vectors to obtain the conversational representation as:
$$
C = \operatorname {c o n c a t} \left(C A _ {u v}, C A _ {v u}, S A _ {u}, S A _ {v}\right) \tag {3}
$$
Classification Layer. To identify one of the mental disorders, the final representation of the
ongoing discussion received from the DAS module is transmitted through a fully-connected layer, which then connects it to the output channel of the classifier consisting of output neurons.
4.2 Mental Illness conditioned Motivational Dialogue Generation (MI-MDG) Framework
In this section, we discuss the details of the proposed MI-MDG framework.