SlowGuess/NAACL_2022_OCR huggingface dataset:
Text Generation. For a long time, Sequence-to-Sequence (Seq2Seq) (Sutskever et al., 2014) and Hierarchical Encoder Decoder (HRED) models (Serban et al., 2017, 2016) were being used for different text generation tasks. However, the main issue with RNN based model is its inability to provide parallelization while processing and is incapable of preserving context at the encoder side for longer sequences, similar to our case as explained above. To counter this, we use the DialogGPT (Zhang et al., 2020) model for our task. DialogGPT is based on the GPT-2 model from OpenAI (Radford et al., 2019), pre-trained on Reddit conversations. We fine-tune the DialogGPT-small model on the summarised version of the MotiVAte dataset. For a given dialogue, we first concatenate the dialog turns till the  $k$ th seeker-VA response pair along with the context available and speaker identifier into a long text,  $m_{1,k}, \ldots, m_{(N-1),k}$  (N-1 is the sequence length), ended by the end-of-text token. To generate responses conditioned on the mental illness of the support seeker, we also concatenate the predicted mental illness category (for the  $k$ th seeker utterance from the MIC framework),  $y_i$  (say) as a mental state identifier after the  $k$ th seeker utterance in the sequence, making the sequence length as  $N$ . With the dialogue history,  $S = m_{1,k}, \ldots, m_{l,k}$  and the VA utterance (ground truth response) as  $T = m_{l+2,k}, \ldots, m_{N,k}$ , the conditional probability  $P(T|S)$  can be written as:
$$
p (T | S) = \prod_ {n = l + 2} ^ {N} p \left(m _ {n} | m _ {1}, \dots , m _ {n - 1}, y _ {i}\right) \tag {4}
$$
To generate semantically acceptable responses, the DialogGPT model is first fine-tuned with the negative log likelihood, i.e., the maximum likelihood estimation (MLE) objective function in a supervised way. This trained model is later initialized to produce motivational and optimistic responses by the VA (explained below).
Reinforcement Learning (RL) based Training. The sequence of tokens in an utterance can
be considered as actions chosen by the DialogGPT model based on a policy it has learned. The model is then tweaked using the MLE parameters to learn a policy that maximises long-term future rewards (Li et al., 2016). The elements of RL based training are addressed below.
State and Action. The state is similar to the input of the DialogGPT model, i.e., context comprising of history and the  $k$ th seeker utterance along with the speaker and mental illness identifiers (explained above),  $[S(H_{h},H_{s},H_{y})]$  where  $h$ ,  $s$  and  $y$  represent history tokens, speaker and mental illness category tokens, respectively. The action  $a$ , is the VA response to be generated in the  $k$ th time-step, i.e.,  $Z_{k}$ . Because the sequence generated might be of any length, the action space is unlimited. As a result, the policy,  $\Pi(Z_{k}|S(H_{h},H_{s},H_{y}))$  is defined by its parameters and is based on learning how to map states to actions.
Reward. Here, we discuss the task-specific reward functions,  $r$ , used to evaluate the predicted output  $Z'$  against the true output.
- BLEU Metric Score  $(r_1)$ : This metric ensures  $n$ -gram content similarity (1-gram here) between the predicted and the true output.
- ROUGE-L Metric Score  $(r_2)$ : This metric ensures the matching of the longest common subsequence between the predicted and the true output.
- Sentiment Score  $(r_3)$ : For the VA to be motivational and optimism inducing, the generated response should exhibit positive sentiment. Since, emotion focuses on a deeper analysis of human sensitivities and is based on a wide spectrum of moods, sentiment provides an overall impression or view people get from consuming a piece of content. So, we quantify optimism with respect to being positively-oriented. The VA should always work towards uplifting the mood of the user, provide reliable suggestions which are positively-oriented. The VA should not oblige with the negative mindset of the support seeker barred from hope and motivation from moving forward in life. This will ensure that the sentiment state of the generated output by the VA is consistent with the true output. Thus, the reward is:
$$
r _ {3} = \left\{ \begin{array}{l l} 1, & \text {i f} S C \left(Z ^ {\prime}\right) = + v e \\ 1 - s s, & \text {i f} S C \left(Z ^ {\prime}\right) = - v e \end{array} \right. \tag {5}
$$
where  $SC$  is the pre-trained distillBERT based uncased model (Sanh et al., 2019), fine-tuned on the SST-2 English dataset for the sentiment classification task.  $ss$  is the sentiment score obtained from the classifier.
Thus, the final reward (R) is the weighted average of all the above terms as given below:
$$
R = \left(r _ {1} * (1 - \alpha - \beta) + r _ {2} * \alpha + r _ {3} * \beta\right) / 3 \tag {6}
$$
where  $\alpha$  and  $\beta$  are parameters of the model. Policy Gradient algorithm (Zaremba and Sutskever, 2015) is used to optimize these rewards. The policy model  $\Pi$  is initialized using the fine-tuned DialoGPT model (using the MLE objective function). So, the final loss back-propagated to the DialoGPT model is a combined objective function as:
$$
L _ {c o m b} = \eta L _ {R L} + (1 - \eta) L _ {M L E} \tag {7}
$$
where  $L_{RL}$  and  $L_{MLE}$  are the losses calculated from the RL and MLE objective, respectively.
Since the MotiVAte dataset is imbalanced for different mental illness categories, we sample  $50\%$  of the dialogue from MDD subforum along with other categories to be utilized in the MIC framework. Thus, the mental illness classification module is trained on 5067 conversations, out of which  $70\%$  of the dialogues were used for training and remaining were utilized as test set. To encode different speaker utterances in the MIC framework, a 300 dimensional Bi-LSTM layer was used.  $d_{f}$  is a dense layer of 100 dimension. The four mental health categories are represented by 4 neurons in the output channels. In the final experiment, a learning rate of 0.01, textitCategorical crossentropy loss function, and Adam optimizer were utilised. All of these parameters were chosen following a thorough sensitivity study.
For training the MI-MDG framework, MIC model is used as a pre-trained classifier providing additional input. For the MI-MDG model, we decode using default temperature and top-k values of the DialoGPT model. Adam optimizer is used to train the model. A learning rate of 0.00004 was found to be optimum. Standard measures, such as the BLEU-1 score (Papineni et al., 2002), perplexity, ROUGE-L score (Lin, 2004) and embedding based metric (Serban et al., 2017) are used to automatically evaluate generation-based models. Three independent human users were recruited to score the quality of 250 simulated conversational responses based on these metrics: (i) Fluency: The VA's generated responses should be grammatically and syntactically acceptable; (ii) Adaptability: An effective VA should generate responses based on the current trajectory of the conversation, i.e.,
Model	k=1	k=2	k=t
Acc.	F1-score	Acc.	F1-score	Acc.	F1-score
CNN (GloVe) (NA+NSenti)	40.82	0.2830	43.75	0.2982	52.46	0.4008
Bi-GRU (GloVe) (NA+NSenti)	41.21	0.2835	44.86	0.3130	53.72	0.4052
Bi-LSTM (GloVe) (NA+NSenti)	41.75	0.2883	45.81	0.3142	56.47	0.4135
BERT+CNN (NA+NSenti)	44.23	0.3238	46.32	0.3315	55.83	0.4112
BERT+CNN+Senti (NA)	44.85	0.3266	46.90	0.3357	58.54	0.5390
BERT+Bi-GRU (NA+NSenti)	41.95	0.3547	45.60	0.3531	56.02	0.4211
BERT+Bi-GRU+Senti (NA)	43.61	0.3624	47.84	0.3715	57.33	0.4677
BERT+Bi-LSTM (NA+NSenti)	44.53	0.3340	46.58	0.3375	56.68	0.4281
BERT+Bi-LSTM+Senti (NA)	46.73	0.3768	47.80	0.3762	59.21	0.5427
BERT+Bi-LSTM+Senti (only SA)	48.73	0.3986	48.23	0.3847	59.61	0.5436
BERT+Bi-LSTM+Senti (only CA)	-	-	48.80	0.3888	59.43	0.5411
MIC Model (BERT+Bi-LSTM+DAS+Senti)	48.73	0.5035	51.33	0.4044	60.49	0.5640
Table 2: Results of all the baselines and the MIC framework. NA represents models without attention (no DAS), NSenti represents models without sentiment score features,  $k$  represents the  $k$  th seeker utterance along with the available context.
what is now being discussed; and (iii) Motivational: The response generated by the VA should be positively-oriented imparting hope and motivation. Finally, we report the average of the human rated scores across different users.
A series of experiments were carried out in order to evaluate the proposed framework.
Evaluation of Summaries. To analyse the quality of the summaries obtained from the state-of-the-art BART-large model, we presented 100 conversations to three human users from authors affiliation to rate the quality of the summaries on a scale of 1 (worst) to 5 (best) based on two metrics, namely, fluency : to ensure that the obtained summary at each time-step of the conversation is syntactically or grammatically correct; content preservation : to ensure that the content of an utterance in the conversation is preserved in the summarised version. We report the average of the human rated scores across different users. Based on the human evaluation, for fluency, we obtained an average score of 4.1, whereas for content preservation, we observed an average score of 3.65.
MIC Framework. Experiments were conducted in three different set-up as: for first  $k$  seeker utterances, where  $k = 1,2$ , and  $t$  (here  $t$  represents the last seeker utterance), along with the available context of the dialogue in order to analyse the competence of the MIC model in assisting the VA as the dialogue progresses. Table 2 summarises the
(b)
Figure 3: (a) Confusion matrix of the MIC model, (b) Human evaluation results of the baselines and the MI-MDG framework, (c) Sentiment polarity results of the generated VA utterances of different models using VSIA
findings of the proposed MIC model, as well as a detailed ablation analysis of its various components. As can be seen, textual encoding based on Bi-LSTM yields the best results in terms of several classification criteria. Also, the conversational level  $(k = t)$  models performed consistently better with different encoding strategies. This advantage is self-evident, as the conversational level models, unlike the other two set-up, have the complete dialogue at their disposal to exploit and learn from. As visible, BERT based embedding features attained better results in all combinations as compared to the GloVe embeddings which is in conformity with the existing literature. The addition of semantic information, such as utterance wise sentiment polarity, improved the models' performance consistently across all model combinations. This demonstrates that the user's sentiment is crucial in determining its mental state. We have also demonstrated the importance of different attentions used for the best performing model, i.e.,  $BERT + Bi - LSTM + Senti$ . The results show that each of these factors aided the proposed MIC framework's performance significantly. The detailed results of the MIC framework and the baseline models in terms of precision and recall is reported in the Appendix section. Welch's t-test (Welch, 1947) at  $5\%$  significance level was conducted to ensure that all of the presented results are statistically significant.
We as well report the confusion matrix of the proposed model for the conversational level ( $k = t$ ) set-up to examine the model's performance in depth and understand its limits in Figure 3a. As can be seen, there was a lot of confusion between MDD and anxiety pairs. The model is limited in its ability to distinguish between these two illnesses at a finer level. Even though people experience these in various ways, they use similar or overlapping terminology to convey their symptoms. The fine-grained
Model(s)	Automatic Evaluation
Embedding	PPL	BLEU	ROUGE-L
Average	Extrema	Greedy
SEQ2SEQ (no MIC+RL)	0.592	0.306	0.392	52.25	0.066	0.050
HRED (no MIC+RL)	0.605	0.301	0.383	65.60	0.069	0.070
DialoGPT (no MIC+RL)	0.697	0.312	0.405	66.82	0.085	0.087
SEQ2SEQ (no RL)	0.610	0.314	0.403	53.81	0.071	0.059
HRED (no RL)	0.681	0.327	0.418	67.20	0.076	0.077
DialoGPT (no RL)	0.702	0.357	0.432	68.03	0.093	0.094
DialoGPT+RL(r1)	0.758	0.374	0.481	69.12	0.118	0.112
DialoGPT+RL(r2)	0.767	0.376	0.488	55.70	0.123	0.115
DialoGPT+RL(r3)	0.751	0.370	0.473	60.34	0.108	0.109
DialoGPT+RL(r1+r2)	0.767	0.375	0.488	59.16	0.127	0.114
DialoGPT+RL(r2+r3)	0.769	0.378	0.491	59.83	0.128	0.116
DialoGPT+RL(r1+r3)	0.767	0.377	0.489	60.20	0.129	0.114
MI-MDG (DialoGPT + RL(r1+r2+r3))	0.769	0.375	0.492	54.27	0.132	0.117
Table 3: Automatic evaluation results of the baselines and the MI-MDG framework. no MIC represents models trained without MIC output, no RL represents model trained without RL objective.
characteristics that identify different illnesses in terms of text must be investigated in depth and discovered, and this will be the subject of future research.
MI-MDG Framework. As the task of motivational response generation has not been studied previously, we compare it to baseline approaches from related tasks such as dialogue generation which are SEQ2SEQ, HRED and DialogGPT and its varying combinations. Table 3 shows the automatic evaluation results of different baselines and MI-MDG framework. All of the fine-tuned DialogGPT-based models outperformed the classic SEQ2SEQ and HRED models, as shown in the table. This shows that long-term assimilation of memory for a particular utterance and across the dialogue was not appropriately learnt by the traditional SEQ2SEQ and HRED models due to their inability to capture context for longer sequences. The models trained to optimize long-term rewards produced better results in comparison to DialogGPT without RL training. This suggests that the RL objective, rather than simply learning to be accurate at the token level, helped generate affirmative responses compatible with the context. This validates the fact that the proposed MI-MDG framework is capable of generating over
Seeker Utterance	Model	Generated Response
I know i have no control, this could have happened to anyone. and when the crisis came i did everything i could. but... i nearly failed, i almost did not make it. and every time i put my little girl to bed i wonder if this will be the last time i see her. it is just the stupid ptsd, anxiety and depression are having a field day.	DialoGPT (no MIC+RL)	it's ok. it's normal. you did the best you could and as you said
DialoGPT (with MIC output)	i'm sorry you are having such a tough time. please don't lose hope. you are a great, caring person.
MI-MDG	you have done everything you can to take care of your child and keep yourself busy. i know this is just perception but sometimes it helps when you focus on something simple that satisfies your mind
In the fall of 2010, i lost a job and my depression became unmanageable. the level of distress i go in and out of now is beyond anything i have ever experienced in past years. i've always had social anxiety and depression are partnered up like a tag team holding me down.	DialoGPT (no MIC+RL)	i'm so sorry to hear that you lost your job. that is awful. i'm glad you have your degree and you are working at a library
DialoGPT (with MIC output)	the first step in dealing with depression is to get in touch with a therapist. you can see your doctor for the same, and they can decide if the meds need to be put right.
MI-MDG	the voices you are describing are the voices of the past, and you can see through them. if you can afford the therapy, then please get in touch with a therapist nearby. you are not alone
Table 4: Examples of responses generated by MI-MDG and the baseline models
all better responses. In Figure 3b, we report the results of the MI-MDG and baselines models during the human evaluation phase. As evident, the MI-MDG framework attained the highest average fluency, adaptability and motivational scores of 3.9, 2.63 and 3.82 respectively. However, all the models generated moderate replies consistent with the context, thus, demonstrating the need to address longer context/sequences more effectively. We present few examples of generated responses from the MI-MDG and baseline model in Table 4. As evident, the baseline DialogGPT model without any MIC input or RL training generated generic responses, devoid of motivation and unaware of seeker's mental state. Whereas the MI-MDG framework learnt a fair trade-off between being consistent with the seeker's mental state and providing optimism.
Additionally, to analyse whether the responses generated are positively-oriented, we report the sentiment polarities of the generated VA utterances for different models using an existing sentiment analyzer, namely, VSIA. The results of the same is shown in Figure 3a. As visible, all the models consistently generated positive sentences, more so for MI-MDG model and for all the baseline models which are trained with sentiment based RL objective. This shows that the addition of sentiment based RL objective aided the model's capability to generate positive-oriented responses of the VA. A thorough qualitative analysis uncovered several common errors made by the MI-MDG framework. In few cases, the model kept on repeating phrases from the ground truth as "glad that you are busy keep busy keep busy and do better". In some instances, responses were mostly generic (without optimism and hope imparting expressions) and unaligned with the mental state of the seeker such as for anxiety, OCD due to their fewer representation in the dataset. Several efforts are being undertaken to increase the scale of the conversations in the MotiVAte dataset after clarifying ambiguities from the
rejected modified conversations in the future (refer to Appendix).
Online mental health support platforms that make use of peer supporters suffers from the biggest challenge of effectively training or scaffolding the peer supporters. In this research, we use AI to propose a virtual assistant (VA) to provide support seekers with comfort and mental health support. As a first step, we created the MotiVAte dataset, which contains dyadic conversations collected from a peerto-peer support network. We mold this system as a combination of two mechanisms: (i) Mental Illness Classification (MIC) Framework: a dual attention classifier that outputs the mental disorder category based on the ongoing dialog between the support seeker and the VA; and (ii) Mental Illness conditioned Motivational Dialogue Generation (MI-MDG) Framework: a sentiment driven RL based motivational response generator conditioned on the mental state of the seeker. Empirical results, both quantitative and qualitative validate the efficacy of the proposed approach. We surmise that this preliminary step will lead to promising direction for developing computational models to assist peer mental health support seekers and allow researchers to extend works on mental-health which is really the need of the hour.
Author, Dr. Sriparna Saha, acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for conducting this research.
Privacy and Ethical Concerns. The use of online posts in health forums for psychiatric research
presents a number of ethical questions about user privacy that must be addressed (Valdez and Keim-Malpass, 2019; Hovy and Spruit, 2016). Following the ethical guidelines established in previous research on various web-based platforms (Benton et al., 2017), we created our dataset using only publicly available discussions without using any personal profile information. Before presenting the data to the annotators, we manually anonymized the profile and removed any disclosure of personal information (if any). Despite the fact that the chats gathered from the online health forum were anonymized by their policy, the annotators pledged not to contact or deanonymize any of the users or share the data with others. This paper makes no therapy recommendations or clinical diagnostic claims. All the copyrights of the data belong to psychcentral.org. Refer to the supplementary section for more details. We also acknowledge that in designing computational models for mental health support, there is a risk that responses trying to aid can have the opposite effect, which can be lethal resulting in self-harm. Thus, risk mitigation steps are appropriate in this context. We stress on the fact that the system does not intend to make any clinical diagnosis or treatment of the disorder. It focuses on distinguishing mental state of the seekers based on semantic and linguistic evidence for the VA to learn a generation policy. In such cases, even if the mental disorder is mis-classified, the VA is focused on providing comfort and motivational support to the seekers. This is perfectly benign.
Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. Trans. Assoc. Comput. Linguistics, 4:463-476.
Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 94-102.
Munmun De Choudhury, Sanket S. Sharma, Tomaz Logar, Wouter Eekhout, and René Clausen Nielsen. 2017. Gender and cross-cultural differences in social media disclosures of mental illness. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2017, Portland, OR, USA, February 25 - March 1, 2017, pages 353-369. ACM.
Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on reddit: Self-disclosure, social support, and anonymity. In *Eighth international AAAI conference on weblogs and social media*.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Mitchell Dowling and Debra Rickwood. 2014. Investigating individual online synchronous chat counselling processes and treatment outcomes for young people. Advances in Mental Health, 12(3):216-224.
Mitchell Dowling and Debra Rickwood. 2016. Exploring hope and expectations in the youth mental health online counselling environment. Computers in Human Behavior, 55:62-68.
Robert Elliott, Arthur C Bohart, Jeanne C Watson, and David Murphy. 2018. Therapist empathy and client outcome: An updated meta-analysis. *Psychotherapy*, 55(4):399.
Gunther Eysenbach, John Powell, Marina Englesakis, Carlos Rizo, and Anita Stern. 2004. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. *Bmj*, 328(7449):1166.
Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): a randomized controlled trial. *JMIR mental health*, 4(2):e19.
Elizabeth A Gage-Bouchard, Susan LaValley, Molli Warunek, Lynda Kwon Beaupin, and Michelle Mollica. 2018. Is cancer information exchanged on social media scientifically accurate? Journal of Cancer Education, 33(6):1328-1332.
Manas Gaur, Ugur Kursuncu, Amanuel Alambo, Amit P. Sheth, Raminta Daniulaityte, Krishnaprasad Thirunarayan, and Jyotishman Pathak. 2018. "let me tell you about your mental health!": Contextualized classification of reddit posts to DSM-5 for web-based intervention. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 753-762. ACM.
Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al. 2014. The distress analysis interview corpus of human and computer interviews. In LREC, pages 3123-3128.