Augmenting Reddit Posts to Determine Wellness Dimensions impacting Mental Health
Year: 2023

Abstract:
Amid ongoing health crisis, there is a growing necessity to discern possible signs of Wellness Dimensions (WD)1 manifested in self-narrated text. As the distribution of WD on social media data is intrinsically imbalanced, we experiment the generative NLP models for data augmentation to enable further improvement in the prescreening task of classifying WD. To this end, we propose a simple yet effective data augmentation approach through promptbased Generative NLP models, and evaluate the ROUGE scores and syntactic/semantic similarity among existing interpretations and augmented data. Our approach with ChatGPT model surpasses all the other methods and achieves improvement over baselines such as Easy-Data Augmentation and Backtranslation. Introducing data augmentation to generate more training samples and balanced dataset, results in the improved F-score and the Matthew’s Correlation Coefficient for upto 13.11% and 15.95%, respectively.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 306–312 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: The social determinants of health (SDOH) refer to various factors present in the surroundings where individuals are born, reside, acquire knowledge, work, engage in leisure activities, practice religion, grow older, impacting a broad range of health-related outcomes, risks and quality-of-life indicators.23 A rapid expansion of research in SDOH 2030 encourages the social NLP research community to design and develop
1The concept of Wellness Dimensions is often used in holistic approaches to health, recognizing that wellbeing encompasses multiple areas of life.
2https://health.gov/healthypeople/ priority-areas/social-determinants-health
3Social Determinants of Health (SDOH) are the social and economic factors that influence an individual’s health outcomes.
computational intelligence models for enhancement of an individual’s well-being (Bompelli et al., 2021). In this work, we choose to prescreen human-writings for biomedical therapy by investigating latent indicators of wellness dimensions in Reddit posts (see illustration in Figure 1). Wellness dimensions (WD) refer to different aspects of an individual’s overall wellbeing that contribute to their physical, spiritual, social, emotional, intellectual, occupational, environmental, and financial well-being. The disturbed WD, if remains unaddressed, have adverse impact on mental health of an individual. As social media becomes integral part of our daily lives (Wang et al., 2020), studies in the past suggest that individuals tend to express their thoughts and emotions impacted by one or more wellness dimensions more easily on social media platforms as compared to during in-person sessions with clinical psychol-
306
ogists and mental healthcare (Zhang et al., 2023; Garg, 2023). We construct, annotate and observe the original (natural) composition of WD dataset as an imbalanced dataset. In this work, we augment a multi-class dataset on WD to facilitate design and development of NLP models for classifying WD impacting mental health in Reddit posts during mental health screening. Pre-screening filters are helpful in biomedical therapy by facilitating early detection of WD impacting mental health, which if left untreated may cause severe mental disorders. Dunn highlights holistic nature of wellness in 1961 as a high-level wellness, denoting a superior level of healthy living (Printz-Markó and Ivancsóné Horváth, 2017).
We reduce multiple WD to four key dimensions of well-being based on the frequency and recognition in human writings: Physical Aspect (PA), Intellectual and Vocational Aspect (IVA), Social Aspect (SA), Spiritual and Emotional Aspect (SEA) (Wickramarathne et al., 2020; Dillette et al., 2021). Our major contributions (as illustrated in Fig. 1) include (i) the applicability of generative NLP models for domain-specific data augmentation, (ii) examining the diversity among generated and original instances through semantic and syntactic similarity measure, (iii) test and validate the efficacy of data augmentation by investigating classifiers’ performance.

heading: 2 Background
text: According to Weiss (1975), sociologists put forth a theory that outlines six social needs to prevent loneliness: attachment, social integration, nurturance, reassurance of worth, sense of reliable alliance, and guidance in stressful situations (Weiss, 1975). The Self-Determination Theory (SDT)4 highlights the importance of balancing relatedness, competency, and autonomy for intrinsic motivation and genuine selfesteem, which contribute to overall well-being. Neglecting mental disturbance can escalate subclinical depression to clinical depression by activating interpersonal risks. This research seeks to examine the origins and outcomes of mental disturbance to mitigate these risks.
4https://en.wikipedia.org/wiki/ Self-determination_theory
Corpus Construction We present a new dataset with 3,092 instances and 72,813 words to identify wellness dimensions impacting mental disturbance: PA, IVA, SA, and SEA. A senior clinical psychologist, a rehabilitation councilor, and a social NLP researcher framed annotation schemes and perplexity guidelines for text annotation through pre-defined wellness dimensions. Our experts trained three postgraduate students to annotate the data based on predefined dimensions. The annotations were validated using Fleiss’ Kappa interobserver agreement, resulting in a kappa score of 74.39%. Final annotations were determined through majority voting and expert verification. The experts achieved a kappa score of 87.32% for the selection of explanatory text spans. Despite slight confusion between PA and SEA, there was a higher agreement for the selection of explanations. To facilitate future research and developments, we publicly release our dataset at Github.5
Problem Formulation: We collect and annotate Reddit data from subreddits r/depression and r/suicidewatch for the task of identifying WD and found imbalanced dataset in its natural composition, suggesting the need of data augmentation. To evaluate the effectiveness of generative NLP models for data augmentation, we frame the task of augmenting Reddit posts as a text generation problem. We compare and contrast the performance of model trained on data augmented with (i) Generative Pre-trained Transformers (GPT) (Yang et al., 2023), and (ii) conventional data augmentation approach for NLP such as Easy Data Augmentation (EDA) (Wei and Zou, 2019) and Back Translation (BT) (Sennrich et al., 2016).

heading: 3 Experiments and Evaluation
text: We first generate the data using two-fold measures: (i) traditional data augmentation methods for NLP - EDA and BT, and (ii) prompt-based Generative Pre-trained Transformer models (Ding et al., 2023). We further investigate the diversity of the generated samples in comparison to the original samples and
5https://github.com/drmuskangarg/ WellnessDimensions
fine-tune BERT language model to observe improvements in WD classification, if any. Hinged on the classification results and similarity measures, we select the best model for augmenting WD dataset.

heading: 3.1 Methods: Data Augmentation
text: We use pre-trained generative models6 (Meng et al., 2021) for this task: (i) ChatGPT models: gpt-3.5-turbo and gpt-3.5-turbo-0301, and (ii) other GPT-3 models: text-curie-001 and text-davinci-003. The original dataset consists of 3092 samples, with 740, 592, 1139 and 621 records from classes PA, IVA, SA, and SEA respectively. We first split the dataset such that we maximize the number of training samples required for each WD.7 After augmentation, the training set comprises a total of 4376 records, with an equal distribution of 1094 records per class.
Prompt Design and Parameter Setup: As shown in Figure 2, we design following prompts to produce, a) text similar to the original text (Topic and text), and b) an explanation of newly generated text (text and explanation).
While designing prompts according to OpenAI prompt design instructions8, we begin with explanation through instructions and examples or both. During the text creation, we only provide instructions to the model. As every text belongs to one of the four pre-defined WD, we provide class name as an input, for example, "Physical Aspect", hypothesizing its contribution towards contextual consciousness required for enhancing similar text generation. Furthermore, the explanation generation is developed as a few-shot learning approach (Brown et al., 2020), where we provide five text-explanation pairs as examples. The selective examples ensure the representation of all four classes and are made static for every call. We keep temperature as 0.7 to preserve the creativity/ randomness of generated text.

heading: 3.2 Method: Similarity Measures
text: First we calculate ROUGE scores <ROUGE-1, ROUGE-2 and ROUGE-L> to examine similar-
6https://platform.openai.com/docs/models 7see Appendix A for more details. 8https://platform.openai.com/docs/guides/
completion/prompt-design
ity9. Next, for semantic similarity, we calculate the embedding for each sentence through eleven pre-trained language models (Ormerod et al., 2021; Reimers et al., 2019).10 The resulting sentence embeddings11 of each original and augmented data instance were then compared using cosine similarity. Lastly, for syntactic similarity, we first parsed given sentences into syntactic trees and then mapped them into vector representations using the "en_core_web_md" English pipeline in the spaCy library12. Next, these vector representations are used to compute the similarity score between sentences. Furthermore, we compute the set overlap between the Part-of-speech (POS) tag sequences of the original and augmented sentences to determine their similarity13.
9https://pypi.org/project/rouge/ 10see Appendix B for more details. 11https://platform.openai.com/docs/guides/
embeddings/what-are-embeddings 12https://spacy.io/models/en 13https://www.nltk.org/api/nltk.tag.pos_tag. html

heading: 3.3 Classification with BERT
text: As the final evaluation, we build BERT (Kenton and Toutanova, 2019), a baseline classifier, with 6 augmented datasets and compare its performance with the BERT classifier built over original data. We used the training data in WD dataset for finetuning for 10 epochs with a batch size of 32 and a learning rate of 3e-5. To preserve the lengths of texts, we set the max_length to 256 during tokenization. We use the validation set (20% of the training set) and testing set (180 samples) to examine the efficiency and effectiveness of a classifier through F-score and Matthew’s Correlation Coefficient (MCC), respectively.

heading: 4 Results and Discussion
text: Similarity Analysis: We report three types of ROUGE scores: ROUGE-1, ROUGE-2 and ROUGE-L between the original and augmented text. The ChatGPT models show the lowest ROUGE scores, and gpt-3.5-turbo versions surpases all other augmentation methods (see Figure 3). We further examine semantic and syntactic similarities through average of all 13 models in Figure 4(a) and 4(b). We observe high diversity and low similarity with GPT based models where ChatGPT based models illustrate the least similarity. However, compared to the other GPT models, text-curie-001 shows a notably higher similarities through all the similarity models.
Classification Performances: We obtain the validation accuracy (Val-A), and testing results with precision (T-P), recall (T-R), Fscore (T-F), accuracy (T-A) and MCC value (TMCC) with experimental results for evaluation (see Table 1). Even though we keep the testing dataset to be a small chunk of 180 samples, we observe significant difference in the results in training on the original imbalanced dataset and augmented dataset. The gpt-3.5-turbo model over testing dataset outperforms all the baseline models, specifically the original dataset by 2.99% F-score and 1.47% Accuracy followed by the second best model: gpt-3.5-turbo-0301. Moreover, compared to the best traditional augmentation method (BT), the top ChatGPT model shows 7.81% improvement in testing accuracy. Notably, the datasets from text-curie001 and EDA which gained higher similarity values have shown lowest performance on all classification measurements.
We further examine the MCC values to determine the effectiveness of the classifier (Boughorbel et al., 2017). MCC values vary between -1 and 1 such that values closer to 0 and 1 suggest increased randomness and perfect prediction towards decision making correspondingly. We found 15.95% improvement in MCC score when model is trained on augmented training samples with gpt-3.5-turbo model. Overall, the augmented text with lowest ROUGE scores, syntactic and semantic similarities showed the highest classification performance on BERT.
Moreover, the following Table 2 compares the class-vise classification performance between the original and M1 (best performed dataset) datasets. We notice a significant improvement in all the measurements of all the classes after augmenting data. Additionally, compared to other classes, the IVA- class with the least
allMin
iLM -L6
-v2
allMin
iLM -L1
2-v 2
allmp
ne t-b
ase -v2
pa rap
hra se-
mp ne
t-b ase
-v2
pa rap
hra se-
alb ert
-sm all-
v2
qu ora
-di stil
be rt-b
ase
alldis
tilr ob
ert a-v
1
pa rap
hra se-
dis tilr
ob ert
a-b ase
-v1
ms ma
rco -ro
be rta
-ba se-
v3
tex t-e
mb ed
din g-a
da -00
2
tex t-s
im ilar
ity -da
vin ci-0
01
spa Cy
_sy nta
cti c_s
im ilar
ity
PoS Tag
_sy nta
cti c_s
im ilar
ity
Similarity Models
0.4
0.5
0.6
0.7
0.8
0.9
Av er
ag e
Se m
an tic
/S yn
ta ct
ic Si
m ila
rit y
0.42 0.44 0.44
0.48
0.44
0.56
0.44 0.45
0.35
0.85
0.81 0.80
0.54
0.43 0.45 0.45
0.48
0.45
0.56
0.45 0.45
0.36
0.85
0.81 0.80
0.54
0.68 0.70 0.70
0.74
0.71
0.76
0.69 0.67 0.67
0.91
0.87 0.86
0.61
0.49 0.52
0.54
0.58
0.52
0.63
0.54
0.48
0.44
0.86
0.82 0.81
0.51
0.73 0.72
0.68
0.76 0.75
0.79
0.70
0.74
0.65
0.92 0.90
0.96
0.80
0.89 0.90 0.88 0.91 0.89 0.92
0.87 0.88 0.87
0.97 0.96 0.94
0.81
gpt-3.5-turbo gpt-3.5-turbo-0301 text-curie-001 text-davinci-003 EDA BT
allMin
iLM -L6
-v2
allMin
iLM -L1
2-v 2
allmp
ne t-b
ase -v2
pa rap
hra se-
mp ne
t-b ase
-v2
pa rap
hra se-
alb ert
-sm all-
v2
qu ora
-di stil
be rt-b
ase
alldis
tilr ob
ert a-v
1
pa rap
hra se-
dis tilr
ob ert
a-b ase
-v1
ms ma
rco -ro
be rta
-ba se-
v3
tex t-e
mb ed
din g-a
da -00
2
tex t-s
im ilar
ity -da
vin ci-0
01
spa Cy
_sy nta
cti c_s
im ilar
ity
PoS Tag
_sy nta
cti c_s
im ilar
ity
Similarity Models
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Av er
ag e
Se m
an tic
/S yn
ta ct
ic Si
m ila
rit y
0.30 0.32 0.32
0.35 0.33
0.48
0.29 0.27
0.22
0.81
0.75
0.52
0.28
0.31 0.33 0.33
0.36 0.34
0.49
0.30 0.27
0.23
0.81
0.75
0.52
0.27
0.49 0.49 0.48 0.51 0.49
0.60
0.47 0.45
0.40
0.86 0.82
0.59
0.41
0.35 0.36 0.37
0.40
0.36
0.51
0.35
0.31
0.24
0.82 0.79
0.51
0.29
0.76 0.76 0.72
0.78 0.77
0.82
0.72 0.74 0.70
0.93 0.92 0.91
0.75
0.83 0.83 0.80
0.84 0.82
0.88
0.79 0.78 0.78
0.93 0.93
0.80
0.67
gpt-3.5-turbo gpt-3.5-turbo-0301 text-curie-001 text-davinci-003 EDA BT
Figure 4: (a): Average Textual Similarity among Original and Augmented Text. (b): Average Textual Similarity among Original and Augmented Explanations.
number of original samples shows a significantly higher improvement in the number of correctly classified samples.

heading: 5 Conclusion and Future Scope
text: In this work, we augment the Reddit posts for a four-class classification problem of determining Wellness Dimensions impacting mental health. The GPT models are outperforming in terms of generating diverse text by preserving the context of the corresponding original text. In future, we plan to experiment with different parameter settings and prompts for generating datasets and develop improved classifiers to determine WD in a well balanced dataset. Furthermore, we will evaluate the classification performance of short explanation text we generated in this dataset.
Ethics and Broader Impact
The data used in this study is obtained from Reddit, a platform designed for anonymous posting, and the user IDs have been anonymized. Furthermore, all sample posts displayed in this study have been obfuscated, paraphrased, and anonymized to protect user privacy and prevent any misuse. As annotation is subjective in nature, we acknowledge that there may be some biases present in our gold-labeled data and the distribution of labels in Wellness Dimensions dataset. We urge researchers to be mindful of the potential risks associated with WD dataset based on personal textual information. To prevent this, human intervention by a moderator is necessary. We acknowledge that we do not release user’s metadata and the augmented samples further increase the privacy. The dataset and the source code required to replicate the baseline results can be accessed at Github.14

heading: Acknowledgement
text: We express our gratitude to Veena Krishnan, a senior clinical psychologist, and Ruchi Joshi, a rehabilitation counselor, for their unwavering support throughout the project. This project was partially supported by NIH R01
14https://github.com/drmuskangarg/ WellnessDimensions
AG068007. This project is funded by NSERC Discovery Grant (RGPIN-2017-05377), held by Vijay Mago, Department of Computer Science, Lakehead University, Canada.

heading: A Training and Testing Split
text: Consider the data containing D documents representing a collection of Reddit posts {D =
d1, d2, ..., dn} where n = 3092. For each document di, there exist a tuple representing < Ei, Ci > where Ei is text-span/ explanation and Ci is the aspect class for ith instance. Thus, the original WD dataset consists of three columns for 3092 samples: < Di, Ei, Ci >. The aspect class Ci ∈ α where α =[PA, IVA, SA, SEA] and the composition of original dataset contains imbalanced distribution of aspect classes (see Table 4). The number of samples for every class α[j] where 1 ≤ j ≤ 4 suggests the need of data augmentation to facilitate development of NLP models over balanced dataset. To this end, we propose the algorithm - Required annotation count to decide the number of samples that needs to generated for each WD. Given an input α as a list of the number of text samples for different WD (PA, IVA, SA, SEA) where PA, IVA, SA, SEA defines the count of instances for each class.
As such, our goal is to achieve a balanced dataset by obtaining 1094 samples for each WD, resulting in 1094 ∗ 4 = 4376 data samples. We observe that all the samples for IVA class must be augmented while no augmentation is required for SA class.

heading: B Semantic Similarity Models
text: We use 11 different semantic similarity models as shown in Table 3. Sentence Transformers are a set of state-of-the-art language models implemented in Python for generating text embeddings. The two different GPT-3 models used for this task accessed the embeddings API endpoint.
Algorithm 1: Required Augmentation Count Result: Return AS=[] // AS: augmented sample Input: α : [PA, IVA, SA, SEA] Set: β= [], R, Red=[], RC=[] Set: minvalue := min(α) // Get the
record count of minority class for j in count(α) do
β[j] := max(α) − α[j] /* For each class, get the
record count difference from majority class */
end // Estimate the size of the test
set R = minvalue − max(β[j]) for j in count(α) do
/* For each class, calculate the Reduction Percentage (percentage of reduction after separating the testing set) */
Red[j] = Rα[j] ∗ 100 /* For each class, calculate
the Reduced Composition (number of training records before augmentation) */
RC[j] = α[j] − Red[j] end /* Get the maximum number of
records per class for augmentation */

heading: Future Work
text: In future, we plan to experiment with different parameter settings and prompts for generating datasets and develop improved classifiers to determine WD in a well balanced dataset. Furthermore, we will evaluate the classification performance of short explanation text we generated in this dataset