SlowGuess/NAACL_2021_OCR huggingface dataset:
Mina Valizadeh* and Pardis Ranjbar-Noiey* and Cornelia Caragea and Natalie Parde
{mvaliz2, pranjb3, cornelia, parde}@uic.edu
Self-disclosure in online health conversations may offer a host of benefits, including earlier detection and treatment of medical issues that may have otherwise gone unaddressed. However, research analyzing medical self-disclosure in online communities is limited. We address this shortcoming by introducing a new dataset of health-related posts collected from online social platforms, categorized into three groups (NO SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and CLEAR SELF-DISCLOSURE) with high inter-annotator agreement  $(\kappa = 0.88)$ . We make this data available to the research community. We also release a predictive model trained on this dataset that achieves an accuracy of  $81.02\%$ , establishing a strong performance benchmark for this task.
Self-disclosure is a communicative act that helps people develop close relationships (Altman and Taylor, 1973) through reciprocal sharing of personal information, promoting maintenance of trust and security (Bruss and Hill, 2010). It is defined as the "process of making the self known to others" (Joinson and Paine, 2007), often by sharing one's personal thoughts, opinions, or experiences. For example:
- When I was 19 years old, I met a man on the internet. He was 21 years old, 2 years older than me.
- My name is Amy and I live in Australia.
- I have suffered from migraines for three years.
In addition to facilitating social bonds, self-disclosure in general produces a wide variety of health benefits and plays a critical role in successful treatment of many physical and psychological
health issues (Ellis and Cromby, 2012). The revelation of private and sensitive information is more widespread online than in face-to-face interactions (Joinson, 2001; Tidwell and Walther, 2002; Wang et al., 2016), perhaps due to the anonymity that online platforms provide, or the ability to avoid the face-to-face stigma of some uncomfortable topics. The benefits of medical self-disclosure (i.e., disclosing symptoms, diagnoses, or other information specifically related to mental or physical health issues) in online settings may be particularly valuable from a clinical perspective, enabling earlier detection and treatment of medical issues that may have otherwise gone unaddressed (Pennebaker and Chung, 2007; Joinson, 2001). However, medical self-disclosure has been under-explored in prior computational work. We set out to address that limitation, making several key contributions.
First, we establish the novel task of medical self-disclosure detection, and create a 6,639-instance dataset comprised of public online social posts covering a wide range of mental and physical health issues, annotated with graded (NO SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and CLEAR SELF-DISCLOSURE) labels. We release this dataset to the research community to facilitate easy replication of our work, as well as rapid entry to this new task by others. Next, we compare a suite of classical machine learning and neural network approaches (including LSTM-, CNN-, and Transformer-based models) for this task, finding that neural approaches typically outperform classical machine learning models. Our highest-performing model, a BERT-based model fine-tuned for the medical self-disclosure task, achieves an accuracy of  $81.02\%$ , establishing a strong performance benchmark for this novel task.
Finally, we find that our highest-performing model outperforms the best existing (general) categorical self-disclosure model (Balani and De Choudhury, 2015), retrained on our new med
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4398-4408 June 6-11, 2021. ©2021 Association for Computational Linguistics
ical self-disclosure dataset and fine-tuned for this task, by relative percentage increases of  $41.81\%$ ,  $32.63\%$ ,  $66.60\%$ , and  $49.76\%$  for accuracy, precision, recall, and F1-measure, respectively. This provides empirical support that detecting medical self-disclosure is a distinct task with unique linguistic nuances, making it impractical to simply apply existing non-medical self-disclosure models to the medical domain with expectations of similarly high performance. In the long term, it is our hope that high-performing medical self-disclosure models can be deployed in clinical settings to support overburdened healthcare workers in understanding, diagnosing, and treating patients' health issues.
Self-disclosure detection has been the focus of prior work in psychology (Meleshko and Alden, 1993; Bridges, 2001; Meissner, 2002) and computer science (Bak et al., 2012; Walton and Rice, 2013; Balani and De Choudhury, 2015). However, research examining self-disclosure in online health discourse specifically has been limited. Existing work in this domain shows that detecting self-disclosure in the areas of health and wellness can be beneficial (Pennebaker and Chung, 2007), with patients often preferring to engage in interviews with computers rather than humans and also providing more candid and honest answers to computers (Joinson, 2001). Thus, detecting illness may be an easier process when taking into account patients' virtual disclosures (Ferriter, 1993; Greist et al., 1973). In fact, Coppersmith et al. (2015) relied on self-reported diagnosis when examining linguistic trends in a wide range of mental health conditions on Twitter.<sup>1</sup>
Most computational work on self-disclosure detection has taken place in the general domain, and specifically on tweets. Bak et al. (2012) presented a computational framework for automatically detecting self-disclosure using text mining techniques applied to Twitter conversations, and Walton and Rice (2013) investigated the roles of gender and social identities and their influences on self-disclosure on Twitter by adult users. Outside of Twitter, Umar et al. (2019) also focused on detecting self-disclosure in news commentaries using dependency parsing and named entity recognition. While these studies involve social posts, they do not specifically focus on health.
Balani and De Choudhury (2015) presented a simple neural network with three classes (No SD, Low SD, and HIGH SD) to predict self-disclosure of mental wellness in Reddit $^{2}$  posts. Their highest-performing approach, a perceptron-based model, achieved an accuracy of  $78.4\%$ . Balani and De Choudhury's work is the closest existing work to ours; however, although mental wellness may be a significant interest when identifying self-disclosure in health domains, limiting work to this precludes other critical health concerns such as psychosomatic (Karasu, 1979; Kellner, 1975) or physical ailments.
We address the limitations of prior work in automated self-disclosure detection by including an extensive range of mental and physical health concerns in our dataset. Like Balani and De Choudhury (2015), we consider three self-disclosure categories (in contrast to, e.g., the two classes employed by Umar et al. (2019)). This facilitates a more precise prediction, and focusing on medical self-disclosure specifically helps to (a) validate the distinction between medical and other types of self-disclosure when building automated models for the task, and (b) develop techniques attuned to the latter.
There are currently no publicly-available medical self-disclosure datasets; thus, a key contribution in this work lies in the creation of such a resource. We downloaded publicly-available English-language posts from randomly-selected forums on patient.info, as well as a random selection of public posts from other popular online platforms (Reddit, Twitter, and Facebook) to avoid overfitting models to site-specific stylistic trends rather than characteristics more closely linked to the presence of medical self-disclosure. We selected patient.info as our primary data source since it is a popular online forum that is well-respected among users from different backgrounds (Lewy, 2013), and it offers publicly available posts on a myriad of
$^{2}$ https://www.reddit.com
3https://patient.info, an online resource that provides information on health, disease, and other medical topics.
4https://www.facebook.com
As the focus in this work is on detecting self-disclosure in health-related posts, most instances in our dataset  $(88.1\%)$  are from patient.info. The rest of the instances are approximately distributed as follows:  $7.1\%$  from Reddit,  $3.3\%$  from Twitter, and  $1.5\%$  from Facebook.
https://twitter.com
No SD	Possible SD	Clear SD
Kappa (κ)	0.9780	0.6829	0.9840
Table 1: Averaged per-class kappa scores.
general and specific mental and physical health concerns. We randomly sampled these posts to avoid learning too strong of a reliance on disease-specific characteristics (e.g., disclosures about COVID-19 specifically). For posts not from patient.info, we scraped data using keywords and hashtags corresponding to frequent unigrams in the patient.info posts that were indicative of medical concerns (e.g., "depression," "sick," and "nausea") and purposely included expressions pertaining to both medical and non-medical senses of those words. This discouraged subsequent models from blindly associating certain keywords with medical self-disclosure. For the Reddit data, no specific subreddits were targeted.
We define instances, or posts, as complete written utterances submitted by users of the respective data sources. In longer source samples, such as those spanning multiple paragraphs on Reddit, Facebook, or patient.info, paragraphs were considered complete utterances. Long samples were thus segmented at the paragraph level, resulting in posts that were approximately equivalent in length to tweets (segmented posts had an average length of 41 tokens, or 214 characters) and thereby avoiding introducing biases associated with post length into the dataset. This resulted in 6,639 instances, each of which were annotated individually. As stipulated by our IRB protocol, we make the dataset available upon request from the authors.
Three trained annotators (computer science graduate and undergraduate students; a mixture of fluent L2 and native English speakers) were provided with guidelines describing different levels of medical self-disclosure, or the absence thereof, ranging from 0-5. They were told to label posts without considering prior or future context. Annotators were compensated for their work as part of assistantships or course credit, and were briefed on annotation procedures and best practices prior to starting the annotation process.
The guidelines instructed annotators to label posts as containing high self-disclosure (label=5) if they contained clear indications that the poster: (a) had been diagnosed with a specific illness by a medical professional; (b) was taking a specific medication; (c) had undergone a surgery, or was undoubtedly about to have one; (d) had visited a doctor, or was undoubtedly about to see one; or other cases disclosing clear, specific medical variables or events. The guidelines directed annotators to assign labels of "4" when the poster indicated specific symptoms they had but did not further specify an illness, medication, or other diagnosis; and labels ranging from 1-3 to instances with very low (ambiguous hinting of possible, non-specific medical concerns) to moderate (clear reference to nonspecific medical concerns) self-disclosure. Finally, the guidelines instructed annotators to assign labels of "0" to instances clearly containing no medical self-disclosure at all.
Each instance was labeled by all three annotators. Annotations were then averaged across all annotators for each instance, and the individual distance between each annotator's label and the average for a given instance was computed. For instances for which the distance between one or more individual annotators and the average was greater than 1.0, the instance was forwarded to a third-party, native English-speaking adjudicator, who determined the gold standard value based on the three annotations and the instance itself. For all other instances, the average label was accepted as the gold standard. These averaged scores were then discretized into the three classes: NO SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and CLEAR SELF-DISCLOSURE.
We measured inter-annotator agreement using averaged pairwise Cohen's kappa, as well as by calculating the percentage of instances that did not require adjudication (91.29%). Averaged pairwise Cohen's kappa across the entire dataset was  $\kappa = 0.88$ , suggesting high agreement (Landis and Koch, 1977). Table 1 shows the averaged pairwise kappa score among annotators for each class. Agreement for the NO SELF-DISCLOSURE and CLEAR SELF-DISCLOSURE classes was extremely high, whereas agreement for POSSIBLE SELF-DISCLOSURE was lower, although still fair (Landis and Koch, 1977). In Table 2 we provide the raw count and percentage distribution across binned gold standard score ranges of  $\{[0 - 1], (1 - 4), [4 - 5]\}$ .
For example, "depression" is in isolation most often a medical term, whereas "the great depression" is not.
Score Ranges	Raw Count	% Distribution
[0-1]	2651	39.93%
(1-4)	1019	15.34%
[4-5]	2969	44.72%
3.3 Categorical Class Labels
Self-disclosure naturally occurs along a spectrum rather than only at two extremes (Farber, 2006), as is evidenced by the distribution in Table 2, which guided our decision to collect annotations along a continuum. Researchers may be able to leverage these continuous annotations directly in future work. However, work to date has framed the problem as a classification rather than regression task (Balani and De Choudhury, 2015; Umar et al., 2019). In following earlier precedent (Balani and De Choudhury, 2015), we frame our self-disclosure task as a multi-class classification problem, facilitating comparison with prior computational work. We binned our score ranges as follows to produce three classes: [0-1] NO SELF-DISCLOSURE, (1-4) POSSIBLE SELF-DISCLOSURE, and [4-5] CLEAR SELF-DISCLOSURE.
Examples from each class are shown in Table 3. We leave the development of true regression models for predicting continuous medical self-disclosure scores to future work, and release both the averaged (and thus continuous) scores and our discretized class labels with our dataset.
3.4 Data Privacy and Permissions
To preserve user privacy, we did not download usernames or other metadata during our data collection process. We further manually reviewed all posts and replaced any names appearing directly within the text with a generic NAME_TOKEN. The patient.info terms and conditions maintain public accessibility of forum posts, and allow use of content in non-commercial contexts.8 Public Facebook posts may be freely downloaded, accessed, and re
Table 2: Raw count and percentage distribution across binned score ranges.
Class	Example	Description
No SD	1. I wish you all the strength x.	No disclosure of medical issue.
2. Cheers and happy new year!
POSSIBLE SD	1. I'm not angry, I'm not even sad as such, I'm just tired...	General, non-specific mention of or allusion to medical issue.
2. I do think my rib pain is from bad posture. I have worked at a computer for years.
CLEAR SD	1. Metoprolol gave me the most horrendous headaches, so I had my doctor take me off.	Clear disclosure of specific symptom, diagnosis, and/or treatment.
2. I did the ultrasound a couple times now since this started 2 yrs ago I'd like to find a good ortho doc.
Table 3: Medical self-disclosure class descriptions and corresponding examples.
shared both on and off the platform, $^{9}$  and the same applies to public Reddit posts. $^{10}$  Twitter's data policy stipulates that only tweet IDs, not fully hydrated tweets, be shared with third parties. $^{11}$  Thus, for Twitter data we provide tweet IDs and corresponding labels, and encourage interested individuals to download the tweet text for their own research use.
To demonstrate efficacy and learnability of our dataset, we created a suite of classification models for comparative analysis. This offered the parallel
$^9$ https://www.facebook.com/policy.php
10https://www.reddit.com/policies/ privacy-policy
11https://developer.twitter.com/en/ developer-terms/policy
Our early pilot experiments suggest that this is a challenging task, due in part to an uneven distribution of labels at that level of granularity for which straightforward solutions (e.g., data augmentation techniques) yield somewhat diminished prediction quality.
8https://patient.info/terms-and-conditions
opportunity to identify a strong performance benchmark for this task. We describe our preprocessing techniques and modeling algorithms below.
4.1 Data Preprocessing
Prior to training our models, we applied the following preprocessing steps to our data:
1. DeEmojifying: Emojis are often used to express emotion on online platforms, and since emotional content may provide valuable clues to the presence of self-disclosure (Eisner et al., 2016; Felbo et al., 2017; Coppersmith et al., 2016), we retained emojis and converted them to text. Each emoji is represented as its CLDR short name. $^{12}$  For example, a happy face with a Unicode of U+1F600 would be converted to [grinning face].
2. Number Replacement: The presence of numbers may likewise be indicative of medical content in a post (e.g., I've always started on 20mg (albeit with side effects for the first few weeks)). However, we hypothesized that retaining value specificity (e.g., "20mg") may produce too much noise to yield high value. We thus replaced all numbers with a single NUMBER_TOKEN.
3. Stopword Removal: We removed stopwords using a modified version of the NLTK (Bird, 2006) English stopwords list. Since some words, such as personal pronouns, may signify the presence ([I, my, myself, me, mine]) or absence ([you, your, yours, yourself, yourselves, he, his, him, himself, she, her, hers, herself)]) of self-disclosure, we retained them. Likewise, auxiliary verbs may not have significant individual importance, but could switch self-disclosure class. For example, I have depression has higher self-disclosure than I might have depression.
4. Punctuation Removal: Since most punctuation marks are unimportant to our task, we removed them, retaining only sentence boundary markers  $([!,..,?])$ . Question marks in particular could change high self-disclosure to a lower category. For example, I have depression could be interpreted quite differently from I have depression?
Technique	Accuracy
Base Model (No Preprocessing)	78.62%
Base + DeEmojifying	80.01%
Base + Number Replacement	80.82%
Base + Stopword Removal	80.79%
Base + Punctuation Removal	79.81%
Base + Spelling Correction	75.62%
Table 4: Model performance in accuracy (%) before and after applying each preprocessing technique. Base model refers to our highest-performing model (§5.2).
We initially experimented with spelling correction as an additional preprocessing step, but ultimately abandoned it since it reduced performance. Inaccurate corrections (e.g.,  $dr \to dry$ ) led to considerable, and often detrimental, changes in predicted class values. We present an empirical analysis of these preprocessing steps in Table 4 to illustrate their relative merits.
We experimented with multiple supervised machine learning methods for our task. We considered the following classification models:
- Support Vector Machine (SVM): SVM is a classical machine learning model that has achieved a very high success rate in text classification (Forman, 2008). We applied a linear kernel and kept the penalty parameter  $C$  at a default value of 1.0.
- Naive Bayes (NB): Naive Bayes is another classical machine learning method that has proven to be useful for a wide range of text classification tasks (Kim et al., 2006).
- Long Short Term Memory (LSTM): Neural networks are capable of achieving strong performance in many text classification problems, with LSTM models being particularly adept at tasks relying on sequential data (Gers et al., 2000). We used the following fine-tuned hyperparameters: learning rate = 0.001, batch size = 64, dropout = 0.5, max sequence length = 286, and optimizer = Adam.
- Bidirectional LSTM (BLSTM): BLSTMs are an extension of traditional LSTMs that
$^{12}$ https://unicode.org/emoji/charts/full-emoji-list.html
consider both prior and forthcoming information in a sequence, allowing them to improve sequential text classification performance (Wollmer et al., 2010). We used the following fine-tuned hyperparameters: learning rate  $= 0.0003$ , batch size  $= 64$ , dropout  $= 0.2$ , max sequence length  $= 286$ , and optimizer  $=$  Adam.
- 1D-Convolutional Neural Network (1D-CNN): Convolutional neural networks have achieved exceptional performance for many text classification problems (Kim, 2014). We used the following fine-tuned hyperparameters: learning rate = 0.0002, batch size = 32, dropout = 0.3, max sequence length = 286, and optimizer = Adam.
- DistilBERT: DistilBERT (Sanh et al., 2019) is a lightweight Transformer-based model. It was designed as a variation of BERT (Devlin et al., 2018) that is well-suited for tasks utilizing smaller datasets. We used the following fine-tuned hyperparameters: learning rate = 0.003, batch size = 32, and epochs = 40.
We also compare these models to two additional approaches:
- **Baseline:** Predicts a constant label (CLEAR SD, the highest frequency label in the dataset) for every record. This allowed us to validate that our models were able to learn to predict medical self-disclosure using our novel dataset at a rate higher than chance.
- Balani and De Choudhury (2015): Our reimplementation of Balani and De Choudhury's best-performing self-disclosure model, fine-tuned for our dataset and task. This allowed us to compare our model performance directly with a high-performing existing model for self-disclosure detection, and subsequently provide empirical justification that detecting self-disclosure within our task domain carries its own uniquely challenging, subtle complexities.
We applied sequence padding for all deep learning models, padding sentences with zeroes to normalize length. The maximum sequence length (maximum number of tokens) of the instances in our dataset is 286, and thus we padded all shorter
instances to reach that length. We used TF-IDF vectors with a vocabulary size of 5000 words (Zhang et al., 2011) for the classical machine learning models, optimizing the vocabulary size on a held-out validation set and retaining the 5000 most-frequent words. We used 100-dimensional GloVe (Pennington et al., 2014) word embeddings pretrained on Wikipedia 2014 and Gigaword 5 for the deep learning models. $^{13}$
We randomly split the data into training  $(80\%)$ , validation  $(10\%)$ , and test  $(10\%)$  subsets, training the models on the training data and fine-tuning them on the validation set to optimize hyperparameters. Since weights for our deep learning models were randomly initialized, we repeated this process multiple times for each model, performing five-fold Monte Carlo cross-validation (Xu and Liang, 2001) and reporting the averaged results. We optimized hyperparameters using grid search.
4.3 Classification Settings
In addition to experimenting with a variety of statistical and neural classification models, we experimented with two classification settings: (1) a binary classification setting, and (2) our target multinomial classification setting. We did so in light of our observation that POSSIBLE SELF-DISCLOSURE exhibited noticeably lower inter-annotator agreement than the two classes at the respective ends of the self-disclosure spectrum (see Table 1). We anticipated that automated self-disclosure models would similarly struggle more with this class.