SlowGuess/NAACL_2021_OCR huggingface dataset:
In the binary setting, we only trained and evaluated our models using data from the NO SELF-DISCLOSURE and CLEAR SELF-DISCLOSURE classes. This had the effect of simplifying the task greatly, but it was also less realistic—in the real world, as shown in the class distribution for our dataset, many instances may be more ambiguous and fall somewhere between the two endpoints of the self-disclosure spectrum. In our more challenging multinomial setting (the setting upon which we placed our primary focus) we retained all three classes: NO SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and CLEAR SELF-DISCLOSURE. We applied the same hyperparameters specified in §4.2 (fine-tuned under the multinomial classification setting) to models in both settings.
Word embeddings represent words as  $n$ -dimensional feature vectors and capture latent patterns in meaning, semantic relationships, and the context in which words are used (Collobert et al., 2011).
Model	Acc.	Precision	Recall	F1
Binary	84.75	0.8938	0.8486	0.8623
Multi	81.02	0.8084	0.8102	0.8089
We evaluated the performance of all models using accuracy, precision, recall, and F1-measure, following prior work on self-disclosure detection (Balani and De Choudhury, 2015; Umar et al., 2019). We provide the results from three separate experiments in the following subsections. In §5.1, we compare performance between the binary and multinomial classification settings. In §5.2, we compare performance between our SVM, NB, LSTM, BLSTM, 1D-CNN, and DistilBERT models for the multiclass setting. Finally, in §5.3, we provide external validation for our highest-performing multinomial model by comparing it to the baseline and Balani and De Choudhury's highest-performing model.
5.1 Binary vs. Multinomial Self-Disclosure Classification
We compare the performance of our binary and multiclass DistilBERT models (the highest-performing models for binary and multinomial classification) in Table 5. Unsurprisingly, the binary DistilBERT model outperforms its multiclass counterpart; as predicted, the model was able to learn to distinguish between NO SELF-DISCLOSURE and CLEAR SELF-DISCLOSURE with relatively little trouble, much like human annotators. The multiclass DistilBERT model struggled slightly more but nonetheless still exhibited strong overall performance, dropping only  $3.73\%$  in absolute accuracy compared to the binary classification setting. We demonstrate later (see Table 8) that a much larger relative percentage of instances from the POSSIBLE SELF-DISCLOSURE class were misclassified than were instances from the other two classes, suggesting ample room for future work that disentangles the nuances of these more ambiguous cases.
We present the results of our model comparison for the multinomial classification setting in Table 6. DistilBERT achieved the best performance overall
Table 5: Comparison between binary and multiclass DistilBERT models. Accuracy shown as a percentage  $(\%)$ .
Model	Acc.	Precision	Recall	F1
SVM	71.18	0.5945	0.5963	0.5632
NB	67.22	0.4510	0.5197	0.4803
LSTM	74.40	0.7937	0.6582	0.7179
BLSTM	72.89	0.7565	0.6621	0.7052
Distil-BERT	81.02	0.8084	0.8102	0.8089
1D-CNN	71.29	0.7493	0.6592	0.7003
Table 6: Model comparison for the multinomial classification setting. Accuracy shown as a percentage  $(\%)$
with an accuracy of  $81.02\%$ , precision of 0.8084, recall of 0.8189, and  $\mathrm{F_1}$ -score of 0.8089. In general, the deep learning models outperformed the standard classification models for this task, with DistilBERT outperforming the highest-performing standard classification model (SVM) by relative percent increases in accuracy, precision, recall, and F1-measure by  $13.82\%$ ,  $35.97\%$ ,  $35.87\%$ , and  $43.62\%$ , respectively.
5.3 External Validation
As mentioned earlier, Balani and De Choudhury (2015) detected three grades of self-disclosure in Reddit posts. Their task has similarities with ours, with ours focusing on medical self-disclosure specifically and theirs targeting more general disclosure of mental wellness. Although we were unable to directly acquire their data or source code, we reimplemented their best model and finetuned it such that it maximized performance on our dataset and task. Our motivation in performing this experiment was to establish that models designed for general self-disclosure do not necessarily generalize to the additional subtle complexities of medical self-disclosure, and correspondingly that different forms of self-disclosure should be managed differently in automated systems. In Table 7 we compare the results achieved by (1) the most frequent class baseline, (2) our best-performing multinomial model, and (3) our reimplementation of Balani and De Choudhury's best-performing model. Our model outperforms both the baseline and Balani and De Choudhury's model by a wide margin, with relative percentage increases of  $41.84\%$ ,  $32.63\%$ ,  $66.60\%$ , and  $49.76\%$  for accuracy, precision, recall,
Model	Acc.	Precision	Recall	F1
Baseline	45.38	0.4471	0.4489	0.4479
Distil-BERT	81.02	0.8084	0.8102	0.8089
Balani and De Choud-hury	57.12	0.6095	0.4863	0.5401
Table 7: Comparison between the baseline, our best (multinomial) model performance, and our replication of Balani and De Choudhury's model (2015). Accuracy is shown as a percentage  $(\%)$ .
Class	# Test Samples	# TP	Accuracy
No SD	398	349	87.68
POSSIBLE SD	153	68	44.44
CLEAR SD	445	390	87.64
Table 8: Total number of test samples per class, number of true positives per class, and overall class accuracy. Accuracy is shown as a percentage  $(\%)$ .
and F1-measure, respectively, over Balani and De Choudhury's model.
Although Balani and De Choudhury's model worked well for their setting, we found that it did not transfer well to our task. It may be that detecting medical self-disclosure inherently carries extra levels of complexity. For example, identifying first-person pronouns could be a decisive indicator of general self-disclosure, whereas for medical self-disclosure, self-identifiers would also need to be accompanied by medical terms, some of which may be obscure (Meystre et al., 2008).
To further disentangle the performance of our highest-performing model, we computed the number of true positives for each class separately, shown alongside per-class accuracy in Table 8. We found that model performance was lowest when predicting POSSIBLE SELF-DISCLOSURE. This was anticipated due to the difficulty of agreeing upon labels for this class even among trained annotators (refer to Table 1 for per-class agreement statistics); in many cases, only one annotator may have felt that an instance clearly disclosed a med
Figure 1: Words most closely associated with CLEAR SELF-DISCLOSURE. The x-axis shows the log odds ratio.
Figure 2: Words most closely associated with POSSIBLE SELF-DISCLOSURE. The x-axis shows the log odds ratio.
ical issue, with others being less certain. Performance was high for NO SELF-DISCLOSURE and CLEAR SELF-DISCLOSURE, with accuracies of  $87.68\%$  and  $87.64\%$ , respectively. Since cases of POSSIBLE SELF-DISCLOSURE may comprise a sizeable contingent of data instances (slightly over  $15\%$  of the dataset in our case), we recommend that this subset of data is examined more closely in follow-up work. Downstream applications may need to handle these more ambiguous cases differently from incidences in which symptoms, diagnoses, or treatments clearly are (or clearly are not) being disclosed.
To develop a further understanding of the linguistic patterns associated with CLEAR SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and NO SELF-DISCLOSURE instances, we computed the log odds ratio with an informative Dirichlet prior (Monroe et al., 2008; Hessel, 2016) for words in these classes to assess which words were most strongly correlated with each, and plot them in Figures 1, 2, and 3. The plots support our hypotheses. The words most closely associated with POSSIBLE SELF-DISCLOSURE have much lower
Figure 3: Words most closely associated with No SELF-DISCLOSURE. The x-axis shows the log odds ratio.
ratios in general than the words most closely associated with CLEAR SELF-DISCLOSURE or NO SELF-DISCLOSURE, suggesting that this class is characterized by fewer strong cues indicating membership. Furthermore, while the words closely associated with CLEAR SELF-DISCLOSURE are a mix of personal pronouns, medical terms, duration, and narrative descriptors, the words most closely associated with POSSIBLE SELF-DISCLOSURE are mostly about others and family, or being scared and in search of hope and support. Words closely associated with NO SELF-DISCLOSURE are less personal or narrative, and more indicative of support or general health interest.
In this work, we introduced a novel medical self-disclosure dataset containing 6,639 instances collected from public online social platforms. Instances in this dataset are triple-annotated with high inter-annotator agreement  $(\kappa = 0.88)$  for NO SELF-DISCLOSURE, POSSIBLE SELF-DISCLOSURE, and CLEAR SELF-DISCLOSURE. We evaluated a wide range of classical machine learning and neural classifiers (including LSTM-, CNN-, and Transformer-based models) to assess their efficacy at learning to predict medical self-disclosure. We examined both a simpler binary classification setting and a more challenging multinomial setting, finding that the highest-performing model in both cases was a fine-tuned DistilBERT model.
We compared our best-performing model to the best existing categorical model for self-disclosure detection (Balani and De Choudhury, 2015), finding that our model outperformed that model by a wide margin for the task of detecting medical self
disclosure (relative percent increases of  $41.84\%$  and  $49.76\%$  for accuracy and F1-measure, respectively). Our findings pave the way for subsequent experiments with other models, moving the dial a necessary step forward by establishing a strong performance benchmark. In the future, we hope to explore medical self-disclosure in the context of goal-oriented dialogue systems, resulting in downstream benefits for both physicians and patients. We make our dataset available to interested researchers to foster further progress on this emerging research task.
This research was approved by the Institutional Review Board at the University of Illinois at Chicago. All data was collected in a manner consistent with the terms and conditions of the respective data sources, as outlined in §3.4. In particular, since Twitter's data policy prohibits direct sharing of tweet text, we release only tweet IDs and corresponding annotations for that subset of the data. Annotations were collected using the process described in §3.2, and annotators were compensated for their work through assistantships and course (independent study) credit. Additional characteristics of the data are provided in §3.1 and §3.3. Instances have been anonymized, with any usernames or other personal names found in the text replaced with a generic NAME_TOKEN, to further promote privacy of content creators when possible (this is not possible with the tweets since they are provided as stand-off annotations). Data is available upon request by emailing the authors, and posts known or assumed to be deleted at the time of request will be removed prior to sharing. We will communicate further data use guidelines as outlined in our IRB protocol directly when sharing the data.
We thank Yasmin Isa for her role in creating the dataset and in participating in helpful research discussions along the way. We also thank the anonymous reviewers for their insightful suggestions, which further strengthened this work. This work was supported in part by a startup grant from the University of Illinois at Chicago.
Irwin Altman and Dalmas A Taylor. 1973. Social penetration: The development of interpersonal relationships. Holt, Rinehart & Winston.
Jin Yeong Bak, Suin Kim, and Alice Oh. 2012. Self-disclosure and relationship strength in twitter conversations. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 60-64, Jeju Island, Korea. Association for Computational Linguistics.
Sairam Balani and Munmun De Choudhury. 2015. Detecting and characterizing mental health related self-disclosure in social media. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pages 1373-1378.
Steven Bird. 2006. NLTK: The Natural Language Toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69-72, Sydney, Australia. Association for Computational Linguistics.
Nancy Bridges. 2001. Therapist's self-disclosure: Expanding the comfort zone. Psychotherapy: Theory, Research, Practice, Training, 38:21-30.
Olivia E Bruss and Jennifer M Hill. 2010. Tell me more: Online versus face-to-face communication and self-disclosure. *Psi Chi Journal of Undergraduate Research*, 15(1).
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493-2537.
Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015. From ADHD to SAD: Analyzing the language of mental health on twitter through self-reported diagnoses. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 1-10, Denver, Colorado. Association for Computational Linguistics.
Glen Coppersmith, Kim Ngo, Ryan Leary, and Anthony Wood. 2016. Exploratory analysis of social media prior to a suicide attempt. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pages 106-117, San Diego, CA, USA. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Ben Eisner, Tim Rocttäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth
International Workshop on Natural Language Processing for Social Media, pages 48-54, Austin, TX, USA. Association for Computational Linguistics.
Darren Ellis and John Cromby. 2012. Emotional inhibition: A discourse analysis of disclosure. Psychology & health, 27(5):515-532.
Barry A. Farber. 2006. Self-disclosure in psychotherapy. Guilford Press.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615-1625, Copenhagen, Denmark. Association for Computational Linguistics.
Michael Ferriter. 1993. Computer aided interviewing and the psychiatric social history. Social Work and Social Sciences Review, 4(3):255-263.
George Forman. 2008. Bns feature scaling: an improved representation over tfidf forsvm text classification. In Proceeding of the 17th ACM conference on Information and knowledge management, CIKM '08, pages 263-270, New York, NY, USA. ACM.
Felix A. Gers, Jürgen Schmidhuber, and Fred A. Cummins. 2000. Learning to forget: Continual prediction with LSTM. *Neural Computation*, 12(10):2451-2471.
John H. Greist, Marjorie H. Klein, and L J Van Cura. 1973. A computer interview for psychiatric patient target symptoms. Archives of general psychiatry, 29 2:247-53.
Jake Hessel. 2016. Implementation: Fightin' words. https://github.com/jmhessel/FightingWords.
Adam N Joinson. 2001. Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European journal of social psychology, 31(2):177-192.
Adam N Joinson and Carina B Paine. 2007. Self-disclosure, privacy and the internet. The Oxford handbook of Internet psychology, 2374252.
Toksoz B Karasu. 1979. Psychotherapy of the psychosomatic patient. American journal of psychotherapy, 33(3):354-364.
Robert Kellner. 1975. Psychotherapy in psychosomatic disorders: a survey of controlled studies. Archives of General Psychiatry, 32(8):1021-1028.
Sang-Bum Kim, Kyoung-Soo Han, Hae-Chang Rim, and Sung-Hyon Myaeng. 2006. Some effective techniques for naive bayes text classification. IEEE Trans. Knowl. Data Eng., 18(11):1457-1466.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
Ruth Lewy. 2013. 50 top websites you can't live without. The Times. Accessed: 2021-03-31.
W.W. Meissner. 2002. The problem of self-disclosure in psychoanalysis. Journal of the American Psychoanalytic Association, 50:827-67.
Kenneth George Andrew Meleshko and Lynn E. Alden. 1993. Anxiety and self-disclosure: toward a motivational model. Journal of personality and social psychology, 64 6:1000-9.
Stéphane M Meystre, Guergana K Savova, Karin C Kipper-Schuler, and John F Hurdle. 2008. Extracting information from textual documents in the electronic health record: a review of recent research. Yearbook of medical informatics, 17(01):128-144.
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372-403.
James W. Pennebaker and Cindy K. Chung. 2007. Expressive writing, emotional upheavals, and health. In Foundations of Health Psychology, pages 263-284. Oxford University Press.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Lisa Collins Tidwell and Joseph B Walther. 2002. Computer-mediated communication effects on disclosure, impressions, and interpersonal evaluations: Getting to know one another a bit at a time. Human communication research, 28(3):317-348.
Prasanna Umar, Anna Squicciarini, and Sarah Rajtmajer. 2019. Detection and analysis of self-disclosure in online news commentaries. In *The World Wide Web Conference*, pages 3272–3278.
S. Courtney Walton and Ronald E. Rice. 2013. Mediated disclosure on twitter: The roles of gender and identity in boundary impermeability, valence, disclosure, and stage. Comput. Hum. Behav., 29(4):1465-1474.
Yi-Chia Wang, Moira Burke, and Robert Kraut. 2016. Modeling self-disclosure in social networking sites. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW '16, page 74-85, New York, NY, USA. Association for Computing Machinery.
Martin Wöllmer, Florian Eyben, Alex Graves, Björn W. Schuller, and Gerhard Rigoll. 2010. Bidirectional LSTM networks for context-sensitive keyword detection in a cognitive virtual agent framework. Cogn. Comput., 2(3):180-190.
Qingsong Xu and Yi-Zeng Liang. 2001. Monte carlo cross validation. Chemometrics and Intelligent Laboratory Systems, 56:1-11.
Wen Zhang, Taketoshi Yoshida, and Xijin Tang. 2011. A comparative study of tf*idf, LSI and multi-words for text classification. Expert Syst. Appl., 38(3):2758-2765.