SlowGuess/ACL_2020_OCR huggingface dataset:
proposed model incorporates attention at two different levels of analysis: word and message level. We examine each attention mechanism's impact on the overall performance of the model.
Table 4 shows ablation results for word and message attentions. As expected, adding any attention results in improvements over the No Attn model. In addition, using only message-level attention generally outperforms using only word-level attention. This may be because message-level attention oc
Figure 4: Performance of our model when keeping only the top  $n$  percent highest or lowest weighted messages.
curs later in the model, where its impacts are less likely to get washed out by downstream layers.
While adding message attention provides the single largest boost, in 3 out of 5 cases combining it with word attention results in additional gains. This may be because the word-level attention helped the model to better encode longer messages: the average message length for the top  $5\%$  highest-weighted messages were, on average, 4.4 tokens longer for Word+Msg than for Msg Only.
The inclusion of message-level attention appears to have little direct impact on the word-level attention. On examination, Word+Msg and Word Only typically assigned roughly the same word-level attention weights to the same sentences. This suggests the strength of adding message-level attention is in learning how best to weight messages, rather than how to better represent each individual message.
We further explore the impact of the learned message-level attention weights. Figure 4 shows our proposed model's performance when evaluated over the top  $n$  percent highest or lowest weighted messages, as learned by our model. We see that performance is much better when using high-attention messages than low-attention ones in all cases but  $CON$ , which we saw in Table 4 did not benefit much from message-level attention. Another note of interest is that  $AGR$  plateaus very quickly for high attention messages, which suggests that high-signal messages are rare but extremely predictive.
In conclusion, while adding any attention is helpful, message-level attention provides overall larger gains than word-level attention.
7 Qualitative Value of Identifying Informative Text
The high-signal text identified by our attention-based models potentially provides additional, qualitative value for researchers interested in the rela
tionship between language and personality. Bag-of-words approaches to language modeling can identify attribute-relevant words (e.g. word clouds), but this can be limiting as it lacks the context in which the words appear. By contrast, a personality researcher interested in how high extraversion, for example, manifests itself in one's language use can use our learned attention weights to identify whole messages that may warrant further study.
Table 5 shows examples of messages that received high and low attention weights from the  $\mathrm{SN + }$  Attn model for users at the extreme ends of each personality dimension. Overall, the highattention messages are thematically relevant to the target personality dimension. For example, the messages for conscientiousness focus on work and school responsibilities, while those for extraversion discuss social interactions. The high-attention words, highlighted in green, are also consistent with each personality dimension. For example, openness to experience highlights philosophical words (weird, nothingness, trippy) while agreeableness favors swear words (shit). In contrast, the low-attention messages have little relevance.
To test whether our high-signal text might be of qualitative value to researchers, we asked two experts on personality (psychologists with past research in the area) to view 100 paired messages sets (20 per dimension) and select which set was more informative of the individual's personality. Each paired set consisted of 5 messages within the top third of message weights and 5 in the bottom third for a given user. To reduce the frequency of long messages, we only selected messages whose length was at most 20 characters above or below that user's average message length. The users themselves were randomly sampled from those in the top or bottom 10th percentile of each dimension and who had at least 20 messages total. Note that personality psychologists, though experts in how
Table 5: Random selection of messages that received high (top) and low (bottom) attention weights from the SN+Attn model. Blue shades indicate strength of message-level attention and green indicates word-level attention. Each set of messages is from a single user, with that user having a personality score in the top or bottom 10th percentile. For brevity, only messages with 70 or fewer characters were included.
personality manifests in behaviors like language, are not trained necessarily to identify it from microblog posts. The goal here is not to simply validate the attention, but to shed some light on where message attention helps and whether it is consistent with expectations from personality theory.
Table 6 shows the percentage of instances where each expert identified the high-attention set as most informative, and their inter-rater agreement. Judges showed a preference towards the high-attention messages for OPE and AGR, while CON and NEU were no better than chance. These findings are somewhat consistent with Table 4, which showed that OPE and AGR benefited from message-level attention more than CON. Not only were EXT judgements no better than chance, but there was virtually no agreement among experts. This suggests that
for some personality dimensions, individual messages have more or less relevance for personality, while for other dimensions there is little difference between messages (or at least it is difficult for both experts and our approach to capture differences).
In general, our proposed model seems to identify text that is informative of one's personality, both in terms of individual words and the overarching themes of the message as a whole, though this is easier for some dimensions than others. Modeling document relevance is useful, then, not just as a means to boost performance but as a tool to aid those seeking to better understand language.
Personality modeling from language is becoming increasingly important for many social scientific
Percent Preferred High	Cohen's κ
Expert 1	Expert 2
OPE	75%	75%	.60
CON	55%	55%	.60
EXT	55%	45%	.08
AGR	75%	75%	.76
NEU	40%	55%	.79
Table 6: Personality experts picked which of a pair of message sets were most informative for prediction. Each pair contained five of the highest and five of the lowest-weighted messages for a user. Table shows the percentage of instances where the expert selected the high-attention message set as most informative, as well as Cohen's  $\kappa$  inter-rater agreement.
applications. For example, Preoiciuc-Pietro et al. (2015) found personality features to be highly predictive of depression and PTSD. Lynn et al. (2017) demonstrated that the performance of document classification models can be improved by adapting to a variety of human factors, including personality. Personality has also been shown to be useful for deception detection (Fornaciari et al., 2013) and recommendation systems (Roshchina et al., 2011).
Most research on personality modeling focuses on the Big Five, or Five-Factor Model (Costa and McCrae, 1992). Personality is traditionally measured using questionnaires, but cost and scalability issues make computational methods preferable.
Linguistic Inquiry and Word Count (LIWC) (Pennebaker et al., 2001) features are popular for personality modeling (Yarkoni, 2010; Schwartz et al., 2013; Gjurković and Šnajder, 2018), as they readily provide insight into the type of language that correlates with certain personality dimensions. However, using predefined lexica is limiting; Schwartz et al. (2013) and Park et al. (2015) showed significantly improved prediction when using topics and n-grams extracted from their training set. When working with a very limited amount of data, Arnoux et al. (2017) found pre-trained word embeddings to be effective.
Deep learning approaches to personality prediction are limited. Majumder et al. (2017) used a convolutional neural network (CNN) with max pooling, alongside traditional document features (e.g. word count). Their best results were obtained when they filtered out sentences that did not contain strong emotion words (as determined via lexica) during preprocessing. This supports our intuition that some messages contain stronger signal than others, though our approach allows the model to identify such cases.
Yu and Markov (2017) also used CNNs with max- and average-pooling to predict personality over Facebook statuses. They experimented with fully-connected neural networks and bidirectional recurrent neural networks, but ultimately CNNs performed best. Both Majumder et al. (2017) and Yu and Markov (2017) used datasets that were significantly smaller than ours ( $n = 2467$  and  $n = 9917$ , respectively) and their problems were framed as binary classification rather than regression<sup>5</sup>.
Language-based personality prediction is an important task with many applications in social science and natural language processing. We presented a hierarchical sequence model with message- and word-level attention that learns to differentiate high- and low-signal messages. Our approach, which novelly models the idea that all messages are not equally valuable for psychological regression tasks, achieves new state-of-the-art results for personality prediction and provides insight into the relationship between language and personality. Our analysis demonstrates that the level of abstraction at which attention is applied can have a significant impact on a model's overall performance. Finally, this work highlights the critical role of document relevance as we progress with further human-centered natural language processing.
This work is supported in part by the National Science Foundation under Grant IIS-1815358. Data set used in grateful collaboration with Michal Kosinski and David Stillwell. We thank Google for supporting this research through the Google Cloud Platform credits. Thanks also to social and personality psychologists Sandra Matz and David Yaden for their help with the expert evaluation task.
Pierre-Hadrien Arnoux, Anbang Xu, Neil Boyette, Jalal Mahmud, Rama Akkiraju, and Vibha Sinha. 2017. 25 tweets to know you: A new model to predict personality with social media. ICWSM.
R Michael Bagby, Russell T Joffe, James DA Parker, Valery Kalemba, and Kate L Harkness. 1995. Major
5Personality theory suggests factors are better represented as continuous dimensions than discrete types (McCrae and Costa Jr, 1989).
depression and the five-factor model of personality. Journal of Personality Disorders, 9(3):224-234.
Benjamin P Chapman, Brent Roberts, and Paul Duberstein. 2011. Personality and longevity: knowns, unknowns, and implications for public health and personalized medicine. Journal of aging research, 2011.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
P.T. Costa and R.R. McCrae. 1992. Revised NEO Personality Inventory (Neo-PI-R) and NEO Five-Factor Inventory (NEO-FFI). Psychological Assessment Resources.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tommaso Fornaciari, Fabio Celli, and Massimo Poesio. 2013. The effect of personality type on deceptive communication style. In Intelligence and Security Informatics Conference (EISIC), 2013 European, pages 1-6. IEEE.
Howard S Friedman and Margaret L Kern. 2014. Personality, well-being, and health. Annual review of psychology, 65:719-742.
Matej Gjurković and Jan Šnajder. 2018. Reddit: A gold mine for personality prediction. In Proceedings of the Second Workshop on Computational Modeling of Peoples Opinions, Personality, and Emotions in Social Media, pages 87-97.
Lewis R Goldberg, John A Johnson, Herbert W Eber, Robert Hogan, Michael C Ashton, C Robert Cloninger, and Harrison G Gough. 2006. The international personality item pool and the future of public-domain personality measures. Journal of Research in personality, 40(1):84-96.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Association for Computational Linguistics.
Margaret L Kern, Johannes C Eichstaedt, H Andrew Schwartz, Lukasz Dziurzynski, Lyle H Ungar, David J Stillwell, Michal Kosinski, Stephanie M Ramones, and Martin EP Seligman. 2014. The online social self: An open vocabulary approach to personality. Assessment, 21(2):158-169.
Michal Kosinski, David Stillwell, and Thore Graepel. 2013. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15):5802-5805.
Vivek Kulkarni, Margaret L Kern, David Stillwell, Michal Kosinski, Sandra Matz, Lyle Ungar, Steven Skiena, and H Andrew Schwartz. 2018. Latent human traits in the language of social media: An open-vocabulary approach. *PloS one*, 13(11):e0201703.
Veronica E. Lynn, Alissa Goodman, Kate Niederhoffer, Kate Loveys, Philip Resnik, and H. Andrew Schwartz. 2018. CLPsych 2018 shared task: Predicting current and future psychological health from childhood essays. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 37-46, New Orleans, LA. Association for Computational Linguistics.
Veronica E. Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2017. Human centered NLP with user-factor adaptation. In Empirical Methods in Natural Language Processing, pages 1146-1155.
François Mairesse, Marilyn A Walker, Matthias R Mehl, and Roger K Moore. 2007. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of artificial intelligence research, 30:457-500.
Navonil Majumder, Soujanya Poria, Alexander Gelbukh, and Erik Cambria. 2017. Deep learning-based document modeling for personality detection from text. IEEE Intelligent Systems, 32(2):74-79.
Matthew Matero, Akash Idnani, Youngseo Son, Salvatore Giorgi, Huy Vu, Mohammad Zamani, Parth Limbachiya, Sharath Chandra Guntuku, and H. Andrew Schwartz. 2019. Suicide risk assessment with multi-level dual-context language and BERT. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 39-44, Minneapolis, Minnesota. Association for Computational Linguistics.
Sandra C Matz, Michal Kosinski, Gideon Nave, and David J Stillwell. 2017. Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the national academy of sciences, 114(48):12714-12719.
Robert R McCrae and Paul T Costa Jr. 1989. Reinterpreting the myers-briggs type indicator from the perspective of the five-factor model of personality. Journal of personality, 57(1):17-40.
Robert R McCrae and Paul T Costa Jr. 1997. Personality trait structure as a human universal. American psychologist, 52(5):509.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS.
Walter Mischel, Yuichi Shoda, and Ozlem Ayduk. 2007. Introduction to personality: Toward an integrative science of the person. John Wiley & Sons.
Kevin R Murphy and Charles O Davidshofer. 1988. Psychological Testing: Principles and Applications. Pearson.
Gregory Park, H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Michal Kosinski, David J Stillwell, Lyle H Ungar, and Martin EP Seligman. 2015. Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. *Mahway: Lawrence Erlbaum Associates*, 71(2001):2001.
Daniel Preotciuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura Smith, Victoria Tobolsky, H Andrew Schwartz, and Lyle Ungar. 2015. The role of personality, age, and gender in tweeting about mental illness. In Proceedings of the 2nd workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality, pages 21-30.
Alexandra Roshchina, John Cardiff, and Paolo Rosso. 2011. A comparative evaluation of personality estimation algorithms for the twin recommender system. In Proceedings of the 3rd international workshop on Search and mining user-generated contents, pages 11-18. ACM.
H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9):e73791.
Nakatani Shuyo. 2010. Language detection library for java.
Louis Leon Thurstone. 1934. The vectors of mind. Psychological review, 41(1):1.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489.
Tal Yarkoni. 2010. Personality in 100,000 words: A large-scale analysis of personality and word use among bloggers. Journal of research in personality, 44(3):363-373.
Jianguo Yu and Konstantin Markov. 2017. Deep learning based personality recognition from Facebook status updates. In 2017 IEEE 8th International Conference on Awareness Science and Technology (iCAST), pages 383-387. IEEE.