SlowGuess/ACL_2020_OCR huggingface dataset:
Veronica E. Lynn, Niranjan Balasubramanian, and H. Andrew Schwartz  
Stony Brook University
{velynn, niranjan, has}@cs.stonybrook.edu
Not all documents are equally important. Language processing is increasingly finding use as a supplement for questionnaires to assess psychological attributes of consenting individuals, but most approaches neglect to consider whether all documents of an individual are equally informative. In this paper, we present a novel model that uses message-level attention to learn the relative weight of users' social media posts for assessing their five factor personality traits. We demonstrate that models with message-level attention outperform those with word-level attention, and ultimately yield state-of-the-art accuracies for all five traits by using both word and message attention in combination with past approaches (an average increase in Pearson  $r$  of  $2.5\%$ ). In addition, examination of the high-signal posts identified by our model provides insight into the relationship between language and personality, helping to inform future work.
Most language-based methods for human attribute prediction assume all documents generated by a person are equally informative. However, this is not necessarily true. Figure 1 gives examples of high and low signal messages for predicting extraversion — one's tendency to be energized by social interaction. The high signal messages contain words relating to social interaction (hangin out, chillin), whereas the low signal messages, while still containing social-related words, have little clear relevance to extraversion. The former examples would ideally be weighted higher by a personality prediction model than the latter.
This paper applies the idea of modeling document relevance to the task of personality prediction. Inferring an individual's personality traits is a fundamental task in psychology (McCrae and Costa Jr,
(a) High signal messages
(b) Low signal messages
Figure 1: Examples of high and low signal messages identified by our proposed model for predicting extraversion. All examples are from the same highly-extroverted user. Shading indicates strength of message-level (blue) and word-level (green) attention.
1997; Mischel et al., 2007), with social scientific applications ranging from public health (Friedman and Kern, 2014) and marketing (Matz et al., 2017) to personalized medicine (Chapman et al., 2011), mental health care (Bagby et al., 1995), and even providing useful information for downstream NLP tasks (Preoticiuc-Pietro et al., 2015; Lynn et al., 2017). Recently, researchers from both NLP and psychology have turned toward more accurately assessing personality and other human attributes via language (Mairesse et al., 2007; Schwartz et al., 2013; Park et al., 2015; Kulkarni et al., 2018). The idea behind "language-based assessments" (Park et al., 2015) is that language use patterns can supplement and, in part, replace traditional and expensive questionnaire-based human assessments.
Here, we present a hierarchical neural sequence model over both the words and messages of the user and correspondingly applies attention to each level. The document-level attention learns the relative importance of each social media post for predicting personality.
Contributions. Our main contributions include:
1. A neural model for personality prediction that uses message-level attention to recover high-signal messages from noisy data.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5306-5316 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2. An empirical demonstration that shows models with message-level attention outperform those without.
3. State-of-the-art performance for language-based assessment of personality.
4. Insight into the relationship between message-level language use and personality.
Our goal is to encode user messages into a representation that can be used to predict the personality of the user. We can use a two-step process to produce such a representation: First encode the sequences of words in each message to form message-level representations and then encode the message-level representations to form a user-level representation. Social media users write hundreds or even thousands of messages; while the messages, and the words within them, contain valuable clues to their personality, not all of it is equally valuable. An ideal representation of user text, therefore, should pay particular attention to personality-revealing portions of a user's text. Hierarchical attention is a natural fit for this problem. At the message level, a word-attention model can learn to emphasize personality related words in the message representation, while at the user-level, a message attention model can learn to emphasize personality-related messages in the overall user representation. We instantiate this idea using a hierarchical sequence architecture shown in Figure 2.
Given a set of  $n$  messages from a user  $u$ , the first step of the model is to produce an encoding for each message  $m_i$ . Each word  $w_j^i$  in message  $m_i$  is fed through a Gated Recurrent Unit (GRU) (Cho et al., 2014) to produce a hidden state:
$$
h _ {j} ^ {i} = \operatorname {G R U} \left(w _ {j} ^ {i}\right) \tag {1}
$$
We then apply an attention mechanism over the sequence of hidden states  $[h_1^i, h_2^i, \dots, h_l^i]$ :
$$
d _ {j} ^ {i} = \tanh  \left(W _ {\text {w o r d}} h _ {j} ^ {i} + b _ {\text {w o r d}}\right) \tag {2}
$$
$$
\alpha_ {j} ^ {i} = \frac {\exp \left(d _ {j} ^ {i} {} ^ {\top} d _ {w o r d}\right)}{\Sigma_ {k = 0} ^ {l} \exp \left(d _ {k} ^ {i} {} ^ {\top} d _ {w o r d}\right)} \tag {3}
$$
$$
s _ {i} = \sum_ {k = 0} ^ {l} \alpha_ {k} ^ {i} h _ {k} ^ {i} \tag {4}
$$
where  $d_{word}$  is a learned context vector for word-level attention,  $b_{word}$  is a bias term, and  $\alpha_j^i$  is a
Figure 2: Diagram of our proposed model for personality prediction. (A) Each post is passed through a GRU to produce a message-level encoding. (B) A word-level attention mechanism learns weights for each of the words in the message. (C) All message representations are passed to a second GRU to produce a user-level encoding. (D) A message-level attention mechanism learns weights for each of that user's posts. (E) The user representation passes through two hidden layers and a final prediction layer.
normalized attention weight for  $h_j^i$ .  $s_i$  is thus a weighted combination of the hidden states representing  $\{w_1^i, w_2^i, \dots, w_l^i\}$ .
Once we have these message representations, the next step is to encode each sequence of messages into a user representation. Each message representation  $s_i$  is passed through another encoder, also using Gated Recurrent Units:
$$
h _ {i} = \operatorname {G R U} \left(s _ {i}\right) \tag {5}
$$
As before, the hidden states are then passed through another message-level attention mechanism:
$$
e _ {i} = \tanh  (W _ {m e s s a g e} h _ {i} + b _ {m e s s a g e}) \tag {6}
$$
$$
\beta_ {i} = \frac {\exp \left(e _ {i} ^ {\top} e _ {\text {m e s s a g e}}\right)}{\sum_ {k = 0} ^ {n} \exp \left(e _ {k} ^ {\top} e _ {\text {m e s s a g e}}\right)} \tag {7}
$$
$$
u = \sum_ {k = 0} ^ {n} \beta_ {k} h _ {k} \tag {8}
$$
As before,  $e_{message}$  is a learned context vector for message-level attention. The representation for a user  $u$  is thus a weighted combination of the hidden states representing that person's messages. Once the user representation has been produced,  $u$  is further passed through some fully-connected layers before being used for prediction at the final layer.
In this way, important words and messages don't get lost to noise and are instead carried through to later portions of the model, where they can have a greater impact on the final prediction. Our model is similar in structure and motivation to the Hierarchical Attention Network proposed by Yang et al. (2016). However, our work focuses on a different level of analysis: whereas Yang et al. (2016) encode words  $\rightarrow$  sentences  $\rightarrow$  documents, our work seeks to encode words  $\rightarrow$  documents  $\rightarrow$  users. This idea of applying attention at a document level when modeling user-level attributes is, to the best of our knowledge, entirely novel. We hypothesize that where attention is applied is crucial and that message-level attention is of particular importance for modeling personality.
We draw our data from consenting users of a Facebook application (Kosinski et al., 2013), which allowed users to take various psychological assessments and voluntarily share their data with researchers. Following the work of Schwartz et al. (2013) and Park et al. (2015), the current state of the art on this dataset, we filtered the users to those who shared their Facebook status posts, wrote at least 1,000 words across those statuses, provided their age and gender, and were less than 65 years old.
All users completed psychological measures, ranging from 20 to 100 items, that assessed their Big Five personality traits (Costa and McCrae, 1992): conscientiousness, agreeableness, neuroticism, openness to experience, and extraversion. Each of the five dimensions is represented by a normalized, continuous score representing the degree to which trait is exhibited. We refer to these as personality scores. The Big Five personality traits are described more fully in Section 4.
Overall, our dataset contains Facebook statuses and personality scores for 68,687 users. To allow for direct comparisons, we use the same test set  $(n = 1,943)$  as Park et al. (2015). Each of these test users completed a longer 100-item questionnaire, ensuring higher-quality scores. We sample an additional 4,998 for use as a development set, and leave the remaining 61,746 for training.
On average, users in our dataset are 23 years old and  $63\%$  are female. Users had an average of 3,619 words and 165 messages, all posted to Facebook between 2009 and 2011.
Ethical Research Statement. All participants consented to sharing their status updates and personality questionnaire results for research purposes, and the study has been approved by an academic institutional review board.
4 Big Five Personality Traits
Discovery of the "Big Five" personality traits began nearly a century ago with some of the first data-driven, statistical latent variable modeling techniques (Thurstone, 1934). The goal in this decades-long pursuit was not very different from that of producing latent vector embeddings of words: to use latent factor analysis to reveal underlying, stable dimensional vectors that distinguish people. However, rather than finding latent semantic dimensions of words, the models (run by hand at first) focused on how individuals answered questions about themselves. For example, modern questions include: "How much do you agree with these statements? (1) I am the life of the party; (2) I have difficulty understanding abstract ideas; (3) I like order; (4) I worry about things" (Goldberg et al., 2006).
The idea behind this data-driven approach was that if such latent dimensions could be found to be stable across time and differing populations, that suggests they are fundamental to what makes each of us different. Such work continued for decades, documented across thousands of studies to eventually arrive at the acceptance of five such factors being fundamental and consistent across time and populations (Costa and McCrae, 1992). Those fundamental human factors, the target of our human language predictive task, are described below.
The big five often goes by the acronym "OCEAN", standing for openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. High scores for openness to experience are correlated with philosophical and free thought, as well as an interest in the arts, music, and cinema (Schwartz et al., 2013; Kern et al., 2014). Those who score low here may be more practical, realistic, or close-minded (Costa and McCrae, 1992).
Individuals with high conscientiousness tend to be well organized and have a lot of self-discipline, which may be expressed through discussions of work or school-related responsibilities (Yarkoni, 2010; Kern et al., 2014). Those who score low
<sup>1</sup>In fact Thurstone referred to the latent variables as “vectors of the mind”.
on this dimension may appear impulsive, disorganized, or unreliable. Those with high extraversion are likely to talk about friends, social situations, and interpersonal interaction. On the other hand, those with low extraversion may be more independent and may focus more on solo activities (e.g. watching television) (Costa and McCrae, 1992; Park et al., 2015).
Agreeableness is associated with being friendly and good-natured, while those who score low may be selfish or rude. Swearing is highly correlated with low agreeableness (Yarkoni, 2010; Schwartz et al., 2013). High neuroticism is strongly linked to anxiety and depression, while low neuroticism is linked to emotional stability. This dimension may be expressed through feelings such as fear, sadness, or frustration (Costa and McCrae, 1992; Kern et al., 2014).
In this section, we describe the method for training and evaluating our proposed model, along with the various baseline models we compared against.
5.1 Features
Each user was represented as a sequence of their messages, from most to least recent, which were themselves represented as a sequence of word embeddings. To do so, we pre-trained 200-dimensional word2vec embeddings (Mikolov et al., 2013) over all messages belonging to the training set users. The vocabulary was limited to words that appear in at least 50 messages. Words that occurred fewer times were replaced by an out-of-vocabulary token. The Language Detection Library (Shuyo, 2010) was used to filter out non-English texts.
5.2 Baseline Models
Ridge Regression (N-Grams/Topics). We compare against Park et al. (2015), which is the current state of the art on this dataset and, to the best of our knowledge, demonstrated the best published regression predictions over a Big Five personality factors from language alone. Their model uses a combination of n-gram features and LDA-based topics extracted from the training data. These features then undergo dimensionality reduction in the
form of univariate feature selection and randomized principal component analysis, resulting in a total of 5106 features. These features are then used to train ridge regression models, one per personality dimension, for prediction. Because we use the same test set users as Park et al. (2015), we compare directly against their reported results.
Ridge Regression (Embeddings). In addition to the n-gram and topic-based ridge models of Park et al. (2015), we train ridge regression models using the word embeddings described in Section 5.1. These embeddings are averaged first per-message and then per-user, creating a 200-dimensional embedding per user to input to the model.
DAN. We modify the model proposed in Section 2 to use a Deep Averaging Network (Iyyer et al., 2015), rather than a GRU, at the word and/or message level. This takes the average across all word (or message) embeddings to produce a message (or user-) level representation.
DAN + Attn. Identical to the DAN variant except takes the weighted (rather than unweighted) average using learned attention weights.
Sequence Network (SN). Similar to our proposed model but using the final state of each GRU, rather than word or message attention.
Transformer (TN). This variant of our proposed model uses a two-layer transformer (Vaswani et al., 2017) with double-headed attention, rather than a GRU, at the message or word level.
BERT. Whereas our proposed model learns message-level representations, we instead experiment with using pre-trained BERT embeddings (Devlin et al., 2019) as our message representations. These 768-dimension message embeddings are produced by averaging across all BERT token embeddings for each message (Matero et al., 2019).
5.3 Training
All models were implemented using PyTorch (Paszke et al., 2017), with the exception of Ridge Regression which used scikit-learn (Pedregosa et al., 2011). One model was trained for each of the five personality dimensions. All deep learning models use two feed-forward layers with 512 hidden units each, followed by a final prediction layer. The GRU layers have a hidden size of 200 to match the number of embedding dimensions. Similarly, we learn a projection down to 200 dimensions for our BERT embeddings.
All hyperparameters (dropout and learning rate
<sup>2</sup>Some versions of the Big Five flip this dimension and call it "emotional stability".
<sup>3</sup>Even without this step, the models tended to artificially exclude non-English texts by assigning them very low attention weights.
word-to-message	message-to-user	OPE	CON	EXT	AGR	NEU
DAN	DAN	.579	.516	.509	.474†	.516
SN	SN	.601	.506	.512	.431	.523
DAN + Attn	DAN + Attn	.615†	.506	.530†	.499†	.528†
DAN + Attn	SN + Attn	.605	.510	.535†	.501†	.560†
SN + Attn	DAN + Attn	.625	.497	.539†	.519†	.532†
SN + Attn	SN + Attn	.626	.521	.552†	.509†	.541
TN (Attn)	SN + Attn	.544	.474	.513†	.483†	.526
Table 1: Comparison of Disattenuated Pearson R of different models for personality prediction on the test set users  $(n = 1943)$ , using different architectures to aggregate from word to message level and message to user level.  $^{\dagger}$  indicates statistically significant improves over the SN (No Attention) baseline, based on a paired t-test on the errors of each model.
word-to-message	message-to-user	OPE	CON	EXT	AGR	NEU
SN + Attn	SN + Attn	.626	.521	.552	.509	.541
BERT	DAN	.602	.512	.537	.505	.520
BERT	SN	.597	.511	.520	.522	.507
BERT	DAN + Attn	.613	.511	.570†	.533†	.536
BERT	SN + Attn	.610	.519	.544	.538†	.547†
BERT	TN (Attn)	.590	.501	.526	.523	.516
Table 2: Performance as Disattenuated Pearson R measures when using pre-trained BERT embeddings (Devlin et al., 2019) at the message level, compared to our proposed model which learns message-level representations.  $\dagger$  indicates statistically significant improvement over the SN + Attn model based on a paired t-test on the errors of each approach.
for deep models; alpha for ridge) were tuned over the development set for a single personality dimension (OPE), with the best parameters being used to train models for the remaining dimensions. The deep models were trained using a batch size of 64. Training lasted for a maximum of 20 epochs, with most models stopping after around 10 epochs due to early stopping with a patience of two epochs. To reduce memory requirements during training, each user's post history was "chunked" into sequences of at most 500 messages each. For example, a user with 1250 messages total would be divided into three instances with 500, 500, and 250 messages. This was only done for the training set; the testing and tuning sets used all messages at once.
Our evaluation aims to answer the following:
1. How successful are attention-based models at predicting personality?
2. What is the distribution of high signal versus low signal messages?
3. What is the relative importance of message-level attention over word-level attention?
6.1 Attention for Personality Prediction
Table 1 compares the performance of our proposed model, SN+Attn, against variations using different architectures to aggregate from the word to
message level and message to user level. Model performance is given as the disattenuated Pearson correlation coefficient<sup>4</sup> between the predicted and questionnaire-based personality scores.
Overall the models with attention outperform those without. Perhaps surprisingly, the  $\mathrm{SN} + \mathrm{Attn}$  at the message level typically outperformed the DAN+Attn, which may be due to the messages forming a sort of personal narrative, containing repeated themes and follow-ups to previous messages. The  $\mathrm{SN} + \mathrm{Attn}$  also tended to outperform the DAN+Attn at the word level. Our proposed model, using  $\mathrm{SN} + \mathrm{Attn}$  at both word and message level, is best for three out of five dimensions.
Table 2 shows the performance when using pretrained BERT embeddings (Devlin et al., 2019) as our message representations, rather than learning them as part of the model. As before, we see that message-level attention is generally beneficial, and additionally we find that the BERT-based models outperform our proposed model in 3 out of 5 cases.
Table 3 compares our proposed model against the state-of-the-art. Unsurprisingly, Ridge (Embeddings) is the worst-performing model overall. Although Park et al. (2015) also used ridge
4Disattenuated Pearson correlation helps account for the error of the measurement instrument (Murphy and Davidshofer, 1988; Kosinski et al., 2013). Following Lynn et al. (2018), we use reliabilities:  $r_{xx} = 0.70$  and  $r_{yy} = 0.77$ .
d	OPE	CON	EXT	AGR	NEU
Ridge (Embeddings)	200	.538	.500	.505	.444	.505
Our Proposed Model	200	.626	.521	.552	.509	.541†
Ridge with PCA (N-Grams/Topics) (Park et al., 2015)	5106	.627	.518	.558	.545	.531
Ridge with PCA (N-Grams/Topics) + Our Proposed Model	5306	.657†	.538†	.583†	.557†	.564†
regression, their models used significantly more features ( $d = 5106$  (dimensionally reduced, supervised, from an original of over  $d > 50$ , 000) compared to our  $d = 200$ ). Finally, we find that by averaging the z-scored predictions of our proposed model and Ridge (N-Grams/Topics), we obtain the overall best performance, outperforming current state-of-the-art. This suggests that the models are able to learn complementary information.
These results show that neural models with attention are better able to predict personality than those without. Because some messages are of more relevance than others, attention allows the model to better separate the signal from noise. In addition, combining the predictions of the best attention-based model, SN+Attn, with those from Park et al. (2015), the previous best, advances the state-of-the-art results over all 5 factors by a significant margin  $(p < .05$  from a paired t-test on error) and an average increase of .025, demonstrating the complementary value in these methods.
6.2 Message Attention Distribution
Results suggest not all text is equally informative when it comes to personality prediction, which is why attention helps. Figure 3 shows the distribution of standardized message-level attention weights, obtained from our proposed model, for 100 randomly-sampled test set users. Sampled users had 742 messages on average. The figure shows that any single user's messages encompass a range of relative importance. OPE skews negative, indicating that most messages of a user are of little relevance with a few being very relevant, while NEU was slightly more likely to mark messages as relevant but with less variance. By incorporating that concept of message (and word) importance via attention, we can produce better user-level representations from which to predict personality.
6.3 Effects of Word and Message Attention
Thus far we have demonstrated the importance of attention for personality prediction. However, our
Figure 3: Standardized distribution of message-level attention weights for 100 randomly-sampled test set users with at least 20 messages. The black dot indicates the max density per user (i.e. the most frequent attention weight for that person).
Table 3: Combining our best model with that of Park et al. (2015) obtains new state-of-the-art performance in terms of Disattenuated Pearson R. Number of input dimensions  $(d)$  is shown for each model.  $\dagger$  indicates a statistically significant improvement over Park et al. (2015) based on a paired t-test on the errors of each approach.
OPE	CON	EXT	AGR	NEU
No Attn	.601	.506	.512	.431	.523
Word Only	.612†	.510	.516†	.456†	.541†
Msg Only	.621†	.511	.535†	.521†	.544†
Word + Msg	.626	.521	.552†	.509†	.541
Table 4: Ablation demonstrating the importance of using word- and message-level attention. All models are sequence networks (SNs) with or without attention at the word and message levels.  $\dagger$  indicates statistically significant improvements  $(p < 0.05)$  over the No Attention baseline based on a paired t-test on the errors of each approach.