SlowGuess/NAACL_2021_OCR huggingface dataset:
Adithya V Ganesan, Matthew Matero, Aravind Reddy Ravula, Huy Vu, and H. Andrew Schwartz
Department of Computer Science, Stony Brook University
{avirinchipur, mmatero, aravula, hvu, has}@cs.stonybrook.edu
In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard  $768+$  hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effectively leverage transformers. Here, we provide a systematic study on the role of dimension reduction methods (principal components analysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a limited amount of data pose a significant difficulty which can be overcome with a pre-trained dimension reduction regime. RoBERTa consistently achieves top performance in human-level tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results comparable to the best performance with just  $\frac{1}{12}$  of the embedding dimensions.
Transformer based language models (LMs) have quickly become the foundation for accurately approaching many tasks in natural language processing (Vaswani et al., 2017; Devlin et al., 2019). Owing to their success is their ability to capture both syntactic and semantic information (Tenney et al., 2019), modeled over large, deep attention-based networks (transformers) with hidden state sizes on the order of 1000 over 10s of layers (Liu et al., 2019; Gururangan et al., 2020). In total such models typically have from hundreds of millions (Devlin et al., 2019) to a few billion parameters (Raffel et al., 2020). However, the size of such models presents a challenge for tasks involving small numbers of observations, such as for the growing number of tasks focused on human-level NLP.
Human-level NLP tasks, rooted in computational social science, focus on making predictions about people from their language use patterns. Some of the more common tasks include age and gender prediction (Sap et al., 2014; Morgan-Lopez et al., 2017), personality (Park et al., 2015; Lynn et al., 2020), and mental health prediction (Coppersmith et al., 2014; Guntuku et al., 2017; Lynn et al., 2018). Such tasks present an interesting challenge for the NLP community to model the people behind the language rather than the language itself, and the social scientific community has begun to see success of such approaches as an alternative or supplement to standard psychological assessment techniques like questionnaires (Kern et al., 2016; Eichstaedt et al., 2018). Generally, such work is helping to embed NLP in a greater social and human context (Hovy and Spruit, 2016; Lynn et al., 2019).
Despite the simultaneous growth of both (1) the use of transformers and (2) human-level NLP, the effective merging of transformers for human-level tasks has received little attention. In a recent human-level shared task on mental health, most participants did not utilize transformers (Zirikly et al., 2019). A central challenge for their utilization in such scenarios is that the number of training examples (i.e. sample size) is often only hundreds while the parameters for such deep models are in the hundreds of millions. For example, recent human-level NLP shared tasks focused on mental health have had  $N = 947$  (Milne et al., 2016),  $N = 9,146$  (Lynn et al., 2018) and  $N = 993$  (Zirikly et al., 2019) training examples. Such sizes all but rules out the increasingly popular approach of fine-tuning transformers whereby all its millions of parameters are allowed to be updated toward the specific task one is trying to achieve (Devlin et al., 2019; Mayfield and Black, 2020). Recent research not only highlights the difficulty in fine-tuning with few samples (Jiang et al., 2020) but it also becomes unreliable even with thousands
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4515-4532 June 6-11, 2021. ©2021 Association for Computational Linguistics
of training examples (Mosbach et al., 2020).
On the other hand, some of the common transformer-based approaches of deriving contextual embeddings from the top layers of a pretrained model (Devlin et al., 2019; Clark et al., 2019) still leaves one with approximately an equal number of embedding dimensions as training size. In fact, in one of the few successful cases of using transformers for a human-level task, further dimensionality reduction was used to avoid overfit (Matero et al., 2019), but an empirical understanding of the application of transformers for human-level tasks — which models are best and the relationship between embedding dimensions, sample size, and accuracy — has yet to be established.
In this work, we empirically explore strategies to effectively utilize transformer-based LMs for relatively small sample-size human-level tasks. We provide the first systematic comparison of the most widely used transformer models for demographic, personality, and mental health prediction tasks. Then, we consider the role of dimension reduction to address the challenge of applying such models on small sample sizes, yielding a suggested minimum number of dimensions necessary given a sample size for each of demographic, personality, and mental health tasks<sup>1</sup>. While it is suspected that transformer LMs contain more dimensions than necessary for document- or word-level NLP (Li and Eisner, 2019; Bao and Qiao, 2019), this represents the first study on transformer dimensionality for human-level tasks.
Recently, NLP has taken to human-level predictive tasks using increasingly sophisticated techniques. The most common approaches use n-grams and LDA (Blei et al., 2003) to model a person's language and behaviors (Resnik et al., 2013; Kern et al., 2016). Other approaches utilize word embeddings (Mikolov et al., 2013; Pennington et al., 2014) and more recently, contextual word representations (Ambalavanan et al., 2019).
Our work is inspired by one of the top performing systems at a recent mental health prediction shared task (Zirikly et al., 2019) that utilized transformer-based contextualized word embeddings fed through a non-negative matrix fac
torization to reduce dimensionality (Matero et al., 2019). While the approach seems reasonable for addressing the dimensionality challenge in using transformers, many critical questions remain unanswered: (a) Which type of transformer model is best? (b) Would fine-tuning have worked instead? and (c) Does such an approach generalize to other human-level tasks? Most of the time, one does not have a luxury of a shared task for their problem at hand to determine a best approach. Here, we look across many human-level tasks, some of which with the luxury of having relatively large sample sizes (in the thousands) from which to establish upper-bounds, and ultimately to draw generalizable information on how to approach a human-level task given its domain (demographic, personality, mental health) and sample size.
Our work also falls in line with a rising trend in AI and NLP to quantify the number of dimensions necessary. While this has not been considered for human-level tasks, it has been explored in other domains. The post processing algorithm (Mu and Viswanath, 2018) of the static word embeddings motivated by the power law distribution of maximum explained variance and the domination of mean vector turned out to be very effective in making these embeddings more discriminative. The analysis of contextual embedding models (Ethayarajh, 2019) suggest that the static embeddings contribute to less than  $5\%$  to the explained variance, the contribution of the mean vector starts dominating when contextual embedding models are used for human-level tasks. This is an effect of averaging the message embeddings to form user representations in human-level tasks. This further motivates the need to process these contextual embeddings into more discriminative features.
Lastly, our work weighs into the discussion on just which type of model is best in order to produce effective contextual embedding models. A majority of the models fall under two broad categories based on how they are pre-trained - auto-encoders (AE) and auto-regressive (AR) models. We compare the performance of AE and AR style LMs by comparing the performance of two widely used models from each category with comparable number of parameters. From the experiments involving BERT, RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) and GPT-2 (Radford et al., 2019), we find that AE based models perform better than AR style models (with comparable model sizes), and
<sup>1</sup>dimension reduction techniques can also be pre-trained leveraging larger sets of unlabeled data
RoBERTa is the best choice amongst these four widely used models.
3 Data & Tasks
We evaluate approaches over 7 human-level tasks spanning Demographics, Mental Health, and personality prediction. The 3 datasets used for these tasks are described below.
FB-Demogs. (age, gen, ope, ext) One of our goals was to leverage one of the largest human-level datasets in order to evaluate over subsamples of sizes. For this, we used the Facebook demographic and personality dataset of Kosinski et al. (2013). The data was collected from approximately 71k consenting participants who shared Facebook posts along with demographic and personality scores from Jan-2009 through Oct-2011. The users in this sample had written at least a 1000 words and had selected English as their primary language. Age (age) was self-reported and limited to those 65 years or younger (data beyond this age becomes very sparse) as in (Sap et al., 2014). Gender (gen) was only provided as a limited single binary, male-female classification.
Personality was derived from the Big 5 personality traits questionnaires, including both extraversion (ext-one's tendency to be energized by social interaction) and openness (ope, one's tendency to be open to new ideas) (Schwartz et al., 2013). Disattenuated Pearson correlation $^{2}$  ( $r_{dis}$ ) was used to measure the performance of these two personality prediction tasks.
CLPsych-2018. (bsag, gen2) The CLPsych 2018 shared task (Lynn et al., 2018) consisted of sub-tasks aimed at early prediction of mental health scores (depression, anxiety and  $\mathrm{BSAG}^3$  score) based on their language. The data for this shared task (Power and Elliott, 2005) comprised of English essays written by 11 year old students along with their gender (gen2) and income classes. There were 9217 students' essays for training and 1000 for testing. The average word count in an essay was less than 200. Each essay was annotated with the student's psychological health measure,
BSAG (when 11 years old) and distress scores at ages 23, 33, 42 and 50. This task used a disattenuated pearson correlation as the metric  $(r_{dis})$ .
CLPsych-2019. (sui) This 2019 shared task (Zirikly et al., 2019) comprised of 3 sub-tasks for predicting the suicide risk level in reddit users. This included a history of user posts on r/SuicideWatch (SW), a subreddit dedicated to those wanting to seek outside help for processing their current state of emotions. Their posts on other subreddits (NonSuicideWatch) were also collected. The users were annotated with one of the 4 risk levels: none, low, moderate and severe risk based on their history of posts. In total this task spans 496 users in training and 125 in testing. We focused on Task A, predicting suicide risk of a user by evaluating their (English) posts across SW, measured via macro-F1.
FB-Demogs	CLPsych 2018	CLPsych 2019
Npt	Sap et al.	Lynn et al.	Zirikly et al.
56,764	9,217	496
Nmax	10,000	9,217	496
Nte	5,000	1,000	125
Table 1: Summary of the datasets.  $N_{pt}$  is the number of users available for pre-training the dimension reduction model;  $N_{max}$  is the maximum number of users available for task training. For CLPsych 2018 and CLPsych 2019, this would be the same sample as pre-training data. For Facebook, a disjoint set of 10k users was available for task training;  $N_{te}$  is the number of test users. This is always a disjoint set of users from the pre-training and task training samples.
Here we discuss how we utilized representations from transformers, our approaches to dimensionality reduction, and our technique for robust evaluation using bootstrapped sampling.
4.1 Transformer Representations
The second to last layer representation of all the messages was averaged to produce a 768 dimensional feature for each user. These user representations are reduced to lower dimensions as described in the following paragraphs. The message representation from a layer was attained by averaging the token embeddings of that layer. To con
4The second to last layer was chosen owing to its consistent performance in capturing semantic and syntactic structures (Jawahar et al., 2019).
$^{2}$ Disattenuated Pearson correlation helps account for the error of the measurement instrument (Kosinski et al., 2013; Murphy and Davidshofer, 1988). Following (Lynn et al., 2020), we use reliabilities:  $r_{xx} = 0.70$  and  $r_{yy} = 0.77$ .  
 $^{3}$ Bristol Social Adjustment Guide (Ghodsian, 1977) scores contains twelve sub-scales that measures different aspects of childhood behavior.
sider a variety of transformer LM architectures, we explored two popular auto-encoder (BERT and RoBERTa) and two auto-regressive (XLNet and GPT-2) transformer-based models.
For fine-tuning evaluations, we used the transformer based model that performs best across the majority of our task suite. Transformers are typically trained on single messages or pairs of messages, at a time. Since we are tuning towards a human-level task, we label each user's message with their human-level attribute and treat it as a standard document-level task (Morales et al., 2019). Since we are interested in relative differences in performance, we limit each user to at most 20 messages - approximately the median number of messages, randomly sampled, to save compute time for the fine tuning experiments.
Algorithm 1 Dimension Reduction and Evaluation
Notation:  $h_D$ : hidden size,  $f(\cdot)$ : function to train dimension reduction,  $\theta$ : Linear Model,  $g(\cdot, \cdot)$ : Logistic loss function for classification and L2 loss for regression,  $\eta$ : learning rate,  $T$ : Number of iterations (100).
Data:  $D_{pt} \in R^{N_{pt} \times h_D}$ : Pre-training embeddings,  $D_{max} \in R^{N_{max} \times h_D}$ : Task training embeddings,  $D_{te} \in R^{N_{te} \times h_D}$ : Test embeddings,  $Y_{max}$ : Outcome for train set,  $Y_{te}$ : Outcome for test set.
1:  $W\gets f(D_{pt})$
2:  $\bar{D}_{max}\gets D_{max}W$
3:  $\bar{D}_{te}\gets D_{te}W$
4: for  $i = 1,\dots ,10$  do
5:  $\theta_{i}^{(0)}\gets \vec{0}$
6: Sample  $(\bar{D}_{ta},Y_{ta})$  from  $(\bar{D}_{max},Y_{max})$
7: for  $j = 1,\dots ,T$  do
8:  $\theta_{i}^{(j)}\gets \theta_{i}^{(j - 1)} - \eta \nabla g(\bar{D}_{ta},Y_{ta})$
9: end for
10:  $\hat{Y}_{te_i} \gets \bar{D}_{te}\theta_i^{(T)}$
11: end for
12: Evaluate  $(\hat{Y}_{te}, Y_{te})$
4.2 Dimension Reduction
We explore singular value decomposition-based methods such as Principal components analysis (PCA) (Halko et al., 2011), Non-negative matrix factorization (NMF) (Févotte and Idier, 2011) and Factor analysis (FA) as well as a deep learning approach: multi-layer non linear auto encoders (NLAE) (Hinton and Salakhutdinov, 2006). We
also considered the post processing algorithm (PPA) of word embeddings $^5$  (Mu and Viswanath, 2018) that has shown effectiveness with PCA on word level (Raunak et al., 2019). Importantly, besides transformer LMs being pre-trained, so too can dimension reduction. Therefore, we distinguish: (1) learning the transformation from higher dimension to lower dimensions (preferably on a large data sample from the same domain) and (2) applying the learned transformation (on the task's train/test set). For the first step, we used a separate set of 56k unlabeled user data in the case of FB-demog $^6$ . For CLPsych-2018 and -2019 (where separate data from the exact domains was not readily available), we used the task training data to train the dimension reduction. Since variance explained in factor analysis typically follows a power law, these methods transformed the 768 original embedding dimensions down to  $k$ , in powers of 2: 16, 32, 64, 128, 256 or 512.
4.3 Bootstrapped Sampling & Training
We systematically evaluate the role of training sample  $(N_{ta})$  versus embedding dimensions  $(k)$  for human-level prediction tasks. The approach is described in algorithm 1. Varying  $N_{ta}$ , the task-specific train data (after dimension reduction) is sampled randomly (with replacement) to get ten training samples with  $N_{ta}$  users each. Small  $N_{ta}$  values simulate a low-data regime and were used to understand its relationship with the least number of dimensions required to perform the best  $(N_{ta}$  vs  $k)$ . Bootstrapped sampling was done to arrive at a conservative estimate of performance. Each of the bootstrapped samples was used to train either an L2 penalized (ridge) regression model or logistic regression for the regression and classification tasks respectively. The performance on the test set using models from each bootstrapped training sample was recorded in order to derive a mean and standard error for each  $N_{ta}$  and  $k$  for each task.
To summarize results over the many tasks and possible  $k$  and  $N_{ta}$  values in a useful fashion, we propose a 'first  $k$  to peak (fkp)' metric. For each  $N_{ta}$ , this is the first observed  $k$  value for which the mean score is within the 95% confidence interval of the peak performance. This quantifies the minimum number of dimensions required for peak performance.
5The'D'value was set to  $\lfloor \frac{\text{number of dimensions}}{100}\rfloor$
these pre-trained dimension reduction models are made available.
Nta	type	LM name	demographics	personality	mental health
age (r)	gen (F1)	gen2 (F1)	ext (r_dis)	ope (r_dis)	bsag (r_dis)	sui (F1)
100	AE	BERT	0.533	0.703	0.761	0.163	0.184	0.424	0.360
AE	RoBERTa	0.589	0.712	0.761	0.123	0.203	0.455	0.363
AR	XLNet	0.582	0.582	0.744	0.130	0.203	0.441	0.315
AR	GPT-2	0.517	0.584	0.624	0.082	0.157	0.397	0.349
500	AE	BERT	0.686	0.810	0.837	0.278	0.354	0.484	0.466
AE	RoBERTa	0.700	0.802	0.852	0.283	0.361	0.490	0.432
AR	XLNet	0.697	0.796	0.821	0.261	0.336	0.508	0.439
AR	GPT-2	0.646	0.756	0.762	0.211	0.280	0.481	0.397
5.1 Best LM for Human-Level Tasks
We start by comparing transformer LMs, replicating the setup of one of the state-of-the-art systems for the CLPsych-2019 task in which embeddings were reduced from BERT-base to approximately 100 dimensions using NMF (Matero et al., 2019). Specifically, we used 128 dimensions (to stick with powers of 2 that we use throughout this work) as we explore the other LMs over multiple tasks (we will explore other dimensions next) and otherwise use the bootstrapped evaluation described in the method.
Table 2 shows the comparison of the four transformer LMs when varying the sample size  $(N_{ta})$  between two low data regimes: 100 and  $500^{7}$ . RoBERTa and BERT were the best performing models in almost all the tasks, suggesting autoencoders based LMs are better than auto-regressive models for these human-level tasks. Further, RoBERTa performed better than BERT in the majority of cases. Since the number of model parameters are comparable, this may be attributable to RoBERTa's increased pre-training corpus, which is inclusive of more human discourse and larger vocabularies in comparison to BERT.
Table 2: Comparison of most commonly used auto-encoders (AE) and auto-regressor (AR) language models after reducing the 768 dimensions to 128 using NMF and trained on 100 and 500 samples  $(N_{ta})$  for each task.  $(N_{ta})$  pertains to the number of samples used for training each task. Classification tasks (gen, gen2 and sui) were scored using macro-F1 (F1); the remaining regression tasks were scored using pearson-r (r)/disattenuated pearson-r  $(r_{dis})$ . AE models predominantly perform the best. RoBERTa and BERT show consistent performance, with the former performing the best in most tasks. The LMs in the table were base models (approx. 110M parameters).
Nta	Method	Age	Gen
100	Fine-tuned	0.54	0.54
Pre-trained	0.56	0.63
500	Fine-tuned	0.64	0.60
Pre-trained	0.66	0.74
Table 3: Comparison of task specific fine tuning of RoBERTa (top 2 layers) and pre-trained RoBERTa embeddings (second to last layer) for age and gender prediction tasks. Results are averaged across 5 trials randomly sampling users equal to  $N_{ta}$  from the Facebook data and reducing messages to maximum of 20 per user.
5.2 Fine-Tuning Best LM
We next evaluate fine-tuning in these low data situations<sup>8</sup>. Utilizing RoBERTa, the best performing transformer from the previous experiments, we perform fine-tuning across the age and gender tasks. Following (Sun et al., 2019; Mosbach et al., 2020), we freeze layers 0-9 and fine-tune layers 10 and 11. Even these top 2 layers alone of RoBERTa still result in a model that is updating tens of millions of parameters while being tuned to a dataset of hundreds of users and at most 10,000 messages.
In table 3, results for age and gender are shown for both sample sizes of 100 and 500. For Age, the average prediction across all of a user's messages was used as the user's prediction and for gender the mode was used. Overall, we find that fine-tuning
7The performance of all transformer embeddings without any dimension reduction along with smaller sized models can be found in the appendix section D.3.
As we are focused on readily available models, we consider substantial changes to the architecture or training as outside the scope of this systematic evaluation of existing techniques.
Npt	Nta	Reduction	demographics	personality	mental health
age (r)	gen (F1)	gen2 (F1)	ext (rdis)	ope (rdis)	bsag (rdis)	sui (F1)
56k*	100	PCA	0.650	0.747	0.777	0.189	0.248	0.466	0.392
PCA-PPA	0.517	0.715	0.729	0.173	0.176	0.183	0.358
FA	0.534	0.722	0.729	0.171	0.183	0.210	0.360
NMF	0.589	0.712	0.761	0.123	0.203	0.455	0.363
NLAE	0.654	0.744	0.782	0.188	0.263	0.447	0.367
500	PCA	0.729	0.821	0.856	0.346	0.384	0.514	0.416
PCA-PPA	0.707	0.814	0.849	0.317	0.349	0.337	0.415
FA	0.713	0.819	0.849	0.322	0.361	0.400	0.415
NMF	0.700	0.802	0.852	0.283	0.361	0.490	0.432
NLAE	0.725	0.820	0.843	0.340	0.394	0.485	0.409
500	100	PCA	0.644	0.749	0.788	0.186	0.248	0.412	0.392
NLAE	0.634	0.743	0.744	0.159	0.230	0.433	0.367
500	PCA	0.726	0.819	0.850	0.344	0.382	0.509	0.416
NLAE	0.715	0.798	0.811	0.312	0.360	0.490	0.409
Table 4: Comparison of different dimension reduction techniques of RoBERTa embeddings (penultimate layer) reduced down to 128 dimensions and  $N_{ta} = 100$  and 500. Number of user samples for pre-trianing the dimension reduction model,  $N_{pt}$  was 56k except for gen2, bsag (which had 9k users) and sui (which had 496 users). PCA performs the best overall and NLAE performs as good as PCA consistently. With uniform pre-training size ( $N_{pt} = 500$ ), PCA performs better than NLAE.
offers lower performance with increased overhead for both train time and modeling complexity (hyperparameter tuning, layer selection, etc).
We did robustness checks for hyper-parameters to offer more confidence that this result was not simply due to the fastidious nature of fine-tuning. The process is described in Appendix B, including an extensive exploration of hyper-parameters, which never resulted in improvements over the pretrained setup. We are left to conclude that fine-tuning over such small user samples, at least with current typical techniques, is not able to produce results on par with using transformers to produce pre-trained embeddings.
5.3 Best Reduction technique for Human-Level Tasks