SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiaqi Mu and Pramod Viswanath. 2018. All-but-thetop: Simple and effective postprocessing for word representations. In International Conference on Learning Representations.
K. Murphy and C. Davidshofer. 1988. Psychological testing: Principles and applications.
Gregory Park, H. A. Schwartz, J. Eichstaedt, M. Kern, M. Kosinski, D. Stillwell, Lyle H. Ungar, and M. Seligman. 2015. Automatic personality assessment through social media language. Journal of personality and social psychology, 108 6:934-52.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Chris Power and Jane Elliott. 2005. Cohort profile: 1958 British birth cohort (National Child Development Study). International Journal of Epidemiology, 35(1):34-41.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 235–243, Florence, Italy. Association for Computational Linguistics.
Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013. Using topic modeling to improve prediction of neuroticism and depression in college students. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1348-1353, Seattle, Washington, USA. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Association for Computational Linguistics.
Maarten Sap, Greg Park, Johannes C Eichstaedt, Margaret L Kern, David J Stillwell, Michal Kosinski, Lyle H Ungar, and H Andrew Schwartz. 2014. Developing age and gender predictive lexica over social media. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Benedetto Saraceno, Mark van Ommeren, Rajaie Batniji, Alex Cohen, Oye Gureje, John Mahoney, Devi Sridhar, and Chris Underhill. 2007. Barriers to improvement of mental health services in low-income and middle-income countries. The Lancet, 370(9593):1164-1174.
H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9):e73791.
H. Andrew Schwartz, Salvatore Giorgi, Maarten Sap, Patrick Crutchley, Lyle Ungar, and Johannes Eichstaedt. 2017. DLATK: Differential language analysis ToolKit. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 55-60, Copenhagen, Denmark. Association for Computational Linguistics.
Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5248-5264, Online. Association for Computational Linguistics.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194-206. Springer.
Y. Tausczik and J. Pennebaker. 2010. The psychological meaning of words: Liwc and computerized text analysis methods. Journal of Language and Social Psychology, 29:24 - 54.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Philip S Wang, Michael Lane, Mark Olfson, Harold A Pincus, Kenneth B Wells, and Ronald C Kessler. 2005. Twelve-month use of mental health services in the united states: results from the national comorbidity survey replication. Archives of general psychiatry, 62(6):629-640.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
Ayah Zirikly, Philip Resnik, Özlem Uzuner, and Kristy Hollingshead. 2019. CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 24-33, Minneapolis, Minnesota. Association for Computational Linguistics.
Implementation. All the experiments were implemented using Python, DLATK (Schwartz et al., 2017), HuggingFace Transformers (Wolf et al., 2019), and PyTorch (Paszke et al., 2019). The environments were instantiated with a seed value of 42, except for fine-tuning which used 1337. Code to reproduce all results is available in our github page: github.com/adithya8/ContextualEmbeddingDR/
Infrastructure. The deep learning models such as stacked-transformers and NLAE were run on single GPU with batch size given by:
$$
b a t c h s i z e = \left\lfloor \frac {G P U m e m o r y - m o d e l s i z e}{(f l o a t i n g p r e c i s i o n / 8) * (\delta)} \right\rfloor
$$
$$
\delta = \left\{ \begin{array}{l} \text {t r a i n a b l e} _ {\text {p a r a m s}}; \quad \text {f o r f i n e t u n i n g} \\ \text {(l a y e r s * h i d d e n _ {s i z e} * m a x _ {t o k e n s})}; \\ \text {f o r e m b e d d i n g r e t r i e v a l} \end{array} \right.
$$
where GPU memory and model sizes (space occupied by the model) are in bytes, trainableparams corresponds to number of trainable parameters during fine tuning and layers corresponds to the number of layers of embeddings required, the hidden_size is the number of dimensions in the hidden state and max_tokens is the maximum number of tokens (after tokenization) in any batch. We carried out the experiments with 1 NVIDIA Titan Xp GPU which has around 12 GB of memory. All the other methods were implemented on CPU.
Figure A1: Depiction of Dimension Reduction method - Transformer embeddings of domain data ( $N_{pt}$  users' embeddings $^9$ ) is used to pre-train a dimension reduction model that transforms the embeddings down to  $k$  dimensions. This step is followed by applying this learned reduction model on task's train and test data embeddings. These reduced train features ( $N_{max}$  users) are then bootstrap sampled to produce 10 sets of  $N_{ta}$  users each for training task specific models. All these 10 task specific models are evaluated on the reduced test features consisting of  $N_{te}$  users during task evaluation. The mean and standard deviation of the task specific metric are collected.
NLAE architecture. The model architecture for the Non-linear auto-encoders in Table 4 was a twin network taking inputs of 768 dimensions and reducing it to 128 dimensions through 2 layers and
<sup>9</sup>Generation of user embeddings explained in detail under methods.
reconstructs the original 768 dimensional representation with 2 layers. This architecture was chosen balancing the constraints of enabling the non-linear associations while keeping total parameters low given the low sample size context. The formal definition of the model is:
$$
x _ {c o m p} = f \left(W _ {2} ^ {T} f \left(W _ {1} ^ {T} x + b _ {1}\right) + b _ {2}\right)
$$
$$
x _ {d e c o m p} = \overline {{W _ {1}}} ^ {T} f (\overline {{W _ {2}}} ^ {T} x _ {c o m p} + \overline {{b _ {2}}}) + \overline {{b _ {1}}}
$$
$$
x, x _ {d e c o m p} \in R ^ {7 6 8}, x _ {c o m p} \in R ^ {1 2 8}
$$
$$
W _ {1} \in R ^ {7 6 8 * 4 4 8}, W _ {2} \in R ^ {4 4 8 * 1 2 8}
$$
$$
b _ {1}, \overline {{b _ {2}}} \in R ^ {4 4 8}, b _ {2} \in R ^ {1 2 8}, \overline {{b _ {1}}} \in R ^ {7 6 8}
$$
$$
\overline {{W _ {1}}} \in R ^ {4 4 8 * 7 6 8}, \overline {{W _ {2}}} \in R ^ {1 2 8 * 4 4 8}
$$
$$
f (a) = \max  (a, 0); \forall a \in R
$$
NLAE Training. The data for domain pretraining of dimension reduction was split into 2 sets for NLAE alone: training and validation sets.  $90\%$  of the domain data was randomly sampled for training the NLAE and the remaining  $10\%$  of pretraining data was used to validate hyper-parameters after every epoch. This model was trained with an objective to minimise the reconstruction mean squared loss over multiple epochs. It was trained until the validation loss increased over 3 consecutive epochs. AdamW was the optimizer used with the learning rate set to 0.001. This took around 30-40 epochs depending upon the dataset.
Fine-tuning. In our fine-tuning configuration we freeze all but the top 2 layers of the best LM, to prevent over fitting and vanishing gradients at the lower layers (Sun et al., 2019; Mosbach et al., 2020). We also apply early stopping (varied the patience between 3 and 6 depending upon the task). Other hyperparameters for this experiment include L2-regularization (in the form of weight-decay on AdamW optimizer, set to 1), dropout set to 0.3, batch size set to 10, learning rate initialized to 5e-5, and the number of epochs was set to max of 15, which was limited by early stopping between 5-10 depending on the task and early stopping patience.
We arrived at these hyperparameter values after an extensive search. The weight decay param was searched in [100, 0.01], dropout within [0.1, 0.5], and learning rate between [5e-4, 5e-5].
Due to human subjects privacy constraints, most data are not able to be publicly distributed but they are available from the original data owners via requests for research purposes (e.g. CLPsych-2018 and CLPsych-2019 shared tasks).
D.1 Results on higher  $N_{ta}$
We can see that reduction still helps in majority of tasks in higher  $N_{ta}$  from Figure A2. As expected, the performance starts to plateau at higher  $N_{ta}$  values and it is visibly consistent across most tasks. With the exception of age and gender prediction using facebook data, all the other tasks benefit from reduction.
D.2 Results on classification tasks
Figure A3 compares the performance of reduced dimensions at low samples size scenario ( $N_{ta} \leq 1000$ ) in classification tasks. Except for a few  $N_{ta}$  values in gender prediction using the facebook data, all the other tasks benefit from reduction in achieving the best performance.
D.3 LM comparison for no reduction & Smaller models.
Table A1 compares the performance of the language models without applying any dimension reduction of the embeddings and the performance of the best transformer models is also compared with smaller models (and distil version) after reducing second to last lasyer representation to 128 dimensions in table A2.
D.4 Least dimensions required: Higher  $N_{ta}$
The 'fkp' plateaus as the number of training samples grow as seen in table A3.
E.1 Pre-trained vs Fine-Tuned models
We also find that fine-tuned model doesn't perform better than the pre-trained model for users with typical message lengths, but is better at handling longer sequences upon training it on the tasks' data. This is evident from the graphs in figure A4.
E.2 PCA vs NMF.
From figure A5, we can see that LIWC variables like ARTICLE, INSIGHT, PERCEPT (perceptual
Nta	type	LM name	demographics	personality	mental health
age (r)	gen (f1)	gen2 (f1)	ext (r_dis)	ope (r_dis)	bsag (r_dis)	sui (f1)
100	AE	BERT	0.615	0.754	0.758	0.176	0.225	0.457	0.400
AE	RoBERTa	0.649	0.753	0.788	0.167	0.213	0.443	0.381
AR	XLNet	0.625	0.698	0.755	0.144	0.152	0.457	0.357
AR	GPT-2	0.579	0.708	0.681	0.090	0.110	0.361	0.335
500	AE	BERT	0.721	0.831	0.849	0.332	0.395	0.507	0.489
AE	RoBERTa	0.737	0.830	0.859	0.331	0.382	0.519	0.447
AR	XLNet	0.715	0.810	0.828	0.314	0.364	0.506	0.424
AR	GPT-2	0.693	0.794	0.790	0.242	0.307	0.508	0.371
Table A1: Comparison of various auto-encoders(AE) and auto-regressor(AR) language models trained on 100 and 500 samples ( $N_{ta}$ ) for each task using all the dimensions of transformer embeddings. RoBERTa and BERT show consistent performance.
Nta	LM	demographics	personality	mental health
age (r)	gen (F1)	gen2 (F1)	ext (r_dis)	ope (r_dis)	bsag (r_dis)	sui (F1)
100	BERT	0.533	0.703	0.761	0.163	0.184	0.424	0.360
RoBERTa	0.589	0.712	0.761	0.123	0.203	0.455	0.363
DistilRoBERTa	0.568	0.640	0.731	0.130	0.207	0.446	0.355
ALBERT	0.525	0.689	0.710	0.111	0.218	0.413	0.355
500	BERT	0.686	0.810	0.837	0.278	0.354	0.484	0.466
RoBERTa	0.700	0.802	0.852	0.283	0.361	0.490	0.432
DistilRoBERTa	0.687	0.796	0.826	0.246	0.346	0.503	0.410
ALBERT	0.668	0.792	0.799	0.237	0.337	0.453	0.385
Table A2: Comparison of the best performing auto-encoder models with a smaller LMs (like ALBERT (Lan et al., 2019) and DistilRoBERTa (Sanh et al., 2019) after reduction to 128 dimensions. These results suggest that the reduction of the larger counterparts produce better results than reducing these smaller LMs' representations.
process), COGPROC (cognitive process) negatively correlates to the difference in absolute error of PCA and NMF. These variables also happen to have higher correlation with the openness scores (Schwartz et al., 2013). We also see that characteristics typical of an open person like interest in arts, music, and writing (Kern et al., 2014) appear in the word clouds.
The divergence of the absolute errors in NMF and PCA is seen in bsag and ope tasks as well. From graphs in figure A6 we can see that the sequence length at which we see this behavior is close to the previously observed value in age and ext tasks.
E.3 PCA vs Pre-trained.
PCA models overall perform better than pre-trained model in low sample regime and from figure A7, we can see that PCA captures slang, affect and standard social media abbreviations better than the
pre-trained models. The task specific linear layer is better able to capture social media language with fewer dimensions (reduced by PCA) than from the original 768 features produced by the pre-trained models.
Dimension reduced (mean ± std err)
All dimensions (mean)
Figure A2: Performance recorded for reduced dimensions for all tasks at higher  $N_{ta}$  values (≥ 1000). Reduction continues to help in performing the best in personality and mental-health tasks. The 'fkp' is observed to be shifting to a higher value, due to the rise in performance of no reduction and the reduction of standard error.
Figure A3: Comparison of performance in gen, gen2 and sui tasks for varying  $N_{ta}$  between 50 and 1000.
Dimension reduced All dimensions (mean ± std err) (mean)
Figure A4: The absolute error in age prediction for the fine-tuned model is higher than pre-trained models for users with short messages. Fine-tuned models have smaller errors for users with longer messages.
Figure A6: Comparison of the absolute error of NMF and PCA with the average number of 1 grams per message. We see that the absolute error of NMF models starts diverging at longer text sequences for the bsag and the ope tasks as well.
Figure A5: The word cloud of the LIWC variables (left) and the 1 grams (right) having negative correlation with the difference in the absolute error of PCA and NMF in Openness prediction. Benjamini-Hochberg FDR.  $p < .05$ . We can see that LIWC variables and 1 grams more correlated of a person exhibiting more openness are better captured by the PCA model than the NMF.
Figure A7: Terms having negative (left) and positive (right) correlations with the difference in the absolute error of the PCA and pre-trained model in age prediction. Benjamini-Hochberg FDR.  $p < .05$ . The error in the PCA model is lesser than pre-trained models when messages contain more slang, affect words and social media abbreviations.
Nta	demographics	personality	mental health
2000	768	90	64
5000	768	181	64
10000	768	181	64
Table A3: First  $k$  to peak for each set of tasks: the least value of  $k$  that performed statistically equivalent  $\left( {p > {.05}}\right)$  to the best performing setup (peak). Integer shown is the exponential median of the set of tasks.