A Systematic Evaluation of Transformer-LM Representations for Capturing Author States and Traits
Authors: Khushboo Singh, Vasudha Varadarajan, Adithya V Ganesan, August Håkan Nilsson, Nikita Soni, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle Ungar, Richard N Rosenthal, H. Schwartz
Accepted at: acl25

Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits

Khushboo Singh1, Vasudha Varadarajan1, Adithya V. Ganesan1, August Håkan Nilsson 4, Nikita Soni1, Syeda Mahwish1, Pranav Chitale1, Ryan L. Boyd2, Lyle Ungar3, Richard N. Rosenthal1, H. Andrew Schwartz1
1Stony Brook University, 2University of Texas at Dallas, 3University of Pennsylvania 4Oslo Business School at Oslo Metropolitan University {khusingh, has}@cs.stonybrook.edu,

Abstract
Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. In this study, we systematically evaluate a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. Analyses span three temporal levels—short daily text responses about current affect, text aggregated over two-weeks, and user-level text collected over two years—allowing us to examine how each model’s strengths align with the underlying stability of different constructs. The findings show that mental health signals emerge as the most accurately predicted dimensions (\( r \approx 0.6 \)) across all temporal scales. At the daily scale, smaller models like DeBERTa and HaRT often performed better, whereas, at longer scales or with greater context, larger model like Llama3-8B performed the best. Also, aggregating text over the entire study period yielded stronger correlations for outcomes, such as age and income. Overall, these results suggest the importance of selecting appropriate model architectures and temporal aggregation techniques based on the stability and nature of the target variable.

1 Introduction
Recently, language model representations (that is, embeddings) have shown strong promise in improving psychological assessments of mental health and well-being now approaching the theoretical upper limit in accuracy for some outcomes (Kjell et al., 2022). However, their utility in different constructs is hitherto inconsistent. A systematic evaluation is yet to be performed to determine what types of psychological attributes can best be captured in language (Boyd and Markowitz, 2024), and by which LM. Psychological variables differ by many factors, fundamentally including (a) their stability – from being more state-like (i.e. changing frequently) to more trait-like (i.e. changing slowly) as well as (b) their construct domains – encompassing areas such as emotional states, personality traits, cognitive functions, and behavioral tendencies.

In this study, we systematically evaluate the effectiveness of Transformer-based LM representations for capturing human psychological states and traits from textual data. Unlike prior work that has predominantly focused on inferring the mental states of conversational participants (Jara-Ettinger and Rubio-Fernandez, 2021), our approach centers on self-expressed narratives—treating participants as the authors of their own psychological profiles. By leveraging a unique dataset consisting of language collected in bursts over daily intervals – Ecological Momentary Assessments (EMA) – and aggregated over longer periods (wave-level) across a two-year span, we directly compare the capability of various LLM-based embeddings to predict standardized psychological scores. These scores span multiple domains, including affect/emotion, personality, mental health, sociodemographics, and health behaviors, thereby offering a comprehensive assessment of language models in psychological

Figure 1: Conceptual framework illustrating how Language Models (LMs) capture temporal dynamics of psychological constructs across varying levels of stability. Constructs are categorized into states (highly variable, e.g., mood), dispositions (moderately stable, e.g., stress), and traits (highly stable, e.g., personality). This figure underscores the study’s focus on aligning LM architectures with psychological stability to enhance predictive performance across temporal granularities.
evaluation (Nilsson et al., 2024).

Our key contributions are: (1) We provide a systematic comparison of different LLM representations’ ability to capture 43 psychological variables; (2) We characterize LLMs by the domains they capture and their ability to capture more stable to less stable attributes; (3) We introduce a method based on measurement theory to determine the stability of psychological constructs, characterizing the temporal granularity at which it is best measured for downstream analysis; (4) We evaluate different outcome aggregation methods to assess their impact on capturing psychological constructs accurately; and finally, (5) We offer best practices for leveraging LLMs in psychology-related tasks, recommending which models and aggregation strategies are best suited for different types of variables.

2 Related Work

Understanding psychological states and traits through language has been a longstanding focus of both psychology and computational linguistics. The evolution of analyzing psychological states through language has progressed from simple lexicon-based tools such as LIWC (Boyd et al., 2022) to more sophisticated approaches using distributed word embeddings (Carducci et al., 2018) and ultimately to advanced contextualized models for capturing personality (Safdari et al., 2023), emotions (Al-Twairesh, 2021), and substance use (Shah-Mohammadi and Finkelstein, 2024). Concurrently, researchers have examined stable individual differences by initially employing bag-of-words and topic modeling methods to predict personality traits, gender, and socioeconomic status (Schwartz et al., 2013).

Modern transformer-based architectures have enhanced capturing the nuanced relationships between words and their broader context, enabling more accurate psychological modeling through language analysis (Kjell et al., 2022). Encoder-based models have emerged as a popular choice for modeling psychological constructs within natural language processing (NLP) since their bidirectional nature helps capture richer semantics (Qorib et al., 2024; Saattrup Nielsen et al., 2024), with significantly fewer parameters than decoder-only counterparts. They have been applied to identify various psychological factors ranging from mental health (Greco et al., 2023; Juhng et al., 2023; Bauer et al., 2024), cognitive distortions and thinking patterns (Varadarajan et al., 2025a; Zong et al., 2020), dissonance and belief changes (Varadarajan et al., 2022, 2023, 2025b; Lahnala et al., 2025), affective factors (Salmerón-Ríos et al., 2024; Li et al., 2024; Hasan et al., 2024; Fabunmi et al., 2025; Xu and Jiang, 2024), substance behaviors (Mahbub et al., 2025) to stable factors like personality (Lynn et al., 2020; Santos and Paraboni, 2022) and demographic attributes (Benton et al., 2016). However, previous studies have also demonstrated the effectiveness of other types of model architectures - human-level LMs like HaRT (Soni et al., 2022, 2024b) in capturing subtle mood changes, valence, arousal, empathy and distress in text (Ganesan et al., 2022; Soni et al., 2025); auto-regressive LMs like GPT-2 for assessing stable traits like personality (Soni et al., 2024a); prompting-based methods have also been successfully applied to identify language patterns indicative of mental health, personality factors (V Ganesan et al., 2023; Ganesan et al., 2024; Varadarajan et al., 2024). Given the high degree of variability of architectures in state-of-the-art mental health and psychology-related tasks, in this work we specifically focus on evaluating the modern LLM architectures.

Unlike prior studies, we assess LLMs performance across multiple temporal granularities (daily messages, biweekly waves, and aggregated user histories) using a longitudinal dataset of self-described feelings, which are particularly suited for authors’ own descriptions, as feelings are inherently subjective.

3 Dataset

The dataset for this study was collected by recruiting service industry workers in the United States Nilsson et al. (2024). The participant population predominantly comprises middle-aged individuals who identify as female, representing approximately 75% of the participants. Their ages range from 18 to 68 years, with a mean age of 35 years (standard deviation: 9.47). These individuals were recruited through professional and online groups within the United States and are known to be at high risk for excessive alcohol use (Jose et al., 2022). This demographic choice is particularly relevant for studies focusing on occupational stressors prevalent within the service industry.

Over a two-year period, data were collected in six 14-day “waves” via a custom smartphone application. The participants were prompted three
times a day for Ecological Momentary Assessments (EMAs), where they provided a textual response (with minimum of 200 characters) in English to the prompt eliciting the current affective state: “Please describe in 2 to 3 sentences how you are currently feeling.” (See Table A1 in the Appendix for additional illustrative EMA responses). Each wave thus yielded up to 42 responses per participant, totaling a potential 42 x 6 = 252 responses across all six waves. To ensure data quality, only individuals who completed at least two waves with a minimum of two responses per wave were included, resulting in 10,108 EMAs from 120 participants across 406 total user waves. (See Table 1). Alongside textual responses, an EMA also consisted of ratings for affect, stress level, number of drinks during the past 24 hours and craving for alcohol. Additionally, at the start of each wave, participants completed comprehensive questionnaires evaluating personality traits, mental health status, affective states, stress, and alcohol use. After decades of research, EMAs have recently emerged as the standard in ambulatory psychology – the study of psychological states in everyday life (Slivinski et al., 2018). This design integrates dynamic, day-to-day EMA data with baseline wave-level assessments, offering a holistic longitudinal perspective on participants’ static characteristics and evolving emotional and behavioral patterns.

<table>
  <tr>
    <th>Statistic</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>Total EMAs</td>
    <td>10,108</td>
  </tr>
  <tr>
    <td>EMAs per user-wave (mean ± SD)</td>
    <td>25 ± 13</td>
  </tr>
  <tr>
    <td>EMAs per user (mean ± SD)</td>
    <td>84.9 ± 50</td>
  </tr>
  <tr>
    <td>Sentences per EMA (mean ± SD)</td>
    <td>3.9 ± 4.3</td>
  </tr>
  <tr>
    <td>Tokens per EMA (mean ± SD)</td>
    <td>50.51 ± 14</td>
  </tr>
  <tr>
    <td>Tokens per user-wave (mean ± SD)</td>
    <td>1,267 ± 729</td>
  </tr>
  <tr>
    <td>Tokens per user (mean ± SD)</td>
    <td>4,288 ± 2,745</td>
  </tr>
  <tr>
    <td>Total users</td>
    <td>120</td>
  </tr>
  <tr>
    <td>Total user-waves</td>
    <td>406</td>
  </tr>
</table>

Table 1: Descriptive statistics of dataset for all 6 waves over 2 years. A user-wave refers to pairing of a participant and a wave (14-day period).

4 Methodology

This research explores three hierarchical levels of data analysis: the message-level, which focuses on individual EMA responses; the wave-level, which involves analysis of aggregated EMA responses over 14-day periods; and the user-level, which aggregates EMA data across all waves to examine long-term psychological trends.

4.1 Models

We systematically evaluate a broad set of Transformer-based architectures autoencoder models—such as BERT (Devlin, 2018), RoBERTa (Liu et al., 2019), and DeBERTa (He et al., 2020); encoder-decoder architectures like T5 (Raffel et al., 2020) and FLAN-T5 (Chung et al., 2022); as well as autoregressive models including GPT-2 (Radford et al., 2019), HaRT (Soni et al., 2022) - specifically designed for human language modeling, XLNet (Yang et al., 2019), Llama2 (Touvron et al., 2023b), and Llama3 (Touvron et al., 2023a). We further examined the influence of architectural scaling by incorporating both base and large variants of these models. We also benchmark Llama3-8B-Instruct via zero-shot and few-shots prompting, comparing its performance against embedding-based pipelines to capture psychological constructs.

4.2 Outcomes

In this research, we assessed a broad set of psychological variables spanning five dimensions: Affect such as Valence and Arousal (Remmington et al., 2000), and Positive and Negative Affect (PANAS) (Thompson, 2007); Substance behavior like Number of Drinks, AUDIT-C (Bradley et al., 2007), Craving, and MACE (Lange et al., 2017); Mental health indicators including PHQ9 for depression (Kroenke et al., 2003), GAD7 for anxiety (Plummer et al., 2016), general stress (PSS) (Cohen et al., 1983), and daily stress (PSS Nervous Stress Agreement); Socio-demographics such as Income and Age; and Personality traits like Openness, Neuroticism, Conscientiousness, Extraversion, and Agreeableness from the BIG-5 model (Soto and Jackson, 2013). These variables were selected to cover a broad range of psychological states and traits, allowing for a precise evaluation of LLMs’ ability to model these constructs effectively. All outcomes were self-reported. Valence, Arousal, Number of Drinks, Craving, and Daily Stress were collected at the EMA level, while the remaining variables were obtained once per wave.

4.3 Stability

To assess the stability of psychological measures across different temporal dimensions, this study utilized two statistical metrics:
1. Intra-class Correlation Coefficient (ICC) (Liljequist et al., 2019) is calculated to assess the reliability of measurements by quantifying the proportion of total variance attributable to differences between individuals. The total variance of a variable is decomposed into two components: the between-individual variance (\( \sigma^2_{\text{between}} \)), representing variability in measurements across different individuals, and the within-individual variance (\( \sigma^2_{\text{within}} \)), capturing variations in repeated measurements for the same individual over time. The ICC is computed as:

\[
\text{ICC} = \frac{\sigma^2_{\text{between}}}{\sigma^2_{\text{between}} + \sigma^2_{\text{within}}}
\]

Higher ICC values indicate greater consistency and reliability of measurements over time.

2. Test-Retest Reliability Each variable in the dataset was analyzed to assess the consistency of measurements across multiple time points. A Pearson correlation matrix was constructed to evaluate pairwise correlations between all waves for each variable. To isolate inter-wave correlations, the lower triangular portion of the matrix (excluding the diagonal) was extracted. The mean of these inter-wave correlations was then calculated, providing a single summary metric for each variable to quantify its test-retest reliability (Weir, 2005).

The test-retest reliability metric is computed as:

\[
\text{retest} = \frac{2}{n(n-1)} \sum_{i=1}^{n-1} \sum_{j=i+1}^{n} r_{ij}
\]

where \( n \): the number of temporal points; and \( r_{ij} \): the Pearson correlation coefficient between temporal points \( i \) and \( j \).

Although ICC scores provide insight into inter-individual stability, our stability classification focused on test-retest scores, as they more directly reflect temporal consistency and are better suited for categorizing constructs along the state-disposition-trait continuum.

Table 2 presents the stability metrics assessed at both EMA-level and Wave-level. Attributes such as Age and Personality Traits (e.g., Extraversion, Conscientiousness) demonstrate high stability, with ICCs and test-retest correlation coefficients (\( r \)) exceeding 0.7, indicating their enduring nature. Conversely, moderately stable measures like Stress (PSS) and Positive Affect display stability indices between 0.5 and 0.7. More dynamic metrics, including Valence, Arousal, and Craving, show lower stability (< 0.5), reflecting their sensitivity to immediate environmental and situational changes. These results illustrate the need to customize modeling strategies according to the temporal stability of each metric. High-stability metrics benefit from aggregation methods across waves or individuals, enhancing model accuracy. Conversely, metrics with low stability require approaches that can capture transient, moment-specific dynamics.

<table>
  <tr>
    <th>Variables</th>
    <th colspan="2">EMA (daily)</th>
    <th colspan="2">Wave (quarterly)</th>
  </tr>
  <tr>
    <th></th>
    <th>ICC</th>
    <th>retest</th>
    <th>ICC</th>
    <th>retest</th>
  </tr>
  <tr>
    <td>Arousal (ARO)</td>
    <td>.108</td>
    <td>.173</td>
    <td>.443</td>
    <td>.490</td>
  </tr>
  <tr>
    <td>Valence (VAL)</td>
    <td>.295</td>
    <td>.313</td>
    <td>.712</td>
    <td>.724</td>
  </tr>
  <tr>
    <td>No of Drinks (DRI)</td>
    <td>.391</td>
    <td>.423</td>
    <td>.776</td>
    <td>.738</td>
  </tr>
  <tr>
    <td>Craving (CRA)</td>
    <td>.412</td>
    <td>.481</td>
    <td>.607</td>
    <td>.726</td>
  </tr>
  <tr>
    <td>Daily Stress (PSS1)</td>
    <td>.547</td>
    <td>.622</td>
    <td>.776</td>
    <td>.738</td>
  </tr>
  <tr>
    <td>Stress (PSS)</td>
    <td>-</td>
    <td>.580</td>
    <td>.586</td>
    <td>-</td>
  </tr>
  <tr>
    <td>AUDIT C (AUC)</td>
    <td>-</td>
    <td>.710</td>
    <td>.698</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Positive Affect (PAF)</td>
    <td>-</td>
    <td>.668</td>
    <td>.710</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Negative Affect (NAF)</td>
    <td>-</td>
    <td>.602</td>
    <td>.642</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Openness (OPE)</td>
    <td>-</td>
    <td>.680</td>
    <td>.660</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Conscientiousness (CON)</td>
    <td>-</td>
    <td>.693</td>
    <td>.682</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Extraversion (EXT)</td>
    <td>-</td>
    <td>.778</td>
    <td>.810</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Agreeableness (AGR)</td>
    <td>-</td>
    <td>.638</td>
    <td>.617</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Neuroticism (NEU)</td>
    <td>-</td>
    <td>.747</td>
    <td>.767</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GAD7 (GAD)</td>
    <td>-</td>
    <td>.720</td>
    <td>.736</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PHQ9 (PHQ)</td>
    <td>-</td>
    <td>.753</td>
    <td>.739</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MACE (MAC)</td>
    <td>-</td>
    <td>.775</td>
    <td>.775</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Individual Income (INC)</td>
    <td>-</td>
    <td>.768</td>
    <td>.793</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Age (AGE)</td>
    <td>-</td>
    <td>.995</td>
    <td>.997</td>
    <td>-</td>
  </tr>
</table>

Table 2: Stability Metrics of Psychological Variables at EMA and Wave Levels. This table reports two key stability measures for each variable: the intra-class correlation coefficient (ICC) and the average test-retest Pearson correlation (retest). Higher values indicate greater stability over time (i.e., less fluctuation between assessments). Cells are color-coded to denote standard stability categories: > 0.7: High (blue); 0.5 to 0.7: Medium (yellow); < 0.5: Low (green) (Koo and Li, 2016). Variables with dashes were not assessed at the EMA level.

4.4 Experimental Design

We generated average token embeddings from each LLM to predict psychological outcomes, employing a 10-fold cross-validation scheme with ridge regression—selected for its ability to address multicollinearity and mitigate overfitting. Data stratification was applied to ensure balanced outcome distributions across folds. Model performance was then assessed using Pearson correlation coefficient (\( r \)) and Mean Square Error (MSE). Figure 2 illustrates our multi-level embedding aggregation strategy. We first generated token embeddings for each EMA message, capturing immediate linguisti-
tic and emotional cues. Next, we aggregated these embeddings at the wave level (i.e., over each 14-day period) by mean pooling to capture mid-range patterns within users. Finally, all wave-level embeddings were averaged at the user level, creating a holistic profile of each participant over the full study duration. This progressive approach enables a nuanced examination of daily, mid-range, and long-term psychological dynamics through the corresponding text signals.

![Hierarchical Embedding Generation Model for Psychological State and Trait Analysis. This diagram depicts the process of generating embeddings at different levels: EMA-level (message-level), wave-level, and user-level, using a large language model (LLM).](page_328_670_755_312.png)

Figure 2: Hierarchical Embedding Generation Model for Psychological State and Trait Analysis. This diagram depicts the process of generating embeddings at different levels: EMA-level (message-level), wave-level, and user-level, using a large language model (LLM).

<table>
  <tr>
    <th>Variable Group: Model \ (stability)</th>
    <th colspan="2">States (low)</th>
    <th colspan="2">Dispositions (mid)</th>
    <th colspan="2">Traits (high)</th>
    <th>User Level</th>
  </tr>
  <tr>
    <th></th>
    <th>r</th>
    <th>MSE</th>
    <th>r</th>
    <th>MSE</th>
    <th>r</th>
    <th>MSE</th>
    <th>r</th>
    <th>MSE</th>
  </tr>
  <tr>
    <td>Auto Encoder</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>RoBERTa-base</td>
    <td>0.36** 3.10</td>
    <td>0.39</td>
    <td>7.98</td>
    <td>0.40</td>
    <td>30.49</td>
    <td>0.39</td>
    <td>19.93</td>
  </tr>
  <tr>
    <td>RoBERTa-large</td>
    <td>0.38* 3.04</td>
    <td>0.27</td>
    <td>8.07</td>
    <td>0.39</td>
    <td>30.85</td>
    <td>0.39</td>
    <td>19.37</td>
  </tr>
  <tr>
    <td>BERT-base</td>
    <td>0.35* 3.11</td>
    <td>0.36</td>
    <td>8.54</td>
    <td>0.38</td>
    <td>32.57</td>
    <td>0.39</td>
    <td>20.19</td>
  </tr>
  <tr>
    <td>DeBERTa-base</td>
    <td>0.37* 3.06</td>
    <td>0.35</td>
    <td>8.12</td>
    <td>0.42</td>
    <td>29.90</td>
    <td>0.41</td>
    <td>19.01</td>
  </tr>
  <tr>
    <td>DeBERTa-large</td>
    <td>0.39* 3.00</td>
    <td>0.39</td>
    <td>8.27</td>
    <td>0.41</td>
    <td>30.70</td>
    <td>0.43</td>
    <td>19.44</td>
  </tr>
  <tr>
    <td>Encoder-Decoder</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>T5-large</td>
    <td>0.37* 3.03</td>
    <td>0.38</td>
    <td>8.05</td>
    <td>0.39</td>
    <td>29.97</td>
    <td>0.44</td>
    <td>18.79</td>
  </tr>
  <tr>
    <td>FLAN-T5-large</td>
    <td>0.38* 2.96</td>
    <td>0.37</td>
    <td>8.20</td>
    <td>0.38</td>
    <td>30.88</td>
    <td>0.40</td>
    <td>19.29</td>
  </tr>
  <tr>
    <td>AutoRegressive</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>GPT2-medium</td>
    <td>0.36* 3.04</td>
    <td>0.36</td>
    <td>8.15</td>
    <td>0.40</td>
    <td>30.07</td>
    <td>0.41</td>
    <td>18.98</td>
  </tr>
  <tr>
    <td>GPT2-HL6</td>
    <td>0.36* 3.04</td>
    <td>0.36</td>
    <td>8.09</td>
    <td>0.41</td>
    <td>30.61</td>
    <td>0.41</td>
    <td>20.19</td>
  </tr>
  <tr>
    <td>Xlnet-large</td>
    <td>0.39* 2.90</td>
    <td>0.38</td>
    <td>7.87</td>
    <td>0.42</td>
    <td>29.20</td>
    <td>0.45</td>
    <td>18.07</td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>0.35* 3.04</td>
    <td>0.36</td>
    <td>8.52</td>
    <td>0.37</td>
    <td>31.97</td>
    <td>0.37</td>
    <td>20.66</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>0.34* 3.03</td>
    <td>0.33</td>
    <td>8.68</td>
    <td>0.36</td>
    <td>31.86</td>
    <td>0.34</td>
    <td>20.29</td>
  </tr>
  <tr>
    <td>Human LM</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>HaRT</td>
    <td>0.34* 2.92</td>
    <td>0.39**</td>
    <td>7.65</td>
    <td>0.44</td>
    <td>28.13</td>
    <td>0.46</td>
    <td>17.91</td>
  </tr>
  <tr>
    <td>Zero Shot</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>0.28</td>
    <td>4.48</td>
    <td>0.35</td>
    <td>28.14</td>
    <td>0.44</td>
    <td>42.12</td>
    <td>0.43</td>
    <td>32.29</td>
  </tr>
  <tr>
    <td>Few Shot</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>0.35</td>
    <td>4.08</td>
    <td>0.40</td>
    <td>23.55</td>
    <td>0.48</td>
    <td>37.79</td>
    <td>0.48</td>
    <td>29.05</td>
  </tr>
</table>

Table 3: Accuracy (as average Pearson r) and Mean Squared Error (MSE) of model embeddings for capturing states, dispositions, and traits, as calculated in Table 2. States are variables with low stability (high variability) across time, while dispositions have moderate stability and traits have high stability. User-Level refers to variables averaged across each user. Statistically significant differences from zero-shot Llama3-8B baseline: * (p < .05) and ** (p < .001), computed using boot-strapped resampling across individuals over 1000 trials. Models tend to perform progressively better in predicting more stable attributes.

4.5 Computational Framework

The computational framework leveraged PyTorch, HuggingFace, and the Differential Language Analysis Toolkit (DLATK) (Schwartz et al., 2017) for feature extraction and model evaluation. To support these operations, 2 NVIDIA RTX A6000 GPUs with 48GB of VRAM, were used for generating embeddings and executing prompting tasks.

5 Results

Capturing States, Dispositions, and Traits. Table 3 presents the average Pearson correlation coefficients (r) that measure the accuracy of various LLMs in capturing three distinct categories of psychological constructs: states, dispositions, and traits, as inferred from the methodology described in §4.3. In the autoencoder category, DeBERTa-large embeddings achieved the highest correlations across all stability levels—\( r = .39 \) for states, \( r = .39 \) for dispositions, and \( r = .41 \) for traits—demonstrating its effectiveness in capturing both fleeting emotional cues and more stable psychological characteristics. Among the encoder-decoder models, T5-large and flanT5-large embeddings show competitive performance, indicating that these models are balanced in capturing both short-term and enduring psychological signals. For the autoregressive models, XLNet-large embeddings stands out, achieving an \( r = .39 \) for states, \( r = .38 \) for dispositions, and \( r = .42 \) for traits, with the highest user-level correlation \( r = .45 \). The human-aware model HaRT also showed strong performance for stable constructs (\( r = .44 \)), highlighting the benefits of its hierarchical attention mechanism. The zero-shot configuration of Llama3-8B demonstrates relatively low performance for states \( r = .28 \) but improves for traits \( r = .44 \) and user-level measures \( r = .43 \). However, with few-shot prompting using between 2 to 7 examples per outcome Llama3-8B shows marked improvement across the board, with correlations rising to \( r = .35 \) for states, \( r = .40 \) for dispositions, and notably \( r = .48 \) for traits
<table>
  <tr>
    <th>Dimensions</th>
    <th>Aff</th>
    <th>Sub</th>
    <th>Mnt</th>
    <th>SDe</th>
    <th>Per</th>
  </tr>
  <tr>
    <th colspan="6">Auto Encoder</th>
  </tr>
  <tr>
    <td>RoBERTa-base</td>
    <td>.487**</td>
    <td>.220</td>
    <td>.600</td>
    <td>.346</td>
    <td>.302</td>
  </tr>
  <tr>
    <td>RoBERTa-large</td>
    <td>.489**</td>
    <td>.222</td>
    <td>.581</td>
    <td>.349</td>
    <td>.296</td>
  </tr>
  <tr>
    <td>BERT-base</td>
    <td>.499**</td>
    <td>.172</td>
    <td>.585</td>
    <td>.333</td>
    <td>.296</td>
  </tr>
  <tr>
    <td>DeBERTa-base</td>
    <td>.467**</td>
    <td>.233</td>
    <td>.603</td>
    <td>.406</td>
    <td>.300</td>
  </tr>
  <tr>
    <td>Deberta-large</td>
    <td>.509**</td>
    <td>.261</td>
    <td>.590</td>
    <td>.425</td>
    <td>.317</td>
  </tr>
  <tr>
    <th colspan="6">Encoder-Decoder</th>
  </tr>
  <tr>
    <td>T5-large</td>
    <td>.502**</td>
    <td>.229</td>
    <td>.595</td>
    <td>.421</td>
    <td>.309</td>
  </tr>
  <tr>
    <td>FLAN T5-large</td>
    <td>.495**</td>
    <td>.195</td>
    <td>.604</td>
    <td>.355</td>
    <td>.297</td>
  </tr>
  <tr>
    <th colspan="6">Auto Regressive</th>
  </tr>
  <tr>
    <td>GPT2-medium</td>
    <td>.490**</td>
    <td>.248</td>
    <td>.595</td>
    <td>.339</td>
    <td>.289</td>
  </tr>
  <tr>
    <td>GPT-2nc</td>
    <td>.501**</td>
    <td>.246</td>
    <td>.601</td>
    <td>.323</td>
    <td>.278</td>
  </tr>
  <tr>
    <td>XLnet-large</td>
    <td>.503**</td>
    <td>.285</td>
    <td>.599</td>
    <td>.439</td>
    <td>.315</td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>.459*</td>
    <td>.168</td>
    <td>.586</td>
    <td>.308</td>
    <td>.299</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.456*</td>
    <td>.132</td>
    <td>.567</td>
    <td>.261</td>
    <td>.268</td>
  </tr>
  <tr>
    <th colspan="6">Human LM</th>
  </tr>
  <tr>
    <td>HaRT</td>
    <td>.512*</td>
    <td>.321</td>
    <td>.615**</td>
    <td>.395</td>
    <td>.342</td>
  </tr>
  <tr>
    <th colspan="6">Zero Shot Prompting</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.456</td>
    <td>.397</td>
    <td>.535</td>
    <td>.315</td>
    <td>.285</td>
  </tr>
  <tr>
    <th colspan="6">Few Shot Prompting</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.492</td>
    <td>.446</td>
    <td>.597</td>
    <td>.382</td>
    <td>.324</td>
  </tr>
</table>

Table 4: Performance Evaluation of LMs across different dimensions of psychology. Statistically significant differences from zero-shot Llama3-8B baseline: * (p < .05) and ** (p < .001), computed using boot-strapped resampling across individuals over 1000 trials. (Aff stands for affective variables; Sub for substance behavior variables; Mnt stands for mental health variables; Sde stands for socio-demographics and Per stands for personality). Similar trends are observed across models, with mental health dimensions being captured more effectively than all other dimensions.

<table>
  <tr>
    <th>Parameter</th>
    <th>VAL</th>
    <th>ARO</th>
    <th>PSS1</th>
    <th>DRI</th>
    <th>CRA</th>
  </tr>
  <tr>
    <td>N</td>
    <td>10,108</td>
    <td>10,108</td>
    <td>4,638</td>
    <td>8,185</td>
    <td>4,909</td>
  </tr>
  <tr>
    <th colspan="6">AutoEncoder</th>
  </tr>
  <tr>
    <td>RoBERTa-base</td>
    <td>.624*</td>
    <td>.373**</td>
    <td>.562</td>
    <td>.211</td>
    <td>.246</td>
  </tr>
  <tr>
    <td>RoBERTa-large</td>
    <td>.635*</td>
    <td>.389**</td>
    <td>.530</td>
    <td>.260</td>
    <td>.242</td>
  </tr>
  <tr>
    <td>BERT-base</td>
    <td>.602</td>
    <td>.354**</td>
    <td>.545</td>
    <td>.178</td>
    <td>.249</td>
  </tr>
  <tr>
    <td>DeBERTa-base</td>
    <td>.624*</td>
    <td>.375**</td>
    <td>.554</td>
    <td>.222</td>
    <td>.277</td>
  </tr>
  <tr>
    <td>DeBERTa-large</td>
    <td>.648**</td>
    <td>.404**</td>
    <td>.562</td>
    <td>.285</td>
    <td>.218</td>
  </tr>
  <tr>
    <th colspan="6">Encoder-Decoder</th>
  </tr>
  <tr>
    <td>T5-large</td>
    <td>.633**</td>
    <td>.390**</td>
    <td>.556</td>
    <td>.244</td>
    <td>.239</td>
  </tr>
  <tr>
    <td>FLAN T5-large</td>
    <td>.642**</td>
    <td>.392**</td>
    <td>.573*</td>
    <td>.259</td>
    <td>.281</td>
  </tr>
  <tr>
    <th colspan="6">AutoRegressive</th>
  </tr>
  <tr>
    <td>GPT2-medium</td>
    <td>.602</td>
    <td>.355**</td>
    <td>.524</td>
    <td>.226</td>
    <td>.250</td>
  </tr>
  <tr>
    <td>GPT-2nc</td>
    <td>.615</td>
    <td>.358**</td>
    <td>.542</td>
    <td>.222</td>
    <td>.239</td>
  </tr>
  <tr>
    <td>XLNet-large</td>
    <td>.627*</td>
    <td>.392**</td>
    <td>.545</td>
    <td>.271</td>
    <td>.281</td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>.596</td>
    <td>.365*</td>
    <td>.529</td>
    <td>.228</td>
    <td>.210</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.588</td>
    <td>.361*</td>
    <td>.490</td>
    <td>.225</td>
    <td>.214</td>
  </tr>
  <tr>
    <th colspan="6">Human LM</th>
  </tr>
  <tr>
    <td>HaRT</td>
    <td>.632*</td>
    <td>.331*</td>
    <td>.583**</td>
    <td>.296</td>
    <td>.317</td>
  </tr>
  <tr>
    <th colspan="6">Zero Shot Prompting</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.470</td>
    <td>.161</td>
    <td>.377</td>
    <td>.225</td>
    <td>.245</td>
  </tr>
  <tr>
    <th colspan="6">Few Shot Prompting</th>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>.606</td>
    <td>.315</td>
    <td>.531</td>
    <td>.273</td>
    <td>.261</td>
  </tr>
</table>

Table 5: Pearson Correlation Coefficients for EMA Level Analysis. Highlighted cells indicate which model excels for the corresponding outcome. Statistically significant differences from zero-shot Llama3-8B baseline: * (p < .05) and ** (p < .001), computed using boot-strapped resampling across individuals over 1000 trials. Overall, EMA-level analysis highlights how certain architectures excel at capturing fleeting emotional cues, while others better align with daily behavioral variables. Although prompting can enhance performance to some extent, many embedding-based models still provide a stronger and more consistent signal.

and user-level outcomes. This improvement suggests that minimal task-specific prompts can significantly enhance the model’s ability to capture subtle psychological signals, particularly for more stable constructs.

Across all models, there is a clear trend: constructs with greater stability—such as traits and user-level aggregates—are predicted with higher accuracy than dynamic, state-like variables. These results imply that language signals associated with enduring characteristics are more consistent and easier to model, whereas the fleeting nature of transient states poses greater challenges.

Performance Evaluation of LMs across various psychological dimensions. Table 4 evaluates the effectiveness of LLMs in capturing five psychological dimensions—Affective, Substance Behavior, Mental Health, Socio-Demographics, and Personality—using average Pearson correlation coefficients. The results highlight distinct strengths among architectures. However, a common pattern emerges across nearly all models, with mental health variables generally yielding stronger correlations than the other four dimensions. This suggests that language-based signals for mental health may be more readily detected or more consistently expressed in the text, enabling LLMs to track these constructs more effectively.

Predictive Performance Across Temporal Granularities. Figure 3 illustrates trends in the predictive performance of different LLMs for Valence, Arousal, and Stress across different temporal granularities. For Valence, performance is consistently low across all models at the EMA level, reflecting the challenge of capturing this construct in momentary assessments. However, predictive accuracy improves markedly at the wave and user levels, indicating that temporal aggregation enhances stability and predictive reliability for this construct.
Figure 3: The predictive performance of different transformer-LM models across varying temporal granularities—EMA, Wave, and User. Valence and stress are more accurately predicted at the wave or user level while arousal has greater accuracy at the EMA (i.e. document) level.

In contrast, Arousal demonstrates an inverse trend, with higher predictive performance at the EMA level due to its immediate and dynamic nature. As temporal aggregation progresses to wave and user levels, performance diminishes, highlighting the difficulty of capturing this transient construct in aggregated representations. For Stress, the predictive performance shows a balanced progression across all levels, with models performing moderately at the EMA level and improving at the wave and user levels. This indicates that Stress encompasses both dynamic and stable components, benefiting from temporal aggregation to capture broader patterns while retaining its sensitivity to momentary fluctuations.

Longitudinal Analysis of Model Performance. Across the three temporal granularity levels—EMA, wave, and user—our findings reveal notable shifts in which models excel and how reliably they can capture each psychological construct (Tables 5, 6, and 7). At the EMA level, where text mirrors daily, state-like fluctuations (e.g., valence, arousal, or sudden stress spikes), context-rich embeddings such as those from DeBERTa-large and HaRT are especially effective (Table 5). These architectures appear to capture subtle linguistic indicators tied to momentary emotional swings and daily behaviors, reflected in their high Pearson correlations. By contrast, when aggregating data over two-week intervals (wave level) (Table 6), we model more disposition-like constructs (e.g., short-term stress, negative affect) that are neither purely momentary nor completely stable. Here, most embedding-based approaches continue to perform well (RoBERTa, DeBERTa), whereas Llama3-8B few-shot prompting often proves advantageous, particularly for capturing substance use behaviors like alcohol intake and craving (Table 6). This suggests that targeted prompting can highlight context-specific cues about consumption patterns or cravings, thereby boosting performance beyond what embedding-only pipelines achieve. Nonetheless, personality-related signals remain harder to extract from these mid-length narratives, producing only moderate correlations overall—likely because users’ short-term text lacks the introspective detail needed to reveal deeper trait-like characteristics. Finally, at the user level (Table 7), which spans up to two years of participants’ collected text, the gains in predicting personality remain modest, reflecting how everyday narratives not fully include explicit trait-related content. Meanwhile, sociodemographic factors such as age and income exhibit clearer signals when enough text is aggregated to reveal contextual mentions—references to life events, employment histories, or financial concerns. Across Tables 6 and 7, it is also evident that outcomes with sparse data or low prevalence sometimes register near-zero or slightly negative correlations (refer to Appendix figure A1 for an illustration of the effect of sample size).

Overall, these results emphasize the importance of aligning model architecture and aggregation strategies with the stability and nature of psychological dimensions across temporal levels. Additionally, while encoder-based models are well accepted to be the best representation, our results indicate that this is not always the case—other architectures and prompting methods can outperform them, depending on the psychological construct and its temporal stability.
<table>
  <tr>
    <th rowspan="2">Dimension</th>
    <th colspan="4">Affective</th>
    <th colspan="4">Substance Behavior</th>
    <th colspan="4">Mental Health</th>
    <th colspan="4">Personality</th>
    <th colspan="4">Sociodemo.</th>
  </tr>
  <tr>
    <th>VAL</th><th>ARO</th><th>PAF</th><th>NAF</th>
    <th>DRI</th><th>CRA</th><th>AUC</th><th>MAC</th>
    <th>GAD</th><th>PHQ</th><th>PSS</th><th>PSSI</th>
    <th>OPE</th><th>CON</th><th>EXT</th><th>AGR</th><th>NEU</th><th>INC</th><th>AGE</th>
  </tr>
  <tr>
    <td>Variable N</td>
    <td>406</td><td>406</td><td>133</td><td>133</td>
    <td>406</td><td>179</td><td>406</td><td>126</td>
    <td>406</td><td>179</td><td>406</td><td>406</td>
    <td>345</td><td>345</td><td>345</td><td>345</td><td>406</td><td>132</td>
  </tr>
  <tr>
    <td>Auto Encoder</td>
    <td>RoBERTa-base</td><td>769</td><td>360</td><td>364</td><td>551</td><td>247</td><td>398</td><td>316</td><td>.017</td><td>.550</td><td>731</td><td>498</td><td>550</td><td>130</td><td>279</td><td>187</td><td>322</td><td>.464</td><td>252</td><td>371</td>
  </tr>
  <tr>
    <td></td><td>DeBERTa-large</td><td>797</td><td>351</td><td>353</td><td>577</td><td>261</td><td>259</td><td>381</td><td>.038</td><td>.538</td><td>677</td><td>495</td><td>477</td><td>.098</td><td>254</td><td>236</td><td>238</td><td>.538</td><td>240</td><td>355</td>
  </tr>
  <tr>
    <td></td><td>BERT-base</td><td>746</td><td>361</td><td>347</td><td>556</td><td>201</td><td>305</td><td>290</td><td>.081</td><td>.511</td><td>732</td><td>480</td><td>488</td><td>.081</td><td>268</td><td>267</td><td>251</td><td>.487</td><td>268</td><td>334</td>
  </tr>
  <tr>
    <td></td><td>DeBERTa-base</td><td>780</td><td>337</td><td>348</td><td>348</td><td>251</td><td>.459</td><td>293</td><td>.019</td><td>.545</td><td>740</td><td>490</td><td>516</td><td>151</td><td>259</td><td>252</td><td>273</td><td>.480</td><td>323</td><td>382</td>
  </tr>
  <tr>
    <td></td><td>DeBERTa-large</td><td>777</td><td>.398</td><td>326</td><td>555</td><td>309</td><td>.288</td><td>346</td><td>.037</td><td>.526</td><td>692</td><td>493</td><td>506</td><td>181</td><td>268</td><td>234</td><td>302</td><td>.520*</td><td>352</td><td>403</td>
  </tr>
  <tr>
    <td>Encoder-Decoder</td>
    <td>FLAN-T5-base</td><td>771</td><td>343</td><td>292</td><td>580</td><td>243</td><td>292</td><td>302</td><td>.027</td><td>.542</td><td>714</td><td>462</td><td>508</td><td>.087</td><td>314</td><td>217</td><td>314</td><td>.446</td><td>324</td><td>430</td>
  </tr>
  <tr>
    <td></td><td>FLAN-T5-large</td><td>764</td><td>344</td><td>313</td><td>568</td><td>228</td><td>.336</td><td>338</td><td>.124</td><td>.550</td><td>705</td><td>.488*</td><td>.535</td><td>.066</td><td>273</td><td>279</td><td>251</td><td>.411</td><td>244</td><td>352</td>
  </tr>
  <tr>
    <td>Auto Regressive</td>
    <td>GPT2-medium</td><td>770</td><td>366</td><td>369</td><td>579</td><td>244</td><td>.418</td><td>283</td><td>.025</td><td>.525</td><td>709</td><td>503</td><td>518</td><td>.018</td><td>300</td><td>231</td><td>276</td><td>.452</td><td>226</td><td>372</td>
  </tr>
  <tr>
    <td></td><td>GPT2 HLC</td><td>772</td><td>352</td><td>.433</td><td>576</td><td>269</td><td>.479</td><td>320</td><td>.039</td><td>.538</td><td>722</td><td>487</td><td>514</td><td>.029</td><td>268</td><td>256</td><td>271</td><td>.466</td><td>287</td><td>290</td>
  </tr>
  <tr>
    <td></td><td>Xlnet-large</td><td>788</td><td>.386</td><td>.308</td><td>566</td><td>322</td><td>.355</td><td>344</td><td>.110</td><td>.547</td><td>684</td><td>501</td><td>522</td><td>.127</td><td>290</td><td>273</td><td>270</td><td>.474</td><td>302</td><td>427</td>
  </tr>
  <tr>
    <td></td><td>Llama3-7B</td><td>780</td><td>291</td><td>272</td><td>502</td><td>206</td><td>.306</td><td>296</td><td>.048</td><td>.543</td><td>697</td><td>436</td><td>.466</td><td>.066</td><td>180</td><td>110</td><td>323</td><td>.526</td><td>253</td><td>329</td>
  </tr>
  <tr>
    <td></td><td>Llama3-8B</td><td>785</td><td>291</td><td>.319</td><td>.491</td><td>190</td><td>.338</td><td>248</td><td>.048</td><td>.543</td><td>697</td><td>436</td><td>.466</td><td>.066</td><td>180</td><td>110</td><td>323</td><td>.526</td><td>253</td><td>329</td>
  </tr>
  <tr>
    <td>Human LM</td>
    <td>HaRT</td><td>771</td><td>243</td><td>.410</td><td>.646</td><td>339</td><td>.346</td><td>387</td><td>.228</td><td>.536</td><td>732</td><td>495</td><td>.563</td><td>.188</td><td>277</td><td>252</td><td>304</td><td>.495</td><td>.366*</td><td>.317</td>
  </tr>
  <tr>
    <td>Zero Shot</td>
    <td>Llama3-8B</td><td>753</td><td>292</td><td>.414</td><td>505</td><td>.351</td><td>.470</td><td>471</td><td>.336</td><td>.495</td><td>527</td><td>.442</td><td>.660</td><td>.050</td><td>291</td><td>201</td><td>296</td><td>.428</td><td>190</td><td>426</td>
  </tr>
  <tr>
    <td>Few Shot</td>
    <td>Llama3-8B</td><td>768</td><td>294</td><td>.426</td><td>510</td><td>.400</td><td>.498</td><td>.473</td><td>.477</td><td>.493</td><td>.552</td><td>.533</td><td>.696</td><td>.055</td><td>.400*</td><td>.322</td><td>308</td><td>.443</td><td>203</td><td>.531</td>
  </tr>
</table>

Table 6: Pearson Correlation Coefficients for Wave-Level Analysis. Highlighted cells indicate which model excels for the corresponding outcome. Higher values indicate stronger predictive ability. Statistically significant differences from zero-shot Llama3-8B baseline: * (p < .05) and ** (p < .001), computed using boot-strapped resampling across individuals over 1000 trials. Different model families appear stronger for capturing mental health signals through embeddings, while prompting approaches are more effective in modeling substance behaviors. In contrast, personality traits remain less directly inferred from two-week text bursts, yielding only moderate correlations overall.

6 Conclusion

This study systematically evaluated the capabilities of many Transformer-based language models, spanning BERT-style autoencoders, as well as generative large LMs, for capturing human factors across different levels of temporal stability—low, medium, and high as well as different psychological dimensions. The findings reveal that model performance is highly influenced by the temporal granularity of the data, the stability of the outcomes, and the constructs being modeled. While aggregation strategies proved instrumental for enhancing predictive reliability for stable constructs (Traits), low-stability constructs that undergo a lot of dynamic fluctuations on the daily might not be best represented through averages over time, an effect we specifically observe for some States.

Additionally, we introduced a framework that determines the preferred temporal granularities at which these constructs should be analyzed. This framework not only aids improving models of psychological constructs but it also has practical implications for data collection design in future studies. By identifying the optimal data collection frequency for such experiments, it offers the potential to eliminate the need for costly daily surveys when evaluating LLMs’ capabilities in psychological assessment. Together, these insights emphasize the importance of tailoring model selection, data aggregation strategies, and experimental design to align with the unique temporal characteristics and stability of psychological constructs, paving the way for more efficient and reliable LLM-based approaches to psychological evaluation.

7 Limitations

This study has few limitations that stem from the ecological nature of its design. The participant population, although socioeconomically diverse, was restricted to service industry workers who spoke English as the primary language recruited from professional organizations and online groups within the United States. The resulting population represents the service industry, consisting of 75% females. The mean age of the participants is 35 years, with a standard deviation of 9.47 years. The ages range from a minimum of 18 to a maximum of 68 years. The population is known to be at high risk for excessive alcohol use (Jose et al., 2022). Thus, the results found here should not be assumed to generalize to other populations.

Furthermore, limitations exist related to the scope of LLMs and the outcomes assessed. Relative to the largest LLMs, the models used here were smaller (~ 8 billion parameters), so findings may not generalize to larger models. We suggest
<table>
  <tr>
    <th rowspan="2">Dimension</th>
    <th colspan="4">Affective</th>
    <th colspan="4">Substance Behaviour</th>
    <th colspan="4">Mental Health</th>
    <th colspan="4">Personality</th>
    <th colspan="4">SocioDemog.</th>
  </tr>
  <tr>
    <th>VAL N</th>
    <th>ARO N</th>
    <th>PAF N</th>
    <th>NAF N</th>
    <th>DRI 120</th>
    <th>CRA 94</th>
    <th>AUC 120</th>
    <th>MAC 99</th>
    <th>GAD 120</th>
    <th>PHQ 120</th>
    <th>PSS 120</th>
    <th>PSS1 92</th>
    <th>OPE 120</th>
    <th>CON 120</th>
    <th>EXT 120</th>
    <th>AGR 120</th>
    <th>NEU 120</th>
    <th>INC 120</th>
    <th>AGE 103</th>
  </tr>
  <tr>
    <td>RoBERTa-base</td><td>.769</td><td>.148</td><td>.371</td><td>.543</td><td>.218</td><td>.383</td><td>.123</td><td>.073</td><td>.571</td><td>.587</td><td>.648</td><td>.706</td><td>.031</td><td>.369</td><td>.257</td><td>.409</td><td>.569</td><td>.398</td><td>.364</td>
  </tr>
  <tr>
    <td>RoBERTa-large</td><td>.778</td><td>.142</td><td>.332</td><td>.538</td><td>.260</td><td>.390</td><td>.173</td><td>.033</td><td>.587</td><td>.578</td><td>.659</td><td>.691</td><td>.055</td><td>.343</td><td>.258</td><td>.357</td><td>.583</td><td>.393</td><td>.406</td>
  </tr>
  <tr>
    <td>BERT-base</td><td>.758</td><td>.344</td><td>.388</td><td>.534</td><td>.179</td><td>.359</td><td>.034</td><td>.007</td><td>.579</td><td>.598</td><td>.634</td><td>.702</td><td>.046</td><td>.360</td><td>.226</td><td>.367</td><td>.611</td><td>.384</td><td>.346</td>
  </tr>
  <tr>
    <td>DeBERTa-base</td><td>.768</td><td>.202</td><td>.340</td><td>.550</td><td>.207</td><td>.406</td><td>.180</td><td>.054</td><td>.590</td><td>.591</td><td>.662</td><td>.739</td><td>.045</td><td>.390</td><td>.278</td><td>.372</td><td>.502</td><td>.454*</td><td>.467</td>
  </tr>
  <tr>
    <td>DeBERTa-large</td><td>.766</td><td>.309</td><td>.369</td><td>.539</td><td>.304</td><td>.433</td><td>.301</td><td>.085</td><td>.573</td><td>.592</td><td>.657</td><td>.709</td><td>.068</td><td>.370</td><td>.254</td><td>.383</td><td>.589</td><td>.470*</td><td>.477</td>
  </tr>
  <tr>
    <td>Encoder-Decoder</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>PTB-large</td><td>.773</td><td>.346</td><td>.361</td><td>.559</td><td>.215</td><td>.449</td><td>.220</td><td>.112</td><td>.565</td><td>.601</td><td>.658</td><td>.749</td><td>.172</td><td>.374</td><td>.271</td><td>.402</td><td>.525</td><td>.421</td><td>.508</td>
  </tr>
  <tr>
    <td>FLAN-T5-large</td><td>.771</td><td>.277</td><td>.339</td><td>.539</td><td>.172</td><td>.432</td><td>.141</td><td>.114</td><td>.592</td><td>.605</td><td>.648</td><td>.736</td><td>.166</td><td>.325</td><td>.284</td><td>.395</td><td>.523</td><td>.448*</td><td>.377</td>
  </tr>
  <tr>
    <td>Auto Regressive</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>GPT2-medium</td><td>.767</td><td>.199</td><td>.355</td><td>.533</td><td>.243</td><td>.431</td><td>.224</td><td>.139</td><td>.601</td><td>.614</td><td>.660</td><td>.703</td><td>.002</td><td>.388</td><td>.247</td><td>.400</td><td>.580</td><td>.396</td><td>.361</td>
  </tr>
  <tr>
    <td>GPT2 HLC</td><td>.784</td><td>.218</td><td>.390</td><td>.514</td><td>.280</td><td>.478</td><td>.234</td><td>.020</td><td>.599</td><td>.604</td><td>.670</td><td>.733</td><td>.019</td><td>.360</td><td>.242</td><td>.380</td><td>.589</td><td>.440*</td><td>.274</td>
  </tr>
  <tr>
    <td>Xbert-large</td><td>.766</td><td>.328</td><td>.331</td><td>.536</td><td>.331</td><td>.399</td><td>.126</td><td>.024</td><td>.596</td><td>.613</td><td>.688</td><td>.708</td><td>.084</td><td>.367</td><td>.317</td><td>.399</td><td>.548</td><td>.440</td><td>.586</td>
  </tr>
  <tr>
    <td>LLama-7B</td><td>.764</td><td>.280</td><td>.380</td><td>.535</td><td>.355</td><td>.407</td><td>.024</td><td>.180</td><td>.565</td><td>.579</td><td>.644</td><td>.714</td><td>.076</td><td>.337</td><td>.217</td><td>.375</td><td>.578</td><td>.328</td><td>.325</td>
  </tr>
  <tr>
    <td>LLama3-SB</td><td>.771</td><td>.242</td><td>.269</td><td>.441</td><td>.040</td><td>.341</td><td>.053</td><td>.174</td><td>.565</td><td>.579</td><td>.637</td><td>.694</td><td>.061</td><td>.353</td><td>.163</td><td>.375</td><td>.650</td><td>.343</td><td>.219</td>
  </tr>
  <tr>
    <td>Human LM</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>HaRT</td><td>.774</td><td>.377</td><td>.410</td><td>.531</td><td>.313</td><td>.483</td><td>.282</td><td>.218</td><td>.604</td><td>.630</td><td>.680</td><td>.708</td><td>.174</td><td>.393</td><td>.320</td><td>.424</td><td>.592</td><td>.447*</td><td>.450</td>
  </tr>
  <tr>
    <td>Zero Shot</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB</td><td>.734</td><td>.174</td><td>.512</td><td>.550</td><td>.427</td><td>.473</td><td>.568*</td><td>.408</td><td>.562</td><td>.564</td><td>.570</td><td>.620</td><td>.065</td><td>.377</td><td>.234</td><td>.419</td><td>.486</td><td>.135</td><td>.507</td>
  </tr>
  <tr>
    <td>Few Shot</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama3-SB</td><td>.741</td><td>.184</td><td>.528</td><td>.546</td><td>.453</td><td>.521</td><td>.611</td><td>.492</td><td>.739</td><td>.609</td><td>.583</td><td>.640</td><td>.093</td><td>.424</td><td>.251</td><td>.440</td><td>.501</td><td>.228</td><td>.565</td>
  </tr>
</table>

Table 7: Pearson Correlation Coefficients for User-Level Analysis. Highlighted cells indicate which model excels for the corresponding outcome. Higher values indicate stronger predictive ability. Statistically significant differences from zero-shot Llama3-8B baseline: * (p < .05) and ** (p < .001), computed using boot-strapped resampling across individuals over 1000 trials. When aggregating all participant text at the user level, personality traits see modest gains but remain moderate overall, reflecting that everyday writing rarely includes explicit statements about one’s personality. Socio-demographic outcomes (age, income) show improved correlations when there is enough text to reveal contextual signals.

that further work should test this for those with the fortune of such computational resources. Our study specifically aimed to evaluate a diverse set of model architectures and types (autoencoders, encoder-decoders, autoregressive) at a tractable scale to provide insights into their comparative performance across temporal granularities and psychological constructs. Although this study uniquely assessed embeddings across a wide range of specific and general psychological outcomes, it certainly does not represent the whole range of possible psychological factors. Future research should expand these findings to additional psychological variables such as in the cognitive domain to enhance their applicability.

8 Ethical Considerations

This study adhered to rigorous ethical guidelines to ensure the responsible application of artificial intelligence in mental health research with procedures approved by an independent academic Institutional Review Board (IRB). All participants provided informed consent for the use of their data in this study, with no agreement to share their non-anonymized individual data beyond the scope of this research. This work is aimed at advancing interdisciplinary NLP-psychology research to better understand human behaviors as reflected in language. Importantly, the models and methods developed in this study are not intended or validated for deployment in clinical settings or for other commercial applications, such as targeted marketing. Instead, the focus is on contributing to the development of more accurate and ethically sound techniques that benefit society and promote human health.

Acknowledgments

This work was supported in part by a grant from the NIH-NIAAA (R01 AA028032), and a grant from the CDC/NIOSH (U01 OH012476), Developing and Evaluating Artificial Intelligence-based Longitudinal Assessments. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing any funding organization.

References

Nora Al-Twairesh. 2021. The evolution of language models applied to emotion analysis of arabic tweets. Information, 12(2):84.

Brian Bauer, Raquel Norel, Alex Leow, Zad Abi Rached, Bo Wen, and Guillermo Cecchi. 2024. Using large language models to understand suicidality in a social media–based taxonomy of mental health disorders: Linguistic analysis of reddit posts. JMIR mental health, 11:e57234.
Adrian Benton, Raman Arora, and Mark Dredze. 2016. Learning multiview embeddings of twitter users. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19.

Ryan L. Boyd and David M. Markowitz. 2024. Verbal behavior and the future of social science. American Psychologist, pages 1–23. Place: US Publisher: American Psychological Association.

Ryan L. Boyd et al. 2022. The development and psychometric properties of liwc-22. Behavior Research Methods, 54:2502–2520.

Katharine A Bradley, Anna F DeBenedetti, Robert J Volk, Emily C Williams, Danielle Frank, and Daniel R Kivlahan. 2007. Audit-c as a brief screen for alcohol misuse in primary care. Alcoholism: Clinical and Experimental Research, 31(7):1208–1217.

Giulio Carducci, Giuseppe Rizzo, Diego Monti, Enrico Palumbo, and Maurizio Morisio. 2018. Twitpersonality: Computing personality traits from tweets using word embeddings and supervised learning. Information, 9(5):127.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharar Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. Preprint, arXiv:2210.11416.

Sheldon Cohen, Tom Kamarck, and Robin Mermelstein. 1983. A global measure of perceived stress. Journal of Health and Social Behavior, 24(4):385–396.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Oluwatoba Fabunmi, Saman Halgamuge, Daniel Beck, and Katja Holtt-Otto. 2025. Large language models for predicting empathic accuracy between a designer and a user. Journal of Mechanical Design, 147(4).

Adithya V Ganesan, Vasudha Varadarajan, Yash Kumar Lal, Veerle C Eijssbroek, Katarina Kjell, Oscar NE Kjell, Tanuja Dhanasekaran, Elizabeth C Stade, Johannes C Eichstaedt, Ryan L Boyd, et al. 2024. Explaining gpt-4’s schema of depression using machine behavior analysis. arXiv preprint arXiv:2411.13800.

Adithya V Ganesan, Vasudha Varadarajan, Juhi Mittal, Shashanka Subrahmanya, Matthew Matero, Nikita Soni, Sharath Chandra Guntuku, Johannes Eichstaedt, and H Andrew Schwartz. 2022. Wwbp-sqt-lite: Multi-level models and difference embeddings for moments of change identification in mental health forums. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 251–258.

Candida M Greco, Andrea Simeri, Andrea Tagarelli, and Ester Zumpano. 2023. Transformer-based language models for mental health issues: a survey. Pattern Recognition Letters, 167:204–211.

Md Rakibul Hasan, Md Zakir Hossain, Tom Gedeon, and Shafin Rahman. 2024. Llm-gem: Large language model-guided prediction of people’s empathy levels towards newspaper article. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2215–2231.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.

Julian Jara-Ettinger and Paula Rubio-Fernandez. 2021. Quantitative mental state attributions in language understanding. Science advances, 7(47):eabj0970.

Rupa Jose, Matthew Matero, Garrick Sherman, Brenda Curtis, Salvatore Giorgi, Hansen Andrew Schwartz, and Lyle H. Ungar. 2022. Using facebook language to predict and describe excessive alcohol use. Alcoholism: Clinical and Experimental Research, 46(5):836–847.

Swanie Juhng, Matthew Matero, Vasudha Varadarajan, Johannes Eichstaedt, Adithya V Ganesan, and H Andrew Schwartz. 2023. Discourse-level representations can improve prediction of degree of anxiety. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1500–1511.

Oscar NE Kjell, Sverker Sikström, Katarina Kjell, and H Andrew Schwartz. 2022. Natural language analyzed with ai-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. Scientific reports, 12(1):3918.

Terry K. Koo and Mae Y. Li. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2):155–163.

Kurt Kroenke, Robert L. Spitzer, and Janet B. W. Williams. 2003. The patient health questionnaire-2: Validity of a two-item depression screener. Medical Care, 41(11):1284–1292.

Allison Lahnala, Vasudha Varadarajan, Lucie Flek, H Andrew Schwartz, and Ryan L Boyd. 2025. Unifying the extremes: Developing a unified model for detecting and predicting extremist traits and radicalization. Proceedings of the International AAAI Conference on Web and Social Media.
Shannon Lange, Charlotte Probst, Kevin D. Gmel, Jürgen Rehm, and Svetlana Popova. 2017. Worldwide prevalence of fetal alcohol spectrum disorders: A systematic literature review including meta-analysis. Addiction, 112(9):1520–1532.

Ming Li, Yusheng Su, Hsiu-Yuan Huang, Jiali Cheng, Xin Hu, Xinniao Zhang, Huadong Wang, Yujia Qin, Xiaozhi Wang, Kristen A Lindquist, et al. 2024. Language-specific representation of emotion-concept knowledge causally supports emotion inference. iScience, 27(12).

David Liljequist, Björn Elfving, and Kirsten Skavberg Roaldsen. 2019. Intraclass correlation—a discussion and demonstration of basic features. PloS one, 14(7):e0219854.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. In arXiv preprint arXiv:1907.11692.

Veronica Lynn, Niranjan Balasubramanian, and H Andrew Schwartz. 2020. Hierarchical modeling for user personality prediction: The role of message-level attention. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5306–5316.

Maria Mahbub, Gregory M Dams, Sudarshan Srinivasan, Caitlin Rizy, Ioana Danciu, Jodie Trafton, and Kathryn Knight. 2025. Decoding substance use disorder severity from clinical notes using a large language model. npj Mental Health Research, 4(1):5.

August Håkan Nilsson, Hansen Andrew Schwartz, Richard N Rosenthal, James R McKay, Huy Vu, Young-Min Cho, Syeda Mahwish, Adithya V Ganesan, and Lyle Ungar. 2024. Language-based ema assessments help understand problematic alcohol consumption. Plos one, 19(3):e0298300.

Faye Plummer, Laura Manea, Dominic Trepel, and Dean McMillan. 2016. Screening for anxiety disorders with the gad-7 and gad-2: a systematic review and diagnostic metaanalysis. General Hospital Psychiatry, 39:24–31.

Muhammad Qorib, Geonsik Moon, and Hewe Tou Ng. 2024. Are decoder-only language models better than encoder-only language models in understanding word meaning? In Findings of the Association for Computational Linguistics ACL 2024, pages 16339–16347.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report.

Colin Raffel, Noam Shazeer, Adam Roberts, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

Nicholas A. Remmington, Leandre R. Fabrigar, and Penny S. Visser. 2000. Reexamining the circumplex model of affect. Journal of Personality and Social Psychology, 79(2):286–300.

Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2024. Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual nlu tasks. arXiv e-prints, pages arXiv–2406.

Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184.

Alejandro Salmerón-Ríos, José Antonio García-Díaz, Ronghao Pan, and Rafael Valencia-García. 2024. Fine grain emotion analysis in spanish using linguistic features and transformers. PeerJ Computer Science, 10:e1992.

Vitor Garcia dos Santos and Ivandré Paraboni. 2022. Myers-briggs personality classification from social media text using pre-trained language models. arXiv preprint arXiv:2207.04476.

H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, S Ramones, Megha Agrawal, Achal Shah, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. In PloS one, volume 8.

H. Andrew Schwartz, Salvatore Giorgi, Maarten Sap, Patrick Crutchley, Johannes Eichstaedt, and Lyle Ungar. 2017. Dlatk: Differential language analysis toolkit. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 55–60. Association for Computational Linguistics.

Fatemeh Shah-Mohammadi and Joseph Finkelstein. 2024. Extraction of substance use information from clinical notes: Generative pretrained transformer–based investigation. JMIR Medical Informatics, 12:e56243.

Martin J. Slivinski, Jacqueline A. Mogle, Jinshil Hyun, Elizabeth Munoz, Joshua M. Smyth, and Richard B. Lipton. 2018. Reliability and validity of ambulatory cognitive assessments. Assessment, 25(1):14–30. Publisher Copyright: © 2016, © The Author(s) 2016.

Nikita Soni, Niranjan Balasubramanian, H Schwartz, and Dirk Howy. 2024a. Comparing pre-trained human language models: Is it better with human context as groups, individual traits, or both? In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 316–328.

Nikita Soni, Pranav Chitale, Khushboo Singh, Niranjan Balasubramanian, and H. Schwartz. 2025. Evaluation of LLMs-based hidden states as author representations for psychological human-centered NLP
tasks. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7658–7667, Albuquerque, New Mexico. Association for Computational Linguistics.

Nikita Soni, Matthew Matero, Niranjan Balasubramanian, and H. Andrew Schwartz. 2022. Human language modeling. In Findings of the Association for Computational Linguistics: ACL 2022, pages 622–636, Dublin, Ireland. Association for Computational Linguistics.

Nikita Soni, H. Andrew Schwartz, João Sedoc, and Niranjan Balasubramanian. 2024b. Large human language models: A need and the challenges. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8631–8646, Mexico City, Mexico. Association for Computational Linguistics.

Christopher J Soto and Joshua J Jackson. 2013. Five-factor model of personality. Journal of Research in Personality, 42:1285–1302.

Edmund R. Thompson. 2007. Development and validation of an internationally reliable short-form of the positive and negative affect schedule (panas). Journal of Cross-Cultural Psychology, 38(2):227–242.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023a. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Pierre Albert, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Adithya V Ganesan, Yash Kumar Lal, August Nilsson, and H. Andrew Schwartz. 2023. Systematic evaluation of GPT-3 for zero-shot personality estimation. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 390–400, Toronto, Canada. Association for Computational Linguistics.

Vasudha Varadarajan, Swanie Juhng, Syeda Mahwish, Xiaoran Liu, Jonah Luby, Christian Luhmann, and H. Andrew Schwartz. 2023. Transfer and active learning for dissonance detection: Addressing the rare-class challenge. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11923–11936, Toronto, Canada. Association for Computational Linguistics.

Vasudha Varadarajan, Allison Lahnala, Adithya V Ganesan, Gourab Dey, Siddharth Mangalik, Ana-Maria Bucur, Nikita Soni, Rajath Rao, Kevin Lanning, Isabella Vallejo, et al. 2024. Archetypes and entropy: Theory-driven extraction of evidence for suicide risk. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 278–291.

Vasudha Varadarajan, Allison Lahnala, Sujeeth Vankudari, Akshay Raghavan, Scott Feltman, Syeda Mahwish, Camilo Ruggero, Roman Kotov, and H Andrew Schwartz. 2025a. Linking language-based distortion detection to mental health outcomes. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 62–68.

Vasudha Varadarajan, Syeda Mahwish, Xiaoran Liu, Julia Buffolino, Christian Luhmann, Ryan L Boyd, and H Schwartz. 2025b. Capturing human cognitive styles with language: Towards an experimental evaluation paradigm. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 966–979.

Vasudha Varadarajan, Nikita Soni, Weixi Wang, Christian Luhmann, H. Andrew Schwartz, and Naoya Inoue. 2022. Detecting dissonant stance in social media: The role of topic exposure. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 151–156, Abu Dhabi, UAE. Association for Computational Linguistics.

Joseph P Weir. 2005. Quantifying test-retest reliability using the intraclass correlation coefficient and the sem. Journal of Strength and Conditioning Research, 19(1):231–240.

Zhichao Xu and Jiepu Jiang. 2024. Multi-dimensional evaluation of empathetic dialog responses. arXiv preprint arXiv:2402.11409.

Zhilin Yang, Zihang Dai, Yiming Yang, et al. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), pages 5753–5763.

Shi Zong, Alan Ritter, and Eduard Hovy. 2020. Measuring forecasting skill from text. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5317–5331, Online. Association for Computational Linguistics.
A Appendix

A.1 Zero-shot Prompting
The prompt template below was provided to the model without any additional in-context examples. This encouraged it to generate responses solely based on the given instructions and the input text.

    <begin_of_text>
    <start_header_id>
    system
    <end_header_id>
    You are helpful assistant.
    </eot_id>
    <start_header_id>
    user
    <end_header_id>
    {prompt}
    Provide your assessment by responding with "Score: "
    followed by the corresponding number.
    </eot_id>
    <start_header_id>
    assistant
    <end_header_id>

A.2 Few-shot Prompting
The prompt template below was provided to the model along with 2 to 7 in-context examples from the dataset for different outcomes. This encouraged it to generate responses by leveraging both the given instructions and the provided examples in addition to the input text.

    <begin_of_text>
    <start_header_id>
    system
    <end_header_id>
    You are helpful assistant.
    </eot_id>
    <start_header_id>
    user
    <end_header_id>
    {prompt}
    Provide your assessment by responding with "Score: "
    followed by a single integer considering the following examples:
    {few_shot_examples}.
    </eot_id>
    <start_header_id>
    assistant
    <end_header_id>

Tables A2 - A7 contain the prompts designed for each psychological variable analyzed in this study. These prompts were tailored to elicit meaningful responses from the language models (LLMs) by framing the tasks in a clear, context-specific manner. While these prompts provide a solid starting point, they can be further refined to enhance clarity and alignment with the constructs of interest, thereby improving the reliability and generalizability of zero- and few-shot prompting methodologies.
EMA Essay

I feel somewhat content today. We went to store early. Got pizzas. We made those pizzas and are watching football all day. I got to enjoy some rest and the weather has been a bit chilly today so that’s been nice not sweating. Hoping to enjoy some more football and make some dessert after dinner

I’m doing ok, I am stressed and had some issues with my relationship and and still very underwhelmed by the current place I am employed but for today anyway I have the kind frame of it is what it is. I can’t change what is not in my control, I can’t undo anything that has already come to pass, and I can’t help others fix themselves if they can’t even see their toxic behavior. Tomorrow may be different and I might never really be ok with what’s happened the past year but today im gna try to give myself a break.

Decided to go out for a little while and meet a friend . I don’t was up on the air bit then decided it was nice to get out a little since I worked all week. Was contacted by a golf course about a job just still up in the air about it. Just don’t know if I want to commit to a weekly shift or not . Supposed to go Monday to meet with the owner so we’ll see.

Table A1: Illustrative examples of daily EMA text. Here we see an essay combining early direct affect mentions (“content”) with the day’s positive leisure activities—like making pizzas and watching football (top row); an entry discussing a complex mix of stress, relationship concerns, and an acceptance mindset amid employment worries (middle row); and one focused largely on daily outings and job-related decisions—implying affect through described activities rather than stating it outright (bottom row).

![Bar charts showing Pearson r and Min-ACE correlations for various LLM models as a function of number of users](page_324_670_1097_388.png)

Figure A1: Model Performance vs Sample Size for Selected Outcomes. This figure illustrates how the predictive performance of different LLM-based embeddings (represented by distinct colored bars) varies as the number of users increases. For smaller sample sizes (shown on the left side of each plot), correlations often hover near zero or even become slightly negative, reflecting instability and potential noise due to insufficient data. As the number of users increases, most models’ correlations tend to improve, highlighting that more data generally leads to more reliable embeddings and more accurate predictions of psychological constructs.
<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>Valence</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the emotional valence of the writer based by analyzing their essays. The valence scale measures the degree of pleasantness or unpleasantness, with 0 representing very low levels of pleasantness and 4 representing very high levels of pleasantness.</td>
  </tr>
  <tr>
    <td>Arousal</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the arousal of the writer by analyzing their essays. The arousal scale measures the energy of the writer, with 0 representing calm or lethargic energy, and 2 representing active or excited energy.</td>
  </tr>
  <tr>
    <td>Positive Affect</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the Positive Affect score of the writer by analyzing their message. Positive Affect (from PANAS) refers to the extent to which an individual experiences positive emotional states such as being interested, excited, enthusiastic, proud, or inspired. Look for explicit mentions of these emotions, descriptions of situations that evoke positive feelings, and implicit cues in the tone, choice of words, or overall mood conveyed in the text. Estimate the Positive Affect score experienced by the writer with a number between 5-25, with 5 representing low positive affect, while 25 denotes very high positive affect.</td>
  </tr>
  <tr>
    <td>Negative Affect</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the Negative Affect score of the writer by analyzing their essays. Negative Affect (from PANAS) refers to the extent to which an individual feels negative emotional states such as distress, fear, anger, guilt, or nervousness. Look for explicit mentions of emotions, descriptions of situations that might evoke negative feelings, and implicit cues in the tone, choice of words, or overall mood conveyed in the text. Estimate the Negative Affect score experienced by the writer with a number between 5-25, with 5 representing low negative affect, while 25 denotes very high negative affect.</td>
  </tr>
</table>

Table A2: Zero- and few-shot LLM instruction prompts used to elicit ratings of Valence, Arousal, Positive Affect and Negative Affect (PANAS) from participants’ text.
<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>Individual Income</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the income of the writer based by analyzing their essays. Use your judgment to evaluate references to occupation, lifestyle, education, and financial indicators mentioned in the text. Chose the most closes income range of this individual from the following categories. (A) < $10,000 (B) $10,000-$20,000 (C) $20,000-$30,000 (D) $30,000-$40,000 (E) $40,000-$50,000 (F) $50,000-$60,000 (G) $60,000-$70,000 (H) $70,000-$80,000 (I) $90,000-$100,000 (J) >$100,000. Choose only one option from the above, and respond with "Income Category: ", followed by the alphabet indicative of the corresponding income range.</td>
  </tr>
  <tr>
    <td>Age</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the age of the writer based by analyzing their essays, which can range from 18 to 65. Based on linguistic patterns, cultural references, mentions of life events, maturity of the writing and the overall tone, estimate the age of the writer.</td>
  </tr>
</table>

Table A3: Zero- and few-shot LLM instruction prompts used to categorize participants’ Income bracket and estimate their Age from contextual cues in their text.

<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>PSS1</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the nervousness/stress levels of the writer by analyzing their essays. The level of stress ranges from 1 to 5 where 1 means very low or no stress/nervousness, and 5 means extremely high stress/nervousness.</td>
  </tr>
  <tr>
    <td>PSS</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the severity of stress experienced by the writer based off their essays. Note that stressed individuals are overwhelmed by difficulties in their lives, while individuals who are not stressed are confident in solving their personal problems. Estimate the stress level of the writer based on the Percieved Stress Scale with a number between 0-16, with 0 representing no stress and 16 representing very high levels of stress.</td>
  </tr>
</table>

Table A4: Zero- and few-shot LLM instruction prompts used to estimate daily nervousness/stress (PSS1) and overall perceived stress (PSS) from participants’ text.
<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>GAD7</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the anxiety levels experienced by the writer based off their essays. Note that anxious individuals feel nervous, worry too much about different things, have trouble relaxing, or can be easily annoyed. Estimate the anxiety level of the writer based on the Generalized Anxiety Disorder scale with a number between 0-21, with 0 representing no anxiety and 21 representing high levels of anxiety.</td>
  </tr>
  <tr>
    <td>PHQ9</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the depression levels experienced by the writer based off their essays. Note that depressed individuals feel hopeless, have little interest in doing everyday things, suffer somatic symptoms like abnormal sleep, abnormal appetite, fatigue or psychomotor agitation/retardation. They might also experience trouble with concentrating on things, feelings of worthlessness/guilt or suicidal ideation. Estimate the depression severity of the writer based on the Patient Health Questionnaire scale with a number between 0-27, with 0 representing no anxiety and 27 representing high levels of depression.</td>
  </tr>
</table>

Table A5: Zero- and few-shot LLM instruction prompts used to assess Generalized Anxiety Disorder (GAD-7) and depression severity (PHQ-9) based on participants’ text.
<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>No of Drinks</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to analyze the essays and determine the likely number of alcoholic drinks the author of these essays had consumed. Consider any direct mentions of drinking, contextual hints about social settings, behaviors associated with drinking and any indirect references that may imply the consumption of alcohol. Use your expertise to gauge the number of drinks based on the narrative provided.</td>
  </tr>
  <tr>
    <td>AUDITC</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to estimate the individual’s level of alcohol use based on their essays. Pay close attention to any direct mentions of drinking, contextual clues about social settings, behaviors commonly associated with alcohol consumption, and indirect references that suggest the frequency and quantity of drinking. Assign a score between 0 and 12 using the AUDIT-C scale, where 0 indicates no alcohol use or minimal risk, and 12 indicates a high risk for harmful drinking behaviors or potential alcohol use disorder.</td>
  </tr>
  <tr>
    <td>Craving</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to analyze the essays and and determine the intensity of the author’s craving for alcohol. Use your judgment to analyze descriptions of feelings, situations triggering desire, any direct mentions of wanting to consume alcohol, or behaviors associated with drinking. Based on the essays, determine how strong the craving is on a scale from 0 to 10, where 0 indicates no craving at all and 10 indicates an extremely high craving.</td>
  </tr>
  <tr>
    <td>MACE</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to estimate the individual’s level of alcohol cravings based on their message. Pay close attention to any strong urges to drink, descriptions of picturing alcohol or drinking, mentions of imagining the taste of alcohol, reflections on how the body might feel after drinking, and intrusive thoughts about alcohol. Assign a score between 0 and 50 using the Mini-ACE scale, where 0 indicates no cravings or minimal risk, and 50 indicates a high level of persistent cravings or potential risk for harmful drinking behaviors.</td>
  </tr>
</table>

Table A6: Zero- and few-shot LLM instruction prompts used to predict number of alcoholic drinks, AUDIT-C score, alcohol craving, and Mini-ACE (MACE) scores from participants’ text.
<table>
  <tr>
    <th>VARIABLE</th>
    <th>PROMPT</th>
  </tr>
  <tr>
    <td>Openness</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the openness score of the writer by analyzing their essays. Note that individuals who are open to experiences tend to be intellectual, imaginative, sensitive and open-minded while individuals that are not open to experiences tend to be down to earth, insensitive and conventional. The openness scale ranges from 0 to 20, where 0 indicates very low levels of openness and 20 indicates extremely high levels of openness .</td>
  </tr>
  <tr>
    <td>Conscientiousness</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the conscientiousness score of the writer by analyzing their essays. Note that individuals who are conscientious tend to be careful, thorough, organized and scrupulous while individuals that are not conscientious tend to be irresponsible, disorganized and unscrupulous. The conscientiousness scale ranges from 0 to 20, where 0 indicates very low levels of conscientiousness and 20 indicates extremely high levels of conscientiousness .</td>
  </tr>
  <tr>
    <td>Extraversion</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the extraversion score of the writer by analyzing their essays. Note that individuals who are extraverted tend to be sociable, talkative, assertive and active while individuals that are not extraverted tend to be retiring, reserved and cautious. The conscientiousness scale ranges from 0 to 20, where 0 indicates very low levels of extraversion and 20 indicates extremely high levels of extraversion .</td>
  </tr>
  <tr>
    <td>Agreeableness</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the agreeableness of the writer based by analyzing their essays. Note that individuals who are agreeable tend to be good-natured, compliant, modest, gentle, and cooperative while individuals that are not agreeable tend to be irritable, ruthless, suspicious and inflexible. The agreeableness scale ranges from 0 to 20, where 0 indicates very low levels of agreeableness and 20 indicates extremely high levels of agreeableness.</td>
  </tr>
  <tr>
    <td>Neuroticism</td>
    <td>Carefully read the series of essays posted below, written by a person describing how they felt each day. Each day’s essay is separated by a new line, with the most recent one in the bottom. Essays: message Your task is to determine the neuroticism score of the writer by analyzing their essays. Note that individuals who are neurotic tend to be anxious, depressed, angry and insecure while individuals that are not neurotic tend to be calm, poised and emotionally stable. The neuroticism scale ranges from 0 to 20, where 0 indicates very low levels of neuroticism and 20 indicates extremely high levels of neuroticism .</td>
  </tr>
</table>

Table A7: Zero- and few-shot LLM instruction prompts used to infer BIG-5 Personality Traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism from participants’ text.

Reviewer paper summaries: This paper systematically evaluates Transformer-based LLMs for predicting psychological variables across five key dimensions. The study analyzes text data at three temporal levels—daily, biweekly, and user-level (two years)—to determine how well different models capture short-term states versus long-term traits. Findings show that mental health variables are the most reliably captured, with models like DeBERTa and HaRT excelling in short-term emotional fluctuations, while Llama3-8B performs well in modeling substance use behaviors over longer periods. Aggregating data over time improves predictive accuracy for stable constructs like personality and demographics, but fleeting psychological states remain harder to capture.

This paper evaluates the ability of LLMs to capture various psychological variables across different temporal scales. The authors compared 15 different models of varying scales, assessing their effectiveness in predicting affect, substance use, mental health, sociodemographic factors, and personality traits over daily, biweekly, and user-level timescales.

The paper evaluates whether the representations of users' emotional descriptions (EMA messages) expressed in natural language at different times, encoded by language models (LMs), can be used to predict the corresponding users' psychological variables.
Summary of strengths: + Analysis in different time granularities, daily, bi-weekly, and annually.
+ Five distinct psychological dimensions, including affect, substance use, mental health, sociodemographics, and personality
+ Important research direction: this study can help researchers bulid LLMs that can help in people's mental health

1. The study covers a broad experimental scope, involving 15 different models and conducting a robust and comprehensive analysis. While the largest model tested hold only 8B parameters, and I am not sure whether models like BERT can truly measure complex human behaviors.
2. The approach is well-grounded in a human-centered perspective, as it examines LLM perception in relation to both temporal stability and psychological constructs, making it a valuable contribution to interdisciplinary research.

1. The paper assesses three different LM architectures, including approaches that directly utilize prompts for prediction.
    
2. The motivation and research question are meaningful, as reflected in the title of the paper.
Summary of weaknesses: - The evaluation is limited to smaller models and open-source models. I would like to see the results of LLaMA3-70B, GPT-4o, and Claude-3.5.
- As the study highlights, the mental health variable is easiest to capture since participants use more specific languages. This brings an issue into attention that a lower performance of an LLM does not necessarily indicate its lower ability. It can be due to the low data quality, making it difficult to compare the performance among the five psychological dimensions.
- The user data are all based on self-reported questionnaires. The reliability of these self-reported is not discussed in the paper. Will unreliable self-reported data influence the analysis and conclusions?

I have concerns regarding the practical significance of their findings. Many numerical differences in tables are extremely small (e.g., in Table 2, several values differ by only 0.1). While the authors conducted significance testing, many p-values exceed 0.5, raising doubts about whether these differences are meaningful in real-world applications.

Additionally, the study focuses exclusively on U.S. service industry workers, which limits its generalizability. It remains unclear whether the findings would hold for other demographic groups.

1. Line 199: BERT, RoBERTa, and DeBERTa are not “autoencoder” models
    
2. From Figure 2, I understand that the paper first obtains EMA message representations using an LM, aggregates them through pooling to derive user-level representations, and then trains a ridge regression model to predict the variables presented in Table 1 for each user. This is much clearer than the previous version. However, on average, how many EMA messages are there per wave? Are all the EMA messages from a wave concatenated and input into the LLM together, or are they processed individually?
    
3. Lines 309-314 describe the process, but the description is quite unclear. What are the dimensions of the embeddings at each step? What specific pooling method was used? For example, the statement, “We first generated token embeddings for each EMA message,” raises the question: Is the representation of an EMA message a matrix in this case?
    
4.  No example of the dataset in the paper. Missing data statistics regarding waves and messages, such as the number of sentences, word counts, etc.
Comments, suggestions and typos: 1. My biggest question about this paper is that whether the psychological variables can truly be predicted using some short texts. That is to say, is it solid to use some simple texts to predict the personality of a complex individual? This part lacks some support from established psychological research.
2. For prompting based method on LLaMA3, is the model sensitive to different prompt designs?
3. The paper includes many encoder-decoder models, like BERT, RoBERTa, etc. What is the criteria of choosing these models?
4. If I am a psychology researcher, which model should I use to monitor the mental states of a group through their texts (e.g., on social media like X)?
5. Is there any bias in models' prediction towards different gender? That is, whether female has a higher performance?

Typo:

1. Line 074: the reference should be before the full stop.
2. The caption of Fig. 2: a missing space "EMA-level(message-level)"
3. Figure 1 is not referenced in the main text.
4. A missing space in Table 1: "Daily Stress(PSS1)"

1. The abbreviation LLM is introduced repeatedly.
2. Figure 2 is referenced too far from the relevant text, making it difficult to follow. Consider repositioning it for better readability.

1. The variables in Table 1 could be categorized and displayed according to the descriptions provided in Lines 216-238.
    
2. It is recommended that Figure 2 not only illustrate the process of obtaining representations but also incorporate the specific tasks of the experiment.
Overall assessment scores: 3

2.5

3
Metareview score: 3.5
Metareview: Experiments with various LLMs automating psychological assessment using an existing fine-grained and extensive dataset from Nilsson et al., 2024, of American restaurant workers. Results show performance varies greatly based on temporal granularity and aggregation strategies.

The main weaknesses in the previous submission were presentation related. They improved greatly, but some intuitive demonstration of the data is still missing, as well as better argumentation/evidence for its reliability.
Summary of reasons to publish: Exceptionally extensive evaluation of LLMs for psychological tasks, covering fine-grained, temporally rich and multi aspect data grounded in psychological research.

Experiments with various types of LLMs and aggregation strategies, providing useful insights for best practices for using LLMs in psychology.