A Systematic Evaluation of Transformer-LM Representations for Capturing Author States and Traits
Authors: Khushboo Singh, Vasudha Varadarajan, Adithya V Ganesan, August Håkan Nilsson, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle Ungar, Richard N Rosenthal, H. Schwartz
Accepted at: acl25


Reviewer paper summaries: This paper presents a systematic evaluation of many LLMs (autoencoders, encoder-decoders, and autoregressive models) for capturing human psychological states, dispositions, and traits from textual data, across varying temporal granularities. They collected Ecological Momentary Assessments (EMAs) data for over two years. They find that model performance varies significantly based on the temporal stability of psychological constructs, with aggregation strategies improving predictions for more stable traits. The study offers practical insights on model selection and data aggregation methods to enhance the use of LLMs in psychological assessments.

This paper evaluates a variaty of language models in modeling human psychological constructs including states, dispositions, and traits. The author used a unique dataset of Ecological Momentary Assessments (EMAs) different levels of temporal stability—low, medium, and high. The results indicate that model performance is significantly influenced by the temporal granularity of the data, the stability of the outcomes, and the specific constructs being modeled. Additionally, the aggregation strategies play a crucial role in enhancing the reliability of predictions. The author advocates for further exploration of large language models (LLMs) in psychological assessments.

This paper evaluates LLMs' ability to capture psychological variables, introduces methods for assessing their stability, and provides best practices for selecting models and aggregation strategies in psychology tasks.
Summary of strengths: + Multi-granularity. Daily, Biweekly, and annually data were collected and used in model evaluations.
+ Robust evaluation method by the 10-fold cross-validation.
+ Diverse psychological domains are all evaluated, unlike many studies where only traits or emotions are investigated.

A wide range of LMs from different sources and types were included in the experimentation.

The motivation and research question are meaningful, as reflected in the title of the paper.
Summary of weaknesses: - The models included in this paper is relatively small (<8B). We are not sure whether the conclusions can generalize to larger, like 70B, models. Additionally, embeddings from some commercial models, such as the text-embedding-3-large from OpenAI can also be included.
- The dataset is limited to US ,English, restaurant worker context, limiting the generalizability of the conclusion.
- There are some technical details missing in the current manuscript (please see comments below). The presentation of this paper can be improved.

1. The dataset collected and tested is not provided in the paper, nor did the author show an example of the dataset in the paper. Therefore, I am curious that whether the author is willing to make the data available after the publication. In addition, the process on how to collect the data and the stats of the data should be included and introduced in the paper. 

2. The process of the methodology is not well detailed. Throughout the section 4, it is still unclear to me that how exactly the models are probed in those psychological traits. Did the author takes the input from human data and extract the embedding spaces or the token logits of the model output? I think more details of the evaluation pipeline should be provided.

1. The writing of the paper is abstract and unclear. It's difficult to understand the exact experimental setup, the specific tasks, how the metrics are applied, and what the inputs are. In short, I can only grasp a broad understanding of what the authors are doing, but not the detailed steps or methodology behind it.
    
2. Line 091-097: The statement "Historically, studies on language-psychosocial connections employed traditional methods like …" lacks supporting references. Suggest to divide the related work into smaller sections for clearer and more organized presentation.
    
3. Section 3: Is the data you used collected by yourself, or is it pre-existing? The description is unclear and indirect. This is crucial for understanding the task of the paper. For instance, what is the exact format of the data? Are there any case examples provided? Also, what are the statistical details of the data? If the process described in Section 3 for data collection is not part of the paper’s original work, it does not deserve such extensive coverage. How does this relate to the research questions of the paper? How does the data illustrate the authors' stated traits and characteristics in the study?
    
4. Line 159: What does "six waves" refer to?
Comments, suggestions and typos: 1.	How do you deal with participants’ privacy issue? Are you able to include demographic information of the participants, such as race, age, and gender?
2.	It will be better to include the statistics of the dataset. For example, what is the average length of their responses? How many responses do they have in a wave?
3.	Valence, Arousal, Number of Drinks, Craving, and Daily Stress are reported daily, while other scales are reported once during a wave. Is the understanding correct?
4.	Should MSE be the lower the better? It is a bit confused that Table 2 bold the largest numbers in both r and MSE, where r is the larger the better.
5.	For wave-level Craving, AUDIT C, and Positive Affect, the ICC and test-retest scores show different stability levels (mid and high). How do you determine the stability level of these three variables?

Minor comments and typos

1.	Spaces are missing before the brackets at Line 258, 259: coefficients(ICC) and correlations(Pearson’s r).
2.	For Table 1, could you group sub-scales from a same scale together? Currently, the five (OCEAN) sub-scales of the BFI are separate.

Question: why did you phrase the title as Capturing "Author" States and Traits? How has it to do with "author"?

see Weaknesses
Overall assessment scores: 2.5

2.5

2
Metareview score: 3
Metareview: Experiments with various LLMs automating psychological assessment using an existing fine-grained and extensive dataset from Nilsson et al., 2024, of American restaurant workers. Results show performance varies greatly based on temporal granularity and aggregation strategies.

While the contribution is valuable, reviewers all express their concern about the poor presentation, making it difficult to understand both the dataset and methodology of the paper.
Summary of reasons to publish: Exceptionally extensive evaluation of LLMs for psychological tasks, covering fine-grained, temporally rich and multi aspect data grounded in psychological research.

Experiments with various types of LLMs and aggregation strategies, providing useful insights for best practices for using LLMs in psychology.