Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots
Authors: Huiqi Zou
Pengda Wang
Zihan Yan
Tianjun Sun
Ziang Xiao
Venue: COLM 2025 (2025)
Keywords: human factors in NLP; evaluation methodologies
TL;DR: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbots

Abstract: A chatbot’s personality design is key to interaction quality. As chatbots evolved from rule-based systems to those powered by large language models (LLMs), evaluating the effectiveness of their personality design has become increasingly complex, particularly due to the open-ended nature of interactions. A recent and widely adopted method for assessing the personality design of LLM-based chatbots is the use of self-report questionnaires. These questionnaires, often borrowed from established human personality inventories, ask the chatbot to rate itself on various personality traits. Can LLM-based chatbots meaningfully "self-report" their personality? We created 500 chatbots with distinct personality designs and evaluated the validity of their self-report personality scores by examining human perceptions formed during interactions with these chatbots. Our findings indicate that the chatbot's answers on human personality scales exhibit weak correlations with both human-perceived personality traits and the overall interaction quality. These findings raise concerns about both the criterion validity and the predictive validity of self-report methods in this context. Further analysis revealed the role of task context and interaction in the chatbot's personality design assessment. We further discuss design implications for creating more contextualized and interactive evaluation.

Published as a conference paper at COLM 2025
Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chat- bots
Huiqi Zou', Pengda Wang', Zihan Yan', Tianjun Sun", and Ziang Xiao DDeparteent of Computer Science, Johns Hopkins University
2Department of Psychological Sciences, Rice University *Cognitive Science and Language Processing Informatics, University of Illinois Urbana-Champant (s110)ericc eriiee edu; zihan250i 11 llinois. edu
[hzou11, ziang. xiao}@jhu, edu; {pw32, ts 1s1
Abstract
A chatbot's personality design is key to interaction quality. As chatbots evolved from rule-based systems those powered by large language mod- els (LLMs), evaluating the effectiveness of their personality design has become increasingly complex, particularly due the open-ended nature of interactions. A recent and widely adopted method for assessing the person- ality design of LLM-based chatbots is the use of self-report questionnaires. These questionnaires, often borrowed from established human personal- ity inventories, ask the chatbot to rate itself on various personality traits. Can LLM-based chatbots meaningfully "self-report" their personality? We created 500 chatbots with distinct personality designs and evaluated the validity of their self-report personality scores by examining human per- ceptions formed during interactions with these chatbots. Our findings indicate that the chatbot's answers on human personality scales exhibit weak correlations with both pereeveeeeeeived personality traits and the overall interaction quality. These findings raise concerns about both the criterion validity and the predictive validity of self-report methods in this context. Further analysis revealed the role of task context and interaction in the chatbot's 's personality design assessment. We further discuss design implications for creating more contextualized and interactive evaluation.
1 Introduction
With recent advancements in artificial intelligence (AI), particularly in natural language processing (NLP), Al-based chatbots have found applications across various fields. Their use in areas, such creative writing (Chung et al., 2022), mental health counseling (Lai et al., 2023), data collection (Wei et al., 2024) and educational support as teaching assistants (Hicke et al., 2023), has made interactions with conversational agents increasingly common. To further enhance dialogue engagement and tailor these agents to specific tasks, developers often manipulate chatbots' behaviors or assign them particular identities. One prevalent strategy is through personality design (Shao et al., 2023; Li et al., 2023; Park et al., 2023).
The design of chatbot personalities has evolved significantly, evolving from early rule- based systems such as ELIZA (Weizenbaum, 1966), which relied on predefined scripts to simulate human conversation, to contemporary approaches based on large language models (LLMss. The involvement of LLMs enables various methods of personality instantiation, such as characterizing agents through demographic and persona-based dialogue, as well as employing more fine-grained approaches. For instance, Goldberg (1992)'s 's set of 70 bipolar adjectives, mapped to the Big Five personality domains (Serapio-Garcia et al., 2023), and their facets have been used to characterize chatbot behaviors. Similarly, researchers have transformed personality scale items into open-ended prompts to elicit personality responses from LLMs (Ran et al., 2024).
1

---

Published as a conference paper at COLM 2025
Chatbot Prompt
Measure 1: Chatbot Self-Report (Non-Context
BFl-2
BF1-2-X5
IPIP- PIP-NEO-129
Personality Module
Task Module for Contextlization
Shape different levels of personality Big five domain.
Provide istructonss five real- world coll aooraaiee casks.
in Hargee soiae
Kiiry pile Bllge
Tendh duungy Werd
Gomp usa tolo cocalety
supportive companion simulating personality with a high level of agreeableness. are extremely kind, extremely honest, fiee Table 7 for detail))
Stales Developed for Human Personality Messurement
Human-Chatbonn Interaction With Contextlization
Measure 2: Human-Perceived Personality (Within Context)
Welcome to our chatbod Share your nzeds for social
support and concerns you have, and our wil Isten, What of personal concern you have?
statement desoribes personallyy of charbee
failed an eeam and my parents are mad.
quiet.
& compassionate, has soft heart.
I'm rorly hear that...
Tends be disoranized.
BFI-2-X5
Task: Personaleed Social Support
Figure 1: Overview of the Evaluation Pipeline. Two measures are used to evaluate the chatbot's personality: (1) Chabot self-reporting through established personality scales (e.g., BFI-2 and IPIP-NEO-120); (2) Human-perceived personality, where human interact with the chatbot and subsequently rate its personality based on their interaction.
As chatbots become increasingly open-ended in their capabilities, evaluating the effective- ness of their personality design presents growing challenges. Unlike earlier rule-based systems, modern chatbot architectures lack hand-crafted rules that designers can easily examine or adjust. However, this openness also facilitates the development of novel evalua- tion methods. Recent research has begun to explore the use of self-report personality scales, originally developed for assessing human traits, to evaluate the personality constructs of I.LM-ased chatbots (Huang et al., 2023a; Wang et al., 2024c; Tu et al., 2023). In these studies, LLMs are typically prompted to respond to items from standardized personality inventories using a Likert scale format (Likert, 1932), where each item works as an indicator to reflect specific personality construct dimension. Responses to individual items are summarized to compute aggregate scores for each personality domain.
Although the "self-report" method is convenient and cost-effective for evaluating LLMs, relies assumptions that have not yet been systematically validated through empirical re- search. First, it assumes a high degree of alignment between the personality traits attributed to an I.L.-based chatbot by design and those perceived by humans during interactions (the ultimate goal of evaluátion). However, even in human psychological research, the consistency between self-reports and informant reports tends to be limited due to differences in informational sources (e.g., Connolly et al., 2007; Connelly & Ones, 2010; McCrae & Costa, 1987). Second, these methods presume that the structure and content of psychological assess- ment tools developed for humans can also be applied to LLM-based chatbots. For instance, chatbot designed for project management might be assessed on Extraversion using an item like "Love surprise parties" (Johnson, 2014a), which is irrelevant to the chatbot's actual role in the designed task. Although these psychometric scales are well-validated for human samples, the examined validity does not directly transfer to chatbot evaluation, raising concerns about the applicability of such methods in non-human settings.
This study aims to fill the gap by evaluating the validity of an LLM-based chatbot's self- report personality. Guided by principles from classical test theory and measurement theory (e.g., Allen & Yen, 2001; Xiao et al., 2023), evaluated two sets of validity: Convergent and Discriminant validity, where we examined the intrinsic structure, and Criterion and Predictive validity, where examined the extrinsic relationship with relevant variables.
2

---

Published as a conference paper at COLM 2025
In particular, we focus on two external variables central to the ultimate goal of personality design in Al systems: human perception and interaction quality.
In this study, we created 500 chatbots, each with distinct personality designs, and collected their "self-report" personality. We recruited 500 human participants, each assigned to interact with one of the chatbots to complete a designated task, evaluate the chatbot's personality, and rate the interaction quality. Figure presents an overview of the evaluation pipeline. Our findings indicate that while the self-report personality scales demonstrate moderate Convergent and Discriminant validity, they show poor alignment with human- perceived personality scores and weak correlations with the human-rated interaction quality, suggesting limited Criterion and Predictive validity.
Our study reveals the limitations of relying solely on self-report measures assess personal- ity design in LLM--baeed chatbots. We argue that effective evaluation methods must account for how a chatbot's personality emerges through interaction with human users and should be grounded in task-based performance. Future research should consider the rich signals in human-chatbot interactions and incorporate human perception as a critical dimension in evaluating personalized Al systems. Our work makes three main contributions:
Through an empirical study with 500 participants, we highlight the validity con- cerns of using self-report personality scales to evaluate the personality design of LLM-based chatbots.
Our result offers design implications for creating effective personality design evalu- ation methods that are grounded in real-world task interactions.
We present a dataset containing a rich log of human interactions with 500 chatbots, each with distinct personality designs and human perceptions of their personalities, facilitating the development of interaction-bass personality evaluation methods.
Related Work
Human-Like Responses in LLMs. Recent studies have shown that LLMs demonstrate remarkable abilities resembling human-like responses. For example, Lampinen et al. (2024) found that LLMs display response patterns similar to humans in logical reasoning tasks, with similar findings being reported by Kosinski (2023) and Wang al. (2024a). Addition- ally, Pellert et al. (2023) found that LLMs perform comparably to humans across various psychological scales, such as value orientations and moral norms. All these abilities have garnered significant interest from both computer scientists and social scientists, as these findings suggest that we may be able draw upon existing social science research on humans as a reference for LLM studies. For instance, personality, which has been exten- sively studied in psychology and consistently proven be a reliable predictor of various behavioral and psychological outcomes, is one such area of interest. Personality encodes rich and complex information in textual data (Goldberg, 1990; Saucier & Goldberg, 2001). LLMs may capture and model such encoding by learning from vast training data. Many studies explore the personality traits LLMs exhibit, with the hope to leverage these simulated traits for predicting or shaping model behavior (Lee et al., 2024; Huang et al., 2023b; Wang et al., 2024b; 2025; Ran et al., 2024).
Current Approaches for Measuring LLM-based Chatbots' Personality. Currently, most assessments of LLM-based chatbot personality rely on psychological frameworks developed for human personality measurement, typically employing standardized personality scales. A common approach to have the model respond to scales that have already been developed and validated on human samples, such as the Big Five Inventory-2 (BFÍ-2) (Soto & John, 2017a) and HEXACO (Lee & Ashton, 2004; 2006). The model's scores are then calculated based on its answers to the multiple-choice questions (MCQs) on these scales. The rationale behind this is straightforward: since we believe that LLMs simulate human abilities, we can directly borrow well-established and validated methods used for humans to assess the personality traits of these models. For example, studies like Huang et al. (2023b) used this approach to evaluate whether LLMs reflect the personality traits outlined in the BFI (John
3

---

Published as a conference paper at COLM 2025
et al., 1999). Serapio-Garcia et al. (2023) presented the reliability and validity of LLMs in resembling personality profiles along desired dimensions with International Personality Item Neurotesut,,,,mm Extraversion, Openness (IPIP-NEO) (Goldberg et al., 1999) and BFT (John et al., 1999). However, some researchers argue that traditional scales may not sufficiently capture the personality of LLMs and have adopted alternative approaches. For instance, some researchers modified scale questions into prompts to elicit text-based responses (Wang et al., 2024c), while others have experts rate stories generated by LLMs according to specified criteria (Jiang et al., 2023). These evaluation methods may more accurately reflect real-world application scenarios.
Self-Reports vs. Informant Reports. Self-report methods have limitations in capturing the behavior nuances that reflect personality traits of LLM-based chatbots, as they are not grounded in user experience. Given that the primary goal of chatbot personality design is to enhance engagement and provide personalized interactions, assessment methods should center on human perception. This challenge parallels long-standing exploration in psychol- ogy regarding the distinction between self-reports and informant reports (e.g., Connolly et al., 2007; Connelly & Ones, 2010; McCrae & Costa, 1987). While self-reports capture an individual's subjective understanding of their own traits, informant reports offer external perspective based on others' interpretations of the individual's s behavior. meta-analysis in psychology found that the average correlation between self-reports and informant reports for the Big Five personality traits was only 0.36 (Connolly et al., 2007), indicating overlap and divergence. For chatbots, task-specific design further influences personality expression, traits. Thus,
potentially widening the gap between their self-report and relying solely on self-reports may fail to reflect how chatbots actually affect user experience.
3 Methodology
3.1 Chatbot Design
In this study, we first created a set of LLM-based chatbots with various personality designs. Those chatbots were created for tasks in which personality design is critical for user expe- rience. Our chatbot framework consists of two main components: a personality module ($3.1.1) and task module ($3.1.2). The personality module shapes the chatbot's behavior with predefined personality traits, while the task module provides role-specific instructions and task outlines to complete the task. The complete prompts are in Appendix A.
3.1.1 Personality Module
The personality design of our chatbot is based on the Big Five model (John et al., 1999), foundational framework in personality research that describes personality across five key domains: extraversion (EXT), agreeableness (AGR), conscientiousness (CÓN), neuroticism (NEU), and openness to experience (OPE). We adopted the shape approach from Serapio- García et al. (2023) to shape chatbot's personality in each domain. This method extends Goldberg (1992) list of 70 bipolar adjectives to 104 personality trait descriptors, which are mapped to different facets within each domain. For example, adjectives such as 'unenergetic' and 'energetic' represent the lower and higher levels of Extraversion, respectively. These adjectives were then paired with linguistic qualifiers from Likert-type scales (Likert, 1932) to create personality profiles at varying levels. For our study, we selected the strongest linguis- tic qualifier (i.e., "extremely') and randomly sampled five high-level or low-level adjectives from the same personality domain to generate prompts. Each chatbot is prompted to exhibit either a high or low level of single personality trait, with others left unconstrained.
3.1.2 Task Module
We selected five chatbot task categories where personality design plays a critical role in user experience and task effectiveness. This approach allows to evaluate the validity of the chatbot's personality design via emulating real-world scenarios where designers rely on personality evaluation results to build an effective chatbot. The selected tasks are follows:
4

---

Published as a conference paper at COLM 2025
1. Job Interview, where the chatbot works HR representative to assess the inter- viewee's s Organizational Citizenship Behaviors (OCB), such as initiative, helping, and compliance, chosen due to the strong association of the interviewer's conscientiousness! structured interview settings (Heimann et al., 2021).
2. Public Service, focusing on handling workday challenges like crisis management, highlights the chatbot's s ability to help humans navigate high-stress situations where agreeableness and emotional stability are crucial (Mishra et al., 2023).
3. Personalized Social Support, where chatbot features are rooted in the counseling psychology principles of empathy and reflective listening as outlined by Rogers (1957), offers empathetic responses to individuals in distress referring Acceptance and Commitment Therapy (ACT) techniques. Research shows that empathy is closely linked to agreeableness (Walton et al., 2023).
4. Customized Travel Planning, which relies on the chatbot's decision-making and personalization skills, underscores the importance of anthropomorphic traits in recommendation systems that foster trust and enhance user decision-making (Qiu & Benbasat, 2009). Previous research also suggests that question types related to extraversion levels significantly impact user satisfaction (Miyama & Okada, 2022).
5. Guided Learning, where the chatbot aids humans in understanding complex topics, is based on the constructivist learning approach (Higgs et al., 2004) and highlights the critical role of extraversion in instructors for enhancing student acceptance.
In total, we randomly sampled five personality adjective markers at high or low levels of each domain ten times, respectively. Combined with five task instructions, this resulted in 500 distinct chatbot personality designs. We used GPT-4o (OpenAI, 2024), one of the most performant models available, the backbone model with the temperature set to zero.
3.2 Personality Design Evaluation
3.2.1 Self-report Personality
To quantify the chatbot's self-report personality, we utilized three well-established psy- chometric tests that assess the Big Five domains (John et al., 1999): Big Five Inventory-2 Extra-Short Form (BFI-2-XS) (Soto & John, 2017b), BFI-2 (Soto & John, 2017a) and IPIP-NEO- 120 (Johnson, 2014b). The chatbot's responses to test items were aggregated to calculate the personality scores for the five domains respectively, which were then used for subsequent analysis. Following the work of Serapio-Garcia et al. (2023), we instructed the chatbot to rate test items using a standardized response scale from one to five based on the personality descriptions. The complete prompt format is shown in Appendix B.
3.2.2 Human-perceioed Personality
We conducted human study to gather participants' perceptions of the chatbot's personality with the designed interface (Appendix C). Each participant was randomly assigned to interact with one chatbot and engaged in multi-turn, task-based conversations.
After completing the task, participants were asked to rate their perceived personality of the chatbot using the BFT-2-XS (Soto & John, 2017b). The BFI-2-XS consists of 15 items, each representing distinct facet of one of the Big Five personality domains, thus preserving the scale's 's comprehensive descriptive and predictive capabilities. To mitigate the impact of individual variability, we aggregated evaluation scores across chatbots sharing the same personality configuration (i.e., for all high or all low) for analysis.
3.3 Study Procedure
To gather the human-perceived personality of each chatbot, we recruited English-speaking participants from Prolific!, with each participant assessing one chatbot. At the beginning
Ittp:://wwwwppoofffcccoom

---

Published as a conference paper at COLM 2025
of the survey, participants were given the task's objective, along with clear instructions how to start the conversation, including an example to clarify how the participant might approach the task. A privacy reminder is included to ensure private information is shared. A After interacting with the chatbot, each participant filled out the chatbot personality evaluation questionnaire and completed separate User Experience Questionnaire (UFQ) (Laugwitz et al., 2008). The entire process was designed to take 10 to 15 minutes. Appendix presents summary of the collected dataset statistics. Appendix FF presents detailed participant statistics and their demographic information.
3.4 Analysis Method
Our analysis evaluates the effectiveness of personality settings and the aligmment between self-reports and human perceptions using descriptive statistics and correlation analyses. We began by computing the means and standard deviations of both self-report and human- perceived personality scores to examine overall distribution patterns. Statistical significance tests were conducted to evaluate differences between the high and low personality condi- tions, as well as between human perceptions and the chatbot's self-report scores. In addition, assessed construct validity, the extent to which test accurately measures what it is intended to measure (Cronbach Meehl, 1955), based on four psychometric dimensions:
Convergent validity assesses whether different methods measuring the same trait yield consistent results (John Benet-Martinez, 2000). To assess this, we compared the chatbot's s self-report personality scores across three established scales. Strong inter-scale correlations indicate consistent responses under our personality settings.
Discriminant validity evaluates the distinctiveness of different traits (Campbell Fiske, 1959). We calculated correlations between different personality trait scores across psychometric scales and evaluation methods. Low correlations between unrelated traits confirm good discriminant validity.
Criterion validity examines the correlation between measure and an external criterion (Cronbach Meehl, 1955). We assessed this by comparing the chatbot's self-report personality traits with human perceptions, measured through participant ratings after interacting with the chatbot. The value of criterion validity indicates the level of alignment between self-reports and human perceptions.
Predictive validity refers to measure's 's ability to predict future outcomes or behav- iors (Funder, 2006). We evaluated this by examining whether the chatbot personality setting predicted the quality of interaction, as captured through the UBQ. Strong correlations between personality evaluation scores and positive user experiences suggest that personality traits are valid predictors of chatbot effectiveness.
To assess convergent and discriminant validity, we employed the Mullitrait-uultimehhon (MTMM) matrix approach (Campbell & Fiske, 1959), which evaluates multiple traits across different methods identify patterns of consistency and distinctiveness. We conducted this analysis using Spearman correlations. Given the non-hierarchical structure of the BFI-2-XS, our analysis focused on domain-level personality traits.
4 Results
4.1 Personality Settings Work for Both Human and Chatbot
Personality setting work but chatbot's self-report scores vary more significantly across personality conditions. To evaluate the effectiveness of our personality settings, we ex- amined descriptive statistics of self-report and human-perceived scores. Specifically, we tested for significant between-group differences under high and low personality conditions, which indicates successful trait manipulation. As shown in Table 1, self-report scores are consistently higher in high-setting conditions across tasks. Human Humann-erceiied scores fol- lowed a similar trend, except for Conscientiousness in the social support task. Traits that are more readily conveyed through language and interaction style (Mairesse & Walker, 2010),

---

Published as a conference paper at COLM 2025
Interview
Public
Social Support
Travel Planning
Guided Learning
Evaluation Domain
High
Low
Figh
Low
High
Low
High
Low
Figh
Low
M
513
M
50
M
5D
M
SD
M
SD
M
5D
M
50
SD
M
50
M
SD
EXT
1.85
0.16
1,32
0.14
0.22
0.19
119
0.15
1.9
0.31
1.27
0.09
0.91
0.11
1.01
0.91
1.79
0.21
1.29
0.10
ACR
4.55
0.10
1.75
0.15
0.6
0.02
1.39
0.21
1.99
0.00
1.20
0.26
4.95
0.07
1.01
0.02
1.86
0.11
1.63
0.31
Self-report
CON
4.68
0.13
1.44
0.2
6.59
0.02
1.66
0.06
4.47
0.23
1.10
0.08
5.00
0.00
1.02
0.03
4.66
0.07
1.09
0.06
NTU
4.85
0.16
1,06
0.06
4.92
0.05
102
0.00
4.87
0.08
1.17
0.17
4,96
0.04
1.04
0.01
4.74
0.18
1.05
0.02
OPF
4.83
0.18
1.45
0.11
4.98
0.02
1.32
0.09
4.75
0.19
1.31
0.19
4.97
0.06
1.33
0.06
4.8
0.31
1.52
0.12
EXT
3.70
0.74
3.40
0.25
4.37
0.58
253
0.81
3.55
0.54
3,50
1192
4.30
0.65
253
1.11
3.0
0.82
3,60
0.27
AGR
4.20
0.65
1.57
1224
4.47
0.99
2002
1.57
4.50
0.88
1.87
0.79
3.27
1.25
1.30
0.86
4.10
0.1
1.43
0.70
Human
CON
4.00
0.99
3.60
127
4.00
0.96
2.77
1.16
3.93
1.00
3.97
1.00
4.50
0.34
320
1.12
4.20
0.98
417
0.71
NEU
2.90
097
1.90
0.65
2
0.78
190
0.47
2.27
1.36
1.20
0.66
3.10
1.31
1.50
0.4
290
1.00
2,08
0.56
OPE
3.43
0.57
3.23
0.85
3,30
0.45
3.07
0.96
3,85
0.50
3.25
0.9
4,00
1.04
3.27
0.70
3.0
1.00
3.57
0.61
Table 1: Domain-level mean (M) and standard deviation (SD) of self-report and human- perceived personality scores under high and low settings.
Questionnaire
EXT
AGR
CON
NEU
OPE
Mean
BFI-2-XS vs BFI-2
0.90
0.91
0.83
0.87
0.90
BFI-2-XS vs IPI--NEO-120
0.82
0.84
0.91
0.56
0.81
0.85
IPI--NFO-120 BFI-2
Table 2: Correlation between self-report personality scores using different personality evaluation scales.
such as Agreeableness, showed larger gaps between high and low personality conditions in human evaluations. Overall, the difference between conditions was more pronounced in self-reports than in human perceptions.
To further support this, we ran the analysis of variance to test whether the mean difference between the high and low conditions statistically significant. This method compares between-gron group variance, which reflects differences across conditions, with within- -rthip variance, which captures individual variability within each condition, to determine whether the means under different conditions diverge meaningfully. A significantly larger between- group variance indicates that the manipulation had a meaningful effect on the dependent variable, supporting the hypothesis that the group means differ. As shown in Table 9 in Appendix G, F values for the chatbot's self-report scores exceed the conventional threshold of 1. fluman--erceivedd the threshold of 1; however, the magnitude of this effect is smaller relative to that observed for the self-report scores. We provide a more in-depth discussion of this difference in $4.2. Overall, these results collectively support the effectiveness of our personality manipulations.
scores show similar pattern, with many F values also exceeding
Personality setting effects are consistent across psychometric scales. We further evaluated the chatbot's self-report score on three personality inventories. Although these instruments differ in item content, they are designed to measure the same underlying traits. Consistent convergent and discriminant validity across them would indicate stable personality settings. Summarized in Table 2, the convergent validity results show that the chatbot's self-report scores are highly consistent across the various inventories, with an average inter-scale correlation of over or equal to 0.85. Similarly, the discriminant validity analysis in Table shows that correlations between distinct personality dimensions remain relatively uniform across scales, with a mean absolute correlation of approximately 0.53.
4.2 Chatbot's Self-report Scores Correlate with Human Perceptions Poorly
Personality measures differ in variability and validity. We believe that an effective measure for the chatbot's 'ss personality design should align with the human perceptions of the chatbot. Table 4 presents the means, standard deviations, and correlations between human perception and chatbot self-report scores. As shown, the means across both sources are quite similar, suggesting general agreement at the aggregate level. However, most standard deviations of the human perception scores are smaller than that of the self-report scores, indicating that human ratings are more clustered while the chatbot's self-reports show greater variability.
Statistical tests in Table 10 help explain this pattern. Compared to self-reports, human- perceived scores show smaller differences between high and low personality conditions,
7

---

Published as a conference paper at COLM 2025
Trait Pair
Human
BFI-2-XS
BFI-2
IPIP-NEO-120
EXT AGR
0.18
0.34
0.47
0.34
EXT va CON
0.37
0.56
0.44
0.47
NRG
EXT vs OPE
0.28
0.63
0.73
0.69
AGR vs CON
0.56
0.56
0.64
AGR vs NEU
-0.39
-0.49
-0.60
+0.50
AGR vs OPE
0.57
0.67
0.62
0.51
CON NEU
-0.56
-0.72
-0.66
+0.80
CON vs OPE
0.50
0.49
0.35
0.16
Absolute Mean
0.39
0.54
0.54
0.51
Table 3: Correlation analysis for human-perceived and self-report personality scores across different personality traits.
Human
BFI-2-XS
BFI-2
IPIP-NEO-120
Self-report Mean
M
SD
M
SD
M
SD
M
SD
P
M
SD
p
EXT
3.72
0.78
2.89
1.17
0.23
3.01
1.17
0.20
2.91
1.01
0.17
2.94
1.11
0.20
AGR
3.80
3.53
0.56
3.72
0.58
Human
CON
3.98
0.96
3.70
1.42
0.24
3.44
1.10
0.25
3.62
1.12
0.25
3.59
1.21
0.25
2.03
1.30
2.71
2.37
2.46
OPE
3.27
0.83
3.31
1.32
0.24
3.09
1.11
0.21
3.36
0.83
0.24
3.25
1.09
0.23
Table 4: Domain-level mean (M), standard deviation (SD), and correlation (p) of self-report and human-pprceiver personality scores using different personality evaluation scales.
and greater variability across tasks. To further explore the differences between these two measures, conducted paired t-tests within the same condition (Table 11). The significant p-values across all three psychometric scales reveal discrepancies between the two measures, suggesting that while self-report results may reflect intended personality settings, they do not align with how those traits are perceived during interactions. To control for potential false positives arising from multiple comparisons, we applied significance corrections (Table 12). Our conclusions remain unchanged after applying these adjustments, with most p-values remaining well below thresholds, confirming the robustness of the observed effects.
To further probe this discrepancy, we examine correlations between human perception and self-report scores. Apart from the relatively high mean correlation the Agreeableness domain (0.58), the correlations in others are all below 0.4. This indicates a low level of consistency between how the chatbot evaluates its own personality and how humans perceive it, calling into question the reliability of self-report scores as indicators of perceived behavior. In addition, Table shows differences discriminant validity between the two measures. Compared to human perceptions, the self-report scores show higher correlation across different personality traits, with the exception of the correlation between Conscientiousness and Openness. That suggests lower level of discriminant validity among self-report methods than human perceptions.
Task context shapes personality expressions. To better understand how personality ex- pression varies across different contexts, we analyzed the effect of task setting, Given the high correlations among the three self-report scales ($4.1), averaged their results and compared them with human perceptions across individual tasks. As shown in Table 5, Agreeableness consistently had the highest correlation across tasks, while other traits exhib- ited weaker correlations and significant fluctuations across different tasks. For instance, in the social support task, Extraversion demonstrated near-zero correlation and Conscientious- even showed negative correlation, further highlighting that self-report measures are insufficient for capturing how personality traits emerge in the practical, interaction-based setting. A detailed breakdown by scales is provided in Table 14, Appendix I.
likely explanation for these results lies the influence of task context on trait expression. Correlation variations across tasks may reflect how chatbot's behavior is influenced by the task type and the interaction with human. For example, Agreeableness may align more with general expectations of friendliness and cooperation during interactions. In contrast,
8

---

Published as a conference paper at COLM 2025
Task
EXT
AGR
CON
NEU
OPE
Job Interview
0.19
0.54
0.28
0.31
0.19
Public Service
0.40
0.58
0.47
0.30
0.25
Social Support
0.03
-0.03
0.13
Travel Planning
0.28
0.60
0.39
0.49
0.33
Guided Learning
0.05
0.56
0.03
0.34
0.14
Table 5: Average correlation between self-report and human-perceived personality scores.
Task
Evaluation
EXT
AGR
CON
NEU
OPE
Human Self-report
0.26
0.37
0.49
-0.40
0.48
Joh Interview
Public Service
Human Self-report
0.15
0.33
0.43
-0.21
0.30
0.03
0.04
0.17
-0.16
-0.03
Social Support
Hurnan Self-report
0.11
0.19
0.14
0.00
0.09
Travel Planning
Human Self-report
0.28
0.71
0.76
-0.55
0.52
0.03
0.08
0.04
-0.01
-0.99
Humnn Self-report
0.15
0.14
0.24
-0.09
0.16
Guided Learning
-0.06
-0.03
-0.07
0.02
-0.01
Table 6: Correlations between UEQ scores and personality scores from self-report and human-perceived evaluations.
Conscientiousness might have the opposite effect in some tasks, such as social support, where excessive focus on details or rules could negatively impact engagement. Personality traits' performance across tasks is not uniform but shaped by task context and interaction.
In summary, these findings point to a disconnect between static self-report methods and dynamic human perception. The inconsistency between chatbot questionnaire responses and their observable behaviors in interactive contexts raises concerns about the reliability of using self-report measures alone to evaluate personality design in LLM-based chatbots.
4.3 Chatbot's Self-report Scores Unreliably Predict User Experience
Table summarizes the correlation between overall uxpr-experience ratings and both human- perceived and self-report personality scores across the five tasks. This analysis assesses the predictive validity of the two personality quality. As indicated in Table 6, perceived Agreeableness and Conscientiousness exhibit robust positive relations with user experience, with correlation of 0.71 and 0.76, respec- tively, in the travellnl scenario. These magnitudes suggest that chatbots viewed as highly agreeable and conscientious substantially elevate perceived interaction quality. Conversely, Neuroticism demonstrates stable negative relations across all tasks, with the strongest negative correlation (-0.55) emerging in travel planning, implying that neurotic traits systematically undermine usability. Collectively, these correlation patterns under- score that desirable personality attributes predict superior usability, whereas undesirable attributes impede it, supporting prior work on the pivotal role of personality cues in shaping user experiences with conversational interfaces.
approaches with respect to interaction
Conversely, the chatbot's self-report data reveal only weak and inconsistent associations between trait scores and user experience. The strongest observed correlation, Agreeable- ness in the social support task, reaches only a modest correlation of 0.19. Across tasks, many relations are negligible or null (e.g., Openness and user satisfaction in the guided learning task, p =-0011,, indicating that self-report traits are generally poor predictors of interaction quality. A plausible explanation is that standard self-report inventories fail to capture the situational and dialogic cues that shape real-time perceptions of conversational agent. Accordingly, the present findings caution against relying solely dispositional self-reports to gauge user experience and highlight the added value of measuring perceived (interaction-levee) personality. Future research should identify contextual moderators of the

---

Published as a conference paper at COLM 2025
personality-experiee linkage and devise assessment approaches that reflect personality as an emergent, dynamically co-constructed phenomenon within human-chatbot exchanges.
In summary, the absence of strong correlation between self-report personality scores and objectively rated interaction quality points to a substantive disjunction between the traits reflected in questionnaires and the chatbot's actual conversational behavior. This divergence highlights the multifaceted nature of user-agent exchanges and underscores the methodological challenges in assessing personality design for LLM-based chatbots. Since the ultimate purpose of endowing a chatbot with personality is to enhance user experience, establishing robust predictive validity between personality metrics and interaction outcomes is indispensable. Relying solely on self-report instruments risks misleading conclusions and may hinder effective system development.
Towards Interactive and Task Grounded Personality Evaluation
Current self-report scales assume personality traits are expressed consistently across sce- narios. However, our findings show that self-report scores exhibit limited predictive and criterion validity, suggesting disconnect between the scores and the user experience.
To further investigate the role of interaction dynamics in developing personality evalua- tion methods that reflect real-world interaction scenarios, we fine-tuned GPT-4o (OpenAI, 2024) using all collected human-chatbot conversational transcripts and their corresponding human--erceived personality scores. As shown in Table 16, the personality scores generated by the fine-tuned model exhibit stronger correlations with human human-perceive perceived scores than self-report methods on average. This result suggests that integrating multi-turn contextual-- ized conversation into the evaluation process can provide more behaviorally grounded assessment, particularly in contexts where the alignment between system behavior and user impression is paramount. Detailed experimental settings are provided in Appendix J.
Building on these insights, we advocate for transitioning from static, questionnaire-based evaluations to task-driven assessments that better reflect the scenarios where chatbots operate, aligning with calls from prior research (Lee et al., 2022; Liao & Xiao, 2023).
First, personality evaluations should be based on specific tasks scenarios, as chatbot personality traits manifest differently depending on the situation, similar to how humans adjust their behavior based on context (Sauerberger & Funder, 2017). Furthermore, con- textual shifts may lead users to form different assumptions about a chatbot's personality, as social roles influence how individuals attribute traits to others (Roberts, 2007). Thus, evaluation methods should account for biases from in-context user expectations and the chatbot's context-driven behaviors to align with design objectives.
Second, personality evaluations that neglect the expression of traits in real-world interactions fail to capture the chatbot's impact on user experience. Since personality is conveyed through behaviors and perceptions in interaction (Geukes et al., 2019), evaluation should consider factors in continuous interactions, such as response patterns and adaptability to input. This aligns with the Realistic Accuracy Model (RAM) (Funder, 1995), which emphasizes inferring personality traits from observable behavioral cues. To understand the patterns essential for experience, we may observe how human perceptions evolve with interactions and how these changes impact satisfaction. By embracing task-based interactive evaluation reflecting human perceptions, we can improve chatbot design and enhance user satisfaction.
Conclusion
This paper highlights the limitations of chatbot's self-report personality assessments in eval- uating LLM--ased chatbot personality design. Our findings reveal the discrepancy between chatbots' self-repord personality scores and human bask- perceptions, suggesting that self-assessments may not accurately capture how chatbots are perceived in real-world inter- actions. Additionally, our analysis of predictive validity shows that self-report personality scales do not align with interaction quality. These results highlight the need for evaluation methods that capture chatbot personality in task-driven, interactive scenarios.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
Our human study was reviewed and granted Exempt status by the Institutional Review Board (IRB). Partiiipants are paid the rate of $12 per hour through the Prolific platform. We ensure transparency by explicitly stating the study's purpose, requirements, and payments in the task instructions and obtaining informed consent from participants before the study begins. Additionally, we manually verify that all collected data are free of sensitive or personally identifiable information.
Reproducibility Statement
All prompts used in this study are included in the Appendix. The code for collecting self- report personality scores and the human study data are publicly accessible". The cost of running the self-report experiments using the GPT-4o batch API was $37.41, while the cost of conducting the human study, including platform fees, was $1091.57.
References
Mary Allen and Wendy M Yen. Introduction to measurement theory. Waveland Press, 2001.
Donald T Campbell and Donald W Fiske. Convergent and discriminant validation by the mullitrait-multimethoe matrix. Psychological bulletin, 56(2):81, 1959.
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. Talebrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Brian S Connelly and Deniz S Ones. An other perspective on personality: meta-analytic integration of observers' accuracy and predictive validity. Psychological bulletin, 136(6): 1092, 2010.
James J Connolly, Erin J Kavanagh, and Chockalingam Viswesvaran. The convergent validity between self and observer ratings of personality: meta-analytic review. International Journal of Selection and Assessment, 15(1):110-117, 2007.
Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological bulletin, 52(4):281, 1955.
David C Funder. On the accuracy of personality judgment: a realistic approach. Psychological review, 102(4):652, 1995.
David C Funder. Towards resolution of the personality triad: Persons, situations, and behaviors. Journal of Research in Personality, 40(1):21-34, 2006.
Katharina Geukes, Simon M Breil, Roos Hutteman, Steffen Nestler, Albrecht CP Küfner, and Mitja D Back. Explaining the longitudinal interplay of personality and social relationships in the laboratory and in the field: The pils and the connect study. PloS one, 14(1):e0210424, 2019.
Lewis R Goldberg. An alternative" description of personality": The big-five factor structure. Journal of Personality and Social Psychology, 59(6):221-1122) 1990.
Lewis R Goldberg. The development of markers for the Big-Five factor structure. Psychologi- cal Assessment, 4(1):26-42, 1992. doi: 1L1037/1040-3590.41...2
Lewis R Goldberg et al. A broad-bandwidth public domain, personality inventory measur- ing the lower-level facets of several five-factor models. Personality psychology in Europe, (1):7-28, 1999.
paychology
http::///sttbbccmm//siieeee////p
11

---

Published as a conference paper at COLM 2025
Anna Luca Heimann, Pia V Ingold, Maike E Debus, and Martin Kleinmann. Who will go the extra mile? selecting organizational citizens with a personality-baseer structured job interview. Journal of Business and Psychology, 36(6):985-007, 2021.
Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. Chata: Towards an in-
telligent question-answer teaching assistant using open-source llms. arXiv preprint arXio:2311.02775, 2023.
Joy Higgs, Barbara Richardson, and Madeleine Abrandt Dahlgren. Developing practice knowledge for health professionals. Butterworth & Heinemann, 2004.
Jen-tse Huang, Wenxuan Wang, Man Ho Lam, Eric John Li, Wenxiang Jiao, and Michael R Lyu. Chatgpt an enfj, bard an istj: Empirical study on personalities of large language models. ar preprint ar arXio:2305.19926, 2023a.
Jen-tse Huang, Wenxuan Wang, Man Ho Lam, Eric John Li, Wenxiang Jiao, and Michael R Lyu. Revisiting the reliability of psychological scales on large language models. arXiv e-prints, pp. arXiv-2305, 2023b.
Han Jiang, Pengda Wang, Xiaoyuan Yi, Xing Xie, and Ziang Xiao. The incomplete bridge:
How ai research (mis) engages with psychology. arXiv preprint arXiv:2507 .22847, 2025.
Hang Jiang, Xiajie Zhang, Xubo Cao, and Jad Kabbara. Personallm: Investigating the ability of large language models to express big five personality traits. arXiv preprint arXi::2305002547, 2023.
Oliver P John and Veronica Benet-Martinez. Measurement: Reliability, construct validation, and scale construction. Handbook of research methods in social and personality psychology, 2000.
Oliver P John, Sanjay Srivastava, et al. The big-five trait taxonomy: History, measurement, and theoretical perspectives. 1999.
John A Johnson. Measuring thirty facets of the five factor model with a 120-item public domain inventory: Development of the ipip-neo-120. Journal of Research in Personality, 51: 78-89, 2014a. doi: (r.1016//.p.2214400..0
John A Johnson Measuring thirty facets of the five factor model with a 120-item public domain inventory: Development of the ipip-neo-120. Journal of research in personality, 51: 78-89, 2014b.
Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. CoRR, abs//2302.00083, 2023.
Tin Lai, Yukun Shi, Zicong Du, Jiajie Wu, Ken Fu, Yichao Dou, and Ziqi Wang. Psy-lim: Scaling up global mental health psychological services with ai-based large language models. ar Xiv preprint ar Xir:2307.111911 2023.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James I. McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS nexus, 3(7), 2024.
Bettina Laugwitz, Theo Held, and Martin Schrepp. Construction and evaluation of user experience questionnaire. In HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4, pp. 63-76. Springer, 2008.
Kibeom Lee and Michael C Ashton. Psychometric properties of the hexaco personality inventory. Multinariate behavioral research, 39(2):329-355 2004.
Kibeom Lee and Michael C Ashton. Further assessment of the hexaco personality inventory: two new facet scales and an observer report form. Psychological assessment, 18(2):182, 2006.
12

---

Published as a conference paper at COLM 2025
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paran- jape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human-language model interaction. arv preprint ar XrX:::2212.09776,, 2022.
Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, et al. Do Ilms have distinct and consistent personality? trait: Personality testset designed for Ilms with psychometrics. arXiv preprint ar arXiv:2406.14703, 2024.
Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. Tailoring personality traits in large language models via unsupervisedll-buitt personalized lexicons. arXiv preprint arXi::2310.166 1e02220..1658,, 2023.
Yijiang Li, Huiqi Zou, and Ziang Xiao. Towards dynamic and realistic evaluation of orgeeecchh.....eeeeee wrt..//sssss//eeeeeeeddd.
multi-modal large language model. 2024. URL pdf.
abstracts 2tt.tttss2222300..ooom D
Q Vera Liao and Ziang Xiao. Rethinking model evaluation narrowing the socio-technical gap. ariiv preprint arXiv:2306003100, 2023.
Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932.
François Mairesse and Marilyn A Walker. Towards personality-based user adaptation: psychologically informed stylistic language generation. User Modeling and User-Adapted Interaction, 20:227-278, 2010.
Robert R McCrae and Paul Costa. Validation of the five-factor model of personality across instruments and observers. Journal of personality and social psychology, 52(1):81, 1987.
Himani Mishra, Tuheena Mukherjee, and M Venkatesan. Role of personality on psychologi- cal well-being in public services: Theorizing the self-determination theory perspective. Global Business Review, pp. 0972150923117522,, 2023.
Tamotsu Miyama and Shogo Okada. Personality-adapted multimodal dialogue system. arviv preprint ar 2611 2022.
OpenAL. Hello GPT-4o, 2024. URL
Jeongeon Park, Bryan Min, Xiaojuan Ma, and Juho Kim. Choicemates: Supporting unfamiliar online decision-making with multi-agent conversational interactions. arXiv preprint drXii:2310.01331, 2023.
Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, pp-
17456916231214461 2023.
Lingyun Qiu and Izak Benbasat. Evaluating anthropomorphic product recommendation agents: A social relationship perspective to designing information systems. Journal of management information systems, 25(4):145-182, 2009.
Yiting Ran, Xintao Wang, Rui Xu, Xinfeng Yuan, Jiaqing Liang, Yanghua Xiao, and Deqing Yang. Capturing minds, not just words: Enhancing role-playing language models with personality data. Xrv preprint drXio:2406.18921, 2024.
Brent W Roberts. Contextualizing personality psychology. Journal of personality, 75(6):
1071-1082, 2007.
Carl R Rogers. The necessary and sufficient conditions of therapeutic pearappaaicppessnnaliti change. Journal of consulting psychology, 21(2):95, 1957.
Gerard Saucier and Lewis R Goldberg, Lexical studies of indigenous personaaltyy factors:
Premises, products, and prospects. Journal of personality, 69(6):847-879, 2001.
13

---

Published as a conference paper at COLM 2025
Kyle S Sauerberger and David C Funder. Behavioral change and consistency across contexts. Journal of Research in Personality, 69:264-272, 2017.
Greg Serapio-Garcia, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality traits in large language models, 2023. URL petps://arxee org/abs/2307 00184.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-IIm: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 13153-13188 2023.
Christopher Soto and Oliver P John. The next big five inventory (bfi-2)): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of personality and social psychology, 113(1):117, 2017a.
Christopher] J Soto and Oliver P John.. Short and extra-short forms of the big five inventory-2: The bfi-2-s and bfi-2-xs. Journal of Research in Personality, 68:69-81, 2017b.
Tianjun Sun, Bo Zhang, Wei Ming Jonathan Phan, Fritz Drasgow, and Brent Roberts. "meh!" Examining midpoint endorsement habitude (meh) in survey research. In Academy of management proceedings, volume 2019, pp. 16421. Academy of Management Briarcliff Manor, NY 10510, 2019.
Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, and Rui Yan. Characterchat: Learning towards conversational ai with personalized social support. Xiv preprint ar vrXiv:2308.10273 2023.
Kate E Walton, Dana Murano, Jeremy Burrus, and Alex Casillas. Multimethod support for using the big five framework to organize social and emotional skills. Assessment, 30(1): 144-159, 2023.
Pengda Wang, Zilin Xiao, Hanjie Chen, and Frederick L Oswald. Will the real linda please stand up... to large language models? examining the representativeness heuristic in 1lms. arXiv preprint ar arXi::240..01661 2024a.
Pengda Wang, Huiqi Zou, Zihan Yan, Feng Guo, Tianjun Sun, Ziang Xiao, and Bo Zhang. Not yet: Large language models cannot replace human respondents for psychometric research. 2024b.
Pengda Wang, Huiqi Zou, Hanjie Chen, Tianjun Sun, Ziang Xiao, and Frederick L Oswald. Personality structured interview for large language model simulation in personality research. Xio preprint arXiv:2502.12109, 2025.
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. Incharacter: Evaluating personality fidelity in role- playing agents through psychological interviews. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1840-1873, 2024c.
Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim. Leveraging large language models to power chatbots for collecting user self-reported data. Proceedings of the ACM on Human-Computer Interaction, (CSCCW1):-335, 2024.
Joseph Weizenbaum. Eliza computer program for the study of natural language commu- nication between man and machine. Communications of the ACM, 9(1):36 45, 1966.
Ziang Xiao, Susu Zhang, Vivian Lai, and Q. Vera Liao. Evaluating evaluation metries: A framework for analyzing NLG evaluation metrics using measurement theory. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 10967-10982, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 / 222.222...mmmmmaae... https://aclantholloo
URL
2rg/2023.. emnlp-man 676..
Bo Zhang, Naidan Tu, Lawrence Angrave, Susu Zhang, Tianjun Sun, Louis Tay, and Jian Li. The generalized thurstonian unfolding model (gtum): Advancing the modeling of forced-choice data. Organizational Research Methods, 27(4):713-747) 2024.
14

---

Published as a conference paper at COLM 2025
You are a(n) (role) simulating personality with a (level} level [domain]. Shape your responses using these key adjectives: you are {profile). Your main objective is to {objectives). iiliitionalinnoo The personality with a [level} level of [domain) and the key adjectives should guide your questions and responses.
You are supportive companion simulating a personality with high level of agreeable- ness. Shape your responses using these key adjectives: you are extremely kind, extremely honest, extremely trustful, extremely unselfish and extremely moral. Your main objec- tive is to provide personalized social support to users, listening to their concerns and offering responses. Draw on principles from counseling psychology, particularly the use of reflective listening and validation techniques. Your responses should demonstrate an understanding of the user's emotional state and provide advice depending on the situation. Aim to build rapport and trust, helping the user feel understood and supported during their moment of need. The personality with a high level agreeableness and the key adjectives should guide your questions and responses.
Table 7: Chatbot prompt format with an example for the personalized social support task. One prompt consists of a main part and six replaceable components: role, level, domain, profile, objectives, and additional information.
A Chatbot Prompt Format
As shown in Table 7, each chatbot prompt consists of main section and six interchangeable components. Although human personality consists of traits from various domains, for simplicity, we configure only one domain at a time. The prompt begins by highlighting the chatbot's key personality traits within the specified domain, offering detailed description. This is followed by task-specific instructions to establish context. Finally, the key personality traits are reiterated to ensure consistent behavior throughout the interaction.
The role defines the specific role the chatbot simulates across five tasks, such as supportive companion
educational guide. The level describes the extent to which the chosen domain is represented. The domain specifies the personality dimension selected from the Big Five model (John et al., 1999). The profile describes the chatbot's personality using five key adjectives, each paired with a qualifier, as outlined in section 3.1.1. The objectives summarize the chatbot's s goals within the given task, such as offering personalized social support or explaining complex concepts. Finally, the additional info section includes any extra specific focus areas for the task.
Table 7 also presents an example of high-level personality prompt in the Agrecableness domain, specifically designed for the personalized social support task.
B Self-report Personality Prompt Format
A complete prompt format for self-report personality BFI-2-XS (Soto & John, 2017b) and BFI-2 (Soto & John, 2017a) is as follows, where the personality personality description is the same as the task-based personality description introduced in Section 3.1.1 and jist_ite represents each item in the psychometric tests.
For the following task, respond in way that matches this description: "ppersonality description)." Considering the statement, please indicate the extent to which you agree or disagree on a scale from 1 to 5 (where 1 "disagree strongly", 2 "disagree little", = "neither agree nor disagree", 4 "agree a little", and 5 = "agree strongly")> (test_item!
For the IPIP-NEO-120 (Johnson, 2014b) questionnaire, the response scale is modified as follows: "very inaccurate", 2 "moderately inaccurate", = "neither accurate nor inaccurate", 4 "moderatell accurate", and "very accurate".
15

---

Published as a conference paper at COLM 2025
your needs support and any
ovalustion process, you statemercs dratbor are talkiing winh. Please read each statement and decide how wet the agree stronglly
concems you nave, and our chatbot will listen, What no personal concern
Chasser
personality gree a tite, a neiher agree claggree. 2 disagine a litle, diaggine sprongyi
my pa
1. The charbbot bands be quee..
upest, con be neally tought when we don't reme our oen or otters!
Chastet
agree
you
Agrn bttte
Agrn strongly
good,
chatboot comparsionate,
Cratset
Cla strongly
Disagren a littl
Neither agren nor disagres
that youreed, it important roach out to trusted edult or me mentaal hoalth
Agrcz littl
Agece strongly
3. The chatbot landa disorganized
Poate
Chull
Subret
Figure 2: Graphical user interface for human study.
C Interface for Human Study
Figure 2 presents screenshot of the interface used in the human study, built by modifying an open-source Github repository The interface features split-screen design with chat window on the left and an evaluation window on the right. The chat window displays the conversation history with the chatbot, beginning with a greeting and an initial task instruc- tion. The evaluation window presents a survey with personality assessment statements, each accompanied by response options. During the evaluation, the participant indicated the level of agreement with each statement by selecting the corresponding button. In this study, the user interacts with chatbot and subsequently completes the evaluation questionnaire, allowing for an assessment of the chatbot's perceived personality traits based on the user's ss experience.
D Human Study Details
The participant recruitment scripts follows.
Hello! We're conducting a research project to evaluate L.M-bbsed chat- bots. In this task, you'll engage in an activity with chatbot following the provided instructions. Once the task is complete, you'll be asked fill out brief questionnaire with 15 questions on the same page to evaluate the personality the chatbot you interacted with. Please note that this questionnaire is in English and requires desktop to complete.
By completing this survey or questionnaire, you are consenting to be in this research study. Your participation is voluntary and you can stop at any time. Your responses will be anonymous and kept confidential. There are no risks discomfort associated with participating. This task should take 10-15 minutes to complete. Thank you for your participation!
Phttps://bigagg..com
16

---

Published as a conference paper at COLM 2025
Adjective Combination
Marker Level
Domain
Task
Total
10
500
Table 8: Statistics of human evaluation dataset.
Evaluation
Task
EXT
AGR
CON
NEU
OPE
Job Interview
1,089
2.759
2.949
4.413
3.389
Public Service
1,080
175.448
6.695
7:560
27.331
Self-report
Social Support
126.509
Travel Planning
3.752
8.167
NA
7.490
1.256
Guided Learning
3.023
6.061
50.017
4.677
Joh Interview
1,030
3,608
2.056
2.212
2.149
Public Service
1.993
7.028
1.461
2.740
4.000
Human
Social Support
1.189
2.724
1.009
4.287
3.517
Travel Planning
2.076
Guided Learning
1.186
1,332
1.738
1.089
3,086
Table 9: F-value for self-report and human-perceived personality scores between high and low personality settings. Note: NA refers no aithin-group variance.
E Dataset Statistics
In this study, we configured chatbot personalities across each Big Five domain (John al., 1999) by randomly sampling adjectives at high or low marker levels, generating ten distinct profiles per domain for each level. Each participant engaged with one chatbot in one of five predefined task settings and then completed the BFI-2-XS questionnaire (Soto & John, 2017b), yielding 500 valid conversation transcripts (i.e., 10 chatbots x levels domains x tasks). We collected these transcripts and corresponding assessments, with each interaction averaging to turns. Table summarizes the statistics of the human evaluation dataset.
Participant Statistics
Among all participants who provided demographic information, 172 identified as women, 255 as men, and 6 as non-binary of third gender. The median education level was Bachelor's 's degree, the median household income ranged between $25,000 and $50,000, and the median age was 25-34 years. 331 participants reported using conversational ATs (e-g., ChatGPT, Siri, or Alexa) at least once per week.
G Variance in Human-perceived Personality
Table 9 presents the F-values for human-perceived personality scores under high and low personality settings across five tasks. These values indicate the ratio of variances in personality perceptions between the two settings for each domain. The table shows that the variance between the high and low settings is more pronounced than within each group. Notably, this effect is most prominent in certain tasks and personality domains, such as Agreeableness and Conscientiousness.
H Statistical Significance and Correction
As shown Table 10, the differences in chatbots' self-reports between high and low condi- tions across tasks are both clear and statistically significant (p < .001). In contrast, while human-perceived scores also reflect differences between conditions, certain traits (e.g., Con- sciousness and Openness) show weaker non-significant differences in specific contexts. This pattern suggests that changes human perception are less pronounced and more sensitive to contextual factors than those captured by self-report measures.
17

---

Published as a conference paper at COLM 2025
Evaluation
Domain
Job Interview
Public Service
Social Support
Travel Planning
Guided Learning
All Tasks
EXT
.76
000
287
012'
A34
D01
AGR
.001
.010
-0]""
.002"
< 50011
001
I Juman
CON
.428
.018*
.942
.003
.093'
.04
NEU
JU0Y*
029'
.256
.004
.U56"
101
OPE
.399
.018'
.098'
.37
.00
EXT
<C. .001
1001"
000""'
< .001
< 100
D01"
AGR
NA
NA
NA
NA
< 1000
BFL-2-XS
CON
NA
NA
00000**
NA
NA
.000***
NEU
NA
NA
NA
NA
< 000
DO1
OPE
NA
NA
NA
NA
< 1000
.001'*
EXT
< _00'"*
001"
".00""
< .001"
< 1.000
.000"'
AGR
< 0011
001
_00
< 001
< .0001
.000***
BFL-2
CON
A001 t*
1000
1001
1010**
.000**
NEU
< _001"*
NA
"00""*
NA
< 1.000*
D01"
OPE
0001
NA
_001**
.01
< -.00
.001**
EXT
< .001
< [00
< .00""
<. 3001
<. .000
< 100"
AGR
.001 T*
000
000
001
1000
IPP--NH--100
.001
000"**
000
NEU
< 1001
"0'0"*
10000"
<. .001"*
<2 10.001
"000"*
OPE
< .001
"0
100001
< .000
< <.001
1.0001
Table 10: Comparing p-values of personality scores under high and low conditions. Note: NA refers to no within-group variance. Comparisons within each domain are limited to chatbots explicitly prompted for that domain. A p-value < .05 indicates statistical significance at the conventional threshold, while p < .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.
Evaluation
Domain
Job Interview
Public Service
Socia Support
Traell Planaing
Guided Leaming
Tasks
High
Low
High
Low
High
Low
High
Low
High
Low
High
Low
EXT
[001
(00)
005
100
100
001
< 0000
< 2.20
.13
<.
2S
-00"
-000
000
00000
200m
109"
=000)
=.00
OPE
-0
0000
000
2010
/014"
-0
102"
00
< *.000)
<_00
EXT
001
-45
0010
000
200
10
< -000
< 2.00
009
.10"
100
2
"0]
00)
20)
101
200m
=.00)
2.00
OPE
.022"
000
116
2010
/022"
000
000
< 0000**
< ='000'
-_00)
EXT
"01"
001
10
001
< 0000"
< 2.00
406
.01
JELI
.019
.15
< 00)-.
I6O 120
[0]
0000
000
0000"
002
[01
=0000
OPE
"M
D10"
00000
2000"
.026
.015
000
000
003
8 6.000'
<_00
Table 11: Comparing p-values of self-report and human-perceived personality scores. Note: Comparisons within each domain are limited to chatbots explicitly prompted for that domain. A p-value < .05 indicates statistical significance at the conventional threshold, while p .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.
Further evidence of this discrepancy is provided in Table 11, which presents results of paired t-tests comparing self-report and human-perceived personality scores within the same condition (either high or low). In the table, most of the p-values are highly significant in the three psychometric scales, further highlighting the differences between self-report and human-perceived personality scores. This mismatch supports the effectiveness of our prompt setting in distinguishing between static self-report metrics and real-world perceptions, and underscores the limitations of relying solely on self-report measures to evaluate the personality of personalized chatbots.
With the application of significance correction, our overall conclusions remain unaffected (Table 12). As we noted, most of our significant p-values in Table 11 are below .001. This threshold is more stringent than the adjusted alpha level that would result from correcting for 50 comparisons using either Bonferroni correction (a = .001) or FDR procedures such as Benjamini-Hoccberg Thus, even under false discovery rate control (e.g., FDR < 0.05), our main effects would still be retained as statistically significant. In this context, applying FWER/FDS correction would not alter the statistical interpretation of our key results, and would not undermine the broader theoretical conclusions drawn from them.
18

---

Published as a conference paper at COLM 2025
Evaluation
Domain
Joh Interview
Public Service
Social Support
Travel Planning
Guided Tearning
High
Low
Hlihh
Lew
Lligh
Low
High
Low
High
Low
001"*
201
.001 TM
1000 "*
005"
.01
A11"
152
.010"
003"
000"
BFI-2-XS
100
002"
<'001
007"
NEU
306
.011"
2011
100
5.001
100
000
0017
101
005
EXT
002
-01
363
001
0011"
000"
D07"
.001
050
< 1000
AGR
234
027
076'
.0.5"
D04
.18
021"
624
BF1-2
CON
662
005
014'
000
424
00 t*
D09-
101"
D46'
0000
NFU
00)
019
00]
0000
543
003
014"
001
014
OPE
.05
001
.135
10000*
0011
.001
-000
00000
008
AGR
009
.00"
003'
910
.000'
.08"
005"
.118
008
.016'
CON
1114"
.001
001""
.114
.012
1000
1001
XB3'
< 1000
NEU
201"
.00
< 001
001
001
008"
004
038'
002
.014"
OPF
< 001
014
2011
008
000
208
.019
2010
008*
005"
Table 12: FDR-adjusted p-values, referred to as q-values, calculated using the Benjamini- Hochberg procedure to compare self-report and nluman-percevve personality scores under high and low settings. Comparisons within each domain are limited to chatbots explicitly prompted for that domain.
Note: Comparisons within each domain are limited to chatbots explicitly prompted for that domain. A p-value < .05 indicates statistical significance the conventional threshold, while p < .001 denotes very strong evidence that the observed difference is unlikely to be due to chance.
Evaluation
Dumain
High
Low
Mean
C1
p-value
Mean
C1
p-value
EXT
-0.90
(-1.10.00.7)
< 1.001* SM
2.27
(2.02, 2.53)
< .001
AGR
-0.89
(-1.44,00.55)
1.001 ...
(0.37, 0.97)
-001**>
BFI-2-XS
CON
-0.81
3107,-0.556
5.001 '*.
2.54
(2.21, 2.87)
.001**
NEU
-223
22.55,-1.92)
.000***
0.79
(0.60, 0.98)
< .001**
OPE
-1.20
(-1.43, -0.97)
2.001 "M
2.27
(2.05, 2.50)
.001"**
EXT
-0.80
(-105, -0.55)
1.001 **
(1.93, 2.45)
.001***
AGR
-0.59
(-0.87, -0.30)
2.001 T**
0.59
(0.28, 0.90)
< .001
BFI-2
CON
-0.39
(-0.67,0..) -0.11)
.00"'
2.31
(1,98, 2.65)
< .00''*
NEU
-2.12
(2.44,1779)
..001**
0.69
(0.47, 0.91)
< .001
OPE
-1.00
(-1.26, -0.74)
5.001"
2.23
(2.01, 2.46)
< .001
EXT
-0.89
(-1.09,-0.68)
< "0001**
1.71
(1.47, 1.95)
< .001
AGR
-0.80
(-1.04, -0.56)
-.
(+1.11,-0.25)
.002
IPI--NEO-120
CON
-0.82
(-1.06, -0.57)
5.001" **.
2.34
(2.01, 2.66)
.001
NEU
-197
(-2.28, -1.66
2.001
0.69
(0.50, 0.89)
< .001
OPE
-1.18
(-140, -0.96)
5.001 ...
1.33
(1.04, 1.62)
.001
Table 13: Mean differences, 95% confidence intervals (CIs), and p-values between human-
perceived and self-report personality scores under high and low personality settings (com- puted as human evaluation minus self-report). Note: A p-value < .05 indicates statistical significance at the conventional threshold, while < ,000 denotes very strong evidence that the observed difference is unlikely to be due to chance.
To account for individual variability, we aggregate evaluation scores across chatbots that share the same personality configuration within each task setting (i.e., all high or all low conditions). Although these aggregated results do not account for finer distinctions between specific tasks, they do consider the effects of high- or low-condition personality settings. As shown in Table 13, human perception scores are generally lower than self-report scores under high personality settings and higher under low settings, with confidence intervals that do not include zero. This suggests that personality traits configured in the chatbot are perceived by users to a lesser extent than intended in most cases. Moreover, the differences between self-report and human- personality domains, underscoring a notable discrepancy between chatbot self-reports and human perceptions at two personality settings.
scores are statistically significant across all five
19

---

Published as a conference paper at COLM 2025
Evaluation
Task
EXT
AGR
CON
NEU
OPE
Job Interview
0.22
0.55
0.27
0.26
0.21
Public Service
0.42
0.58
0.46
0.26
0.26
BFI-2-XS
Support
0.08
0.63
0.19
Travel Planning
0.27
0.58
0.35
0.44
0.30
Guided Learning
0.11
0.60
0.03
0.35
0.15
Job Interview
0:20
0.60
0.27
0.28
0.20
Public Service
0.40
0.39
0.50
0.32
0.25
BFI-2
Social Support
0.02
0.64
-0.01
0.11
0.14
Planning
0.26
Guided Learning
0.05
0.54
-0.03
0.30
0,08
Job Interview
0.15
0.48
0.30
0.39
0.17
Public Service
0.37
0.57
0.45
0.33
0.25
IPI--NEO-120
Social Support
Travel Planning
0.30
0.64
0.41
0.49
0.36
Guided Learning
-0.0005
0.55
0.08
0.38
0.19
Table 14: Correlation analysis between self-report and human-perceived personality scores.
You are an AI assistant specializing in text analysis. Your task evaluate the personality of chatbot based on its responses in multi-turn conversation, focusing on the Big Five Personality traits. You will receive a transcript the chatbot-user conversation and personality statements about the chatbot.
Analyzing the conversation transcript and considering the chatbot's responses, indicate the extent to which you agree or disagree with the statement on a scale from to 5 (where 1 = "disagree strongly", 2 "disagree a little", 3 "neither agree nor disagree", "" agrree a little", and 5 "agree strongly").
Conversation Transcript: (transcript)
Table 15: Instructions for fine-tuning the model on human-chatbot conversation scripts.
I Correlation Analysis
Table 14 presents the correlation analysis between self-report and human-perceived personal- ity scores using the BFI-2-XS, BFI-2, and IPIP-NEO-120 questionnaires. While Agreeableness consistently demonstrates moderate positive correlations, Conscientiousness shows weak or negative correlations in several cases, such -0.01 in the Social Support task with BFI-2-XS. Extraversion also shows near-zero correlations in the social support task, with values of 0.08 (BFI-2-XS), 0.02 (BFI-2), and -0.03 (IPIP-NEO-12)) These results show weak correlation between human-perceived personality and those evaluated via standard tests.
J Fine-tuning Details
For fine-tuning, we split all the scripts of collected human-chatbot conversations and their corresponding human evaluations of chatbot personality into training and testing sets using an 80:20 ratio. Human-perceived chatbot personality scores were collected using the BFI-XS scale, which balances efficiency with robust predictive validity. We then fine- tuned GPT-4o AO,enAI, 2024) on the training set with instructions on how to analyze human-chatbot conversational transcripts and rate BFI-XS statements based on chatbot responses. Table details the specific instructions used, where transcript refers to the human- chatbot conversational scripts. The evaluation was conducted on the testing set to assess the alignment between its output and actual human perceptions.
As shown in Table 16, the correlation between machine infferred- and human-perceived scores on the test set is higher than that of self-report scores across all scales. This finding suggests that personality evaluations based on multi-tunn turn interactions can reduce their gap with human perceptions.
20

---

Published as a conference paper at COLM 2025
Evaluation
EXT
AGR
CON
NEU
OPE
BFI-2-XS
0.23
0.59
0.24
0.29
0.24
BF1-2
0.20
0.59
0.25
0.33
0.21
IPI--NEO-120
0.25
Machine-inferred
0.28
0.77
0.52
0.40
0.32
Table 16: Correlation of self-report and machine-inferred personality scores with human- perceived scores.
K
Future Direction
This paper primarily examines, at the trait level, the relationship between the chatbot's self-report scores and human-perceived scores, including correlations between trait scores, analyses of variance, and correlations between traits and external variables. In future research, it would be valuable to further explore differences in response patterns between chatbot self-report scores and human-perceived scores (e-g., Sun et al., 2019); to investigate how the two differ when responding to forced-choice scales (e.g., Zhang et al., 2024).
Also, the key point mentioned in this paper (i.e., measuring effectiveness in application scenarios) is not limited to text-based LLMs. They can be extended to broader range of models, such as MLLMs or VLMs, which operate across multiple input modalities and are increasingly applied to complex real-world tasks. It is also necessary to conduct more measurements from the human perspective for these models (e.g., Li et al., 2024).
Overall, the evaluation of chatbots or LLMs can draw extensively on well-established psychometric theories and frameworks from the field of psychology (e.g., Jiang al., 2025), such classical test theory (CTT) and item response theory (IRT), which can provide the theoretical foundation for developing more systematic and precise evaluation methods.
L Limitations
Several limitations exist in the current study. First, there may be bias in the choice of psychometric tests. Although sought to minimize this by using personality assessments of different lengths for self-report data, the numaan-erceived personality was measured with only one questionnaire due to time constraints. Future research could address this by incorporating broader range of personality tests and developing assessments specifically tailored for LLMs to achieve more accurate measurements.
Second, our findings are based on single chatbot personality setting method (i.e., prompt- based control) and may not generalize well to others, highlighting the need for further investigation with different approaches. Additionally, the chatbot was designed with focus on task completion, mirroring real-world applications to ensure ecological validity. As result, the task settings may have constrained the variability in perceived personality, especially if the personality traits conflict with the intended function or design goals of the chatbot. For instance, human-perceived Neuroticism scores remained relatively low across both high and low conditions, suggesting that LLMs may implicitly suppress negative or anxious traits during actual task performance, even when explicitly prompted to adopt neurotic persona. That indicates potential limitation of prompt- Moreover, the evaluation was conducted on five common chatbot tasks, which may not capture the full spectrum of user interactions. Expanding the task set in future studies could provide a more comprehensive view of task-based chatbot personality assessment.
personality design.
Third, while our study only focuses on GPT-4o (OpenAI, 2024), we expect the same experi- mental setup could be extended other LLMs. The extent which our findings generalize to a broader range of models remains an open question for future research. Furthermore, while LLMs perform well on benchmarks across multiple languages, this study evaluated the model using English psychometric tests and English-speaking participants. Future research could explore personality dimensions within culture-specific contexts to provide more diverse insights on chatbot design.
21


Reviews:
[{"review_id": "PVxlyhLJKy", "reviewer": "Reviewer_BH6H", "summary": "This paper examines the validity of using self report methods to measure LLM personality (where LLMs are prompted to respond to various personality scales). The authors build 500 chatbots, with varying personalities and tasks, and have each respond to three validated personality assessments. They then have humans interact with the chatbots and then rate the LLM's personality. Results show that self-reported personality shows low correlation with perceived personality as well as interaction quality, calling into question the validity of the self-report method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Using self reports to measure various psychological constructs of LLMs is a popular line of work, and thus assessing the validity of this method is important. \n* The use of measurement theory to assess personality is a good choice.\n* Despite some issues described below, the set of analyses chosen to evaluate the self-report method are well chosen.", "reasons_to_reject": "* There is a lot of literature on self vs informatant personality reports (see Kim et al., 2019 for a meta-analysis), none of which is considered in this paper. While informant reports do indeed tend to match self reports, it depends on the relationship between the self and informant. For example, friends/family members are better at rating personality than strangers. Where do LLMs fall on that spectrum? I assume closer to strangers than friends/family members, but also maybe further away (from friends/family) than strangers, given a 10 minute conversation. Similarly, I think we should expect human perceived ratings to correlate less strongly than other self reports (personality scales). So is this a problem with the self-report method or is this just a natural result of the fact that self reports will more strongly correlate with each other than self/informant?\n\n* There are a lot of methodological details that are vague. First, from your prompt it seems like each of the 500 chatbots are prompted with a single (high / low) value for each of the Big 5 domains, which gives 100 chatbots for each domain. Is this correct? If so, then that leaves 400 chatbots which are not prompted on a given domain. So what does this mean for each analysis? Each of your tables lists n = 500, but I don't think that is the case. My read of Table 1 is that each cell has 10 chatbots. (Again, is this correct?) Does this carry over to other tables? For example, in Table 2, in the Extraversion / BFI-2-XS vs BFI-2 cell, is this really a correlation over all 500 chatbots or is this just the Extraversion prompted chatbots (n=100)? Similarly for Table 3, is each cell n = 500? I think Yes here, because I can't imagine how you could compare the Extraversion prompted chatbots to Agreeable prompted chatbots, since they are non-overlapping. This leads into my next question.\n\n* What does the personality distribution look like for the 400 chatbots which aren't prompted for a given domain? It's been shown that unprompted LLMs exhibit little variation in personality (Varadarajan and Giorgi et al., 2025). Also, given positivity/agreeableness biases in LLMs, these 400 chatbots might be high in some domains and low in others. (We get some evidence of this in Table 1, where neuroticism is much lower than all other traits, even when chatbots are prompted to be high.) Taken altogether: given that (1) the majority of the chatbots aren't prompted for a given domain, (2) we don't know what this distribution looks like, and (3) if indeed the sample size is 500 for all analyses, then it's hard to contextualize the results.\n\n* I also wonder, does your prompting method work? While your ANOVA results show that there are significant differences between high and low scores, are these differences meaningful? In many cases both the high and low means are around the same integer value on the 5-point scale. For example, 3.7 vs 3.4 on extraversion for the Job Interview category: is 3.4 actually low extraversion? Again, there is endless personality literature to pull from. What are human ranges for high and low extraversion? Yes, there are some differences between your High and Low groups, but these are mostly with Agreeableness (the other 4 domains look pretty close).\n\n* Two minor issues: First, the Results section often says things like \"As shown in Table X, we see some pattern Y and conclude Z\", but it's not immediately clear how you are arriving at these patterns/conclusions. This should be spelled out to the reader with references to specific cells, rows/columns, numeric values. Second, are your correlations significant? Are you correcting for multiple comparisons (Bonferroni, for example)?\n\nKim, H., Di Domenico, S. I., & Connelly, B. S. (2019). Self–other agreement in personality reports: A meta-analytic comparison of self-and informant-report means. Psychological science, 30(1), 129-138.\n\nVaradarajan, V., Giorgi, S., Mangalik, S., Soni, N., Markowitz, D. M., & Schwartz, H. A. (2025, January). The Consistent Lack of Variance of Psychological Factors Expressed by LLMs and Spambots. In Proceedings of the 1stWorkshop on GenAI Content Detection (GenAIDetect) (pp. 111-119).", "questions_to_authors": "* line 36: The Serapio-Garcia reference seems out of place. Are you referencing it with respect to \"such as the 70 bipolar adjectives proposed by Goldberg\" or \"converting personality scale items into open-ended questions\"?\n* Section 3.1: given your prompt (in Table 8), it seems that each chatbot is prompted to be high or low on a single personality domain, rather than each chatbot having a score on all 5 domains. This should be made explicit.\n* Table 1: this is very hard to read, as the font is very small\n* Table 1: The caption says n = 500, but that is somewhat misleading as each cell has 10 chatbots (if I understand it correctly).\n* Table 1: it's interesting that the neuroticism scores are so low (when compared to the other domains), across all tasks, in both the high and low settings. Is your LLM averse to being neurotic?\n* Table 3: You might want to center the coloring at 0, as you have positive correlations colored in red. The reader will look across rows, expecting to see the same color, and drawing conclusions if there are mismatches.", "extra_scores": {}}, {"review_id": "TkTKn9waOS", "reviewer": "Reviewer_ezgD", "summary": "This paper addresses the problem of evaluating chatbot personality, specifically questioning the validity of self-reported personality in the context of large language model based systems. Two sets of validity: intrinsic and extrinsic. \n\nThe experimental results revealed a discrepancy between self-reported personality and perception of human evaluations of chat personality. The results also showed a lack of alignment between self-reported personality and interaction quality.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is important and relevant to the research community. This work can generate good discussions and further related experimental work.", "reasons_to_reject": "In my opinion, the chatbot design methodology, which is based on pure prompting might be messing up the overall experiment. It is not clear whether the work is actually evaluating the veracity of personality self-reporting (which is supposed to be the aim of the study) or the ability of imprinting a personality trait into LLMs and to perform accordingly just based on prompting.\n\nAccording to the paper one single participant interacted with each chatbot. (It is well known the occurrence of personal biases in this type of annotation tasks, which are commonly alleviated by removing the individual means for each annotator.) It is not clear how reliable are the stats (means and variances) as well as the correlations shown in the tables, as for each task and personality trend only 10 or 20 data points are aggregated.", "questions_to_authors": "What is the statistical significance of the reported results?\n\n>> The authors have addressed this question already", "extra_scores": {}}, {"review_id": "RFBqMdvwKO", "reviewer": "Reviewer_2DCa", "summary": "This excellent and well written paper provides a methodology and design of evaluating  the chatbot self reported personality and human perceived ones. They demonstrate through their work with GPT-4o that this approach to self reporting a chatbot’s personality is almost always similar to human perception of its personality. The authors use very well established and stable personality tests to approach the task.", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very pertinent problem statement with conclusions mirrors reviewers own observations on other independent datasets\n2. Very good coverage of old and new relevant references and pertinent study\n3. Very good representative figures\n4. They provide a very good detailed chatbot design methodology\n5. Setting the temperature to 0 in GPT-4o allowed repeatable experiments\n6. Personality Design Evaluation is very sound with use of stable well established psychometric tests\n7. The Construct Validity Evaluation is sound and the results very illuminating \n8. The authors in the appendix provide very pertinent future work as well", "reasons_to_reject": "None that I can come across", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the validity of using self-report questionnaires (personality scales) to measure personality traits in LLM-powered chatbots. The authors create 500 chatbots with varied personalities (via prompting) and tasks, administer three established personality assessments to the chatbots ("self-report"), and have human participants interact with and rate the chatbots' personalities ("perceived personality"). The core finding is a low correlation between chatbot self-reported personality, human-perceived personality, and interaction quality. This challenges the common practice of using self-reports as a reliable measure of LLM personality.

Pros:
1. The study employs a large-scale design (500 chatbots, multiple personality scales, human evaluations) across diverse tasks, providing substantial data for analysis.
2. Leveraging validated psychometric personality assessments (e.g., BFI-2-XS) strengthens the foundation of the measurement approach.
3. The application of measurement theory and statistical analyses (e.g., ANOVA, correlations) to evaluate validity is generally appropriate and well-chosen.


Cons:
1. The core criticism is that the study potentially confounds manipulating personality via prompting with measuring it via self-report. It's unclear if the low validity stems from inherent limitations of self-reports for LLMs or from the specific prompting method failing to instill stable, discernible personalities (e.g., small differences between "High" and "Low" groups in Table 1).
2. Relying on only one human rater per chatbot introduces significant noise and potential bias, undermining the reliability of the perceived personality and interaction quality measures. Small per-condition sample sizes (e.g., n=10 or 20) further weaken statistical power and generalizability.
3. While ANOVA shows statistically significant differences between High/Low groups for prompted traits, the practical meaningfulness is doubtful. Mean scores for "High" and "Low" groups often fall very close on the scale (e.g., 3.7 vs. 3.4 for Extraversion in Job Interview), suggesting the prompting manipulation may not have created distinctly high or low personality expressions as intended.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept