PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection
Authors: Tao Yang
Tianyuan Shi
Fanqi Wan
Xiaojun Quan
Qifan Wang
Bingzhe Wu
Jiaxiang Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: personality detection
psychological questionnaire
chain-of-thought
TL;DR: We investigate the use of psychological questionnaire as a chain-of-thought for the personality detection task.

Abstract: Recent advances in large language models (LLMs), such as ChatGPT, have showcased remarkable zero-shot performance across various NLP tasks. However, the potential of LLMs in personality detection, which involves identifying an individual's personality from their written texts, remains largely unexplored. Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, we argue that these items can be regarded as a collection of well-structured chain-of-thought (CoT) processes. By incorporating these processes, LLMs can enhance their capabilities to make more reasonable inferences on personality from textual input. In light of this, we propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. In particular, we employ a LLM as an AI assistant with a specialization in text analysis. We prompt the assistant to rate individual items at each turn and leverage the historical rating results to derive a conclusive personality preference. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection, achieving an average F1 score improvement of 4.23/10.63 points on two benchmark datasets compared to the standard prompting method. Our code is available at \url{https://github.com/TaoYang225/PsyCoT.

Psy CoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection
Tao Yang', Tianyuan Shi', Fanqi Wan', Xiaojun Quan'*
Qifan Wang', Bingzhe Wu', Jiaxiang Wu
"School of Computer Science and Engineering, Sun Yat-sen University, China Meta AI, Teencent Al Lab
(yangt225, wanfq, shity6)@mail2.. sysu. edu. en, quanx jumnxj3mmaill sysu. edu. cn wqfcrêfb. com, subingzheagent@@maiinn com, jonathanwwtteneentccm
Abstract
We have text written by an author:
Recent advances in large language models (LLMss) such as ChatGPT, have showcased re- markable zero-shot performance across various NLP tasks. However, the potential of LLMs in personality detection, which involves identify- ing individual's personality from their writ- ten texts, remains largely unexplored. Drawing inspiration from Psychological Questionnaires, which are carefully designed by psychologists to evaluate individual personality traits through a series of targeted items, argue that these items can be regarded as collection of well- structured chain-of- totho-h (CoT) processes. By incorporating these processes, LLMs can enhance their capabilities to make more reason- able inferences on personality from textual in- put. In light of this, we propose novel person- ality detection method, called PoyCoT,, which mimies the way individuals complete psycho- logical questionnaires in multi-turn dialogue manner. In particular, we employ LLM as an AI assistant with specialization in text anal- ysis. We prompt the assistant to rate individ- ual items each turn and leverage the histor- ical rating results to derive a conclusive per- sonality preference. Our experiments demon- strate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in per- sonality detection, achieving an average F1 score improvement of 4.23/10.66 points two benchmark datasets compared to the standard prompting method. Our code is available at https://githuie com/T Ta0Yang225/PsyCoT.
Today has been pretty good day. little worried about.....
Rate items baged on text.
Item1: The author is talkative.
Tum 1
Score: (Agree little)
Item8 :The author sociable.
Tum
Score: 5 (A gree strongly)
CoTl Steps
The author is "Extraversion" or B: "Introversion".
Tum
Personality Inquiry
Choice: (Extraversion)
Figure 1: An illustration of our PsyCoT, where items (1-8) from the personality questionnaire as the CoT to answer the final personality inquiry. We prompt the LLM rating items based on the author's text, which simulates the process of human to complete personality tests through a multi turn dialogue.
employed
Bagby al., 2016; Andrist et al., 2015; Hickman et al., 2022; Matz al., 2017). For instance, per- sonality aids clinical psychologists gaining a bet- ter understanding of psychiatric disorders (Khan et al., 2005) and developing personalized treatment modalities (Bagby et al., 2016); it improves the human-robot interaction, particularly for socially assistive robots (Andrist et al., 2015).
1 Introduction
Previous studies (Mehta et al., 2019; Yang et al., 2021b, 2022) on text-based personality detection have focused on training or fine-tuning specific models. However, their performance is significantly limited by the quality and quantity of training data. The emergence of large language models (LLMs), such as GPT-3 (Brown al., 2020), InstructGPT (Ouyang et al., 2022), and LLaMA (Touvron al., 2023), has recently demonstrated impressive in- context learning (ICL) ability, in which LLMs
Personality, as an important psychological con- struct, refers to individual differences in patterns of thinking, feeling, and behaving (Corr and Matthews, 2009). Consequently, detecting one's personality from their generated textual data has garnered considerable interest from researchers due its wide-ranging applications (Khan al., 2005;
Corresponding authors.

---

make predictions solely based on designed prompts instructions without any parameter modifica- tions, leading to a new paradigm in the NLP com- munity. Building upon these strengths, this study aims to investigate the ability of LLMs to perceive an author's personality from text, an aspect that has not been extensively explored previously.
ods on most personality traits, demonstrating its competitiveness within the fine-tuning paradigm. Furthermore, ablation studies and analysis indi- cate that the form of multi-turn dialogue helps to achieve a more accurate reasoning than the single- turn dialogue, and PsyCoT exhibits strong robust- ness when faced perturbation in option order.
Inspired by the process of human to complete self-assessment personality tests, we approach per- sonality detection as a multi-step reasoning task since psychologists often use a series of assess- ments to measure an individual's personality. To perform complex reasoning, a recently technique named chain-of- (CoT)) has been developed for solving math word problems (Cobbe et al., 2021; Patel et al., 2021) by generating intermediate reasoning steps (Wei et al.,, 2022b; Kojima al., 2022; Zhang et al., 2022b). our scenario, we ar- gue that the items in psychological questionnaires can be considered as set of rigorous reasoning steps. Consequently, we propose a novel personal- ity reasoning framework, PsyCoT, which utilizes psychological questionnaires as COT. Figure 1 illus- trates the framework of PsyCoT, which mimics a human to complete personality test through multi- prompt the LLM (e.g., G act an Al assistant whose task is rate² a given item based on the author's text. At each turn, we sample an item from the psycho- logical questionnaire and present it to the assistant. The Al assistant then returns a specific score for the item. Once all the items have been rated, the AT assistant selects definitive personality trait based on the historical rating results. Additionally, these scores can be aggregated using rules defined by questionnaires to yield an overall score, which sev- ers as a double check and provides confidence for the chosen personality trait.
Our work is the first to explore the ability of LLMs in detecting personality. The proposed Psy- CoT incorporates a well-designed questionnaire CoT steps to facilitate the reasoning of LLM. We highlight that GPT-3.5 yields comparable perfor- mance to some fine-tuning methods by equipped with PsyCoT. Besides, PsyCoT offers a fresh per- spective of using well-designed questionnaires to design prompts in the era of LLMs.
2 Related Work
Personality Detection In early stage, Pen- nebaker et al. (2001) developed Linguistic Inquiry and Word Count (LIWC) to extract psycholinguis- tic features from text, which has been used for feature engineering in machine learning models (Cui and Qi, 2017; Amirhosseini and Kazemian, 2020). However, these feature engineering- based methods have limitations in extracting implicit se- mantic features. Recently, several studies have proposed unbiased user representations for person- ality detection. For instance, Yang et al. (2021a, 2022) addressed the post-order problem in encod- ing posts by introducing a multi-document Trans- former and unordered dynamic deep graph net- work, respectively. Zhu et al. (2022) constructed a fully connected post graph for each user and devel- oped CGTN to consider correlations among traits. Another body of research incorporates additional knowledge into the model. For example, Yang et al. (2021b) proposed TrigNet, which constructs heterogeneous tripartite graph using LIWC and utilizes flow-GAT to operate on this graph. Yang et al. (2021b) introduced PQ-Net, which incorpo- rates psychological questionnaires as additional guidance to capture itee-velevant information from contextual representations. Despite their successes, these methods are rely on a data-driven approach to train the model to capture implicit personality cues, which are differ from our study as we specifically explore the zero-shot performance of the LLM in personality detection.
turn dialogue. Specifically,
To evaluate the effectiveness of PsyCoT, we conduct extensive experiments on two benchmark datasets (i.e., Essays and Kaggle) that employ dif- ferent personality taxonomies (i.c., Big Five (Gold- berg, 1990) and MBTI (Myers-Briggs, 1991)). Ex- perimental results show that PsyCoT significantly increases the zero-shot performance of GPT-3.5 in personality detection. For instance, PsyCoT outper- forms the standard prompting by 4.23/10.63 points in average Fl on the two datasets. Moreover, in the Essays datasct, PsyCoT surpasses finc-tuned meth-
hhtps::/plltform.oo wwwnoo..ooooniiioo com///oss/mmooeesggg--33 WWe follow the original rating rules defined by psychologi- cal questionnaires.
LLMS and Prompting Methods Recent years have witnessed an increasing interest in LLMs,

---

such GPT-3 (Brown et al., 2020), FLAN (Wei al., 2022a), OPT (Zhang et al., 2022a), PaLM (Chowdhery et al., 2022), and LLaMA (Touvron et al., 2023), due to their impressive zero-shot gen- eralization across various tasks. With techniques like instruction tuning (Wei et al., 2022a) and rein- forcement learning with human feedback (RLHF) (Ouyang et al., 2022), ChatGPT3 demonstrates re- markable alignment capabilities when following human instructions. Consequently, to leverage the potential of LLMs in downstream tasks, several works have focused on carefully designing prompts manually (Hendy et al., 2023) or automatically (Gao et al., 2021; Zhou et al., 2022b). Another approach that has gained attention is the Chain-of- Thought (CoT), which explicitly guide LLMs in generating reasoning steps. Wei et al. (2022b) ini- tially introduced the few-shot-CoT, which utilized reasoning steps demonstrations by crafting few- shot exemplars, resulting in significant improve- ments in complex math tasks. Building on this work, other studies have proposed various variants of CoT, including Zero-shot CoT (Kojima et al., 2022), Auto-Cot (Zhang et al., 2022b), Least-to- most prompting (Zhou et al., 2022a), and Synthetic prompting (Shao et al., 2023). Unlike most of these works that focus on how select few-shot exem- plars, we aim use psychological questionnaire as a explicitly CoT to facilitate reasoning.
Algorithm 1: PsyCoT Prompting.
Input: LLM: LLM (+); Author's text: X;
Task description prompt: D;
Inference prompt: I; K Reasoning
prompts: R- (ri)k=]
Output: the personality trait: y'
I: Initialize a dialogue history H = [nall]]
2: for k = 1 to K do
3: Get k-th rating result on under ri:
a² = LLM X,,
4: Append the r4K and 0g into H:
II [r, 01.001..... Tke aj.
5: end for
6: Infer the personality trait:
yf = LLM (D, H,
7: return of
where ÿ' is the inferred personality trait. We show an example of the standard prompting at the top of Figure 2. The initial paragraph introduces a task description prompt D, where we specify the role of LLM as an AI assistant specializing in text analysis. Its objective is to predict an individual's 's personality trait based their written text. The subsequent paragraph contains the text written by the author, which will be analyzed by the LLM. Lastly, the final paragraph depicts the inference prompt I, which includes the provided options and instructions for the LLM to generate the choice using a fixed format. The standard prompting relies solely on the LLM to comprehend the concept of personality and determine how to evaluate it.
3 Psychological Questionnaire as CoT
For the current generative LLM, personality de- tection can be formulated as a reading com- prehensive task. Formally, given a set of texts X= {#1, }n written by an individual, the goal is to determine the correct personality option about this individual based on X. We can achieve the goal by designing the appropri- ate prompt P = (D, f}, in which D represents the task description prompt that informs the LLM about the task's definition, and represents the in- ference prompt that pushes the LLM to select the desired personality trait in specified format.
3.2 PsyCoT Prompting
While standard prompting techniques have demon- strated effectiveness across various tasks (Sanh et al., 2022; Wang et al.. 2023), they fall short when confronted with the complex task of person- ality detection. This limitation arises from the fact that texts authored by individuals are often topic- agnostic, lacking explicit references to their person- ality traits. Motivated by self assessmment personal- ity tests (Digman, 1990; Myers-Brigee Briggs, 1991), we introduce PsyCoT prompting, which utilizes items from questionnaires as chain-of-thought (CoT) framework to enhance reasoning capabilities.
Y= (=(y") [t=
3.1 Standard Prompting
We first describe the standard prompting, which infers the results directly from the given input text X under prompted by D and I:
The example of PsyCoT utilizing the 44-item Big Five Inventory (John al., 2008) depicted at the bottom of Figure 2. In comparison to stan- dard prompting, PsyCoT mimics the process of
LLM II
(1)
"haps://chat.opemaa.comm

---

Standard Prompting
<Task description prompt D> You are an Al assistant who specializes in text analysis. You will complete a text analysis task. The task follows: according text written by an author, predicting whether the author is A: "High Agreeableness" or B; "Low Agreeableness".
<Text content X AURHOR'S TEXT: [Text]
<Inference prompt /> The author is: A: "High Agreeableness" or B: "Low Agreeableness". Provide choice
the format: "CHOICE: <A/B>" and do not give the explanation.
PsyCoI Prompting
<Task description prompt D You are an Al assistant who specializes in text analysis and am Human. We will complete text analysis task together through multi-turn dialogue. The task is follows: we have text written by an author, and at each turn, will give you a statement about the author. According to the author's text, you need 3=neutral, 4-agree a little, and 5=agree strongly. After rating all the statements (SO-S9), I will ask you if the author is more likely to be A: "High Agreeableness" or "Low Agreeableness", and then you need give your choice. Note that S1, S3, S4, S6, S8 are positive statements, with higher scores indicating higher agreeableness, while SO, S2, S5, are reverse-scored statements, with higher scores indicating lower agreeableness.
the statement with score 1-5, where 1-disagree strongly, 2-disagree little,
<Text content X> AUTHOR'S TEXT: [Text]
Human: S0: The author original, comes up with new ideas. Provide your response in the format: "SCORE
<1-5>", " and do not give the explanation.
Assistant: SCORE:
(after rating all the statements)
<Inference prompt >> According to above scores, the author is: A: "High Agreeableness" or B: "Low Agreeableness", Provide choice in the format: 'CHOICE: <A/B>" and do not give the explanation. Assistant:
Human:
Figure 2: Comparison of Standard Prompting (Top) and our PsyCoT Prompting (Bottom). PsyCoT, the dotted box indicates reasoning step derived from a psychological questionnaire. Unlike Standard Prompting, which directly prompts LLM to output the personality preference, PsyCoT incorporates multiple reasoning steps through interactions with the LLM, guiding the LLM infer personality in more reasonable manner.
self-assessment personality tests and enhances stan- dard prompting in the following key aspects: (1) We modify the task description D by instructing the LLM to rate cach statement (item the ques- tionnaire). (2) We include in D a description of the rating rules for the questionnaire, encompass- ing the scoring system (e.g., "I=disagree strongly, ==disagree little, 3=neutral, 4=agree little, and 5=agree strongly") and highlighting the signifi- cance of reversed statements. (3) Most importantly, we introduce K reasoning steps R- (rk) prior to accessing the personality trait 'j² via multi- dialogue, guiding the LLM in inferring the person ality with more reasonable manner. The overall progress of PsyCoT is described in Algorithm 1, including step by step reasoning in Line 2-4. We provide complete dialogue records in Appendix B. A by-produce of PsyCoT Prompting is the rating results (11 42 ... ax), which can be aggregated into an overall score like the self-assessmmnt per- sonality tests. For example, the overall score in 44-item Big Five Inventory is computed as:
8k
(2)
K
1=
where sy. is the transition score, which defined as:
8k -
ax
TE E Rs otherwisc
(3)
Ok
where R. is the set of reversed statements. Since positively correlated with ys, the overall score s' can serve double check and provide confidence for the inferred 0²
Discussion PsyCoT employs multi-turn ap- proach to rating questionnaire items, opposed to providing simultaneous ratings for all items within single turn. There are two main reasons. Firstly, by rating each item individually during each turn, can maintain higher level of focus for the LLM, resulting more accurate outcomes. Sec- ondly, there are inherent correlations between the items. For instance, statements such as "The author

---

is reserved." and "The author tends be quite." exhibit a high level of correlation. Consequently, incorporating the historical rating results when eval- uating the current item within a multi-turn structure will enhance the consistency of the obtained results. We provide more investigations in Section 4.5.
using LIWC (Pennebaker et al., 2001), and then applies SVM as the classifier. TF-IDF+SVM (Cui and Qi, 2017): This method is similar to LIWC+SVM, but it extracts features using TF-IDF.
W2V+CNN (Rahman et al., 2019) and Glove+LSTM (Sun et al., 2018): These two methods are non-pretrained models. The former encodes the context using the word2vec algorithm and then applies CNN (Chen, 2015) to obtain the context representation. The latter uses Glove (Pennington et al.. 2014) for word embeddings and then utilizes LSTM (Hochreiter and Schmidhuber, 1997) to encode the context.
4 Experiment
4.1 Datasets
We evaluate PsyCoT on two public datasets: Es- says (Pennebaker and King, 1999) and Kaggle'. The Essays comprises 2468 anonymous texts writ- ten by volunteers in strict environment. These volunteers were instructed to write whatever came to their mind within a short time-frame. Each essay is associated with the author's Big Five personality traits, namely Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness. The au- thors assessed their own traits using a 44-item Big Five Inventory (John et al., 1991), which is the same questionnaire we employ in PsyCoT. To en- sure fair comparisons with fine-tuned methods, we randomly divided the dataset into training, vali- dation, and testing with proportions of 80-10-10, respectively. For prompt-based methods, we evalu- ate their zero-shot performance on the test set.
Regression (Park et al., 2015): This method trains regression model with two regression scores 0 and 1. The same quantile discretization method from Ganesan et al. (2023) is then used to convert the test set scores into categorical labels.
BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019): These two models are fine-tuned and utilize "bert base--bas-c and troberta-base as backbones, respectively. For Essays, they en- code the context directly, while for Kaggle, they encode each post and then aggregate post represen- tations via mean pooling.
SN+Attn (Lynn et al., 2020): This method adopts a hicrarchical attention network to generate the user representation. Following Yang al. (2021b), the pre-trained BERT is utilized as post-encoder ensure fair comparisons.
The Kaggle dataset is collected from Personality- Cafe², a forum where people share their personality traits and engage in diseussions about personality- related topics. The Kaggle comprises total of 8675 user's data, each consisting of 45-50 posts. The used personality taxonomy is MBTI (Myers- Briggs, 1991), which divides personality into four Introversion or Extroversion (I/E), Sensing or iNtuition (S/N), Thinking or Feeling (T/F), and Perception or Judging (P/J). Limited by API resources, we randomly selected 200 samples form the test set evaluate PsyCoT and baselines.
TrigNet (Yang et al., 2021b): TrigNet constructs a tripartite graph with psycholinguistic knowledge in LIWC to fuse posts.
dimensions:
DDGCN (Yang al., 2022): DDGCN is the latest SOTA method in the Kaggle dataset, which firstly encodes each post using domain-adapter BERT, and then aggregates the posts in a disorderly man- ner by a dynamic deep graph network.
For the Essay dataset, the average token count per sample is 762.38, whereas for Kaggle, stands at 1632.38. More details of the two datasets and the used questionnaires are provided in Appendix and Appendix D, respectively.
For prompt-based methods, in addition to stan- dard prompting, we adopt Zero-shot-ooT (Kojima et al., 2022), which inserts reasoning step with "Let's 's think step by step." before accessing the final personality via the inference prompt.
4.2 Baselines
4.3 Implementation Details
In our experiments, we adopt the following previ- ous methods as baselines.
In this study, we simplify multi-label personality detection into multiple binary classification tasks. For the prompt- methods, we request the GPT- 3.5 API (gpt-3. 5-turbo- 0001)) to obtain results, 'More details are provided in Appendix A
LIWC+SVM (Tighe et al., 2016): This shallow method firstly extracts psycholinguistic features
wtt.://ggwg..oommaaaaaaameeeeeea hhtp::/peesonnalitaaafcc..oom/ com./o/

---

AGR
CON
EXT
NEU
OPN
Average
Methods
Acc.
F1
Acc.
FI
Acc.
F1
Acc.
F1
Acc.
FI
Acc.
F1
LIWC+SVM
51.78
47.50
51.99
52,00
51.22
49.20
51.09
50.90
54.05
52.40
52.03
50.40
Regression
50.96
51.01
54.65
54.66
55.06
55.06
57.08
57.09
59.51
59.51
55.45
55.47
W2V+CNNí
46.16
52,11
39.40
58.14
59.80
51.12
BERT 56.84
54.72
57.57
56.41
58.54
58.42
56.60
56.36
60.00
59.76
57.91
57.13
RoBERTa
59.03
57.62
57.81
56.72
57.98
57.20
56.93
56.80
60.16
59.88
58.38
57.64
Standard
59.11
57.98
57.49
49.55
59.92
54.39
61.13
59.95
55.87
49.11
58,70
54.20
Zemoshot-CoT
58.94
58.09
55.14
42.49
57.55
55.63
57.49
54.63
58.78
54,40
57.58
53.05
PsyCoT 61.13
61.13
59.92
57.41
59.76
59.74
56.68
56.58
60.73
57.30
59.64
58.43
Table 1: Overall results of PsyCoT and baselines on the Essays dataset. We use Accuracy(%) (%) and Macro-FI(9o) as metrics. The symbol means results directly taken from the original papers. Best results are listed in bold and the second best results are shown with underline.
I/E
S/N
T/F
J/P
Average
Methods
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
TF-IDF+SVM
71.00
44.94
79.50
46.38
75.00
74.25
61.50
58.59
71.75
56.04
Regression
61.34
64.00
47.10
54.50
76.34
76.50
65.58
66.00
62.59
65.25
Glove+LSTM
72.50
62.04
80.50
52.78
74.00
73.23
62.50
60.31
72.38
62.09
BERT
77.30
62.50
84,90
54.04
78.30
77.93
69.50
68.80
77.50
65.82
SN+Attn
65.43
62.15
78.05
63.92
67.39
RoBERTa
77.10
61.89
86.50
57.59
79.60
78.69
70.60
70.07
78.45
67.06
TrigNet
77.80
66.64
85.00
56.45
78.70
78.32
73.30
71.74
78.70
68.29
DDGCN
78.10
70.26
84.40
60.66
79.30
78.91
73.30
71.73
78.78
70.39
Standard
52.00
51.52
47.00
43.76
68.00
67.68
55.50
55.41
55.63
54.59
Zero-shot-CoT
76.50
64.27
83.50
55.16
72.50
71.99
57.50
53.14
72.50
61.14
PsyCoT
79.00
66.56
85.00
61.70
75.00
74.80
57.00
57.83
74.00
65.22
Table 2: Overall results of PsyCoT and baselines on the Kaggle dataset.
which is currently the most popular and forms the foundation of ChatGPT. We set the temperature as 0 to ensure deterministic outputs. In the case of Essays, we limited the maximum tokens for the au- thor's text to 3200, while for Kaggle, we restricted it to 80 tokens per post. For the fine-tuning based methods, we set the learning rate to 2e-5, and re- port their test performance (using same data as prompt-based methods) by averaging the results of five runs. The evaluation metries employed in our study include Accuracy and Macro-IFl score.
sults. First, PsyCoT outperforms the baselines on most personality traits, surpassing the fine-tuned models. Specifically, PsyCoT enhances standard prompting with an average increase of 0.94/4.23 points in accuracy and macro-F1. Second, PsyCoT performs worse than standard prompting on the Neuroticism trait. Further analysis reveal that this discrepancy may be attributed to dataset bias. The Essays contains significant amount neg- ative emotions expressed by authors, which mis- leads the LLM into assigning high scores for state- ments such as "The author worries lot.". "The author is depressed, blue.", and "The author can be moody.". Third, although includes a reasoning step with "Let's 's think step by step.", Zero-shot- CoT does not consistently improve the performance of standard prompting. Our investigation shows that Zero-shot-CoT often fails to guide the rea- soning process correctly, resulting in meaningless responses like "Sure, what would you like start with?". . In contrast, PsyCoT directly constructs rea- soning steps with the help of questionnaires.
4.4 Overall Results
The overall results of PsyCoT and several base- lines on Essays are listed Table 1. Data-driven baselines comprise three types of methods: shal- low model (LIWC+SVM and Regression), non- pretrained model (W2V+CNN), and fine-tuned models (BERT and RoBERTa). The prompt-based baselines include Standard prompting and Zero- shot-CoT.
There several key observations from these re-

---

Agrocableness
Conscientiousness
Extraversion
Neuroticism
Openness
Trait Score
3-
2-
S-0.84
S=0.63
S-0.85
S=0.60
S-0.74
Low
High
Low
High
Low
High
Low
High
Low
High
Figure 3: Distributions of the trait scores across five personalities in the Essays dataset. The dashed line represents the neutral value defined by the questionnaire (i.e., 3=Neutral). The value of "S" displayed in figures indicate the Spearman's s Coefficient between trait scores and the personality dimensions. A value closer to suggests stronger positive correlation. Results demonstrate a strong positive correlation between trait scores and personality type. particularly in Agreeableness, Extraversion, and Openness.
Table presents additional results on the Kag- gle dataset. The obeservations are threefold. First, PsyCoT achieves the highest performance among prompt-based methods with 18.37/10.63 points im- provement in accuracy and macro-FI over standard prompting, and 1.50/4.08 points improvement over Zero-shot-CoTi Second, compared to fine-tuned models, PsyCoT achieves a performance very close to that BERT, while still falls behind the cur- rent SOTA method DDGCN by a non-negligible margin. To explore the underlying reasons, we an- alyze the textual input and find that users in the Kaggle dataset directly discuss the MBTI and fre- quently employ abbreviations such "Fii" "Fe," "Ni," "Te." and "IxxP" to describe personality traits. These abbreviations hold potential as strong fea- tures for data-driven methods, but for zero-shot methods, they may be insufficient for capturing the association between these abbreviations and per- sonalities. Third, despite being weaker than the fine-tuned methods on the Kaggle, prompt-based methods have the advantages of not requiring the collection of training data and retraining of mod- els to accommodate changes in personality types and application scenarios (e.g., Essays and Kaggle should be quite different in terms of the text style: freety-written text vs. discussion posts.)
Methods
Accuracy
A
Macro-F1
PsyCoT
59.6c 94.64
58.43
rhv single-tur
59.27
0.37
56.00
2,43
rAv TIPI
58.46
1.18.).
54.73
3.70
rbe Mini-IPIP
59.19
0.45.
57.42
1.01
Table 3: Abalation results of PsyCoT in average Accu- racy and Macro-F1 on the Essays dataset, where "A" indicates the corresponding performance change, and rfw means "replace with".
approach, which involves presenting all the items within a single-turn dialogue and instructing the LLM to assign ratings to all of them simultane- ously. From Table 3, we can observe that reasoming with a single-turn dialogue deteriorates the perfor- mance, demonstrating that PsyCoT with multi-turn for reasoning is preferable.
Other Questionnaire We utilizes the 44-item Big Five Inventory (John et al., 1991) for operating PsyCoT in Essays. However, alternative question- naires also can be applied in PsyCoTT Therefore, we have included other widely used brief measures, such as TIPI (10-item) (Gosling et al., 2003) and Mini-IPIP (20-item) (Donnellan et al., 2006), examine the impact of different questionnaires. Ap- pendix presents a comparison between these three inventories. It is evident that the 44-item inventory offers a broader range of aspects for eval- uating an individual's 's personality, making it po- tentially more effective CoT for guiding reason- ing. The results in Table 3 further confirm that the TIPI is insufficient in eliciting the desired level of reasoning, as it leads a significant decline in performance. While the 20-item scale is better than 10-item scale, but still inferior to the 44-item scale. This experiment underscores the crucial
4.5 Ablation Study
Table 3 presents several ablation results of PsyCoT to facilitate more comprehensive investigations.
Single-turn PsyCoT adopts a multi-turn dialogue structure to facilitate reasoning. This structure al- lows the LLM concentrate on a particular item, enhancing the accuracy of its reasoning process. To verify its effectiveness, we test another alternative

---

selecting appropriate questionnaires for PsyCoT.
Stundard
Zero-shot CoT
PsyCoT
1.0
5
Analysis
0.58
9.86
0.81
0,25982
5.1 Correlation Analysis
Unchanged Rate
0.8
0.73
0.71
0.73
In this study, we explicitly introduce the psycholog- ical questionnaire as COT, which guides the LLM in assessing the author's personality by rating each item. Consequently, as defined by Eq.(2), these ratings can be computed into an overall score (we refer to trait score) that reflects the strength a spe- cific personality trait. To investigate the correlation between the trait score and the chosen personality trait, we firstly visualize the distributions of trait scores across five different dimensions in the Es- says dataset, shown in Figure 3.
0.6
0.52
0,52
0.4
0.30
0.2
0.0
AGR
CON
EXT
NEU
OPN
Figure 4: The robustness testing results for three prompt- based methods on the five traits, PsyCoT demonstrates the highest unchanged rate, indicating its robustness handling option order perturbations.
The observations yield main findings. First, in general, there a significant difference in the dis- tribution between low and high traits, and the trait scores exhibit strong positive correlation with the personality types, particularly in Agreeableness, Extraversion, and Openness. Second, the distribu- tion of scores for Extraversion and Openness can be distinguished by the dashed line, illustrating that the LLM''s ratings for these traits align with the cri- teria of the questionnaire. On the other three traits, however, the distributions intersect the dashed line and shift to one side, suggesting potential bias in the LLM's ratings of the questionnaires. We further apply Spearman's 's Coefficient (Sedgwick, 2014) to quantitatively measure the correlations. From Figure 3, the trait scores for Agreeableness, Ex- traversion, and Openness have higher correlation (> 0.70) with the chosen personality type, which validates our obscrvations.
5.3 Robustness Testing
notorious drawback of LLMs its vulnerability to minor variations in prompts, such as changes in the order of options within choice questions. Intuitively, PsyCoT incorporates questionnaire to facilitate reasoning, potentially improving the ro- bustness of LLMs. To verify this hypothesis, we conduce experiment where swap the option orders in both the task description prompt D and inference prompt (e.g., A: "High Agreeableness" or B: "Low Agreeableness" is exchanged to A: "Low Agreeableness' or B: "High Agreeableness "). and re-test the prompt-based methods. We measure the unchanged rate of j' across 100 samples. The results from the Essays dataset are presented in Fig- ure 4. PsyCoT achieves the highest unchanged rate and significantly superiors other methods, demon- strating that the inclusion of the questionnaire en- hances its robustness.
5.2 Statistical Tests
To test the statistical significance of our method, we conduct a comparison between the Standard prompt and PsyCot using the Essays datasct, and the Fl results are presented in the Table 4. Each point represents the average of 5 runs. The T-test analysis indicates that our improvements are statis- tically significant, with p-values of less than 0.05 for AGR, less than 0.001 for CON, EXT, and OPN.
5.4 Impact of Post Order
The Kaggle dataset contains a of posts for each user. These posts are concatenated in sequence create long document, which serves as the in- put X. However, Yang et al. (2021a, 2022) have demonstrated that encoding posts sequentially is srder-sensitive for the fine-tuned models. To inves- tigate whether the prompt-based methods are also influenced by post order, randomly disrupt the post orders and re-test the prompt-based methods using 100 samples. Similar to Section 5.3, the un- changed rate is used for evaluation. As shown Figure 5, shuffling post orders leads to noticeable changes in the predictions across several personal-
Methods AGR
CON
EXT
NEU
OPN
Standard 58.17
49.52
53.96
59.46
59.46
PsyCoT 60.34o
57.44+
59.23+
59.46
57.88+
Table Significant tests on the Essays dataset. o and * mean p < 0.5 and 0.001, respectively.

---

Standard
Zoco sbot Col
PoyCol
Limitations
0.97
0.52
The potential limitations of PsyCoT are discussed below. Firstly, PsyCoT utilizes multi-tun dia- logue approach to rate items from the questionnaire. Although this approach surpasses the single-tum method (as reported in Section 4.5), it does result in more API requests. Secondly, the performance of PsyCoT is heavily influenced by the selected ques- tionnaire, necessitating additional effort finding the most suitable one. Thirdly, PsyCoT has only been applied to one language, and there is a need to explore its application in a broader range of lan- guages.
0.5
0.24
Unchanged Rate
072
1.69
0.65
0.0
1/
S/N
T/F
J/P
Figure 5: Results of the study on post orders. The three prompt-based methods are influenced by post orders for most of personality traits.
Ethics Statement
ity traits, including PsyCoT. This observation high- lights that the order of posts remains an important issue for the LLM.
Text-based personality detection is a long history research area within psycholinguistics. We clarify that the purpose of this study is to explore and im- prove the ability of the LLM in this particular task, rather than creating tool that invades privacy. The Essays and Kaggle datasets used in our study are public available and all user information has been anonymized. We have strictly adhered to the data usage policy throughout our research. However, it is crucial to acknowledge that the misuse of this technology may pose ethical risks. We state that any research application stemming from this study solely permitted for research purposes, and any attempt exploit this technology to covertly infer individuals' personality traits or other sensi- tive characteristics is strictly prohibited.
6 Conclusion
In this paper, we propose a novel method, Psy- COT, for zero-shot personality detection. PsyCoT prompts the LLM to engage in reasonable per- sonality reasoning. Unlike the standard prompting approach that directly requests the LLM to infer the answer, PsyCoT draws inspiration from self- assessment personality tests that evaluate an indi- vidual's personality through well-designed ques- tionnaires. PsyCoT introduces items from the ques- tionnaire as a set of rigorous COT. It guides the LLM to evaluate each item based on the author's text and then infer the final personality trait. Ex- tensive experiments on two benchmark datasets demonstrate that PsyCoT outperforms Standard prompting and Zero-shot-CoT approaches by large margin. It also achieves comparable perfor- mance to some fine-tuned models. This work rep- resents the first effort leverage psychological questionnaires elicit the LLM's s reasoning abili- ties, providing a new perspective for exploring the use of questionnaires in the era of LLMs.
References
Mohammad Hossein Amirhosseini and Hassan Kazemian. 2020. Machine learning approach to per- sonality type prediction based on the myers myers-briggs type indicator@. Multimodal Technologies and Interaction, 4(1):9.
Sean Andrist, Bilge Mutlu, and Adriana Tapus. 2015. Look like me: matching robot personality via gaze increase motivation. In Proceedings of the 33rd an- nual ACM conference on human factors in computing systems, pages 3603-3612.
Acknowledgements
R Michael Bagby, Tara M Gralnick, Nadia Al-Dajani, and Amanda A Uliaszek. 2016. The role of the five- factor model in personality assessment and treatment planning. Clinical Psychology: Science and Practice, 23(4):365.
This work was supported by the National Natu- ral Science Foundation of China (No. 62176270),
the Guangdong Basic and Applied Basic Research Foundation (No. 2023A15 15502223.,,,, and 2023 Tencent Rhino-Bine Research Elite Program.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda

---

Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901.
Samuel D Gosling, Peter Rentfrow, and William B Swann Jr. 2003. A very brief measure of the big- five personality domains. Journal of Research in personality, 3746):504-528.
Yahui Chen. 2015. Convolutional neural network for sentence classification. Master's thesis, University of Waterloo.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models ma- chine translation? comprehensive evaluation. arXiv preprint arXiv: 2302000000..
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Louis Hickman, Nigel Bosch, Vincent Ng, Rachel Saef, Louis Tay, and Sang Eun Woo. 2022. Automated video interview personality assessments: Reliability, validity, and generalizability investigations. Journal of Applied Psychology, 107(8):1323.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. 2021. Training verifiers solve math word problems. arXiv preprint arXiv:21 14168.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 1735 1780.
Philip Corr and Gerald Ed Matthews. 2009. The Cam- bridge handbook of personality psychology. Cam- bridge University Press.
Oliver P JJohn, Eileen M Donahue, and Robert I. Kentle, 1991. Big five inventory. Journal of Personality and Social Psychology.
Brandon Cui and Calvin Qi. 2017. Survey analysis of machine learning methods for natural language processing for mbli personality type prediction. Final Report Stanford University.
Oliver P John, Laura P Naumann, and Christopher J Soto. 2008. Paradigm shift to the integrative big five trait taxonomy: History, measurement, and concep- tual isses.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
Amir A Khan, Kristen C Jacobson, Charles O Gard- ner, Carol A Prescott, and Kenneth S Kendler. 2005, Personality and comorbidity of common psychi- atric disorders. The British Journal of Psychiatry, 186(3): 190-196.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICML 2022 Workshop Knowledge Retrieval and Language Models.
John M Digman. 1990. Personality structure: Emer- gence of the five-factor model. Annual review of psychology, 4144):417444..
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.1 16692.
M Brent Donnellan, Frederick L Oswald, Brendan M Baird, and Richard E Lucas. 2006. The mini-ipip scales: siny-yet-effective measures the big five factors of personality. Psychological assessment, 18(2):192.
Veronica Lynn, Niranjan Balasubramanian, and H An- drew Schwartz. 2020. Hicrarchical modeling for user personality prediction: The role of message-level at- tention. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5306 5316.
Adithya V Ganesan, Yash Kumar Lal, August Håkan Nilsson, and H Andrew Schwartz. 2023. Systematic evaluation of gpt-3 for zero-shot personality estima- tion. arXiv preprint arXiv:2306.0 183.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making tre-trained language models better few-shot learners. In Pruceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 381 3816-3830.
Sandra C Matz, Michal Kosinski, Gideon Nave, and David J Stillwell. 2017. Psychological targeting effective approach to digital mass persuasion. Proceedings of the national academy of sciences. 1 117444)::214--2277
Yash Mehta, Navonil Majumder, Alexander Gelbukh, and Erik Cambria. 2019. Recent trends in deep leam- ing based personality detection. Artificial Intelli- gence Review, pages 1-27.
Lewis R Goldberg, 1990. An alternative" description of personality": the big- factor structure. Journal of personality and social psychology, 1216)

---

Isabel Mlyers-Briggr 1991. Introduction to type: A de- scription of the theory and applications of the myers- briggs indicator. Consulting Psychologists: Palo Alto.
Xiangguo Sun, Bo Liu, Jiuxin Cao, Junzhou Luo, and Xiaojun Shen. 2018. Who am i? personality detec- tion based on deep learning for texts. In 2018 IEFE international conference on communications (ICC), pages 1-6. IEEE.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Edward P Tighe, Jennifer C Ureta, Bernard Andrei L Pollo, Charibeth K Cheng, and Remedios de Dios Bu- los. 2016. Personality trait classification of essays with the application of feature reduction. In SAAIP@ @ IJCAI, pages 22-28.
Gregory Park, Andrew Schwartz, Johannes C Fich- staedt, Margaret L Kern, Michal Kosinski, David J Stillwell, Lyle H Ungar, and Martin EP Seligman. 2015. Automatic personality assessment through social media language. Journal of personality and social psychology, 108(6):934.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nip models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080 2094.
Jindong Wang, HU Xixu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, et al. 2023. On the robustness of chatgpt: An adversarial and out-of- distribution perspective. In ICLR 2023 Workshop Trustworthy and Reliable Large-Scale Machine Learning Models.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71(0001):2011.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. International Con- ference on Learning Representations.
James W Pennebaker and Laura A King. 1999. Lin- guistic styles: language use as individual differ- ence. Journal of personality and social psychology, 1296).
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 1532-1543.
Feifan Yang. Xiaojun Quan, Yunyi Yang, and Jianxing Yu. 2021a. Multi-document transformer for personal- ity detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 14221- 14229.
Md Abdur Rahman, Asif AI Faisal, Tayeba Khanam, Mahfida Amjad, and Md Saeed Siddik. 2019. Person- ality detection from text using convolutional neural network. In 2019 1st international conference on advances in science, engineering and roboties tech- nology (ICASERT), pages 1-6. IEEE.
Tao Yang, Jinghao Deng, Xiaojun Quan, and Qifan Wang. 2022. Orders are unwanted: Dynamic deep graph convolutional network for personality detec- tion. In Proceedings of AAAI 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach. Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR International Conference on Learning Representa- tions.
Tao Yang, Feifan Yang. Haolan Ouyang, and Xiaojun Quan. 202 Psycholinguistic tripartite graph net- work for personality detection. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- une 1: Long Papers), pages 4229-4239.
Philip Sedgwick. 2014. Spearman's 's rank correlation coefficient. Bmj, 349.
Zhihong Shao, Yeyun Gong, Yelong Shen, Min- lie Huang, Nan Duan, and Weizhu Chen, 2023. Synthetic prompting: Generating thain-of-thought demonstrations for large language models. arXiy preprint arXiv:2302.0061
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066

---

Zhuosheng Zhang. Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompt- ing in large language models. arXiv preprint
Dataset
Types
Train
Validation
Test
AGR
1051 /922
132 115
127 120
CON
1019 954
121 / 126
113 134
arXiv:2210.03493.
Essays
EXT
1019 954
122 125
135 112
NEU
985 988
/
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Lc. and Ed Chi. 2022a. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint
OPN
1011 / 962
127 120
133 114
I/E
4011 / 1194
1326 409
152 / /8
S/N
610 4478
222 1513
27 173
Kaggle
2410 2795
791
115
arXiv:2205.1005
P/J
3096 / 2109
1063 / 672
122 78
Yongchao Zhou, Andrei Toan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022b. Large language models are human-level prompt engineers. In NeurlPS 2022 Foundation Mod- els for Decision Making Workshop.
Table 6: Statistics of the Essays and Kaggle datasets.
D Appendix: Details of Questionnaires
In this study, the 44-item Big Five Inventory and Myers-Briggs Type Indicator are used for our meth- ods. The details of items are listed in Table 7 and Table 10. Note that the original items in Myers- Briggs Type Indicator do not include the option "Not sure whether or B". We include this option to ensure that the LLM has the ability to indicate un- certainty in cases where the provided information is insufficient for making definitive choice. Be- sides, in Section 4.5, we utilize TIPI and Mini-PIPT for comparison. The details of these two scales are listed in Table 8 and Table 9, respectively.
Yangfu Zhu, Linmei Hu, Xinkai Ge, Wanrong Peng, and Bin Wu. 2022. Contrastive graph transformer network for personality detection. In Proc. IJCAI Conf.
A Appendix: Form of the Task
Although the original tasks of the Big 5 are in the regression form, the Essays dataset is further col- lected and processed into the binary label format (Y/N), without releasing the original scores from the raters. Consequently, we have framed this task as classification problem, designating 'High' as "Y" and 'Low' as 'N.' Besides, our initial study indicates that simplifying multi-label personality detection into multiple binary classification tasks leads to performance improvements (the average results on standard prompt are reported in Table 5). Hence, this approach was chosen for our exper- iments.
Forms
Accuracy
Macro-F1
Multi-label
51.74
46.85
Five binary (ours)
58.70
54.20
Table 5: Comparison results of different forms.
B Appendix: Dialogue Record
To better understand our method, provide two complete records in Figure 6 and Figure 7, respec- tively. These records demonstrate how PsyCoT predicts Big Five and MBTI personality traits.
Appendix: Details of Datasets
We provide the statistics of Essays and Kaggle in Table 6.

---

# Record of Big Five
assistant specializes in analysis and Human. will complete text analysis task together through multi-turn dialogue. The task follows: have text written by author, and at each turn I give you a statement about the author. According to the author\'s text, you need to rate the statement with a score 1-5, where I=disagree strongly, 2=disagree little 3=neutral, 4=agree little,
You
5=agree strongly. After rating all statements (58-S8), will ask you the author A: "High Agreeableness" or B: "Low Agreeableness", and then you need give your choice. Note that S1, $3, S4, S6, S8 are positive statements, with higher scores indicate higher agreeableness, while 58, S2, $5, S7 are reverse-scored statements, with higher scores indicate lower agreeableness. AUTHOR'S TEXT:
I'm pretty happy with my first week and half classes. I've met to me. am a very quiet, sort of shy person. was afraid of coming to UT because I'm not the best at making friends. So far though, it hasn't been problem. My big classes have been little intimidating because of there size. I used to have classes about 30 or less. Right now I am happy, uT won its first football game. I'm glad I was able to go. It was lots of fun. Now I feel a little exhausted. I haven't t really done all that much today. I actually got to sleep for 12 hours last night could be exhausted because I am hungry. I think I'm pretty lucky. My parents sent me some food. My brother brought it to Austin. He and I will share it. I'm already starting to miss home-cooked meals. think my 20 minutes are just about up. I probably fix bite to eat, watch little T. V. and go to sleep.
of people. means a lot
Human: S0: "The author sometimes notices other people\'s faults." Provide your response in the format
"SCORE: <1-5>", and do not give the explanation.
Assistant: SCORE: 3 (neutral)
Human: $1: "The author selflessly helps others." Provide your response only with the score.
Assistant: SCORE: (neutral)
Human: $2: "The author sometimes argues with others." Provide your response only with the score.
Assistant: SCORE: (disagree strongly)
Human: S3: "The author forgives others easily." " Provide your response only with the score.
Assistant: SCORE: (agree little)
Human: $4: "The author trusts people easily." Provide your response only with the score.
Assistant: SCORE: (agree little)
Human: 55: "The author be distant with others sometimes." Provide your response only with the
score.
Assistant: SCORE: (agree little)
Human: "The author is kind everyone." Provide your response only with the score.
Assistant: SCORE: (disagree little)
Human: "The author be rude sometimes." " Provide your response only with the score.
Assistant: SCORE: (disagree little)
Human: "The author likes to collaborate with others." " Provide your response only with the score.
Assistant: SCORE: (agree strongly)
Human: According to above scores, the author more likely to be: "High Agreeableness" or B; "Low Agreeableness". Provide a choice in format: "CHOICE: <A/B>" and not give explanation.
Assistant: CHOICE:
Figure 6: A complete record that PsyCoT is applied to predict Big Five personality traits.

---

# Record of MBTI
You are AI assistant who specializes in text analysis and I an Human. We will complete text analysis task together through a multi-turn dialogue. The task follows: have set of posts written by author, nd at each turn will give you Question about the author. According to the author\'s text, you need choose the possible options, After opting all the choices, I will ask you if the author is A: "Extraversion"
"Introversion", and then you need to give your choice.
AUTHOR'S POSTS:
Post1: dreamed a lot of weird stuff but generally my dream always sent back to school for some reason to meet someone haven't met in long while. Previously dreamed of flying
machine...." Post2: Maybe some headless dolls?; Post3: People like thank yous much the thumbs up icon keep sending them.; Post4: Try this. 717794; Post5: No more spam? Nuuununnunn; Posté: For some people life is buffet. For others, their world
set meal. LOL.: Post7: I currently desire pancakes Post8: Fi dom signature.; Post9; Neither heartlessness and indecision for following your guts will get you anywhere.; ; Post10: Hahahahahaha!!!! !!; Post11: What makes you so certain that only and not s can be writers? Both are good with written words. Just a genuine question. thappy:; Post12: running the treadmill not for you, you could either try the rowing machine or there is a public pool available, try swimming. They are relatively low-impact exercises but would still give.... Post14: was so sad I went for a long walk all the way to the gym Good thing though I didn't have to sign up for membership or anything. It was my first time there. I was surrounded by old dude and...; Post15: Being an artist is not just about expressing yourself. It's about trying to understand and connecting life in the way that would make you learn more about yourself. Before one is an artist, one is a...; Post16: I only think of them as sentimental. Not really conventional.; Post17: I'm hungry just looking at them..: Post18: You are a definite NF. Other than that, I May need to get to know you a little more..: Post19: It's almost a.m. here and I've read it as NT's domino club.; Post20: Most of my clothes function in such a way that at any moment, I can go for a jog, jump into a swimming pool and then proceed to bunjee jump off the building. The fabric has to be stretchy and...: (Due to page limitations, we omit the rest of posts.)
Seems likely.; Post13:
Human: Q: "The author is usually: A: "A good mixer with gropus of people", B: "Quiet and reserved", or C: "Not sure whether A or B"_" Provide a choice ID in the format: "CHOICE: <A/B/C>", and do not give the explanation.
Assistant: CHOICE: C
Human: Q: "Among the author's friends, the author is: A: "Full of news about everybody", B: "One of the last to hear what is going on", or C: "Not sure whether A or B"."
Assistant: CHOICE: B
Human: Q: "The author tends to have: A: "A broad range of friendships with many different people", B: "Deep friendship with very few people", or C: "Not sure whether A or B"."
Assistant: CHOICE: C
Human: Q: "When the author is with group of people, the author is usually: A: "Join in the talk of the group", B: "Stand back and listen first", or C: "Not sure whether A 8"."
Assistant: CHOICE: B
Human: Q: "The author is: A: "Talk easily to almost anyone", B: "Find a lot to say only to certain people or under certain conditions", or C: "Not sure whether A or B"."
Assistant: CHOICE:
Human: Q: "In a large group, the author is more often: A: "Introduce others", B: "Get introduced",
C: "Not sure whether or B"."
Assistant: CHOICE:
Human: Q: "When the author meets the new people, the author tells what they interested in: A:
Right away", B: "Only after people to get to know the author", or "Not sure whether or B"."
Assistant: CHOICE:
Human: Q: "The author usually: "Show their feelings freely", "Keep their feelings themselves", or C: "Not sure whether A or B"."
Assistant: CHOICE:
Human: Q: "Which word more suitable for the author: A: "QUIET", B: "HEARTY", , or C: "Not sure
whether or B"."
Assistant: CHOICE: A
Human: Q: "Which word is more suitable for the author: A: "RESERVED", B: "TALAKATIVE", or C: "Not
sure whether or g"."
Assistant: CHOICE; A
Human: Q: "Which word is more suiteble for the author: A: "CALM", B: "LIVELY", or C: "Not sure whether A or B"."
Assistant: CHOICE; A
Human: According to above choices, the author is more likely to be: A: "Extraversion" or B: Introversion". Provide a choice in the format: "CHOICE: <A/B>" and do not give the explanation.
Assistant: CHOICE: B
Figure 7: A complete record that PsyCoT is applied to predict MBTI personality traits.

---

Agroeableness
Conscientiousness
#1: The author sometimes notices other people's faults.
#1: The author does things carefully and completely.
#2: author seiflessly helps others.
#2: The author can kind of careless.
#3: The author sometimes argues with others.
#3: The author good, hard worker.
#4: The author forgives others easily.
#4: The author isn't I very organized.
#5: author trusts people easily.
#5: The author tends to be lazy.
#6: The author be distant with others sometimes.
#6: The author keeps working until things done.
#7: The author is kind to everyone.
#7: The author does things quickly and carefully.
#8: The author can rude sometimes.
The author makes plans sticks to them.
#9: The author likes collaborate with others.
Extraversion
Neuroticism
#1: The author is talkative.
#1: The author is depressed, blue.
#2: author is reserved.
#2: The author relaxed, handles stress well.
#3: The author full of energy.
#3: The author be
#4: The author generates lot of enthusiasm.
#4: The author worries a lot.
#5: author tends to be quiet.
#5: The author steady, not easily upset.
#6: The author has assertive personality.
The author
moody.
#7: The author is sometimes shy, inhibited.
#7: The author stays calm in difficult situations.
#8: The author is outgoing, sociable.
#8: The author gets nervous easily.
Openness
#1: The author is original, comes up with ideas.
#2: The author is inquisitive about many things.
#3: The author is ingenious, a deep thinker.
#4: The author has strong imagination.
#5: The author is good at innovation.
#6: The author values artistic.
#7: The author works routine.
#8: The author often reflects himself.
#9: The author has ordinary artistic interests.
#10: The author is skilled art, music, literature.
Table 7: Items in 44-item Big Five Inventory.
Agrecableness
Conscientiousness
#1: The author be critical and quarrelsome.
#1: The author dependable, self- self-disciplined.
#2: The author be sympathetic and warm. Extraversion
#2: The author disorganized, careless.
Neuroticism
#1: The author extraverted, enthusiastic.
#1: The author anxious, casily upset.
#2: The author reserved, quiet.
#2: The author calm, emotionally stable.
Openness
#1: The author open to new experiences, complex.
#2: The author conventional, uncreative.
Table 8: Items Ten Item Personality Inventory.

---

Agrecableness
Conscientiousness
#1: The author sympathizes with others'feelings, feelings.
#1: The author often forgets put things back their proper place.
#2: The author is not interested in other people's problems.
#2: The author is not interested in other people's ' problems.
#3: The author feels others' emotions.
#3: The author makes mess of things.
#4: The author is not really interested in others.
#4: The author likes order.
Extraversion
Neuroticism
#1: The author is the life of the party.
#1: The author is relaxed most of the time.
#2: The author does not talk lot.
#2: The author seldom feels blue.
#3: The uuthor keeps in the background.
#3: The author has frequent mood swings.
#4: The author talks to a lot different people at purties.
#4: The author gets upset easily.
Openness
#1: The uuthor bus u vivid imagination.
#2: The uuthor is not interested in abstruct ideas.
#3: The uuthor bus difficulty understanding ubstruct ideus.
#4: The author does not have good imagination.
Table 9; Items in Mini-PIPI.
Estovenion (VE)
sanhoria sesally "A good niser groups people", Queer and merved' "Nee cam whether IC.
#2: Among the suthed mendi, the autor is A: "Tull of newr shour everybodty, B: "One of the last hear wtat is going on or "Nor srre whether A II",
When the aithor with a grop of people. the yatheric vaully Join people" "Doep frindship say few people"
undh
range frindships
many
of poup' "Sond and licter first or "Nex amm whater IP", 65: The authrr is At "Tilk rosily almost anyone", J:: Tind lot say only cenain people or under cemain conditions, or C: "Noe sire whether A or
lane gruup auhhri
Tatnsdux
intoduaal C: Nut
When the zuthor momn nox people anthor sile what they am immroadd k "iighht away", li: "Only afer people to pit know anthor", or "Not srre whether or It*
e5: The author istally: A "Shon their feelings treey". Dt KKeep their feelings themoevees" ar "Ner sure whether A or DI", suiible
Which word
ouioor "QUIET, "HEARTY" C. "No whefier AKATIVE Cr "Noe eam whether A Br"
#ID- Which wort mon exitsble vathrr RESEEVEEE
#11: Which word more siltalle for the aathor A: "CALM", B: LIVBLY", ar "Not sure whether A D".
Serning
F1. Uhe autlox was teachers would the caterr teach. A "Fass based courses B: Couses involving opinas o theary" o C. "Nor sure whelier A or B";
#2: In dring sometling that otlas people do would the sulhon rallar. A "Inveat way of this own', B. "Do in the soopped way" or C. "Net sure whetlae A B", "Nomaal acing
Does auther
people
themselves
ancation, B. "Too origital individuall care whether they center
out".
Droe author mosilly get aling bette with A: "Ratistic people" B: "Imogination people', or "Net whither A W.:
AS: ln seading pleauut, does lhs sallon, A. "Eajoy oid or original of laying thinp B. 'Like wills exastly what they meas", uu "Net sure whstles A B". gensar C:
Would
comidker
prattcal gence', cnd
whacher
World the anther rathrr has frient *Somonte whnin shays cemiing up with now sieae' "Semeene who has both firt on the grondd" "Not sore whether
#s: Which weed more amrille for the suthor: A: "DACTS" Dr "IDRAS" or C: "Nor sore whether A Ar
Which word
the sariire: "IMAGINATIVE'' "MATTER-OFFAACT
"Sat whetler
4ID Which word mor exitable aathor "STATEMENT" "CONCFIT"" ar
whether
#11: Which word more csitable for sathor A: "CRUXTTE" D:: "MAKE" C: "Noe eare whether A or
A12 Which wond
vsitable
"CERTTINTTY" B:
C. "Ni whather
41% Which word mom exitaiie aathor "FASCINATING: "SENSRB E or C "Nee cmm whatee
#14 Which word more csttahle for aathor A: "LITTRAL"" (I: "FIUURATVVE, or "Not am whether A II"
Fezling
11: Does lhe author norz otten it. A: TThir boatrale their beod". B: "Thee head rule deel bear", or C. "Not sure wiethe A A
#2: For the audis, which is lights compliment. A: "A pesus of roal fecling" B: conisicaaly resomalie paoual, or C. "Not sure whethes is sure
Dees auhhor maally: Volue emotion more than logic.. "Volue logic than feelings"
64: Which word moce sutable for the authoe: "COOVINNNIGG B: "TO POUCHING", or C: "Norsure whefter Ar
A5: Which word
suiabll fo the axihoe "'BENEFITS", B. "BLESSINGS" or C. "Nit whather A C:
woed
suthor: APPLCCCAAAAI
whether
17: Which word more suiable for the authoe: A: "ANALYZE. "SYMPXTHIZE" or C. "Not sure whether A oc
#5: Which word
sutabll fox the axihoe 'DETERMINED", B: "DEVOTED", or C. "Not sue whetles A
word
suther: ":"GENTLE"" "TIRM"
sure whether
fiD. Wlich word more suitable for the sathar. A: "JUSTICE" B: "MERCY"" or "Not sure whether or B".
#11. Whish word
waitakle for the saller: A. 'FEM MINDED" B. "WARM HEARTED", C: "Not whacher A "THINNINGG
412 Which word
saltalle
author. "TEELLIGG
whether A or
413. Wlich word more suitable for the suthor. A: "ANTICIPNTION" B: COMPASSION". or C: "Nor sure wheller A B".
#14. Wlish wond is more wiitadle for the sallor: A. "HARD" B. 'SOFT: C: Not vur whicher A B
Perception or lodging
P1: When is seated well is advunce But the author will de certais ning at cenuin tine, does the zuher Hid A: "Nice to be able to plas accondingh" B: littl unplessant to tied dowa",
C. "Nat sue whellas A
#2: When the author semeehom. wondd the author rather A "Pas what will do and Whan' B: "Joe go" or C. "Not are whether A it". auther dioald get donz werkend "Bely
Duss ilss miking list
the suiuee". "Posiiecly dapeow the author",
Whene author a special job
auther like A: "Organize it carefully before they sun". "Find
necessary as they go alinn" orr
eore wheher or
A5: Doos the authrr peefer to: A "Acming pieriss partics di, well in advanee' B: "Re froz to whateser to looks like fun when the tine comes" or C. "Nat sarz whether se B*. "Noc wheller Br.
Does following schetute A "Appeal outiee. "Cuup withor'
lae saller uaee ancconful: A." Edloving escffllly worked plui, "An dealing with the encspected and socing quickly what should disee 64: In oothoc's 's daily work does the suhoo. A "Usalll plan thei work so the athor won't ueed to wock under presure", Bi "Rater enjoy as emergency that makes thei: work agaisst time".
suse whiclee A B".
u1 C. 'Hase work udda pecoure" in "Nat aus whatlas A. B, or
#): Which word mare mntbale for the suthre: 'SCHEDULED: B. "UNH ANNELY or C: "Not ume whither or B², "XVSTTMATTI:: "SONTANNIOUS"
esttali A11. Wlide word is more wittade for the autlon: A: "SYSTEMATIC, B. "CASUAL" aa C. "Non sare whecler A ot
orm
Table 10: Items in Myers-Briggs Type Indicator.


Reviews:
[{"review_id": "s90JGpvPSG", "reviewer": "Reviewer_QAsf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your time and effort in reviewing our paper. We are grateful that you find our work to be both interesting and inspiring.\n#### **Q1: Regarding the innovation of this work.** \n**A1:** We appreciate the valuable feedback. While we acknowledge that our work builds upon Chain-of-Thought, we would like to emphasize the distinct contributions and unique value our research brings to the community.\n\n**Limitation and Challenge of Existing Work** Prior studies in personality detection have typically relied on crafting specialized models using datasets sourced from the internet. The limitations of these datasets, both in terms of their quality and their alignment with personality traits, have significantly restricted the attainment of optimal results.\n\n**Our Contribution** We introduce a pioneer approach that leverages the capabilities of Large Language Models to effectively analyze and elucidate personality traits. Our PsyCoT model emulates the cognitive process of human self-assessment related to personality traits. This emulation is accomplished through a dynamic, multi-turn dialogue process, which rigorously employs reasoning steps using items from a psychological questionnaire. This approach results in a significantly enhanced alignment with personality traits.\n\n**Impact of our Work** Our experimental results demonstrate the effectiveness and superiority of our approach over state-of-the-art methods. Additionally, PsyCoT exhibits versatility beyond personality detection, with applications extending to questionnaire assessment and depression assessments. Finally, we believe our research offers valuable insights to the personality detection community, serving as a catalyst for future explorations in this domain.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology is interesting enough to understand the ability of LLMs for personality detection, and the explanations about details are clear. The authors analyze the correlation between scores and traits to predict some patterns between the personality traits and the LLM's behaviors. Overall, it is easy to follow and good enough to inspire researchers in a similar field to analyze or predict authors' personalities. ", "reasons_to_reject": "PsyCoT is basically coming from a popular method, Chain-of-Thought. It is weak to say the proposed method is innovative and original though the application to personality detection has been yet underexplored.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes PsyCoT, a chain-of-thought method specialized in personality detection tasks. The methodology uses chain-of-thought reasoning in the form of answering psychological questionnaires. The authors investigate the correlation\nbetween the trait score and the chosen personality and explore the patterns of score distributions in terms of different traits.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nopuj64i7i", "reviewer": "Reviewer_6BCv", "summary": "", "questions": "A: The Big 5 personality traits elicit outcomes in a dimensional form. When this method elicits the outcome in the most dimensional form, why do you reduce it back to 2 classes?\n\nB: Post order matters (missing ref: Nelson Liu et al., 2023) matters for these LLMs, given that some messages have more information than others for Personality estimation (missing ref: Lynn et al 2020). How do you account for bias introduced due to order of posts in your prompting setup?\n\nC: Missing row in results table: Item informed high/low classification score (What is the performance when you aggregate the scores for each item)\n\nD: 10 vs 44 item scale is expected to have a big drop. Performance of 20 items scale would be more interesting to see since the 20 item scale has shown convergent validity (ref: Table 1 in Park et al., 2015)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your perspective comments. We carefully address each of your concerns point by point as follows.\n\n#### **Q1: Regarding the CoT in personality detection.**\n**A1:** Thank you for the insightful perspective. We acknowledge the inherent difficulty in developing a chain-of-thought solution for personality detection. This is precisely why we have meticulously crafted the PsyCoT approach. It assesses individual personality traits by incorporating a series of psychological questionnaire items into the prompt, effectively harnessing the capabilities of Large Language Models (LLMs) for this task. These items can be viewed as a well-structured collection forming a chain-of-thought (CoT).\n\n#### **Q2: Regarding the form of Big 5 traits.**\n**A2:** Although the original tasks of the Big 5 are in the regression form, this data is further collected and processed into the binary label format (Y/N), without releasing the original scores from the raters. Consequently, we have framed this task as a classification problem, designating 'High' as 'Y' and 'Low' as 'N.' While it is still possible to approach it as a regression task by assigning 'Y' as 1 and 'N' as 0, previous research, such as the work by Ganesan et al. [1], has delved into the distinctions between classification and regression in similar contexts, ultimately suggesting that simplifying the task to classification is more desirable. We appreciate your question and will provide further clarification in the revision.\n\n[1] Ganesan A V, Lal Y K, Nilsson A H, et al. Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation[J]. arXiv preprint arXiv:2306.01183, 2023.\n\n#### **Q3: Regarding the statistical tests.**\n**A3:** Thank you for the valuable suggestion. Regarding PsyCoT and other in-context learning-based baselines, we have indeed set the temperature to 0 to ensure deterministic outputs when interacting with the API, which can be subject to minor randomness. Additionally, we've executed multiple runs of the baselines, including RoBERTa, and consistently observed improved performance with our approach compared to the baselines. To enhance the rigor of our analysis, we will incorporate the results of statistical testing in the revised version.\n\n#### **Q4: Regarding the baselines.**\n**A4:** Thank you for bringing up these two methods. We’d like to provide some clarification here. Firstly, the model proposed by Park et al. (2014) is better suited for regression tasks and is therefore not well-matched to the datasets used in this study, which involve classification labels (as previously mentioned in A2). Secondly, regarding the method presented by Lynn et al. (2020), previous research, as demonstrated by the results of TrigNet (2021) and DDGCN (2023), has consistently shown superior performance when compared to Lynn et al. (2020) (detailed results on Kaggle are provided below). Consequently, we have omitted Lynn et al. (2020) from our selection of baselines in favor of these more stronger methods. To ensure completeness, we will include these additional results in the revised version.\n\n|  Methods | I/E | S/N | T/F | J/P | Average |\n| ---| :-: | :-: | :-: | :-: | :-: |\n| **Lynn et al.** | 65.43 | 62.15 | 78.05 | 63.92 | 67.39 |\n| TrigNet | 69.54 | 67.17 | 79.06 | 67.69 | 70.86 |\n| DDGCN | 69.52 | 67.19 | 80.53 | 68.16 | 71.35 |\n\n#### **Q5: Regarding the form of this task.**\n**A5:** Our study indicates that simplifying multi-label personality detection into multiple binary classification tasks leads to performance improvements (the average results on standard prompt are reported below). Hence, this approach was chosen for our experiments. We will supplement these additional results and provide more discussion in the revision.\n\n|  Forms  | Accuracy | Macro-F1 |\n| --- | :-: | :-: |\n| **Multi-label** | 51.74 | 46.85 |\n| Five binary (ours) | 58.70 | 54.20 |\n\n#### **Q6: Regarding the post order.**\n**A6:** This is an insightful question! We fully concur with the reviewer that post order can affect the model performance. In Section 5.3, we have conducted a preliminary investigation into the influence of post order, and our experimental results highlight that in-context learning-based methods can be affected by it. Nevertheless, we would like to acknowledge that delving into the determination of the optimal post order falls outside the scope of our current study. This aspect stands as one of our future research directions.\n\n#### **Q7: Regarding the aggregation of item scores.**\n **A7:** Thank you for the excellent suggestion. In fact, we have computed the aggregated trait scores and visualized their distributions in Figure 3. Our analysis reveals that the aggregated scores for Agreeableness, Conscientiousness, and Neuroticism cross the dashed line (3=Neutral) and exhibit a shift towards one side. This observation suggests potential bias in the LLM's ratings of the questionnaires. As a result, it becomes imperative to determine a suitable threshold for converting these scores into binary labels. We appreciate your suggestion, and in our revision, we will also incorporate the aggregate scores into the tables.\n\n#### **Q8: Regarding the 20 item scale.**\n**A8:** Thank you for the great suggestion. We evaluate the performance of PsyCoT on the Essays dataset using a 20-item scale from Mini-IPIP [2]. As shown in below table, the results are better than the 10-item scale, but still inferior to the 44-item scale. We will supplement the results in the revised version as suggested.\n\n|  Item-scale  | Accuracy | Macro-F1 |\n| :-: | :-: | :-: |\n| 10 | 58.46 | 54.72 |\n| **20** | 59.19 | 57.42 |\n| 44 | 59.64 | 58.43 |\n\n[2] Donnellan M B, Oswald F L, Baird B M, et al. The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality[J]. Psychological assessment, 2006, 18(2): 192.\n\n#### **Q9: Regarding the missing references.**\n**A9:** We will include these references in the revised paper accordingly. Thank you!\n\nWe sincerely thanks the reviewer again for your valuable suggestions. Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes the first use of item-based personality related questionnaires to establish personality traits exhibited in text. Their method works better than simple standard prompting.", "reasons_to_reject": "CoT does not just guide LLMs through arbitrary reasoning steps, but through a definitive order of reasoning steps. Furthermore, the abstract nature of the personality detection task render the step-by-step nature of the chain-of-thought problem solving approach challenging even for humans.\n\nHow can high and low be defined for the Big 5 traits? In its original form, this is a regression task, not a classification one.\n\nThere are no statistical significance tests b/w PsyCot and Standard baseline, and PsyCoT and RoBERTa.\n\nThe best model for these tasks is a simple regression model - Park et al (mentioned in missing references). However, the paper does not compare to that model and neither does it compare to Lynn et al 2020, which is another strong performing model. It is hard to contextualize the results in this paper and pit them against what the community knows.", "questions_to_authors": "A: The Big 5 personality traits elicit outcomes in a dimensional form. When this method elicits the outcome in the most dimensional form, why do you reduce it back to 2 classes?\n\nB: Post order matters (missing ref: Nelson Liu et al., 2023) matters for these LLMs, given that some messages have more information than others for Personality estimation (missing ref: Lynn et al 2020). How do you account for bias introduced due to order of posts in your prompting setup?\n\nC: Missing row in results table: Item informed high/low classification score (What is the performance when you aggregate the scores for each item)\n\nD: 10 vs 44 item scale is expected to have a big drop. Performance of 20 items scale would be more interesting to see since the 20 item scale has shown convergent validity (ref: Table 1 in Park et al., 2015)", "paper_topic_and_main_contributions": "This paper proposes a method to use item-based psychology questionnaires as a way to improve personality detection capabilities of LLMs in a CoT framework. They prompt the model to rate individual items from the questionnaire at each turn and leverage the historical rating results to determine the underlying personality trait.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 256: By-Produce -> Byproduct"}, "extra_text": {"Missing_References": "The following papers are critically missing:\nAutomatic Personality Assessment Through Social Media Language - Park et al\nVERY similar baseline - Systematic Evaluation of GPT-3 for Zero-Shot Personality Estimation - Ganesan et al"}}, {"review_id": "Mcn5t83XtZ", "reviewer": "Reviewer_3hiz", "summary": "", "questions": "Question A: Why do you not give the results for TrigNet and DDGCN in Table 1 for the Essays dataset, as they work best on the Kaggle dataset?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and dedication in reviewing our paper. We are grateful for your recognition of the contributions made by this work. In the following, we carefully address each of your concerns point by point.\n\n#### **Q1: Regarding the results of TrigNet and DDGCN not applied in Essays datasets.**\n\n**A1:** Thank you for the insightful question. We’d like to provide some clarification on this. TrigNet and DDGCN are both designed to amalgamate multiple posts. TrigNet accomplishes this by leveraging a psycholinguistic dictionary (LIWC) to establish a triplet graph between posts, whereas DDGCN autonomously learns to construct a graph among multiple posts. However, it's worth noting that the Essays dataset comprises only a single paragraph from the user, without multiple posts. Consequently, TrigNet and DDGCN are not directly applicable to this dataset.\n\n#### **Q2: Regarding discussion of limitations.**\n\n**A2:** We agree with the reviewer that these two limitations should be mentioned. We will supplement these discussions of limitations in the revised paper.\n\n#### **Q3: Regarding the details of the two data sets.**\n\n**A3:** Thank you for the excellent suggestion. For the Essay dataset, the average token count per sample is 762.38, whereas for Kaggle, it stands at 1632.38. We will provide additional dataset statistics in the Appendix during revision.\n\n#### **Q4: Regarding the typos and improvements.**\n **A4:** We will revise them accordingly. Thank you for your careful review!\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper describes in a reproducible way an interesting approach to predict aspect of an author's personality from text they have written.\n- The approach is validated in a feasible evaluation.\n- An ablation study shows the effectiveness go the multi-turn approach.\n- The work not only shows the technical possibility to predict personality (i.e., traits) using LLMs, but also the (relative) simplicity, which is in turn a warning for potential misuse.", "reasons_to_reject": "- In the results section the best two (alternative!) approaches which works best for Kaggle are not show for Essays. Thus the results are not complete.\n- A specific discussion is missing and the limitations are very short. Obvious limitations like application to only one language (which is fine, but should be named) or the rather artificial datasets are not mentioned.\n- The description of the two used data sets is very short. I would aspect at least additional information on the mean length of the texts to get a feeling for the needed amount of data per text author.", "questions_to_authors": "Question A: Why do you not give the results for TrigNet and DDGCN in Table 1 for the Essays dataset, as they work best on the Kaggle dataset?", "paper_topic_and_main_contributions": "The paper describes and evaluates an *NLP engineering experiment*. The authors try to predict the personality  of an author based on text written by the author. Their approach is to use a Large Language Model in an instructive way to rate an authors personality in a multi-turn dialog (following the chain-of-thoughts approach). The LLM, here GPT3.5, is gets first the text in question, some instructions on the answering format and finally multiple request about the authors personality. The rating request are based on psychological questionnaires (i.e., MBTI and Big 5). Finally, the LLM is asked to rate the text authors's personality on different dimensions.\n\nFrom my point of view, the paper provides two main contributions. First, it evaluates the usage LLMs in order to predict the personality of a person on the basis of text written by that person. Second, it compares two approaches of prompt engineering, one using single-turn instructions and another one using multi-turn instructions.\n\nFinally, both approaches as well as several other are compared in a formative evaluation.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 118-121: The sentence sounds somehow incomplete.\n- Line 167: the ChatGPT -> remove the\n- Line 356: Implementation\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies Chain of Thought reasoning to personality detection using large language models. Their method is based on commonly used personality questionnaires (MBTI and Big 5), but re-structured for use with  LLMs in a multi-turn dialogue. The reviewers initially raised issues related to comparisons to previous work, which were addressed by subsequent experiments during the discussion period. The reviewers also questioned the framing of the task as Chain of Thought, given the lack of a concrete reasoning process. This point was discussed at length, and the authors promised to reconsider their framing. Overall, reviews are positive.
Decision: Accept-Findings