# Personality Emulation Utilizing Large Language Models
## 4 Results
### Human Perception Validation Results  
The results of the human perception survey allow for an analysis of how well humans were able identify personality traits in the email responses generated by the GPT-4o model. As mentioned, 11 individuals rated each Big Five trait for a random sampling of 37 different emails. Individuals were asked to rate their perception for each trait as high, neutral, or low, without being provided with any information about the distribution of personality traits within the email responses. Overall, humans were 46.48% accurate in trait-level identification across all traits, with a 95% confidence interval of 44.32% to 48.64%. For the 11 individuals, accuracy ranged from 37.1% to 61.82%. Accuracy by individual Big Five characteristic is provided in Table 7.  
Figure 12: Accuracy by trait and model after normalization using median.  
Figure 13: LLM prediction proportions after normalization utilizing median.  
The confusion matrices in Figure 14 compare how individuals rated the levels of personality traits compared to the true level of the trait. The total number of entries for each matrix is 407, representative of the 11 individuals who rated the trait across 37 emails. For agreeableness, shown in Figure 14a, individuals rated responses high 61.9% of the time. However, of the responses they rated high, 39.7% were designated low in agreeableness. Similarly, participants rated 73.5% of responses high in conscientiousness; however, 42.8% of those were designed to be low. Individuals tended to rate these traits higher, suggesting that the responses did not clearly portray low agreeableness and conscientiousness effectively.  
The confusion matrix for the extraversion trait, which was the lowest-accuracy trait for participants, is shown in Figure 14c. Individuals rated 81.2% of the responses that were truly high as either high or neutral and 76.3% of the true low responses as either low or neutral. This suggests that individuals were generally able to determine that the true high and low response were not the opposite; however, a subset of responses were perceived as ambiguous. For neuroticism, shown in Figure 14b, individuals rated 50.6% of responses as low, 19.7% as neutral, and 29.5% as high. This distribution suggests that participants were generally able to identify the true low cases; however, the cases of high neuroticism were not clearly perceived. Finally, for openness to experience, participants were approximately equally likely to correctly identify responses high in openness (46.7% correct) as those that were low (45.6% correct). They were also equally likely to misclassify the true trait as the opposite, occurring 28.6% of the time for true low responses and 28.1% of the time for true high responses. This suggests that a subset of the responses did not clearly portray the characteristics of a highly open personality.  
Overall, individuals achieved a higher accuracy rate than what would be expected from random guessing; however, the results suggest that the level of some traits, like agreeableness and conscientiousness, were more difficult for individuals to accurately perceive in the responses. For extraversion, neuroticism, and openness to experience, individuals tended to be able correctly identify a subset of cases as true high or true low. However, the results also suggest that a subset of emails were more ambiguous, making correct classification by participants more difficult.  
\begin{table}
\begin{tabular}{c c} \hline
**Trait** & **Accuracy** \\ \hline Conscientiousness & 0.479115 \\ Agreeableness & 0.484029 \\ Neuroticism & 0.488943 \\ Openness to Experience & 0.452088 \\ Extraversion & 0.432432 \\ \hline \end{tabular}
\end{table}
Table 7: Human accuracy by Big Five trait.