Title: Revealing Personality Traits: A New Benchmark Dataset for Explainable Personality Recognition on Dialogues
Benchmark: PersonalityEvd
Summary: This paper introduces Explainable Personality Recognition, a novel task requiring models to identify Big-Five personality traits from dialogues and provide supporting evidence. It proposes the Chain-of-Personality-Evidence (COPE) framework, reasoning from dialogue context to short-term states to long-term traits. Based on COPE, the PersonalityEvd dataset is constructed from dialogues, featuring annotated state/trait labels and detailed reasoning evidence. Experiments with LLMs show the task is challenging.
Contribution: Key contributions include: (1) Proposing the novel task of Explainable Personality Recognition. (2) Developing the COPE framework grounded in personality theory for structured explanation. (3) Creating the PersonalityEvd dataset with dialogue-level state and speaker-level trait annotations, including utterance/dialogue IDs and natural language reasoning evidence. (4) Defining two sub-tasks (EPR-S and EPR-T) and providing LLM baselines. (5) Demonstrating the task's difficulty and offering insights for future work.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Personality recognition (Big-Five model); Explainable AI; Reasoning about personality states and traits from dialogue evidence.
Phenomenon defined by authors: Yes
Phenomenon definition: Recognising Big-Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) of a speaker based on aggregating evidence of short-term personality states (patterns of thoughts, feelings, behaviours) observed across multiple dialogues. The task also requires generating an explanation tracing evidence from specific utterances (for states) and dialogues (for traits).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: Addresses the lack of interpretability in existing automatic personality recognition systems by requiring evidence-based explanations grounded in psychological theory (state-trait distinction).
Task definition: Explainable Personality Recognition, comprising two sub-tasks: (1) EPR-S: Given a dialogue, target speaker, and Big-Five dimension, predict the speaker's personality state (high/low/uncertain) and provide evidence (relevant utterance IDs and natural language reasoning). (2) EPR-T: Given multiple dialogues for a speaker and a Big-Five dimension, predict the speaker's personality trait (high/low/uncertain) and provide evidence (relevant dialogue IDs and faceted natural language reasoning).
Task item definition: An EPR-S instance includes a dialogue, target speaker, and target dimension, mapped to a state label and state evidence (utterance IDs + reasoning text). An EPR-T instance includes multiple dialogues for a speaker and a target dimension, mapped to a trait label and trait evidence (dialogue IDs + faceted reasoning text).
Task definition detail: Uses the Big-Five model structured according to the BFI-2 scale (3 facets per dimension). Labels are High/Low/Uncertain. Evidence structure is specific (utterance/dialogue IDs + template-based natural language reasoning).
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Dialogues from CPED corpus (Chinese TV series), translated to English. State labels/evidence pre-annotated by GPT-4, then manually corrected and validated by psychology students/experts. Trait labels/evidence annotated by psychology students via consensus, checked by authors.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Models how personality traits might be inferred and explained based on observed conversational behaviours over time.
Dataset size: Total: 72 speakers, ~1924 dialogues. Test set size (state-level): ~14 speakers / ~370 dialogues. Test set size (trait-level): 24 speakers per fold.
Dataset size notes: State-level: Train ~50 speakers / ~1347 dialogues; Valid ~8 speakers / ~215 dialogues. Trait-level: 48 speakers for train/val per fold.
Dataset metadata available: Yes
Dataset metadata detail: Speaker ID, Dialogue ID, Utterance IDs, Big-Five Dimension Name, Facet Name, State Label, Trait Label, Evidence Utterance IDs, State Reasoning Text, Evidence Dialogue IDs, Trait Reasoning Text.
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Response format detail: Output requires the personality label (High/Low/Uncertain) and the structured evidence (specific utterance/dialogue IDs plus natural language reasoning text).
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Binary F1 for Evidence IDs
Metric definition detail: Label Accuracy (average over 5 dimensions). Evidence ID F1 score. Reasoning Text Quality: BERTScore F1, Claude-3 score (avg 1-5), GPT-4 score (avg 1-5). Human Evaluation: Fluency, Coherence, Plausibility (avg 1-5).
Metric aggregation: Simple Mean
Metric statistics: Accuracy, F1 score, BERTScore F1, Average score (1-5 scale).
Metric subscores: Yes
Metric subscores detail: Results reported for state (EPR-S) vs. trait (EPR-T) tasks. Accuracy reported per Big-Five dimension. Ablation studies analyze impact of evidence and state analysis on trait prediction.
Metric access: Outputs alone
Authorship: Academia and Independent Researcher
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: COPE framework based on personality theories. Annotators were psychology students/graduates. Multi-stage annotation included GPT-4 pre-annotation, human correction, expert inspection (states), and 3 annotators + consensus (traits). Human evaluation confirmed high quality of ground truth explanations (avg scores >4.3/5).
Taxonomy: Psychology / 
New taxonomy: Specific Application / Social Sciences
Procedural notes: Dataset sourced from CPED (Chinese TV dialogues), translated to English. Uses Big Five Inventory-2 (BFI-2) scale. Details GPT-4 pre-annotation prompt. Details human annotation process (training, guidelines, quality checks). Details LoRA fine-tuning parameters. Provides LLM details used for evaluation (Claude, GPT-4). Mentions fair pay for annotators.
Other notes: The paper's strength lies in its psychologically grounded framework (COPE) and the two-level annotation (state and trait) with explicit evidence linking. The results highlight the significant challenge LLMs face in not just predicting personality but justifying it with evidence from dialogue history.