Title: EmoBench: Evaluating the Emotional Intelligence of Large Language Models

Benchmark: EmoBench
Summary: EmoBench evaluates emotional intelligence (EI) covering emotional understanding and application with 400 questions in English and Chinese.

Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Emotional Intelligence

Phenomenon defined by authors: Yes
Phenomenon definition: Emotional Intelligence, ""the ability to monitor feelings of our own and understand
feelings of others, differentiate between them, and leverage this information to guide our thoughts and actions"". Authors use understanding and application as core components.


Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Use both a breakdown of possible emotions and a breakdown into application and understanding
Task definition: Scenario-based multiple-choice questions; some just requiring selection of most effective action, others requiring action selection + definition of emotion in scenario

Task item definition: Scenario + MCQ on emotions, causes, or best action

Task definition detail: Task items are theory-grounded and include taxonomies of emotions

Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: GPT-4 used for inspiration in creating scenarios, final content human-authored

Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Fully fictional; while grounding in psychological theory seems very solid, task is completely from-scratch, not e.g. from existing personality/emotional intelligence exams (should those exist)
Dataset size: 400
Dataset metadata available: Yes
Dataset metadata detail: Emotion type, correct label, category

Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Accuracy, Fleiss’ k for human agreement

Metric subscores: Yes
Metric subscores detail: By task type, language, subcategory

Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Exhaustive human baselining via online survey of 48 participants (humans outperform all LLMs on average)

Taxonomy: Psychology / 
New taxonomy: Specific Application / Social Sciences