Title: CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation
Benchmark: CharacterEval
Summary: A dataset of role-playing dialogues for Chinese characters is used to evaluate agentic role-playing ability
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: role-playing conversational agents
Phenomenon defined by authors: Yes
Phenomenon definition: " Role-Playing Conversational Agent (RPCA), designed to offer emotional value instead of productivity", 'RPCAs engage users in
 dynamic scenarios, where LLM agents are assumed
 as specific characters or roles, often derived from
 existing composition such as novels, films, car
toons, and games."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task is hardly defined. It seems to involve asking the LLMs to conduct a role play, but the prompts given are not described.
Task item definition: Probably a single setting in which to conduct a role play.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The tasks are taken from unspecified texts and parsed with LLMs into a useful format
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 4564
Dataset size detail: these are the number of "Examples" which are probably statement, response pairs from the dataset
Dataset size notes: train - 6811 
Dataset metadata available: No
Sampling method: Unknown
Split reviewed: Test, Train
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric statistics: mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: No, no link is provided
Task face validity: The task is too unclear to know
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction