Title: ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
Benchmark: ValueBench
Summary: This paper introduces ValueBench, a benchmark for evaluating value orientations and value understanding in LLMs, grounded in realistic human-AI interactions to test for value orientations, along with new tasks for evaluating value understanding in an open-ended value space.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Value orientation and value understanding 
Phenomenon defined by authors: Yes
Phenomenon definition: Values are concepts or beliefs about desirable end states or behaviors that transcend specific situations. Value Orientation is the extent to which an LLM exhibits preferences or inclinations toward specific human values — i.e., how aligned its responses are with particular value statements or stances. Value Understanding is the extent to which an LLM can recognise, interpret, and reason about human values — including identifying relationships between values, inferring values from behaviours or statements, and generating expressions that reflect particular values.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: In value orientation, an LLM is given a value loaded question converted from psychometric statements abd asked to respond with advice. In value understanding, an LLM is prompted to identify relevant values on both positive and negative samples. For each value pair, the LLMs are required to sequentially output the definition of both values, a brief explanation of their relationship, the corresponding relationship label, and a final assessment of relevance (1 if relevant and 0 otherwise). An LLM can also be presented with a value name and is requireed to generate a value-reflecting statement or a pairs of values and required to identify semantic relationships. 
Task item definition: For value orientation a single task item is a rephrased value statement to advice-seeking question. For value understanding, the task items can be a behavioural statement, a value name or pairs of values. 
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 44 psychometric inventories, 453 value dimensions and 1989 value orientation questions
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Across each value orientation and subtask dataset
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The benchmark contains questions sourced from psychometric analysis tests and shows correlations in the results for tests that examine similar behaviours.
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment