Title: ANALOBENCH: Benchmarking the Identification of Abstract and Long-context Analogies
Benchmark: ANALOBENCH
Summary: Aims to measure ability of LLMs use analogy, a skill that allows humans to creatively solve problems and articulate ideas more efficiently. They create a dataset of pairs of stories that have an analogous meaning. Given one story, the task is to pick the paired story out of a group of K other non-analogous stories.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Analogy
Phenomenon defined by authors: Yes
Phenomenon definition: "Analogy is the ability to think about relational patterns (Holyoak et al., 2001) and forms an integral aspect of human communication (Hofstadter, 2001; Gentner and Hoyos, 2017). " 

and also

"We assess the ability of LMs to handle components of analogy making. Two important features characterize how humans form analogies in creative pursuits. (1) Humans are able to pinpoint analogies between prolonged experiences (e.g. “obtaining a PhD is like running a marathon”). (2) Humans can recollect relevant analogs from a large collection of past experiences to form analogies (Keane, 1987; Wharton et al., 1994)."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Purpose notes: Not clear how the composite sub-elements map to the task they define
Task definition: The problem setup: given a story, the goal is to identify an analogous story from a story bank. 
Task item definition: Short story variant:

Target: You can't pour from an empty cup.
✓ A fallen tree cannot provide shade.
✗ All that glitters is not gold.
✗ After letting off his rage he sat down like a...
✗ A succession of waves battered the rock.

Long story variant are GPT4 written stories that expand upon the short analogy pairs.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 340
Dataset metadata available: Yes
Dataset metadata detail: Story length
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Accuracy for different lengths of stories
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning