Title: ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
Benchmark: ePiC (Employing Proverbs in Context)
Summary: This paper introduces ePiC, a high-quality crowdsourced dataset designed to benchmark abstract language understanding and analogical reasoning in LLMs. The dataset pairs narratives with proverbs, featuring fine-grained span alignments and minimal lexical overlap. Three tasks are proposed: proverb recommendation/alignment, narrative generation, and identifying similar narrative motifs. Experiments show that current LLMs struggle with these tasks compared to humans, indicating significant challenges in abstract reasoning.
Contribution: Introduced a high-quality, manually curated dataset (ePiC) specifically for benchmarking abstract reasoning using proverbs, featuring fine-grained span alignments and intentionally low lexical overlap. Proposed three challenging tasks (proverb recommendation/alignment, narrative generation, similar motif identification) designed to test reasoning beyond surface patterns. Provided benchmark results for several LLMs, demonstrating a significant performance gap compared to humans on these tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Abstract language understanding, complex analogical reasoning.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability for abstract language understanding and complex analogical reasoning, demonstrated by correctly associating proverbs with illustrative narrative contexts and identifying underlying motifs, requiring reasoning beyond surface lexical features.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The benchmark uses proverbs because they require understanding analogies, cultural context, and reasoning beyond literal meanings, posing a challenge distinct from many standard NLU tasks.
Task definition: The benchmark includes three main tasks: (1) Proverb & Alignment Prediction: Given a narrative, predict the most fitting proverb from 250 options and identify corresponding text spans between the narrative and proverb. (2) Narrative Generation: Given a proverb and topic keywords, generate a relevant narrative. (3) Identifying Similar Motifs: Given a narrative, identify other narratives that illustrate the same underlying proverb/motif.
Task item definition: A proverb paired with 10 distinct, crowdsourced narratives. Each narrative-proverb pair includes annotations of aligned text spans (up to 5) indicating semantic correspondences.
Task definition detail: Narratives are short (avg. 64 words), intended as realistic stories, and intentionally written with minimal lexical overlap with the corresponding proverb to prevent reliance on surface cues.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: Proverbs were collected from public online sources (The Phrase Finder, WikiQuotes) and manually curated. Narratives and alignments were generated by paid crowdworkers on Amazon Mechanical Turk following specific instructions to ensure quality and low lexical overlap.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The dataset consists of narrative stories intended to be realistic, but the tasks themselves (classification, generation from keywords, similarity based on shared proverbs) are primarily evaluation constructs.
Dataset size: Total dataset: 250 proverbs, 2500 narratives. Test set: 1000 proverb-narrative pairs (exact narratives depend on 'seen' vs 'unseen' split setup). 
Dataset size notes: Train set: 1500 proverb-narrative pairs. No validation set mentioned.
Dataset metadata available: Yes
Dataset metadata detail: Fine-grained aligned spans between proverbs and narratives (up to 5 pairs per item, linking contiguous text spans).
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Multiple choice, Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Response format detail: Proverb prediction is classification/MCQ. Alignment prediction involves outputting span indices. Narrative generation produces free text. Motif identification ranks narratives based on similarity.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), Distribution (perplexity, calibration, correlation)
Metric definition detail: Proverb Prediction: Accuracy, MRR. Alignment Prediction: Token-level Precision, Recall, F1. Narrative Generation: BLEU, ROUGE-L, Keyword Recall, Human Likert ratings (1-5) for Relatedness, Interesting/Creative, Fluency, Overall. Motif Identification: Accuracy.
Metric aggregation: Simple Mean
Metric statistics: Accuracy, MRR, Precision, Recall, F1, BLEU, ROUGE-L, Keyword Recall, Mean Likert scores.
Metric subscores: Yes
Metric subscores detail: Results are reported separately for 'seen proverbs' and 'unseen proverbs' test conditions.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Analyses demonstrated minimal lexical overlap between proverbs/narratives and high diversity among narratives for the same proverb. Sentiment analysis showed narrative sentiment diversity. The dataset contains diverse events and reading complexity levels. Human evaluations confirmed high quality for narratives (Overall 3.68/5) and alignments (3.91/5), surpassing prior related datasets. Potential gender bias was identified and discussed.
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Detailed appendices cover additional data analysis (sentiment, gender, complexity, hate speech), human evaluation specifics (MCQ task design, error analysis), generated narrative examples, and detailed training parameters (models, hyperparameters, hardware, software). Ethical considerations including data bias (gender, cultural), turker compensation and selection are discussed.
Other notes: A key feature is the fine-grained span alignment annotations, intended to support interpretability and more sophisticated modeling approaches. The paper explicitly acknowledges the limitation of focusing only on English proverbs and suggests future work to broaden cultural representation. The low performance of models, especially compared to humans, strongly suggests these tasks capture reasoning abilities beyond current LLM capabilities.