Title: Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts
Benchmark: WIKIDOMAINS
Summary: NLP models are useful for aiding comprehension of complex texts from unfamiliar domains, but simplifying entire texts can remove important details. Targeted concept simplification helps readers understand difficult concepts within context, enhancing vocabulary and knowledge. The new WIKIDOMAINS dataset and preliminary benchmarks show human judges prefer explanations over simplifications for difficult concepts, with no single model excelling across all quality dimensions, highlighting the need for personalized reading comprehension support.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLMs' ability to support people in reading complex text from unfamiliar domains
Phenomenon defined by authors: Yes
Phenomenon definition: targeted concept simplification as a task for supporting readers
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: The task of targeted concept simplification is to rewrite an input definition containing a concept to make it understandable to someone unfamiliar with the concept. 
Task item definition: Text, domain, concept, human difficulty rating, evaluation
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 22561
Dataset size notes: 3384 val, 15873 train
Dataset metadata available: Yes
Dataset metadata detail: human difficulty
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum, t-tests
Metric subscores: Yes
Metric subscores detail: simplify, explain, human eval, automatic eval
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization