Title: Paloma: A Benchmark for Evaluating Language Model Fit
Benchmark: Paloma
Summary: Evaluations of language models typically use a single dataset for measuring perplexity, but this dataset comprises various domains with different language distributions. PALOMA introduces a new benchmark to assess language model performance across distinct English and code domains, including two new datasets from top subreddits and popular programming languages, providing a more detailed and domain-specific analysis of model fit.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Perplexity analysis to assess LM fit to different domains
Phenomenon defined by authors: Yes
Phenomenon definition: perplexity
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Predict text from different data sources
Task item definition: Source, domain, val and test tokens, token per split per domain
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 123,683,201 tokens
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: domains, sources
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / 
New taxonomy: General Capability / Adaptability