Title: Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
Benchmark: TASKBENCH500
Summary: We present a large-scale empirical study on LM adaptability using TASKBENCH500, a benchmark of 500 procedurally generated sequence modeling tasks. 

We evaluate three facets of adaptability, finding that: (1) adaptation methods vary in memorizing small datasets; (2) some show compositional adaptability to complex tasks; and (3) label distribution mismatches arise from differences in intrinsic label difficulty. 

Our results show that adaptability to new tasks can be systematically analyzed, similar to generalization.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LM Adaptability
Phenomenon defined by authors: Yes
Phenomenon definition: Adaptability: 'Adapting pre-trained language models (LMs) by finetuning their parameters or input prompts for downstream tasks' (Page 1)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: They define a set of atomic tasks, which are combined using a set of composition functions to produce more complex tasks. 

The atomic tasks include lexical tasks, factual tasks, and random relation tasks. The composition functions are word-level and sequential compositions. 
Task item definition: For each task f, they construct a dataset D(f) = {(x_i, y_i)}, where x_i is sampled from the task's input distribution (e.g., most common words), and y_i is uniformly sampled from the set of valid outputs f(x_i), i.e., y_i ~ Unif(f(x_i)).
Task definition detail: For evaluation, they measure the model’s average per-token accuracy on both training and test splits of the dataset D(f).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Response format detail: Every task takes as input a word or word sequence, and outputs either a boolean value or a set of words/word sequences.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors explicitly justify their synthetic task generation approach as a more controlled, interpretable, and generalizable way to study model adaptability, enabling them to identify which task attributes make learning easier or more difficult, something that is difficult to study with existing real-world datasets.

'For benchmarks built from collections of real-world datasets, the makeup and difficulty of these datasets is often difficult to characterize precisely: differences in annotation standards, annotation quality, and dataset size mean that models often exhibit very different performance on datasets designed to evaluate model performance on the same abstract task.' (Page 3)
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context