Title: Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
Benchmark: TaskBench500
Summary: TaskBench500 is a benchmark designed to systematically measure how LLMs adapt to new tasks. It comprises "500 procedurally generated sequence modeling tasks" spanning "lexical semantics, sequence processing, memorization, logical reasoning, and world knowledge" (4696). The benchmark is publicly available. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: adaptability, compositional adaptability
Phenomenon defined by authors: Yes
Phenomenon definition: Language model (LM) adaptability is how LMs are tuned to "perform" a "new task" they were not trained to complete, with adaption techniques like "fine-tuning or prompt-tuning" (4696). 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: Task adaption is split into the domains of measuring memory, composition, and distribution matching. 
Task definition: The benchmark defines 500 tasks total tasks. It first defines atomic tasks that are then "combined using a set of composition operators to produce more complex tasks" (4698). Atomic tasks span lexical tasks, factual tasks, and random relation tasks, and composition operators include word-level and sequential compositions. 
Task item definition: Although there are 500 distinct tasks, "every task takes as input a word or word sequence, and outputs either a boolean value or a set of words/word sequences" (4698). 
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation), Correlation (Matthew's correlation, Pearson's r)
Metric definition detail: To measure memorization, accuracy is used. To measure compositions of atomic tasks, Pearson correlation is used. Both metrics are referred to as the adaptability metric for their task. To measure how models learn new distributions, the paper defines a custom metric to produce "an aggregated probability mass assigned to all easier task and all harder tasks in a task pair," so it should be easier to adapt to an easier task than a harder task (4703). 
Metric aggregation: Simple Mean
Metric statistics: Simple mean, average
Metric subscores: Yes
Metric subscores detail: Atomic, Word-level Comp, Seq Comp
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors distinguish between previous attempts to study "generalization to new examples" and the paper's "systematic study of adaption to new tasks" (4698). The authors identify that "new pre-raining adaption schemes are evaluated using small suites of curated tasks" which are "poorly suited for answering larger, structural questions" like "can we predict how quickly (and how effectively) pre-trained LMs can be adapted to perform it" (4696). 
Taxonomy: Language Modelling / In-context Learning
New taxonomy: General Capability / In-context Learning