Title: Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
Benchmark: Test Haystack
Summary: The paper introduces lifelong ICL as a new long-context problem setting for LLMs and the Test Haystack evaluation suite to understand how LLMs utilize contexts for the lifelong ICL task. Many long-context LMs are benchmarked, and contributors to failure cases are identified.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: lifelong in-context learning
Phenomenon defined by authors: Yes
Phenomenon definition: "Lifelong ICL, a new problem setting that challenges these models to learn a sequence of tasks via in-context learning"
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a task instruction and test inputs, leverage the relevant demonstrations in the Lifelong ICL prompt, avoid distraction and interference from other tasks, and achieve test accuracies that are not significantly worse than those of the Single-task ICL baseline.
Task item definition: Each sample is represented by the input context and target answer.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), pass rate
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: number of shots
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / In-context Learning
New taxonomy: General Capability / In-context Learning