Title: CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
Benchmark: CLiMB
Summary: The paper introduces CLiMB, a benchmark designed to evaluate continual learning (CL) for multimodal tasks, addressing the challenges of learning both new multimodal and unimodal tasks over time. It shows that while common CL methods can reduce forgetting in multimodal learning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Continual learning, multimodal reasoning, knowledge transfer
Phenomenon defined by authors: Yes
Phenomenon definition: Upstream Continual Learning of Multimodal Tasks: A candidate model 𝑀
M encounters a sequence of vision-language tasks... We evaluate two primary model properties in the upstream phase: upstream knowledge transfer from past learned tasks to new tasks, and withstanding forgetting of previously-seen tasks.

Downstream Transfer to Low-Shot Tasks: We evaluate the low-shot adaptation ability of the model  after learning each upstream vision-language task
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The paper provides specific operational definitions for components like knowledge transfer, forgetting, and low-shot adaptation.
Task definition: Learning from a sequence of different multimodal (vision-and-language) tasks in a continual learning (CL) setting, and then transferring to low-shot multimodal and unimodal tasks.
Task item definition: Each item framed as a classification problem: some input and a target label, where the inputs can be vision-only ( an image), language-only (a sentence or question), or multimodal (an image paired with a question or caption). 
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: N/A 
Dataset size detail: the benchmark consists of the already existent benchmarks  and doesn't provide any numbers in the paper
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), VQAScore
Metric aggregation: Simple Mean
Metric statistics: simple mean, std, relative performance changes
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Updating
New taxonomy: General Application / Updating