Title: DevBench: A multimodal developmental benchmark for language learning
Benchmark: DevBench
Summary: DevBench is a multimodal benchmark for assessing how LLMs compare to human language development across seven language evaluation tasks spanning lexical, syntactic, and semantic domains. Each task contains item-level human baseline data to facilitate human-model language development comparison using a novel metric: softmax-optimized Kullback-Leibler divergence. The goal of the benchmark is to measure whether developmentally realistic data leads to human-like learning in LLMs. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: language evaluation, language development, cognitive evaluation
Phenomenon defined by authors: No
Phenomenon definition: Language development evaluation is assessing whether the language ability gained by machine learning models matches the language ability gained by children when exposed to similar developmental data.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The benchmark consists of 7 multi-modal language evaluations. The lexical tasks consist of Looking-while-listening (LWL) and Visual vocabulary task (VV), the syntatic tasks consist of Test of Receptive Grammar (TROG), Winoground-NoTag (WG), and the semantic tasks consist of Free word association task (WAT), Visual object categorization (VOC), and THINGS similarity ratings. 
Task item definition: For each task, a single sample would consist of the task prompt, a correct label if applicable, and the associated human response and human age range. Several tasks (LWL, VOC) are quantitative and measured by the looking time response, while the rest are categorical. 
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: The experiments are sourced from child development literature, hence the choice of real task examples. Several task samples were modified to ensure that the images used in multimodal prompts had the correct licensing. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 22212
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Distribution (perplexity, calibration, correlation)
Metric statistics: Visual semantic tasks were measured with representational similarity analysis (RSA), while the other tasks were measured with a novel metric: softmax-optimized Kullback-Leibler divergence
Metric subscores: No
Metric subscores detail: Scores are provided per task, and the benchmark itself consists of 7 distinct tasks
Metric access: Model access required (e.g. logits)
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors define the desiderata for an ideal benchmark of developmentally appropriate evaluation of language models as (1) a wide dynamic range of difficulty (2) multiple levels of linguistic representations (3) corresponding data from children, and (4) high similarity in evaluation method between models and humans. These desiderata are based on child development literature and seek to overcome the limitations of existing benchmarks. Namely, current benchmarks are either unimodal, when cognitive language evaluations for children and infants are multimodal to accommodate pointing or looking responses, or current benchmarks compare language models to exclusively adult performance. DevBench seeks to fulfill all four criteria. 
Taxonomy: Language Modelling / 
New taxonomy: General Capability / Adaptability
Procedural notes: For attribution and licensing reasons, not all assets and data are hosted in the repo.