Title: On the generalization capacity of neural networks during generic multimodal reasoning
Benchmark: Generic COG (gCOG)
Summary: The paper introduces gCOG, a multimodal reasoning dataset designed to measure various types of OOD generalisation (distractor generalisation, systematic compositional, and productive compositional). The authors train various encoder architectures from scratch and compare their performances. Transformers can systematically generalise at scale, but no architectures can productively generalise.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal generalisation
Phenomenon defined by authors: Yes
Phenomenon definition: "OOD generalization – the ability to perform tasks beyond the training distribution" (1)
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Models are given an 8x8 grid containing multicoloured letters at different indices, and must follow a binary tree of "if-then-else" instructions to answer a question like "Get the position of the orange 't'".
Task item definition: A query in natural language, an image of an 8x8 grid in some .jpg-like format, and a correct answer, which is either a shape ("d") a colour ("orange") or a location ((5, 4)).
Task definition detail: The concrete dataset used for their evaluation is not provided, only a generator object in python is given.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: Yes
Dataset metadata detail: task tree depth, num distractors
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: IID and OOD accuracy on varying numbers of distractors and tree depths
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: "Identifying neural architectures that can robustly generalize OOD is a central goal in artificial intelligence. Compositional generalization benchmarks, which explicitly evaluate for generalization, provide a good testbed for measuring these capabilities" (9)
Taxonomy: Language Modelling / Adaptability
New taxonomy: General Capability / Adaptability