Title: ClevrSkills: Compositional Language And Visual Reasoning in Robotics
Benchmark: ClevrSkills
Summary: ClevrSkills is a multi-modal benchmark for evaluating compositional reasoning in robotics. It is composed of 33 tasks spanning three cumulative different levels of difficulty. The benchmark is publicly available and built on top of ManiSkill 2. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: compositional reasoning in robotics
Phenomenon defined by authors: Yes
Phenomenon definition: Compositional generalization/reasoning is the ability to "receive instructions in natural language and successfully perform previously unseen tasks with minimal to no task-specific learning or adaption" (1). 
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The benchmark consists of 33 different tasks, spanning low-level execution tasks (L0), intermediate composition tasks (L1), and and higher-level planning tasks (L3). Both zero-shot and fine-tuned models are assessed. The benchmark supports multi-modal and text-only prompts. 
Task item definition: A single item is a ground truth task trajectory for the given task. Each trajectory is annotated, "including task or predicate level (the highest level describing the task, which can also be used as the task specification), sub-task level (a sub-task on a semantic level that needs to be achieved for the high level task to be completed), and step level (a language label for each step that is being taken)" (5). 
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: "ClevrSkills is built within the ManiSkill2 simulator, which allows for realistic physics and graphics. We use a simulated model of the UFACTORY xArm 6 robot with vacuum gripper as our default robot for the environments, with Franka Emika Panda also being available" (3). The trajectories are procedurally generated with oracle policies, which perform "a greedy search" for the next solution step, and "instantiates a solver (oracle) policy for the same" (4). 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 330000
Dataset size detail: The paper mentions a train-test split but does not provide it. 
Dataset metadata available: Yes
Dataset metadata detail: Object bounding boxes, key-step frames, action labels
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Movement trajectory to complete task
Response format detail: Task success is measured through predicates, which can be physical or logical. "Physical predicates specify the target state of the robot and/or the objects in the scene, and how the
agent achieves these states" (4). "Logical predicates can be used to combine physical predicates to specify more complex tasks" (4). 
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: The benchmark uses the task success rate, average reward, and reward per step as metrics. 
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors highlight that reasoning tasks must require "actions, behaviors or planning," otherwise the benchmark would evaluate "compositionally in a purely abstract setting" (2). 
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning