Title: ClevrSkills: Compositional Language And Visual Reasoning in Robotics
Benchmark: ClevrSkills
Summary: We ask the question: if the models are taught the low-level capabilities, can they compose them in novel ways to achieve high-level tasks like cleaning the table without having to be explicitly taught so? 

To this end, we present ClevrSkills - a benchmark suite for compositional reasoning in robotics. The dataset contains trajectories generated on a range of robotics tasks with language and visual annotations as well as multi-modal prompts as task specification. 

We benchmark multiple different VLM baselines on ClevrSkills and show that even after being pre-trained on many tasks, these models fail on compositional reasoning in robotics tasks.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Compositional reasoning/generalization in robotics
Phenomenon defined by authors: Yes
Phenomenon definition: Compositional generalization is a hallmark feature of human intelligence. Unlike any other animals, humans can receive instructions in natural language and successfully perform previously unseen tasks with minimal to no task-specific learning or adaptation.' (Page 1)
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: In ClevrSkills, they benchmark robotic models on a set of simple manipulation tasks, such as pick, place, throw, touch and push, and evaluate their ability to generalize to complex tasks based on these low-level capabilities. 

Tasks are organized into three levels (L0 → L1 → L2), where higher-level tasks build on skills from lower levels to assess compositional reasoning.
Task item definition: A task prompt (plain text or multi-modal), a sequence of action labels (skill traces and language annotations), corresponding RGB observations from multiple camera views, key visual steps, and dense rewards over time. 

It represents one complete trajectory for solving a specific manipulation task.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 330k trajectories on 33 tasks
Dataset metadata available: Yes
Dataset metadata detail: For each of 330k trajectory, it contains many types of annotation, including language, action classes, bounding boxes for objects, visibility annotations, key-steps, rewards (for offline RL), camera parameters and more.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: A series of robot actions
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Per-task success rates
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Success rates on L0, L1 and L2 tasks
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Compositional
New taxonomy: General Capability / Compositional Reasoning