Learning to Grasp Anything By Playing with Random Toys

## Abstract

Abstract Robotic manipulation policies often struggle to generalize to novel objects, limiting their real-world utility.
In contrast, cognitive science suggests that children develop generalizable dexterous manipulation skills by mastering a small set of simple toys and then applying that knowledge to more complex items.
Inspired by this, we study if similar generalization capabilities can also be achieved by robots. Our results indicate robots can learn generalizable grasping using randomly assembled objects that are composed from just four shape primitives —spheres, cuboids, cylinders, and rings.
We show that training on these “toys” enables robust generalization to real-world objects , yielding strong zero-shot performance.
Crucially, we find the key to this generalization is an object-centric visual representation induced by our proposed detection pooling mechanism. Evaluated in both simulation and on physical robots, our model achieves a 67% real-world grasping success rate on the YCB dataset, outperforming state-of-the-art approaches that rely on substantially more in-domain data.
We further study how zero-shot generalization performance scales by varying the number and diversity of training toys and the demonstrations per toy.
We believe this work offers a promising path to scalable and generalizable learning in robotic manipulation. Demonstration videos, code, checkpoints and our dataset are available on our project page: https://lego-grasp.github.io/ .

## 1 Introduction

> *“Treat nature by means of the cylinder, the sphere, the cone, everything brought into proper perspective.”* Paul Cézanne

Robotic manipulation policies have recently achieved impressive progress, solving complex tasks in domains such as dexterous manipulation (Kumar et al., 2016; Chen et al., 2022; Wang et al., 2024; Chen et al., 2023; Qin et al., 2021), robust sim-to-real transfer (Chukwurah et al., 2024; Pinel et al., 2023; Ho et al., 2020), and long-horizon planning for multi-step tasks (Mishra et al., 2023; Simeonov et al., 2020; Pertsch et al., 2020). Yet, a fundamental challenge remains: they often fail to generalize their manipulation skills to novel objects, limiting their practical application. In stark contrast, humans show astonishing generalization capabilities in dexterous manipulation. For example, cognitive literature (Schneiberg et al., 2002; Oztop et al., 2004; Rochat, 1989; Thelen et al., 1993; Needham et al., 2002; Ruff, 1984; Bonaiuto & Arbib, 2015) suggests that children can learn to grasp by mastering only a small set of simple toys and then applying that skill to unseen complex objects. This raises a central question: can robotic manipulation policies generalize similarly?

In this work, we demonstrate that robots can learn to grasp novel real-world objects when trained only on randomly constructed toys.
The design of these toys is inspired by a classic insight from Cézanne: that complex objects can be deconstructed into a vocabulary of simple shape primitives. Specifically, we construct our toys as random compositions of just four shape primitives: spheres, cuboids, cylinders, and rings.
These “Cézanne toys” preserve the compositional structure of real objects while remaining sufficiently out-of-distribution, providing a challenging yet principled testbed for generalization.
Trained on these random toys, our policy learns to grasp complex, unseen real-world objects in a zero-shot manner.
See Figure [1](https://arxiv.org/html/2510.12866v1#S1.F1) for an overview.

The key to this generalization capability, as we empirically show, lies in the usage of object-centric visual representations.
Specifically, we introduce detection pooling (DetPool) to obtain an object-centric visual representation. This method first uses a mask of the target object to constrain the vision encoder’s attention to the object region, and then applies mean pooling on the output features corresponding to the object patches.
In this way, we ensure the final vision representation only contains information about the object and not the background or other distractors. We find this visual representation is the key to enable a grasping policy to generalize between the vastly different objects in training and testing. We name our framework LEGO (*LEarning to Grasp from tOys*).

Figure: Figure 1: Our grasping policy built from just four basic primitives (left), trained exclusively on random toy compositions (middle), zero-shot generalizes to real-world objects (right) and achieves an 67% success rate on 64 objects from the YCB dataset.
Refer to caption: x1.png

To evaluate our model’s generalization capabilities, we conduct a comprehensive experimental evaluation. First, we test its zero-shot performance: trained on a small dataset of 250 “Cézanne toys” with 1,500 demonstrations, our policy achieves a 67% success rate on 64 real-world YCB (Calli et al., 2015) objects, significantly outperforming larger, state-of-the-art models like OpenVLA-OFT (Kim et al., 2025) and $\pi_{0}$-FAST (Black et al., 2024; Pertsch et al., 2025) that are pretrained on much more data. Second, detailed ablations confirm that the key to this success is the object-centric representation induced by our DetPool mechanism, which significantly outperforms standard pooling baselines. Furthermore, we conduct thorough scaling experiments, finding that the zero-shot generalization performance scales with both toy diversity and the number of demonstrations, with the latter being more critical. Finally, we show this generalization capability is robust across robot diverse embodiments, including simple grippers and dexterous hands.

## 2 Related Work

Cognitive Approaches for Manipulation. Developmental psychology shows that infants acquire manipulation skills through exploration and practice (Thelen et al., 1993; Schneiberg et al., 2002; Needham et al., 2002). Several studies (Ruff, 1984; Rochat, 1989; Yoshida & Smith, 2008) have revealed a gradual progression in which they learn to manipulate objects by focusing on increasingly diverse features such as shape and color. Rakison & Butterworth (1998) further demonstrate that infants generalize to unseen objects by applying learned actions to familiar object parts. Motivated by this literature, we investigate achieving similar generalization in robotic manipulation.

Existing object modeling, inspired by infant learning (Farhadi et al., 2009), uses descriptive attributes (Cohen et al., 2019; Sun et al., 2013), explicit segmentation (Liu et al., 2024; Li et al., 2024a; b; Vahrenkamp et al., 2016; Aleotti & Caselli, 2011), or represents objects as 3D primitives (Tulsiani et al., 2016; Monnier et al., 2023; Lin et al., 2025). Our work builds on these ideas and explores whether generalized object representations can emerge from just a few primitives.

Generalization in Robotic Manipulation. While several robotic manipulation models have shown capability of mastering various real-world tasks (Zhao et al., 2023; Fu et al., 2024; Barreiros et al., 2025), they often generalize poorly to novel objects and scenes. One common approach to address this is scaling up the training data (Brohan et al., 2022; Zitkovich et al., 2023; Intelligence et al., 2025; Eppner et al., 2021; Fang et al., 2020; Ye et al., 2025). In contrast, we show that strong generalization capabilities are still achievable even with a few hours of training data. Another line of works improves the generalizability of robotic models through heavy data augmentation (Hansen & Wang, 2021; Tobin et al., 2017; Sadeghi & Levine, 2016), while we achieve strong generalization without any augmentation. Other works enhance transferability by learning better visual representations (Burns et al., 2023; Srirama et al., 2024). Compared to these approaches, which usually require costly visual pretraining, we improve generalizability by obtaining object-centric visual representations via DetPool, which is light-weight and can be applied to any pretrained vision model.

Object-Centric Vision Models.
Object-centric models represent a scene as a collection of discrete object-level entities. They are shown to improve performance and robustness in computer vision. Methods like Burgess et al. (2019); Engelcke et al. (2019); Locatello et al. (2020) learn object-centric representations from 2D images, typically for scene decomposition. Extending object-centric learning to the temporal domain, Herzig et al. (2021) introduce an object-region transformer that learns temporally coherent object-level features across video frames for tasks such as action recognition and dynamics prediction. Other approaches extend this idea into the 3D domain via world models (Ferraro et al., 2023; Jeong et al., 2025). Unlike these works, our approach focuses on the effect of object-centric representations on generalizable robotic manipulation.

When it comes to manipulation, a variety of object-centric grasping approaches have been explored (Chen et al., 2024; Zurbrügg et al., 2024; Mandikal & Grauman, 2020). Papers such as Devin et al. (2017) address generalizable robot learning through object-centric methods by using attention mechanisms. However, they only evaluate the generalization between similar objects and do not explore the limit of generalization between vastly different objects such as real objects. The most similar approach to ours is OTTER (Huang et al., 2025), which uses the vision-language attention map in CLIP to obtain object-centric visual representations. However, it is only limited to CLIP, while our method can be applied to any vision transformer.

## 3 A Cézanne Toy Grasping Dataset

Figure: Figure 2: Our Cézanne toys are composed of different number of primitives. We generate each toy by randomly assembling 1-5 primitives and randomizing dimensions and colors.
Refer to caption: x2.png

To evaluate the generalization of robotic grasping policies, we explore a challenging zero-shot setting: training policies exclusively on a set of out-of-distribution (OOD) objects and testing on common real-world objects. To this end, we develop a systematic approach for generating a diverse set of random, OOD objects. We draw inspiration from Cézanne’s classic idea that complex objects can be abstracted into compositions of simple shape primitives. We thus generate our training objects by randomly combining these primitives.
This process efficiently creates a training set of “Cézanne toys” composed of random primitives, which ensures they are OOD, while still retaining structural and compositional properties that enable generalization.
An overview is presented in Figure [1](https://arxiv.org/html/2510.12866v1#S1.F1). Next, we detail our primitives’ designs, the toy generation process, and the resulting grasping dataset.

Designing the Primitives. Inspired by prior literature (Marr & Nishihara, 1978; Tulsiani et al., 2016; Li et al., 2019), we choose four primitive types: spheres, cuboids, cylinders, and rings (See the left column of Figure [2](https://arxiv.org/html/2510.12866v1#S3.F2)). The primitive’s scale is randomized within specific ranges. Cuboids range from 2–7.2 cm in width, 1–20 cm in height, and 2–28 cm in length; spheres range from 1–8 cm in diameter; cylinders range from 4–7 cm in diameter and 4–12 cm in height; and rings range from 6–20 cm in diameter, 0.6–1.8 cm in wall thickness, and 2–6 cm in height.

Generating Cézanne Toys. We generate Cézanne toys by randomly combining the primitives. Figure [2](https://arxiv.org/html/2510.12866v1#S3.F2) illustrates some examples of the generated toys. We start by choosing a random number of primitives, ranging from 1 to 5. We then randomly select the required number of primitives from the four basic types, allowing repetitions, and the dimensions of each instance are randomized. The sampled primitives are then sequentially assembled to form the final toy. Specifically, the first primitive is placed at the origin, and the centroid of each subsequent primitive is randomly positioned within a previous primitive. This ensures the primitives overlap and form a coherent structure rather than scattered components. Each primitive is also assigned a random 3D rotation. Finally, the toy is assigned one of four colors: blue, red, green or yellow. By repeating this process, we generate a training set of 250 diverse toys, including 27 made of two primitives, 35 of three, 38 of four, and 47 of five, as well as individual primitives such as 46 cuboids, 18 balls, 20 cylinders, and 19 rings. All toys are both simulated and 3D printed for grasping data collection.

Collecting Grasping Data. We collect toy grasping trajectories in both simulation and real. In simulation, we use ManiSkill (Tao et al., 2025) with a Franka arm and gripper; in the real world, we use the same Franka arm with a Robotiq gripper, as well as a Unitree H1-2 humanoid equipped with Inspire RH56DFTP hands. We collect all data via teleoperation, except for grasping individual primitives in simulation, which is performed using motion planning. During collection, we ensure a diverse set of grasping poses per object, since individual objects can be grasped in many different ways. We collect 2,500 trajectories in simulation, 1,500 on the real Franka, and 500 on the H1-2.

## 4 The LEGO Method

To enable a policy trained on our “Cézanne toys” to generalize to real-world objects, we introduce a novel object-centric approach. Our method’s key distinction from full-scene architectures is its use of a detection pooling mechanism to obtain an object-centric visual representation, which we empirically show is the key to robust generalization. This section details our preliminaries (Section [4.1](https://arxiv.org/html/2510.12866v1#S4.SS1)), full architecture (Section [4.2](https://arxiv.org/html/2510.12866v1#S4.SS2)), and the detection pooling method (Section [4.3](https://arxiv.org/html/2510.12866v1#S4.SS3)).

### 4.1 Preliminaries

Robotic Tasks. Robotic tasks can be represented as temporal sequences of observations and actions. The observations typically consist of visual observations $i_{1:T}$ and proprioceptive states $s_{1:T}$, where $T$ is the episode length, $i_{t}\in\mathbb{R}^{N\times H\times W\times 3}$ denotes the images captured by $N$ cameras at time step $t$, and $s_{t}\in\mathbb{R}^{d_{s}}$ is the proprioception (e.g., the joint positions) of the robot at step $t$. The actions $a_{1:T}\in\mathbb{R}^{d_{a}}$ represent how the robot commands its joints (e.g., target joint positions) at step $t$.

Policy Learning.
The training objective is to learn a policy that maps a history of the past $C$ steps—visual inputs $i_{t-C+1:t}$ and robot states $s_{t-C+1:t}$—to a future action sequence of length $K$, in order to successfully complete the task: $\pi(i_{t-C+1:t},s_{t-C+1:t})\to a_{t:t+K-1}$.

Figure: Figure 3: The LEGO architecture with DetPool. (a) LEGO uses a ViT with DetPool to extract features of the target object and uses a transformer to predict future actions based on the visual features and the proprioception. (b) The ViT extracts features that focus on the target object via DetPool which restrains the attention to the object patches using an attention mask and performs mean pooling on the output object patch tokens to get the final object-centric vision feature.
Refer to caption: x3.png

### 4.2 Architecture

Below, we describe the different components of our policy architecture as well as the training objective. Figure [3](https://arxiv.org/html/2510.12866v1#S4.F3) (a) illustrates the overall LEGO architecture.

Vision Encoder. Given the observations $i_{t-C+1:t}$ and $s_{t-C+1:t}$ from past $C$ steps, our model uses a vision encoder to encode each set of visual observations $i_{t}$ into visual embeddings $e_{t}^{1:N}$, where $e_{t}^{n}\in\mathbb{R}^{d_{e}}$ is the embedding of the $n$-th camera image at step $t$, and $d_{e}$ is the hidden dimension of the vision encoder. We use a pretrained MVP (Xiao et al., 2022) as the vision encoder. These resulting features are then input into a transformer-based architecture for further processing.

Transformer Policy. We use a transformer-based architecture as our main policy network. It first concatenates the visual embeddings $e_{t}^{1:N}$ and the proprioception $s_{t}$ along the channel dimension into a single token, and then projects it with an MLP.
The transformer backbone then takes the projected tokens from all past $C$ steps and predicts the concatenated actions $a_{t:t+K-1}$ for the next $K$ steps from the last token. The transformer is designed to have the same size as a ViT-B (Dosovitskiy et al., 2021) to get the best performance (see the ablation in Section [5.5](https://arxiv.org/html/2510.12866v1#S5.SS5)).

Training Objective. Following the regular behavior cloning algorithm, the training loss is the mean $\ell_{1}$ loss between the predicted actions $\hat{a}_{t:t+K-1}$ and the ground-truth actions $a_{t:t+K-1}$, i.e., $\mathcal{L}=\frac{1}{Kd_{a}}\|\hat{a}_{t:t+K-1}-a_{t:t+K-1}\|_{1}$.

To enable generalization to novel objects, we design the vision encoder to be object-centric through detection pooling, introduced in the next section.

### 4.3 Detection Pooling

We design a detection pooling mechanism in the vision encoder such that the extracted visual feature is focused on the object to be grasped, as shown in Figure [3](https://arxiv.org/html/2510.12866v1#S4.F3) (b). Specifically, we first obtain the object segmentation mask for each frame using SAM 2 (Ravi et al., 2024b). We then use the object mask to set the attention mask in the vision encoder such that there is no attention between object patch tokens and non-object patch tokens. In this way, we ensure that the object patch tokens only contain features from the object itself while ignoring features from non-object patch tokens. Note that this method still allows the vision encoder to understand where the object is in the scene due to the use of positional embeddings. At the end of the vision encoder, we obtain the object-centric visual feature by applying mean pooling on the object patch tokens, which is the final visual embedding we use for the policy model. We empirically find that DetPool is crucial for achieving strong zero-shot generalization compared to other pooling methods such as mean and attention pooling that do not restrict the attention mask within the ViT and only pool the final output tokens (Section [5.2](https://arxiv.org/html/2510.12866v1#S5.SS2)).

## 5 Experiments

We evaluate LEGO on the YCB object benchmark (Calli et al., 2015) using the ManiSkill simulator (Tao et al., 2025). For comparison, we include vision-language-action (VLA) models such as $\pi_{0}$-FAST and OpenVLA-OFT, which aim to generalize through large-scale pretraining. We further analyze how performance scales with the number of unique toys and demonstrations. Beyond simulation, we test LEGO on two real-world setups: a 7-DoF Franka Emika Panda with a 1-DoF Robotiq 2F-85 adaptive gripper, where evaluation is done on the YCB benchmark, and an Unitree H1-2 humanoid with Inspire dexterous hands, evaluated on a 13-object set of everyday items. We include demonstration videos in the supplementary materials that show our collected data and the corresponding evaluation settings.

### 5.1 Implementation Details

Model and Training Setup. LEGO is implemented using PyTorch (Paszke et al., 2019). Its architecture consists of a ViT-L encoder from MVP (Xiao et al., 2022) for feature extraction and a ViT-Base transformer backbone. The policy is conditioned on a history of $C=16$ timesteps to predict $K=16$ future actions. For our DetPool mechanism, we use SAM 2 (Ravi et al., 2024a) to obtain object masks for real-world images and use ground-truth masks in simulation. The model is trained on eight NVIDIA A6000 GPUs using the AdamW optimizer (Loshchilov & Hutter, 2019), and evaluated on a single A6000.

State and Action Parameterization. We parameterize the proprioceptive space using the joint angles of the robot arm used and a continuous gripper state (when applicable). This yields an 8-dimensional vector for the Franka setup, and a 40-dimensional vector for our H1-2 setup (which includes feedforward torques and finger joints). The model then conditions on state vectors from past timesteps and predicts action vectors for future timesteps, where state and action vectors are represented using absolute joint angles, rather than relative (delta) angles.

### 5.2 Simulation Evaluation

Experimental Setup. Our training set contains 2,500 demonstrations, comprising 10 successful grasps for each of our 250 unique toys. To analyze scaling laws, all models are also trained on subsets of this data. For evaluation, we use a set of 65 graspable objects from the YCB benchmark, selecting only those feasibly graspable by the Franka robot; each object is tested 16 times on a predefined grid, and we report the mean success rate across all trials.

Baselines. We compare LEGO (86M parameters) against two significantly larger, state-of-the-art VLAs that rely on large-scale pretraining: $\pi_{0}$-FAST (3B) (Black et al., 2024) and OpenVLA-OFT (7B) (Kim et al., 2025). $\pi_{0}$-FAST is a state-of-the-art VLA model trained on a large-scale robotics dataset. OpenVLA-OFT is a 7B-parameter VLA model pretrained on the Open-X Embodiment (OXE) dataset (Collaboration et al., 2023). Both models are fine-tuned on the same data as ours. To validate the contribution of our core DetPool mechanism, we also conduct ablation studies, replacing DetPool with standard alternatives like attention pooling, CLS pooling, and mean pooling.

Results. Our simulation results, summarized in Table [1](https://arxiv.org/html/2510.12866v1#S5.T1), highlight the superior generalization and scalability of LEGO compared to baselines. While LEGO’s performance scales reliably with more data—achieving a top success rate of 80% with 2,500 demonstrations, the state-of-the-art VLA baselines falter. We find that $\pi_{0}$-FAST is too data-hungry for the small dataset and struggles with a real-to-sim domain gap from its pretraining. Similarly, OpenVLA-OFT shows initial promise with 250–500 demonstrations but quickly overfits as more data is added, causing its performance to deteriorate. On the other hand, while attention pooling is the strongest baseline, it is still significantly outperformed by our DetPool mechanism. In contrast, DetPool enables robust and scalable generalization, underscoring the effectiveness of object-centric visual representation for generalizability.

**Table 1: Results of zero-shot grasping in simulation. We compare our model with state-of-the-art models (OpenVLA-OFT and $\pi_{0}$-FAST) finetuned on our dataset in simulation, as well as different pooling baselines. Our model outperforms the finetuned baselines in simulation, with our DetPool proving key to generalization by boosting performance 22-48% over other pooling baselines.**
| Method | # Demos |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 250 | 500 | 1000 | 1500 | 2000 | 2500 |  |
| OpenVLA-OFT (Kim et al., 2025) | 30.10 | 36.35 | 22.31 | 15.38 | 14.71 | 12.79 |
| $\pi_{0}$-FAST (Black et al., 2024) | 8.85 | 7.60 | 7.69 | 8.56 | 4.23 | 4.13 |
| Ours - Attn Pooling | 34.71 | 40.10 | 44.23 | 48.27 | 49.81 | 51.63 |
| Ours - CLS Pooling | 24.71 | 20.29 | 36.92 | 41.44 | 42.40 | 49.81 |
| Ours - Mean Pooling | 32.98 | 30.38 | 36.15 | 39.90 | 40.29 | 40.58 |
| Ours - Det Pooling | 56.63 | 68.17 | 71.15 | 74.62 | 76.83 | 80.00 |

### 5.3 Franka Robot Evaluation

Experimental Setup. For real-world experiments, we use a 7-DoF Franka Emika Panda arm with a Robotiq 2F-85 gripper, consistent with the DROID benchmark. We 3D-print the 250 toys with the highest simulated success rates in simulation and collect 1,500 successful grasp demonstrations. All models are then evaluated on a test set of 64 YCB objects. Following the simulation protocol, each object is tested 16 times on a predefined grid, and we report the mean success rate.

Baselines.
We compare LEGO with strong baselines including $\pi_{0}$-FAST, OpenVLA-OFT, and ShapeGrasp (Li et al., 2024b) which is a training-free, LLM-based approach that uses pretrained language models to decompose objects geometrically before selecting a graspable part.

A common theme across these methods is reliance on large-scale pretraining: either extensive in-domain robot trajectories (for $\pi_{0}$-FAST) or internet-scale multimodal data (for ShapeGrasp). In contrast, LEGO is trained from scratch on only 1,500 demonstrations, yet achieves competitive performance despite being orders of magnitude smaller in both dataset size and model scale.

Results. As shown in Table [2](https://arxiv.org/html/2510.12866v1#S5.T2), LEGO achieves the second-best performance among all models tested, highlighting the effectiveness of our approach. It outperforms OpenVLA-OFT and ShapeGrasp, as well as $\pi_{0}$-FAST in its zero-shot setting, despite these being much larger models and $\pi_{0}$-FAST using in-domain DROID data. This also shows the strength of our object-centric representation, as LEGO attains superior performance using far less data and a smaller model architecture.

The finetuned version of $\pi_{0}$-FAST achieves the best overall performance, which we hypothesize is because finetuning on additional demonstrations from our DROID setup allows it to utilize its pretrained knowledge and adapt to the specific lighting and physical environment, improving performance. In addition, finetuning on the randomized toys dataset likely provides an additional performance boost. In contrast, OpenVLA-OFT is less effective. We observed that minor inaccuracies frequently caused grasp failures, indicating difficulty in generalizing to novel objects and settings.

**Table 2: Zero-shot grasping results on the real Franka robot. We compare our model against ShapeGrasp, OpenVLA-OFT (finetuned), and state-of-the-art $\pi_{0}$-FAST (zero-shot and finetuned). Our model achieves a 66.67% success rate, outperforming all baselines except finetuned $\pi_{0}$-FAST.**
| Method | Pretraining | Tuned on Toys | # Parameters | # Demos |
| --- | --- | --- | --- | --- |
| 1500 |  |  |  |  |
| OpenVLA-OFT (Kim et al., 2025) | OXE |  | 7B | 9.47 |
| $\pi_{0}$-FAST (Black et al., 2024) | $\pi$ Dataset + 75K DROID |  | 3B | 61.82 |
| $\pi_{0}$-FAST (Black et al., 2024) | $\pi$ Dataset + 75K DROID |  | 3B | 76.56 |
| ShapeGrasp (Li et al., 2024b) | GPT4o |  | - | 26.56 |
| Ours |  |  | 86M | 66.67 |

### 5.4 H1-2 Dexterous Hands Evaluation

Experimental Setup.
We perform real-world experiments with the Unitree H1-2, a humanoid robot. Each 7-DoF arm is equipped with a 6-DoF Inspire RH56DFTP hand, which has 12 total joints. The 6 DoFs capture the independent motions of the fingers, while the 12 joints result from each DoF being implemented as a pair of mechanically linked joints driven by a single linear servo. This hand design mimics human-like dexterity more closely than traditional gripper end-effectors, making it well-suited for experiments that require fine-grained grasping.

We evaluate our model and baselines on 13 everyday objects using the left arm and hand. Each object is tested 5 times across a predefined grid, with trials scored identically to the Franka experiments.

Baselines. We compare LEGO with OpenVLA-OFT and $\pi_{0}$-FAST (see Section [5.3](https://arxiv.org/html/2510.12866v1#S5.SS3) for details).

Results. As shown in Table [3](https://arxiv.org/html/2510.12866v1#S5.T3), LEGO achieves the highest success rate of 50.77% in a more challenging setting than the Franka DROID experiments, despite being trained from scratch with only 500 demonstrations.
In contrast, $\pi_{0}$-FAST struggles due to limited demonstrations and the likely absence of this embodiment in its pretraining data. OpenVLA-OFT also underperforms, having been pretrained on robot arm and gripper data without humanoid or dexterous-hand examples. These results underscore LEGO ’s data efficiency and the role of DetPool in enabling robust generalization.

**Table 3: Results on H1-2 humanoid robot. We compare our model with state-of-the-art models ($\pi_{0}$-Fast and OpenVLA-OFT) that are finetuned on our data. We show our model achieves superior performance without any pretraining on real objects.**
| Method | Bell<br>Pepper | Pink<br>Cube | Baton<br>Cookies | Solder<br>Coil | Tomato | Pink<br>Ribbed Ball | Piggles<br>Stuffed Toy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OpenVLA-OFT | 0 | 0 | 40 | 20 | 20 | 0 | 40 |
| $\pi_{0}$-FAST | 20 | 20 | 0 | 20 | 20 | 20 | 40 |
| Ours | 60 | 40 | 60 | 40 | 60 | 60 | 60 |
|  | Mike Wazowski<br>Stuffed Toy | Red<br>Tape Dispenser | Red<br>Solo Cup | Paper<br>Towel Roll | Hand<br>Sanitizer | Yo-Yo | Average |
| OpenVLA-OFT | 60 | 0 | 0 | 60 | 0 | 0 | 18.46 |
| $\pi_{0}$-FAST | 60 | 40 | 40 | 40 | 20 | 0 | 26.15 |
| Ours | 60 | 60 | 20 | 60 | 60 | 20 | 50.77 |

### 5.5 Ablation Studies

We present our ablation studies below, conducted within the ManiSkill simulation environment, which offers a stable and reproducible setting for rigorously evaluating the impact of various factors.

Figure: Figure 4: Scaling studies. Left: The zero-shot success rate scales with both the number of demos and the number of unique toys. We also find that once the number of demos is sufficient, 25 toys is already enough to achieve a robust zero-shot transfer. Right: The performances scales with the size of the policy transformer until it saturates at the size of 86M.
Refer to caption: x4.png

Effect of Number and Diversity of Demonstrations.
We perform an ablation study to examine how both the number of unique toys in the training set and the number of grasping demonstrations influence performance. Specifically, we construct six object sets containing 1, 25, 125, 250, 500, and 1000 unique toys, respectively. For each set, we collect 2,500 total grasping demonstrations and train our model using varying numbers of demonstrations per set. The results (left panel, Figure [4](https://arxiv.org/html/2510.12866v1#S5.F4)) indicate that increasing the number of unique objects improves performance, but with diminishing returns. In contrast, the number of demonstrations has a stronger impact on learning generalizable grasping, a result consistent with findings from cognitive science literature (Smith & Slone, 2017).

Effect of Model Size.
To investigate how the size of the policy’s transformer backbone affects performance, we conduct an ablation study. Using the 250-object set—which yields the best overall performance—we vary the transformer’s size and evaluate the policy across different numbers of demonstrations. The results, shown in the right panel of Figure [4](https://arxiv.org/html/2510.12866v1#S5.F4), indicate that ViT-Base (86M) is the best overall choice: it matches or slightly surpasses ViT-Large (307M) in performance while being significantly smaller and thus allowing for faster inference.

Importance of Individual Primitives.
To assess the relative importance of each of the four primitives, we conduct an ablation study in which the training set excludes toys containing a given primitive. For each case, the model is trained with varying numbers of demonstrations. The results, presented in Table [4](https://arxiv.org/html/2510.12866v1#S5.T4), show that the sphere is the most critical primitive, as its exclusion results in the largest performance degradation. In contrast, the ring and cylinder appear to be less important, with a relatively small performance drop when they are omitted.

Effect of Toy Complexity.
We perform an ablation study to measure the relationship between toy complexity, quantified by the number of constituent primitives, and model performance. The model is trained on demonstrations containing toys with 2-5 primitives. As shown in Table [5](https://arxiv.org/html/2510.12866v1#S5.T5), toys with two primitives contribute the most to performance, while toys with five primitives are still beneficial but less influential. This is likely due to the evaluation set’s distribution of sizes, which contains more toys with two or three distinct parts; highly complex toys with five parts are relatively rare.

**Table 4: Ablation of primitive types. We study the importance of each primitive type by removing each one from the primitive set, and report the mean success rate on the YCB object set for different numbers of total demonstrations.**
| Primitive<br>Removed | 100 | 200 | 500 | 1000 |
| --- | --- | --- | --- | --- |
| Cuboid | 37.88 | 56.35 | 65.38 | 72.12 |
| Sphere | 44.13 | 47.31 | 61.83 | 63.08 |
| Ring | 44.23 | 67.5 | 68.56 | 72.6 |
| Cylinder | 45.29 | 57.6 | 69.52 | 72.31 |

## 6 Conclusion

In this work, we demonstrate that robots can acquire robust general-purpose grasping skills by learning from a simple set of objects composed from just four basic shape primitives: spheres, cuboids, cylinders, and rings. We show that training on these toys enables a policy to generalize to a wide range of real-world objects. Our method learns an object-centric visual representation using a detection pooling and transformer architecture, and is trained on a dataset of 250 toys with 1,500 demonstrations in the real Franka setting. This policy achieves a 67% zero-shot success rate on the YCB dataset, outperforming state-of-the-art models such as $\pi_{0}$-FAST and OpenVLA-OFT despite them being trained on more diverse and larger datasets. Our findings on grasping scaling laws highlight how we can efficiently optimize performance with limited data. Ultimately, this work demonstrates a scalable path to robotic manipulation by showing that real-world grasping generalization can emerge from learning on object composites of a few primitive shapes. We believe this work offers a promising path to scalable and generalizable learning in robotic manipulation.

## 7 Limitations and Future Work

While we show that our method offers a promising path toward generalized grasping, it is important to acknowledge its limitations to guide future research. One key limitation is the diversity of the training domain: the model’s performance may degrade on objects with different physical properties. Furthermore, our current work focuses on simple, single-step grasping. Future work could extend this approach to complex, long-horizon tasks such as cloth folding and manipulation in dynamic scenes. Finally, the computational cost of the model’s architecture presents a challenge for real-world deployment on resource-constrained hardware, pointing toward a need for future optimization.

## Acknowledgments

We would like to thank Caitlin Regan for setting up teleoperation on the Unitree H1-2, Hunter Tsai for his efforts in building a 3D printing farm and supplying the robot toys for real experiments, and Raj Saravanan for helping with initial toy design and data collection with teleoperation. We would like to thank Ilija Radosavovic for helpful discussions and feedback. We also thank Nicole Walters for creating our lovely logo.
Authors, as part of their affiliation with UC Berkeley, were supported in part by the Defense Advanced Research Projects Agency (DARPA) under the TIAMAT program (HR00112490425), Design of Robustly Implementable Autonomous and Intelligent Machines, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program, as well as the Humanoid Intelligence Center program.
Matteo Gioia conducted this work while enrolled in the Data Science Doctorate at Sapienza University of Rome, funded by the European Union—NextGenerationEU, Mission 4, Component 1 (CUP B53C22003870006). Matteo Gioia also acknowledges partial support from Sapienza grant RM1241910E01F571 (V3LI).

## References

- Aleotti & Caselli (2011)
Jacopo Aleotti and Stefano Caselli.
Manipulation planning of similar objects by part correspondence.
*2011 15th International Conference on Advanced Robotics (ICAR)*, pp. 247–252, 2011.
URL [https://api.semanticscholar.org/CorpusID:18127179](https://api.semanticscholar.org/CorpusID:18127179).
- Barreiros et al. (2025)
Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al.
A careful examination of large behavior models for multitask dexterous manipulation.
*arXiv preprint arXiv:2507.05331*, 2025.
- Black et al. (2023)
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine.
Zero-shot robotic manipulation with pretrained image-editing diffusion models.
*arXiv preprint arXiv:2310.10639*, 2023.
- Black et al. (2024)
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.
$\pi$0: A vision-language-action flow model for general robot control.
*ArXiv*, abs/2410.24164, 2024.
URL [https://api.semanticscholar.org/CorpusID:273811174](https://api.semanticscholar.org/CorpusID:273811174).
- Bonaiuto & Arbib (2015)
James J. Bonaiuto and Michael A. Arbib.
Learning to grasp and extract affordances: the integrated learning of grasps and affordances (ilga) model.
*Biological Cybernetics*, 109:639 – 669, 2015.
URL [https://api.semanticscholar.org/CorpusID:17366201](https://api.semanticscholar.org/CorpusID:17366201).
- Brohan et al. (2022)
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.
Rt-1: Robotics transformer for real-world control at scale.
*arXiv preprint arXiv:2212.06817*, 2022.
- Burgess et al. (2019)
Christopher P. Burgess, Loïc Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner.
Monet: Unsupervised scene decomposition and representation.
*ArXiv*, abs/1901.11390, 2019.
URL [https://api.semanticscholar.org/CorpusID:59523721](https://api.semanticscholar.org/CorpusID:59523721).
- Burns et al. (2023)
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman.
What makes pre-trained visual representations successful for robust manipulation?
*arXiv preprint arXiv:2312.12444*, 2023.
- Calli et al. (2015)
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.
The ycb object and model set: Towards common benchmarks for manipulation research.
In *2015 international conference on advanced robotics (ICAR)*, pp. 510–517. IEEE, 2015.
- Chen et al. (2024)
Hanzhi Chen, Binbin Xu, and Stefan Leutenegger.
Funcgrasp: Learning object-centric neural grasp functions from single annotated example object.
*2024 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 1900–1906, 2024.
URL [https://api.semanticscholar.org/CorpusID:267547826](https://api.semanticscholar.org/CorpusID:267547826).
- Chen et al. (2022)
Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuan Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, and Song-Chun Zhu.
Towards human-level bimanual dexterous manipulation with reinforcement learning.
*ArXiv*, abs/2206.08686, 2022.
URL [https://api.semanticscholar.org/CorpusID:249848184](https://api.semanticscholar.org/CorpusID:249848184).
- Chen et al. (2023)
Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jiechuan Jiang, Zongqing Lu, Hao Dong, and Yaodong Yang.
Bi-dexhands: Towards human-level bimanual dexterous manipulation.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 46:2804–2818, 2023.
URL [https://api.semanticscholar.org/CorpusID:265801841](https://api.semanticscholar.org/CorpusID:265801841).
- Chukwurah et al. (2024)
Naomi Chukwurah, Abiodun Sunday Adebayo, and Olanrewaju Oluwaseun Ajayi.
Sim-to-real transfer in robotics: Addressing the gap between simulation and real-world performance.
*Journal of Frontiers in Multidisciplinary Research*, 2024.
URL [https://api.semanticscholar.org/CorpusID:276984724](https://api.semanticscholar.org/CorpusID:276984724).
- Cohen et al. (2019)
Vanya Cohen, Benjamin Burchfiel, Thao Nguyen, Nakul Gopalan, Stefanie Tellex, and George Dimitri Konidaris.
Grounding language attributes to objects using bayesian eigenobjects.
*2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pp. 1187–1194, 2019.
URL [https://api.semanticscholar.org/CorpusID:170078764](https://api.semanticscholar.org/CorpusID:170078764).
- Collaboration et al. (2023)
Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue
Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan
Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi ”Jim” Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick ”Tree” Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart’in-Mart’in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav
Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao,
Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin.
Open X-Embodiment: Robotic learning datasets and RT-X models.
[https://arxiv.org/abs/2310.08864](https://arxiv.org/abs/2310.08864), 2023.
- Devin et al. (2017)
Coline Devin, P. Abbeel, Trevor Darrell, and Sergey Levine.
Deep object-centric representations for generalizable robot learning.
*2018 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 7111–7118, 2017.
URL [https://api.semanticscholar.org/CorpusID:33758357](https://api.semanticscholar.org/CorpusID:33758357).
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
*ICLR*, 2021.
- Engelcke et al. (2019)
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner.
Genesis: Generative scene inference and sampling with object-centric latent representations.
*ArXiv*, abs/1907.13052, 2019.
URL [https://api.semanticscholar.org/CorpusID:198986015](https://api.semanticscholar.org/CorpusID:198986015).
- Eppner et al. (2021)
Clemens Eppner, Arsalan Mousavian, and Dieter Fox.
Acronym: A large-scale grasp dataset based on simulation.
In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 6222–6227. IEEE, 2021.
- Fang et al. (2020)
Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu.
Graspnet-1billion: A large-scale benchmark for general object grasping.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11444–11453, 2020.
- Farhadi et al. (2009)
Ali Farhadi, Ian Endres, Derek Hoiem, and David Alexander Forsyth.
Describing objects by their attributes.
*2009 IEEE Conference on Computer Vision and Pattern Recognition*, pp. 1778–1785, 2009.
URL [https://api.semanticscholar.org/CorpusID:14940757](https://api.semanticscholar.org/CorpusID:14940757).
- Ferraro et al. (2023)
Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, and B. Dhoedt.
Focus: Object-centric world models for robotics manipulation.
*ArXiv*, abs/2307.02427, 2023.
URL [https://api.semanticscholar.org/CorpusID:259342267](https://api.semanticscholar.org/CorpusID:259342267).
- Fu et al. (2024)
Zipeng Fu, Tony Z Zhao, and Chelsea Finn.
Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.
*arXiv preprint arXiv:2401.02117*, 2024.
- Hansen & Wang (2021)
Nicklas Hansen and Xiaolong Wang.
Generalization in reinforcement learning by soft data augmentation.
In *2021 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 13611–13617. IEEE, 2021.
- Herzig et al. (2021)
Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson.
Object-region video transformers.
*2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 3138–3149, 2021.
URL [https://api.semanticscholar.org/CorpusID:238744000](https://api.semanticscholar.org/CorpusID:238744000).
- Ho et al. (2020)
Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai.
Retinagan: An object-aware approach to sim-to-real transfer.
*2021 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 10920–10926, 2020.
URL [https://api.semanticscholar.org/CorpusID:226278453](https://api.semanticscholar.org/CorpusID:226278453).
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2021.
- Huang et al. (2025)
Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, and Pieter Abbeel.
Otter: A vision-language-action model with text-aware visual feature extraction.
*arXiv preprint arXiv:2503.03734*, 2025.
- Intelligence et al. (2025)
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al.
$pi_{0}.5$: a vision-language-action model with open-world generalization.
*arXiv preprint arXiv:2504.16054*, 2025.
- Jeong et al. (2025)
Youngjoon Jeong, Junha Chun, Soonwoo Cha, and Taesup Kim.
Object-centric world model for language-guided manipulation.
*ArXiv*, abs/2503.06170, 2025.
URL [https://api.semanticscholar.org/CorpusID:276903201](https://api.semanticscholar.org/CorpusID:276903201).
- Khazatsky et al. (2024)
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, P Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Ye Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sung Yul Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean-Pierre Mercat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, C. Blake Simpson, Quang Uyen Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Da Ling Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel
Morton, Tony Nguyen, Abigail O’Neill, Rosa Maria Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Mart’in-Mart’in, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn.
Droid: A large-scale in-the-wild robot manipulation dataset.
*ArXiv*, abs/2403.12945, 2024.
URL [https://api.semanticscholar.org/CorpusID:268531351](https://api.semanticscholar.org/CorpusID:268531351).
- Kim et al. (2024)
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn.
Openvla: An open-source vision-language-action model.
*ArXiv*, abs/2406.09246, 2024.
URL [https://api.semanticscholar.org/CorpusID:270440391](https://api.semanticscholar.org/CorpusID:270440391).
- Kim et al. (2025)
Moo Jin Kim, Chelsea Finn, and Percy Liang.
Fine-tuning vision-language-action models: Optimizing speed and success.
*arXiv preprint arXiv:2502.19645*, 2025.
- Kumar et al. (2016)
Vikash Kumar, Emanuel Todorov, and Sergey Levine.
Optimal control with learned local models: Application to dexterous manipulation.
*2016 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 378–383, 2016.
URL [https://api.semanticscholar.org/CorpusID:7586242](https://api.semanticscholar.org/CorpusID:7586242).
- Li et al. (2024a)
Haosheng Li, Weixin Mao, Weipeng Deng, Chenyu Meng, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Hongan Wang, and Xiaoming Deng.
Seggrasp: Zero-shot task-oriented grasping via semantic and geometric guided segmentation.
*ArXiv*, abs/2410.08901, 2024a.
URL [https://api.semanticscholar.org/CorpusID:273323195](https://api.semanticscholar.org/CorpusID:273323195).
- Li et al. (2019)
Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas J Guibas.
Supervised fitting of geometric primitives to 3d point clouds.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 2652–2660, 2019.
- Li et al. (2024b)
Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia P. Sycara, and Simon Stepputtis.
Shapegrasp: Zero-shot task-oriented grasping with large language models through geometric decomposition.
*2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pp. 10527–10534, 2024b.
URL [https://api.semanticscholar.org/CorpusID:268723780](https://api.semanticscholar.org/CorpusID:268723780).
- Lin et al. (2025)
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu.
Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids.
*arXiv preprint arXiv:2502.20396*, 2025.
- Liu et al. (2024)
Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans, Animesh Garg, and Jiajun Wu.
Composable part-based manipulation.
*ArXiv*, abs/2405.05876, 2024.
URL [https://api.semanticscholar.org/CorpusID:265154271](https://api.semanticscholar.org/CorpusID:265154271).
- Locatello et al. (2020)
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.
Object-centric learning with slot attention.
*ArXiv*, abs/2006.15055, 2020.
URL [https://api.semanticscholar.org/CorpusID:220127924](https://api.semanticscholar.org/CorpusID:220127924).
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations (ICLR)*, 2019.
URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7).
- Mandikal & Grauman (2020)
Priyanka Mandikal and Kristen Grauman.
Learning dexterous grasping with object-centric visual affordances.
*2021 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 6169–6176, 2020.
URL [https://api.semanticscholar.org/CorpusID:233439776](https://api.semanticscholar.org/CorpusID:233439776).
- Marr & Nishihara (1978)
David Marr and Herbert Keith Nishihara.
Representation and recognition of the spatial organization of three-dimensional shapes.
*Proceedings of the Royal Society of London. Series B. Biological Sciences*, 200(1140):269–294, 1978.
- Mishra et al. (2023)
Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu.
Generative skill chaining: Long-horizon skill planning with diffusion models.
In *Conference on Robot Learning*, 2023.
URL [https://api.semanticscholar.org/CorpusID:261685884](https://api.semanticscholar.org/CorpusID:261685884).
- Monnier et al. (2023)
Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros, and Mathieu Aubry.
Differentiable blocks world: Qualitative 3d decomposition by rendering primitives.
*ArXiv*, abs/2307.05473, 2023.
URL [https://api.semanticscholar.org/CorpusID:259766537](https://api.semanticscholar.org/CorpusID:259766537).
- Needham et al. (2002)
Amy Work Needham, Tracy M. Barrett, and Karen Peterman.
A pick-me-up for infants’ exploratory skills: Early simulated experiences reaching for objects using ‘sticky mittens’ enhances young infants’ object exploration skills.
*Infant Behavior & Development*, 25:279–295, 2002.
URL [https://api.semanticscholar.org/CorpusID:16427970](https://api.semanticscholar.org/CorpusID:16427970).
- Oztop et al. (2004)
Erhan Oztop, Nina S. Bradley, and Michael A. Arbib.
Infant grasp learning: a computational model.
*Experimental Brain Research*, 158:480–503, 2004.
URL [https://api.semanticscholar.org/CorpusID:8738077](https://api.semanticscholar.org/CorpusID:8738077).
- Paszke et al. (2019)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
*Advances in neural information processing systems*, 32, 2019.
- Pertsch et al. (2020)
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine.
Long-horizon visual planning with goal-conditioned hierarchical predictors.
*ArXiv*, abs/2006.13205, 2020.
URL [https://api.semanticscholar.org/CorpusID:219981151](https://api.semanticscholar.org/CorpusID:219981151).
- Pertsch et al. (2025)
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine.
Fast: Efficient action tokenization for vision-language-action models.
*arXiv preprint arXiv:2501.09747*, 2025.
- Pinel et al. (2023)
Ricardo Garcia Pinel, Robin Strudel, Shizhe Chen, Etienne Arlaud, Ivan Laptev, and Cordelia Schmid.
Robust visual sim-to-real transfer for robotic manipulation.
*2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pp. 992–999, 2023.
URL [https://api.semanticscholar.org/CorpusID:260315942](https://api.semanticscholar.org/CorpusID:260315942).
- Qin et al. (2021)
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang.
Dexmv: Imitation learning for dexterous manipulation from human videos.
In *European Conference on Computer Vision*, 2021.
URL [https://api.semanticscholar.org/CorpusID:236986915](https://api.semanticscholar.org/CorpusID:236986915).
- Radosavovic et al. (2023)
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik.
Robot learning with sensorimotor pre-training.
In *Conference on Robot Learning*, pp. 683–693. PMLR, 2023.
- Rakison & Butterworth (1998)
David H. Rakison and George Butterworth.
Infants’ use of object parts in early categorization.
*Developmental Psychology*, 34:49–62, 1998.
URL [https://api.semanticscholar.org/CorpusID:210399125](https://api.semanticscholar.org/CorpusID:210399125).
- Ravi et al. (2024a)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer.
Sam 2: Segment anything in images and videos.
*arXiv preprint arXiv:2408.00714*, 2024a.
URL [https://arxiv.org/abs/2408.00714](https://arxiv.org/abs/2408.00714).
- Ravi et al. (2024b)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al.
Sam 2: Segment anything in images and videos.
*arXiv preprint arXiv:2408.00714*, 2024b.
- Rochat (1989)
Philippe Rochat.
Object manipulation and exploration in 2- to 5-month-old infants.
*Developmental Psychology*, 25:871–884, 1989.
URL [https://api.semanticscholar.org/CorpusID:197658959](https://api.semanticscholar.org/CorpusID:197658959).
- Ruff (1984)
Holly A. Ruff.
Infants’ manipulative exploration of objects: Effects of age and object characteristics.
*Developmental Psychology*, 20:9–20, 1984.
URL [https://api.semanticscholar.org/CorpusID:201316353](https://api.semanticscholar.org/CorpusID:201316353).
- Sadeghi & Levine (2016)
Fereshteh Sadeghi and Sergey Levine.
Cad2rl: Real single-image flight without a single real image.
*arXiv preprint arXiv:1611.04201*, 2016.
- Schneiberg et al. (2002)
Sheila Schneiberg, Heidi Sveistrup, Bradford J. McFadyen, P. Mckinley, and Mindy F. Levin.
The development of coordination for reach-to-grasp movements in children.
*Experimental Brain Research*, 146:142–154, 2002.
URL [https://api.semanticscholar.org/CorpusID:15714879](https://api.semanticscholar.org/CorpusID:15714879).
- Simeonov et al. (2020)
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Robert Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, and Alberto Rodriguez.
A long horizon planning framework for manipulating rigid pointcloud objects.
In *Conference on Robot Learning*, 2020.
URL [https://api.semanticscholar.org/CorpusID:226964745](https://api.semanticscholar.org/CorpusID:226964745).
- Smith & Slone (2017)
Linda B. Smith and Lauren K Slone.
A developmental approach to machine learning?
*Frontiers in Psychology*, 8, 2017.
URL [https://api.semanticscholar.org/CorpusID:22527330](https://api.semanticscholar.org/CorpusID:22527330).
- Srirama et al. (2024)
Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta.
Hrp: Human affordances for robotic pre-training.
*arXiv preprint arXiv:2407.18911*, 2024.
- Sun et al. (2013)
Yuyin Sun, Liefeng Bo, and Dieter Fox.
Attribute based object identification.
*2013 IEEE International Conference on Robotics and Automation*, pp. 2096–2103, 2013.
URL [https://api.semanticscholar.org/CorpusID:8413785](https://api.semanticscholar.org/CorpusID:8413785).
- Tao et al. (2025)
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su.
Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai.
*Robotics: Science and Systems*, 2025.
- Thelen et al. (1993)
Esther Thelen, Daniela Corbetta, Kathi Kamm, John P. Spencer, Klaus Schneider, and Ronald F. Zernicke.
The transition to reaching: mapping intention and intrinsic dynamics.
*Child development*, 64 4:1058–98, 1993.
URL [https://api.semanticscholar.org/CorpusID:7142337](https://api.semanticscholar.org/CorpusID:7142337).
- Tobin et al. (2017)
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.
Domain randomization for transferring deep neural networks from simulation to the real world.
In *2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)*, pp. 23–30. IEEE, 2017.
- Tulsiani et al. (2016)
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik.
Learning shape abstractions by assembling volumetric primitives.
*2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 1466–1474, 2016.
URL [https://api.semanticscholar.org/CorpusID:2380406](https://api.semanticscholar.org/CorpusID:2380406).
- Vahrenkamp et al. (2016)
Nikolaus Vahrenkamp, Leonard Westkamp, Natsuki Yamanobe, Eren Erdal Aksoy, and Tamim Asfour.
Part-based grasp planning for familiar objects.
*2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)*, pp. 919–925, 2016.
URL [https://api.semanticscholar.org/CorpusID:12049991](https://api.semanticscholar.org/CorpusID:12049991).
- Wang et al. (2024)
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Fei-Fei Li, and Karen Liu.
Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.
*ArXiv*, abs/2403.07788, 2024.
URL [https://api.semanticscholar.org/CorpusID:268363547](https://api.semanticscholar.org/CorpusID:268363547).
- Xiao et al. (2022)
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik.
Masked visual pre-training for motor control.
*arXiv preprint arXiv:2203.06173*, 2022.
- Ye et al. (2025)
Jianglong Ye, Keyi Wang, Chengjing Yuan, Ruihan Yang, Yiquan Li, Jiyue Zhu, Yuzhe Qin, Xueyan Zou, and Xiaolong Wang.
Dex1b: Learning with 1b demonstrations for dexterous manipulation.
*arXiv preprint arXiv:2506.17198*, 2025.
- Yoshida & Smith (2008)
Hanako Yoshida and Linda B. Smith.
What’s in view for toddlers? using a head camera to study visual experience.
*Infancy : the official journal of the International Society on Infant Studies*, 13 3:229–248, 2008.
URL [https://api.semanticscholar.org/CorpusID:18371897](https://api.semanticscholar.org/CorpusID:18371897).
- Zhao et al. (2023)
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.
Learning fine-grained bimanual manipulation with low-cost hardware.
*arXiv preprint arXiv:2304.13705*, 2023.
- Zitkovich et al. (2023)
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al.
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
In *Conference on Robot Learning*, pp. 2165–2183. PMLR, 2023.
- Zurbrügg et al. (2024)
René Zurbrügg, Yifan Liu, Francis Engelmann, Suryansh Kumar, Marco Hutter, Vaishakh Patil, and Fisher Yu.
Icgnet: A unified approach for instance-centric grasping.
*2024 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 4140–4146, 2024.
URL [https://api.semanticscholar.org/CorpusID:267034845](https://api.semanticscholar.org/CorpusID:267034845).

## Appendix A Toys Design

### A.1 Real 3D Toys Design and Manufacturing

Primitive Design. To design the toys, we wrote a Python script that uses the SAPIEN physics engine to generate random dimensions for a set of primitives in the amount desired, such as a cuboid and a cylinder for a two primitive toy. These primitives are assembled into a toy by placing them at random offsets between them that ensure the primitives are still physically connected to each other. Finally, we export the toy mesh into an STL file using the Trimesh library. We list out the dimension ranges of the primitives in Table [6](https://arxiv.org/html/2510.12866v1#A1.T6).

**Table 6: Dimension ranges for primitive shapes.**
| Shape | Diameter/Width (cm) | Height (cm) | Length (cm) |
| --- | --- | --- | --- |
| Cuboid | 2–7.2 | 1–20 | 2–28 |
| Sphere | 1–8 | N/A | N/A |
| Cylinder | 4–7 | 4–12 | N/A |
| Ring | 6–20 | 2–6 | 0.6–1.8 (wall thickness) |

Toy Manufacturing. We printed a total of 250 toys in PLA filament, in addition to multiple test prints to validate the toy geometry and print quality. This was done using a fleet of eight Bambu P1P printers over a span of four weeks, enabling a maximum throughput of 200 toys per week by printing multiple toys on a single print bed (excluding FivePrimitive toys, whose size meant that they took up the entire print bed and took significantly longer to print). The fleet was managed using the Bambu Farm Manager platform.

The biggest challenge with printing the toys was the delicate geometry of the rings. The original designs had very thin ring walls that would snap during removal from the print bed. To compensate, we redesigned the toys to have thicker ring walls to strengthen the print. In addition, the intersection of shape primitives often resulted in large overhanging bodies, which required large amounts of tree supports to be modelled and printed. Toys with larger primitive counts had significantly higher print times due to their increased volume and complexity. Certain FivePrimitive toys had to be scaled down in size by 20% to fit in the 256mm x 256mm x 256mm print volume.

We have provided the full Bambu printer settings used for our prints for ease of reproducibility in Tables [8](https://arxiv.org/html/2510.12866v1#A4.T8), [9](https://arxiv.org/html/2510.12866v1#A4.T9), [10](https://arxiv.org/html/2510.12866v1#A4.T10), and [11](https://arxiv.org/html/2510.12866v1#A4.T11). Any omitted settings are assumed to take the default value. Organizing the toys into boxes and using a label printer to label them with their names is important for keeping track of all the toys, such as if a reprint is needed.

## Appendix B Additional Experiments

**Table 7: Effect of toy colors on zero-shot generalization. We compare the zero-shot performance of model trained on single-color toys with multi-color ones. Training on toys with multiple colors boost performance by about 1%-4% although training on single-color toys still yields a strong generalization to real objects.**
| Toy Colors | 250 | 500 | 1000 | 1500 | 2000 | 2500 |
| --- | --- | --- | --- | --- | --- | --- |
| Red | 50.1 | 66.44 | 68.94 | 72.6 | 75.48 | 76.35 |
| Red + Green + Blue + Yellow | 56.63 | 68.17 | 71.15 | 74.62 | 76.82 | 80 |

Effect of Color.
We measure the impact of toy color on performance by conducting an ablation study comparing a policy trained on a set of only red toys to our original set where toys were randomly assigned one of four colors (red, green, blue, yellow). As shown in Table [7](https://arxiv.org/html/2510.12866v1#A2.T7), color diversity improves performance. This is likely because exposure to toys with varying colors during training helps the model learn more robust visual features so it can generalize better to real-world objects.

## Appendix C Real Robot Hardware Configuration

### C.1 Franka Emika Panda

We deploy our policy on a Franka Panda Robot with 7 DoFs equipped with a RobotiQ gripper and a ZED 2i wrist camera. The 7 DoFs allow for precise and dexterous manipulation of the gripper to grasp various types of objects from every part. Two additional ZED 2i cameras are positioned to the left and right sides of the robot. Each camera provides an RGB stream at 720p and 30 FPS, without depth information. The hardware configuration is shown in Figure [5](https://arxiv.org/html/2510.12866v1#A3.F5).

Figure: Figure 5: Hardware Configuration for Franka Emika Panda with Robotiq Gripper.
Refer to caption: x5.png

### C.2 H1-2 Humanoid with Dexterous Hands

We also deploy our policy to a Unitree H1-2 humanoid robot. The robot is equipped with two Inspire RH56DFTP dexterous hands, each with 6 DoFs, 12 motors and a linear drive design with six miniature linear servo drives and six pressure sensors integrated inside. Given these characteristics, the hands are a good fit to emulate real dexterous operations by a human. The robot is also equipped with a ZED 2i head camera mounted below the original head camera to improve the quality of the egocentric data captured. Two ZED 2i cameras are positioned to the side of the robot, creating a similar setup to the one used for the Franka arm. Each camera provides an RGB stream at 720p resolution and 30 FPS, without depth information. The hardware configuration is shown in Figure [6](https://arxiv.org/html/2510.12866v1#A3.F6).

Figure: Figure 6: Hardware Configuration for H1-2 Humanoid with Inspire Dexterous Hands.
Refer to caption: x6.png

## Appendix D Robot Demonstrations Collection

### D.1 ManiSkill Simulation Motion Planning

ManiSkill (Tao et al., 2025) is a simulation environment built on the SAPIEN framework. We generated a dataset of demonstrations for a Franka arm grasping and lifting single primitive objects using scripted planners.

**Table 8: Print Quality Settings**
| Setting | Value |
| --- | --- |
| Layer Height | 0.3 mm |
| Initial Layer Height | 0.3 mm |
| Line Width (All) | 0.62 mm |
| Seam Position | Aligned |
| Smart Scarf Seam Application | On |
| Scarf Application Angle | 155^∘ |
| Scarf Steps | 10 |
| Scarf Joint for Inner Walls | On |
| Role-based Wipe Speed | On |
| Slice Gap Closing Radius | 0.049 mm |
| Resolution | 0.012 mm |
| Arc Fitting | On |
| Elephant Foot Comp. | 0.15 mm |
| Ironing Type | No Ironing |
| Initial Layer Density | 90% |

### D.2 ManiSkill Simulation Teleoperation

Using ManiSkill, we also designed a simulation environment to collect data via human teleoperation. The teleoperation data collection process was then standardized as follows: the arm was first positioned slightly above the target grasp pose, then moved down to the grasp position, its gripper was closed to secure the object, and the object was then lifted upward. While it is possible to fully automate the data generation pipeline using grasping planners for more complex toys, we encountered engineering challenges that ultimately led us to rely on teleoperation.

### D.3 Franka Real Robot Teleoperation

We teleoperated the Franka robot using a Meta Quest 3 headset, with only the right-hand controller mapped to arm control. Each pick-up demonstration was executed in one smooth motion on a foam-covered table to protect the objects. We recorded videos from the left and right ZED cameras as well as the wrist camera, and additionally logged the robot’s proprioceptive states. We adopt the Franka-DROID robot settings provided by the DROID dataset (Khazatsky et al., 2024).

### D.4 H1-2 with Dexterous Hands Teleoperation

To collect real-world data for the H1-2 humanoid robot, we used a teleoperation setup with the Apple Vision Pro (AVP) VR headset, built on Unitree’s XR Teleoperate platform. The headset provides an RGB 2D view from the head camera, giving the operator a human-like perspective via the Vuer visualization toolkit. Our tracking script controls both dexterous hands and monitors the arms’ poses; however, due to hardware limitations, we restricted data collection to the left arm and hand. Each recorded episode corresponds to a single toy-grasping demonstration.

## Appendix E DetPooling

Creating Attention Masks.
To pool visual features, we first extract the target object’s segmentation mask from camera views. In the ManiSkill Franka simulation, ground truth object masks are directly available and used to identify vision encoder patches overlapping with the object. For the real Franka and H1-2 dexterous hand setups, we manually annotated 200 toy images with bounding boxes to train a Faster R-CNN detector with a ResNet-101 backbone (^1^11[https://github.com/facebookresearch/detectron2](https://github.com/facebookresearch/detectron2))
. The detector’s bounding boxes are then used as input to SAM 2 to obtain segmentation masks, from which the attention masks are constructed in the same manner as in simulation.

Pooling Visual Features.
For detector-based pooling, we follow a standard vision processing pipeline. The image is first patchified and passed through Transformer blocks. From the final block, we obtain spatial feature maps, and then apply the attention mask obtained above to pool the corresponding spatial features, yielding the final pooled features. For the visual encoder, we adopt the off-the-shelf ViT-L MVP model, which was pre-trained with a masked autoencoder objective and has been demonstrated to be effective for robotic control in prior work (Radosavovic et al., 2023).

## Appendix F Robotic Policy Training Details

Observation.
For the simulated Franka robot setting, we use three camera views as visual inputs: two fixed cameras mounted on the tabletop and one wrist-mounted camera. For the real Franka robot, the hardware configuration follows the standard DROID setup, with two tabletop-mounted cameras and one wrist-mounted camera. For LEGO policy training, we use only the two tabletop-mounted cameras as visual inputs.

Action Space.
The LEGO policy is conditioned on the previous and current states, represented by the 7-DoF arm joint positions and the 1-DoF gripper state. The policy is trained to predict future action chunks, consisting of joint poses and gripper states.

Training Details.
We adopt a learning rate of $5\times 10^{-4}$ with a weight decay of 0.01. Training is conducted for 900 epochs with a 30-epoch warm-up and a global batch size of 512. In comparison to foundation VLA models such as $\pi_{0}$-FAST (Black et al., 2023) and OpenVLA-OFT (Kim et al., 2024), our approach demonstrates substantially lower GPU memory requirements and achieves faster convergence, highlighting the efficiency of the proposed architecture.

## Appendix G Baselines Implementation Details

### G.1 π 0 \pi_{0} -FAST

We adopt $\pi_{0}$-FAST (Black et al., 2024) as a baseline for our simulated Franka, real-world Franka, and real-world H1-2 Dexterous Hands experiments, following the official code and instructions (^2^22[https://github.com/Physical-Intelligence/openpi](https://github.com/Physical-Intelligence/openpi))
.

Simulated Franka Robot.
On the ManiSkill simulation platform, we fully finetuned the released base autoregressive $\pi_{0}$-FAST model on our simulated toy dataset. We use joint position control, adapting the pretrained model to predict the absolute 7-DoF joint pose and 1-DoF gripper status. We use left camera view and wrist camera view as visual inputs, and use “pick the toy” as the language instruction. We follow the default learning rate in the original implementation and finetune the model for 10K steps with a batch size of 32 for each setting reported in Table [1](https://arxiv.org/html/2510.12866v1#S5.T1).

Real Franka Robot.
For the real-world Franka robot, we use the DROID setting. Instead of velocity control, we adopt joint position control and finetune the released base autoregressive $\pi_{0}$-FAST on our teleoperated toy dataset. The pretrained model is adapted to predict the absolute 7-DoF joint pose and 1-DoF gripper status. We use left camera view and wrist camera view as visual inputs, and use “pick the toy” as the language instruction. Following the default learning rate, we train for 10K steps under both the 500-demonstration and 1500-demonstration settings shown in Table [2](https://arxiv.org/html/2510.12866v1#S5.T2).

Real H1-2 Robot with Dexterous Hands.
We also extend the setting to include humanoid arms with dexterous hands. Specifically, we finetune the released base autoregressive $\pi_{0}$-FAST on our 500-demonstration teleoperated toy dataset using delta joint control. We experiment with both absolute joint control and delta joint control for the 7-DoF right arm, 6-DoF wrist torque, and 6-DoF finger angles (totally a 20-dim action). We use left camera view and head camera view as visual inputs, and use “pick the toy with dual arms” as the language instruction. Results show that delta control outperforms absolute control.

However, in this new embodiment-specific setting (compared with DROID setting, which the pre-trianing covers it), we find that $\pi_{0}$-FAST tends to overfit with limited data, likely due to its large model size. To mitigate overfitting, we select an early checkpoint where the cross-entropy loss reaches a reasonable value greater than 1 (but for DROID, it will not overfit even with a 1$e-2$ loss probably since its pretrained on large amount of DROID data). For the reported results in this paper, we follow the default learning rate and train for 1K steps using 500 demonstrations, as summarized in Table [2](https://arxiv.org/html/2510.12866v1#S5.T2).

### G.2 OpenVLA-OFT

We use OpenVLA-OFT (Kim et al., 2025) as a baseline for both simulation and real-world experiments, following the official implementation and finetuning instructions (^3^33[https://github.com/moojink/openvla-oft](https://github.com/moojink/openvla-oft)). We use LoRA (Hu et al., 2021) finetuning with a rank of 32 for all experiments.

Simulated Franka Robot.
On the ManiSkill simulation platform, we use delta joint position control and input images from the front, base, and wrist cameras. The model is trained with a batch size of 2 and an initial learning rate of 1.25e-4, decayed to 1.25e-5 after 100,000 steps. Training runs for a total of 150,000 steps, with checkpoints at every 20,000 steps evaluated to select the best-performing model for each experiment.

Real Franka Robot.
In the real Franka DROID setting, we use delta joint position control, consistent with the simulation experiments. The model receives images from the left, right, and wrist cameras. Training uses a batch size of 2 and an initial learning rate of 1.25e-4, decayed to 1.25e-5 after 100,000 steps, for a total of 150,000 steps. We used the last checkpoint for evaluation.

Real H1-2 Robot with Dexterous Hands.
The model is conditioned on images from the left, right, and head cameras. It receives a 26-dimensional state vector—corresponding to 7 DoF per arm and 6 DoF per hand—and predicts a 40-dimensional output, which includes absolute joint targets for all joints as well as feedforward torques for both arms. Training uses a batch size of 2 and an initial learning rate of 1.25e-4, decayed to 1.25e-5 after 100,000 steps, for a total of 150,000 steps. We used the last checkpoint for evaluation.

### G.3 ShapeGrasp

We evaluate ShapeGrasp on our real Franka setup using the official implementation(^4^44[https://github.com/samwli/ShapeGrasp](https://github.com/samwli/ShapeGrasp))
. ShapeGrasp uses GPT-4o to identify a graspable part from a decomposition graph, where nodes represent object parts (modeled as convex shapes) and their spatial relationships. It outputs a pixel location along with a $z$-axis rotation for a top-down grasp. Using a calibrated Intel RealSense D435 camera, we project the pixel-level grasp prediction into 3D space. An executable grasp trajectory is then generated by interpolating between the robot’s current pose and the predicted grasp pose.

## Appendix H Evaluation Details

Figure: Figure 7: Real-world Evaluation Settings. We have DROID Franka setting with YCB dataset on the left and H1-2 robot with dexterous hands and 13 everyday objects.
Refer to caption: x7.png

For the simulated Franka robot, we use the default task environment “PickClutterYCB-v1” for evaluation, with details available in the official documentation. For the real-world experiments, we consider two settings, as shown in Figure [7](https://arxiv.org/html/2510.12866v1#A8.F7). The left panel illustrates the standard DROID setup with the YCB dataset used for evaluation, while the right panel shows the H1-2 robot equipped with Inspired dexterous hands and the 13 everyday objects used for evaluation.

### H.1 ManiSkill Simulation Evaluation

To evaluate policies in simulation, we defined a $0.15\times 0.15$ m square workspace, subdivided into a $4\times 4$ grid. The grid was constructed from the Cartesian product of the sets $\{-0.075,-0.025,0.025,0.075\}$ along both the $x$ and $y$ axes, resulting in 16 evenly spaced placements. For each trial, the object was placed at one grid location with its $z$-rotation initialized using a random seed. Each object was tested across all 16 placements, and success rates were averaged across objects and placements. A trial was considered successful (1) if the robot lifted the object above a height threshold of 0.3 m. For OpenVLA-OFT policies, we reduced the success threshold to 0.15 m, as the gripper would often prematurely open after grasping the object for these policies. Trials in which the object was not lifted above the threshold were marked unsuccessful (0).

### H.2 Franka Robot Evaluation

To evaluate our policy on the Franka Panda arm, we defined a $0.5\times 0.28$ m rectangular workspace on the table, subdivided into a $4\times 4$ grid. For each trial, the object was placed in one of the 16 grid cells, with its $z$-axis orientation randomized. We evaluated policies by testing each object across all 16 placements and averaged the results to compute the final success rate. A trial was considered successful (1) if the robot securely lifted the object above a height threshold of 0.2 m, and unsuccessful (0) otherwise.

### H.3 H1-2 Humanoid Dexterous Hands Evaluation

To evaluate our policy on the H1-2, we defined a grasping workspace by taping off a $40\,\text{cm}\times 36\,\text{cm}$ rectangular zone on the table, positioned within the head-mounted ZED camera’s field of view and centered between the two Inspire hands. This rectangle was subdivided into six equally sized $3\,\text{in}\times 3\,\text{in}$ squares. For each object tested, we conducted five grasping trials, placing the object in a different square for each trial. Performance was scored as 1 if the robot successfully picked up the object and 0 otherwise. All trials were executed using the left arm and hand. During evaluation, we encountered technical issues with the Inspire hands—most notably unresponsive thumb joints on both sides—which limited the scope of humanoid grasping experiments we were able to carry out.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: Please see the Weaknesses section.
soundness: 2
strengths: - The paper executes a significant number of real-world robot experiments, which is excellent.

- The idea of using randomly assembled objects that are composed from just four shape primitives—spheres, cuboids, cylinders, and rings is insightful.
summary: This work introduces LEGO, demonstrating that robots can acquire robust general-purpose grasping skills by learning from a simple set of objects composed of just four basic shape primitives: spheres, cuboids, cylinders, and rings. 

A key contribution of this paper is the use of a detection pooling mechanism to learn a critical object-centric visual representation, enabling the policy to generalize to a wide range of real-world objects in a zero-shot manner.
weaknesses: - The detection pooling proposed in this paper shares similar ideas with those widely applied in object-centric manipulation. The authors should discuss these related works and clearly articulate the differences between them and their own approach. For instance, the following works:

[1] Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

[2] Transferring foundation models for generalizable robotic manipulation

[3] DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping

- Another major shortcoming lies in the paper's evaluation. First, the paper only evaluates the grasp task. Evaluating other contact-rich manipulation tasks is necessary, as this is central to distinguishing the proposed method from pure grasping approaches like Anygrasp. Second, the comparison made against models such as FAST and OpenVLA-OFT is unfair, as the evaluation task is a single-task setup that does not require language understanding or reasoning.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Why only consider Transformer architectures here? It seems to me this could be applicable to CNNs as well?
2. Can you clarify precisely which variant of DetPool is used in your main experiments — masked self-attention plus pooling, or masked pooling only. Were they both compared head-to-head?
3. Since LEGO uses explicit object masks while other baselines do not, did you evaluate variants of the baselines with comparable object-localization inputs (e.g., crops or masks)? If not, how do you justify the fairness of these comparisons?
4. Is the ViT-L MVP encoder frozen or fine-tuned in each experiment? If partially fine-tuned, could you specify which layers receive gradients?
5. The model is described as having 86M parameters. Does this exclude the ViT-L encoder? Could you report both total and trainable parameter counts for transparency?
6. Why is the success threshold for OpenVLA relaxed relative to other methods? Did you perform sensitivity analyses to confirm that this does not alter comparative trends?
7. Some baselines use different control modes and camera setups. Can you quantify or comment on how much these differences might affect performance comparisons?
8. Have you tested how performance degrades under realistic mask noise, detection failures, or segmentation errors? Since DetPool relies on accurate masks, this seems crucial for generalization. 
9. Could you include analyses under more challenging conditions, such as heavy clutter, occlusion, lighting variation, or distractor object, to better support claims of robust object generalization?
10. Have you compared DetPool against simpler alternatives such as crop-and-encode or mask-as-input-channel baselines (e.g., Shariar et al. 2025) to isolate the contribution of attention masking itself? 
11. The abstract claims outperforming state-of-the-art approaches, though π0-FAST surpasses LEGO in one setup. Could you clarify or qualify which baselines and settings support this claim?


**References**
1. Shahriar, F., Wang, C., Azimi, A., Vasan, G., Elanwar, H. H., Mahmood, A. R., & Bellinger, C. (2025). General and Efficient Visual Goal-Conditioned Reinforcement Learning using Object-Agnostic Masks. arXiv preprint arXiv:2510.06277.
soundness: 3
strengths: - Clear and compelling motivation with an interesting set of experiments
- Training only on random compositions of four primitives is a nice way to study object-generalization. 
- The procedure for generating and 3D-printing toys is well specified and reproducible.
- Simple, modular representation idea with strong empirical effect:
- Results in simulation and a real Franka setup on 64 YCB objects, and a humanoid with dexterous hands
- The study disentangles the effects of number of unique toys versus demonstrations per toy, showing stronger gains from more demonstrations once a small toy set is present (Figure 4, left). This is a useful, actionable finding for data collection planning.
- Detailed reporting of implementation and experiment setup in the appendix
summary: The paper proposes training robotic grasping policies exclusively on randomly assembled “toys” made from four shape primitives, then evaluating zero-shot on real objects. A detection pooling mechanism (DetPool) is introduced to induce object-centric visual representations. Experiments in simulation and on two real robot platforms (a Franka with a parallel gripper and a humanoid with dexterous hands) show strong transfer, including notable gains over large pretrained vision-language-action models under several settings. Ablations study the role of toy diversity, number of demonstrations, model size, and primitive/complexity composition.
weaknesses: - Inconsistency/ambiguity in DetPool’s implementation and mechanism:
  - Section 4.3 states that the object mask is used to enforce no attention between object and non-object tokens inside the vision transformer and then mean-pool object tokens. Appendix E, however, describes patchifying, passing through transformer blocks, and then using the mask only at pooling time to select spatial features. This distinction matters: masked self-attention changes the encoder’s computation; masked pooling does not. The paper should explicitly reconcile which variant is used in each experiment (simulation vs. real) and in Table 1’s ablations, and whether both variants were tried and compared head-to-head.
- Mask generation pipeline is split across sections, creating avoidable ambiguity:
  - Section 5.1 states SAM2 for real and ground-truth masks for simulation, while Appendix E explains the two-stage real pipeline (Faster R-CNN boxes → SAM2 masks). The pipeline is present, but unifying its description in the main text would reduce confusion and make the experimental setup easier to follow.
- Unfair evaluation of baselines:
  - LEGO has access to object masks (ground-truth in simulation, detector→SAM2 pipeline in real), whereas competing VLAs (OpenVLA-OFT, π0-FAST) are not given any object cues (e.g., crop/box/mask). Since the paper’s main contribution is about the value of object-centric features, this mismatch likely affects the performance gap. A fair comparison should provide comparable object-centric inputs (or report variants with/without such cues for all methods).
- Mixed messaging on “outperforming state-of-the-art”:
  - The abstract claims outperforming state-of-the-art approaches trained with much more in-domain data. In Table 2, a fine-tuned π0-FAST achieves higher real-world success than LEGO on the Franka setting. The paper discusses this, but the abstract/overall positioning would benefit from qualifying the claim or toning down these claims.
- Evaluation protocol mismatches and thresholds:
  - In simulation, the success threshold for OpenVLA is relaxed to 0.15 m (vs. 0.3 m for others) due to gripper reopening (Appendix H.1). Although documented, thresholds should be unified or the relaxation carefully justified and accompanied by sensitivity analysis.
  - Sensor and control mismatches exist across methods (e.g., camera views, absolute vs. delta joint control). While some of these are inherited from baseline codebases, the paper should quantify or mitigate their impact.
- Vision encoder training status unclear and parameter counts potentially misleading:
  - The paper uses a ViT-L MVP encoder (Section 5.1), but it is not explicitly stated whether it is frozen or fine-tuned in each setting. Reporting whether gradients flow to the encoder is important for reproducibility and for fair comparison to large VLAs that train all vision stacks.
- Missing or limited analyses that would strengthen the generalization claim:
  - Failure-mode and robustness analyses (lighting changes, heavy clutter, occlusion, distractors with similar colors/shapes, heavy texture) are limited. Given the object-centric focus, stress tests around segmentation failures (mask noise, false positives/negatives) would be highly informative.
  - An oracle-cropping or oracle-bounding-box control for baselines would isolate the contribution of precise object localization versus the transformer architecture itself.

**To the authors:** I find the motivation behind this work compelling and see potential for a strong contribution. I’d be glad to reconsider my evaluation if you can effectively address the issues highlighted in the review and outline a concrete plan to strengthen the experimental validation.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. What was the reason for choosing absolute instead of delta joint prediction?
2. Line 307: "our unique 250 toys". Line 341: "We 3D print the 250 toys with the highest". If 250 is the total no. of toys, how many of them were selected for Franka experiment?
3. What is the performance trend when trained between 1 and 25 toys?
4. What were the primary failure modes? What's preventing higher success rates, say 90%, 99% or even 100%?
soundness: 3
strengths: **Positive result**
The authors show results in simulation as well as the real world on 2 robot platforms (Franka robot and Unitree H1-2), comparing their method against state-of-the-art VLAs (both scratch and finetuned). In simulation and Unitree H1-2, the proposed method outperforms all baselines. In real Franka, the proposed method outperforms all baselines except finetuned $\pi_0-FAST$

**Extensive ablations**
1. As they increase no. of toys, performance increases initially but quickly saturates with no/minila difference beyond 25 toys
2. As they increase no. of demonstrations, performance increases
3. As they increase model size, performance improves and then saturates. 86M is the sweet spot for their setting
4. Sphere is the most important primitive shape as removing it leads to significant drop in performance. Ring has the least impact
5. Toy complexity (defined by no. of primitives): 2 primitives contributes the most. Likely due to the level of complexity of toys in the test set
summary: This paper presents a method LEGO (LEarning to Grasp from tOys), to grasp diverse objects by learning to grasp random, synthetically generated toys composed of primitive 3D geometries (spheres, cuboids, cylinders, and rings). For each toy, a random number of primitives of random sizes are fused in random positions and orientations. In total, 250 such toys are generated in simulation as well as 3D printed in the real world. Grasping data is primarily collected via teleoperation except for single primitives, which are collected with motion planning.

A neural network is trained with behavior cloning on the grasp demonstrations. The architecture consists of a Vision Transformer (ViT) for extracting visual features. Authors use a pretrained MVP. SAM 2 is used to get a segmentation mask of the object of interest. Attention is masked between object and background patches using the segmentation mask. Finally, features in object patches are mean-pooled to get a vision feature. This masking mechanism and pooling is coined Detection Pooling (DetPool).

The vision feature is concatenated with proprioception to get a single token at a time step. Multiple tokens from the history of timesteps are passed to a transformer policy to predict the next action token.

Results, both in simulation and real robot, show competitive performance, even when compared to the state-of-the-art VLAs.
weaknesses: **Doesn't capture desired grasp location**
Many objects have desired grasp location, e.g. a cup should be grapsed by it's handle & a vertical cylindrical object is more stably grasped sideways instead of top-down. The proposed method finds a grasp location but doesn't guarantee grasp at desired grasp location since grasping is learnt on random toys.

**Doesn't capture physical properties of objects**
Some objects may have delicate parts necessitating that the robot doesn't apply too much force which grasping. As such, it's a stretch to claim that the method can grasp any object as per the title. A better wording would be - diverse real world objects or something similar.

**Constraints imposed by masking**
This makes it more difficult to extend to complex manipulation tasks involving multiple objects or occlusion

**Minor comments**
1. Line 191: Grammar - "details our including preliminaries"
2. Line 197: Definining visual observations as $i_{1:T}$ is incorrect since at $t=1$, visual observation is $i_{1-C+1:1}$ which includes time $t<1$ which is outside the $1:T$ range. Suggestion - We denote visual observation at time $t$ as $i_t$ and proprioceptive state as $s_t$. Observation at time $t$ consists of $i_{t-C+1:t}, s_{t-C+1:t}$.
3. Line 425: "Effect of Number and Diversity of Demonstrations" - Diversity can come from grasp style variation and not just object variation. Suggestion - Effect of Number Demonstration and Number of unique toys
4. Table 4 and 5: Label the entries. No. of demonstration and success rate.
5. In conclusion, it's not entirely correct to claim that the method outperforms baseline VLAs since for Franka finetuned $\pi_0 FAST$ significantly outperforms.
6. Line 403: Missing citation of ManiSkill
7. Line 425: Missing details on test objects, no. of evals, etc

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Authors use GT masks in simulation and SAM2 masks in the real setup. Could this amplify DetPool’s gains? Please unify to automatic (noisy) masks in both sim and real, report performance as a function of mask quality (e.g., IoU corruption, under/over-segmentation, occlusion levels), and include “box-only” and “no-mask” variants to quantify DetPool’s pure benefit under weak or absent prompts.
2. How is DetPool materially different from existing mask/ROI-guided object-centric representations (e.g., ROI pooling, mask-guided token selection, segmentation feature fusion)? Please compare against these closest alternatives under a fair hyperparameter grid and explain where DetPool’s advantage comes from (attention constraint vs pooling choice).
3. Your policy uses history length C=16 and predicts K=16 future actions. Please provide modular ablations where you hold the controller fixed and swap the visual representation (and vice versa). Also include latency/frame-rate vs success curves to show the gains are due to DetPool rather than increased temporal capacity or compute.
4.On Franka you use ~250 toys and ~1,500 successful demos; on the dexterous hand ~500 demos are needed. Please provide real-world scaling plots (toys count × diversity × demos → success), identify the “minimal viable toy set” and demo thresholds.
5. Quantify geometric similarity between training toys and target objects (e.g., Chamfer/SDF/voxel features), bucket by similarity, and report success with error bars. When targets diverge strongly from the primitives, does DetPool still generalize? Diagnose failure sources (grasp candidate generation, gripper aperture, approach strategy, etc.).
6.When mask boundaries are biased, have holes, or stick to background, how distorted is the representation? Do you need boundary refinement or multi-scale token aggregation to stabilize features? Please provide sensitivity analyses and any corrective post-processing that helps.

If the authors can address these questions convincingly during rebuttal, I am inclined to raise my score.
soundness: 3
strengths: Originality
1.Proposes a “toy-driven” paradigm that composes a few geometric primitives to approximate real-world shape complexity, avoiding reliance on large annotated object corpora.
2.Introduces DetPool to build a target-centric representation during visual encoding (mask-constrained attention + pooling), offering a simple, effective object-centric alternative to CLS/mean/attention pooling.
Quality
Comprehensive evaluation: sim-to-real comparisons, strong VLA baselines, and ablations/scaling on pooling, data scale/diversity, and model size.
Clarity
 Coherent structure from motivation to analysis, clearly explaining why target-centric encoding and primitive-based training support generalization.
Significance
1. Demonstrates zero-shot grasping on complex real objects without heavy real-data/annotation costs, improving practical feasibility.
2.Provides concrete evidence for object-centric representations in embodied manipulation, nudging the field toward “structural priors + target-centric encoding.”
3. Cross-embodiment results suggest portability across manipulators/dexterous hands/mobile settings, indicating broad impact.
summary: This paper studies “cross-object generalization in grasp learning.” Inspired by developmental psychology, the authors train a grasping policy using only four geometric primitives (sphere, cube/rectangular prism, cylinder, and ring) that are randomly combined into “toys,” and achieve zero-shot generalization to real, unseen objects. The core idea is a “detection pooling” method called **DetPool** to build a **target-centric** visual representation: during visual encoding, a target segmentation mask constrains attention so object patches do not attend to background patches; the model then pools over the object patches to obtain a compact representation. A policy network takes this representation together with proprioception and predicts an action sequence (the overall framework is dubbed **LEGO**). Experiments show that training on only a few hundred random “toys” with limited demonstrations attains strong zero-shot grasp success on the real YCB object set. Compared to common pooling baselines (CLS token, mean pooling, attention pooling), DetPool yields markedly better generalization and scales well as data increases. Against several large vision-language-action baselines, LEGO is more robust under small-data and domain-shift settings; it also achieves leading or competitive success rates on a real Franka platform and a humanoid dexterous-hand platform, supporting the “target-centric representation + few demonstrations” path as a data-efficient solution.

Main contributions:

1. Proposes a “toy-driven” grasp learning paradigm and the **LEGO** framework, showing that random combinations of a few geometric primitives suffice to support generalization to complex real-world objects.
2. Introduces **DetPool**, which explicitly constructs a target-centric representation during visual encoding, significantly improving zero-shot and out-of-distribution generalization.
3. Provides systematic evaluation and scaling analyses in both simulation and real robots (including Franka and a dexterous-hand platform), demonstrating consistent gains across data scale, object diversity, and model size.
weaknesses: 1. **Mask dependence and insufficient robustness**
   *Current setup:* GT masks in simulation, SAM2 in the real system; this may inflate DetPool’s gains.
   *Specific gaps:* No sensitivity curves of “mask quality → performance,” and no weak/no-prompt controls (box-only, no-mask).
   **Suggestion:** Use automatic (noisy) masks in both sim and real; plot performance vs IoU corruption, under/over-segmentation, and occlusion levels; add box-only and no-mask variants to quantify DetPool’s intrinsic benefit.

2. **Evaluation too idealized; extrapolation radius unclear**
   *Current setup:* Single-object, regular poses; the dexterous-hand setting uses only 13 objects with ~50% average success and high variance.
   *Specific gaps:* Missing stratified stress tests for clutter, multi-object scenes, heavy occlusion, look-alike distractors, with attribute-bucketed results.
   **Suggestion:** Build layered benchmarks and report attribute-bucketed success and failure attributions; provide “shift strength (viewpoint/material/scale) → success” curves to characterize the effective operating radius.

3. **Insufficient ablations; key claims are questionable**
   *Claim:* DetPool is the key to generalization and outperforms mean/attn/CLS pooling by 22–48%.
   *Specific gaps:*

   * All pooling variants share the same architecture/training recipe, but there’s no evidence they aren’t simply overfitting under limited data.
   * Missing **no-pooling** baseline using full-image features (full-image ViT).
   * No analysis of **SAM2** mask quality on performance (robustness to mask noise unclear).
   * SOTA baselines (e.g., OpenVLA-OFT 7B) overfit in small-data regimes; the paper doesn’t show LEGO remains superior at larger scales.
     **Suggestion:** Add data-regime sweeps (few→many), include a full-image no-pooling baseline, conduct mask-quality sensitivity analyses, and report matched-compute/parameter learning curves to verify whether DetPool’s advantage persists with more data.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: In the experimental section, you are often referring to a pre-defined grid when testing objects multiple times. Could you give more information about that pre-defined grid?
soundness: 3
strengths: * Paper clearly written, well structured ans easy to understand, with good intuitive explanations
Elegant, data-efficient learning setup leveraging primitive-based object construction.
* The proposed method shows strong empirical performance with minimal data (the data efficiency clain is well demonstrated empirically).
* I appreciate the experimental coverage including some ablations and, more importantly, multiple robot platforms and real-world experiments!
summary: The main contribution of this paper consists of the introducing a data-efficient, object-centric learning framework—called LEGO (LEarning to Grasp from tOys)— that enables robots to generalize grasping skills to unseen real-world objects by training only on randomly assembled synthetic toys, built from only 4 primitive shapes. It gets its inspiration from an analogy between human cognitive development and robotic skill acquisition.
As many other papers in the field, it emphasizes the critical role of an object-centric perception model, through a detection pooling (DetPool) mechanism.
weaknesses: * The approach is overall heuristic (based on an analogy with child development) and does not formally support that toy-based training will yield to generalizable features.
In robotic manipulation, it is well known that geometry is not enough: mass distribution, friction/contact forces, affordances, texture are also very important factors to deal with when we want to generalize to a wide class of objects. All these aspects are not sufficiently covered in the paper and it is likely to explain the drop in performance on complex or non-rigid objects.
* Dependence on accurate segmentation, as it is based on reliable object masks (obtained via SAM2). In real world settings, due to lightning conditions and other occlusion factors, this could not be the case.
* The Object-centric representation (DetPool) is not really new. It could be useful to add (and compare with) two recent papers: https://arxiv.org/abs/2505.11563 and https://arxiv.org/abs/2503.11565.


Decision: Accept (Poster)