Learning to Grasp Anything By Playing with Random Toys
Authors: Dantong Niu
Yuvan Sharma
Baifeng Shi
Rachel Ding
Matteo Gioia
Haoru Xue
Henry Tsai
Konstantinos Kallidromitis
Anirudh Pai
S. Shankar Sastry
Trevor Darrell
Jitendra Malik
Roei Herzig
Venue: ICLR 2026 Poster (2026)
Keywords: Generalizable Grasping
Object-centric Representation
Zero-shot Robotic Manipulation
Primary area: applications to robotics, autonomy, planning
TL;DR: Training robots on random toys enables zero-shot grasping of real-world objects.

Abstract: Robotic manipulation policies often struggle to generalize to novel objects, limiting their real-world utility. In contrast, cognitive science suggests that children develop generalizable dexterous manipulation skills by mastering a small set of simple toys and then applying that knowledge to more complex items. Inspired by this, we study if similar generalization capabilities can also be achieved by robots. Our results indicate robots can learn generalizable grasping using randomly assembled objects that are composed from just four shape primitives: spheres, cuboids, cylinders, and rings. We show that training on these "toys" enables robust generalization to real-world objects, yielding strong zero-shot performance. Crucially, we find the key to this generalization is an object-centric visual representation induced by our proposed detection pooling mechanism. Evaluated in both simulation and on physical robots, our model achieves a 67% real-world grasping success rate on the YCB dataset, outperforming state-of-the-art approaches that rely on substantially more in-domain data. We further study how zero-shot generalization performance scales by varying the number and diversity of training toys and the demonstrations per toy. We believe this work offers a promising path to scalable and generalizable learning in robotic manipulation.

Published as a conference paper at ICLR 2026
LEARNING TO GRASP ANYTHING BY PLAYING WITH RANDOM TOYS
Dantong Niu Yuvan Sharma' ' Baifeng Shi Rachel Ding', MMttteo G Haoru Xue', Henry Tsai', Konstantinos Kallidromitis", Anirudh Pai', Shankar Sastry!.1, Trevor Darrell--i, Jitendra Malik Roei Herzigh
'University of California, Berkeley
"Sapienza University of Rome
IItalAI
PPanssonic
ABSTRACT
Robotic manipulation policies often struggle generalize novel objects, lim- iting their real-world utility. In contrast, cognitive science suggests that children develop generalizable dexterous manipulation skills by mastering small set of simple toys and then applying that knowledge to more complex items. Inspired by this, we study similar generalization capabilities can also be achieved by robots. Our results indicate robots can learn generalizable grasping using randomly as- sembled objects that are composed from just four shape primitives -pperees. cuboids, cylinders, and rings. We show that training on these "toys" enables robust generalization to real-world objects, yielding strong zero-shot perfor- mance. Crucially, we find the key to this generalization is an object-centric visual representation induced by our proposed detection pooling mechanism. Evalu- ated in both simulation and physical robots, our model achieves 67% real- world grasping success rate on the YCB dataset, outperforming state-of-the-arr approaches that rely substantially more in-domain data. We further study how zero-shot generalization performance scales by varying the number and diversity of training toys and the demonstrations per toy. We believe this work offers promising path to scalable and generalizable learning in robotic manipulation. Demonstration videos, code, checkpoints and our dataset are available on our project page: https://leoograap.giihbb..
INTRODUCTION
"Treat nature by means of the cylinder, the sphere, the cone, everything brought into proper perspective."
PAUL CÉZANNE
Robotic manipulation policies have recently achieved impressive progress, solving complex tasks in domains such as dexterous manipulation (Kumar et al. 2016 Chen et al., 2022 Wang et al.
2024
Chen al. 2023 Qin et al. 2021). robust sim-to-real transfer (Chukwurah et al. 2024 Pinel
et al. 2023 Ho et al. 2020), and long-horizon planning for multi-step tasks (Mishra et al.) 2023
Simeonov et al.] 2020 Pertsch et al. 2020). Yet, fundamental challenge remains: they often fail to generalize their manipulation skills to novel objects, limiting their practical application. In stark contrast, humans show astonishing generalization capabilities dexterous manipulation. For example, cognitive literature (Schneiberg et al.
et al. [2044 Rochat 1989: Thelen et al., 1993 NNedhhm et al. 2002; Ruff 1984 [Boniiuo & Arbib] 2015) suggests that children can learn to grasp by mastering only small set of simple toys and then applying that skill to unseen complex objects. This raises a central question: can robotic manipulation policies generalize similarly?
Equal contribution.
Equal advising.

---

Published as a conference paper at ICLR 2026
Primitives Set
Training Set
Evaluation Set
Select Random Primitivvas
Create Random Combinations
3D Print Toys
Zero-Shot Grasping with 67% Success on YCB Objects
ORETT
Our grasping policy built from just four basic primitives (left), trained exclusively on random toy compositions (middle), zero-shot generalizes real-world objects (right) and achieves a 67% success rate on 64 objects from the YCB dataset.
Figure I:
In this work, we demonstrate that robots can learn to grasp novel real-world objects when trained only on randomly constructed toys. The design of these toys is inspired by a classic insight from Cézanne: that complex objects can be deconstructed into a vocabulary of simple shape primitives. Specifically, we construct our toys as random compositions of just four shape primitives: spheres, cuboids. cylinders, and rings. These "Cézanne toys" preserve the compositional structure of real ob- jects while remaining sufficiently ouito--distribution, providing challenging yet principled testbed for generalization. Trained on these random toys. our policy learns to grasp complex, unseen real- world objects in a zero-shot manner. See Figure for an overview.
The key to this generalization capability, as we empirically show, lies in the usage of object-centric visual reprexentations. Specifically, we introduce detection pooling (DetPool) to obtain an object- centric visual representation. This method first uses mask of the target object to constrain the vision encoder's attention to the object region, and then applies mean pooling on the output features corresponding to the object patches. In this way, ensure the final vision representation only contains information about the object and not the background or other distractors. We find this visual representation is the key to enable grasping policy to generalize between the vastly different objects in training and testing. We name our framework LEGO (LEarning to Grasp from 10ys).
To evaluate our model's generalization capabilities, we conduct a comprehensive experimental eval- uation. First, we test its zero-shot performance: trained on a small dataset of 250 "Cézanne toys" with 1,500 demonstrations, our policy achieves 67% success rate on 64 real-world YCB (Calli cet al. 2015) objects, significantly outperforming larger, statc-of- thettrrtrr models like Vpen VLA- OFT (Kim et al. 2025) and To -FST (Black al.) 2024 Pertsch et 2025) that are pretrained on much more data. Second, detailed ablations confirm that the key to this success is the object-centric representation induced by our DetPool mechanism, which significantly outperforms standard pool- ing baselines. Furthermore, we conduct thorough scaling experiments, finding that the zero-shot generalization performance scales with both toy diversity and the number of demonstrations, with the latter being more critical. Finally, we show this generalization capability is robust across robot diverse embodiments, including simple grippers and dexterous hands.
2 A CÉZANNE TOY GRASPING DATASET
To evaluate the generalization of robotic grasping policies, we explore challenging zero-shot set- ting: training policies exclusively on a set of out-of-disttibution (OOD) objects and testing on com- mon real-world objects. To this end, we develop systematic approach for generating diverse set of random, OOD objects. We draw inspiration from Cézanne's 's classic idea that complex objects can be abstracted into compositions of simple shape primitives. We thus generate our training objects by randomly combining these primitives. This process efficiently creates a training set of "Cézanne toys" composed of random primitives, which ensures they OOD, while still retaining structural
2

---

Published as a conference paper at ICLR 2026
One Primitive
Two Primitives
Three Primitives
Four Primitives
Five Primitives
Figure 2: Our Cézanne toys are composed of different number of primitives. We generate each toy by randomly assembling 1-5 primitives and randomizing dimensions and colors.
and compositional properties that enable generalization. An overview is presented in Figure Next, we detail our primitives' designs, the toy generation process, and the resulting grasping dataset.
Designing the Primitives. Inspired by prior literature (Marr & Nishihara, 1978 Tulsiani et al. 2016 Li et al. 2019), we choose four primitive types: spheres, cuboids, cylinders, and rings (See the left column of Figure ( The primitives' scale is randomized within specific ranges. Cuboids range from 2-7.2 em in width, 1-20 em in height, and 2-28 em in length; spheres range from 1-8 in diameter; cylinders range from 4-7 em in diameter and 4-12 em in height; and rings range from 6-20 cm in diameter, 0.6-1.8 em in wall thickness, and 2-6 em in height.
Generating Cézanne Toys. We generate Cézanne toys by randomly combining the primitives. Figure 2 iillsstrates some examples of the generated toys. We start by choosing a random number of primitives, ranging from 1 to 5. We then randomly select the required number of primitives from the four basic types, allowing repetitions, and the dimensions of each instance are randomized. then sequentially assembled to form the final toy. Specifically, the first
The sampled primitives primitive is placed at the origin, and the centroid of each subsequent primitive is randomly positioned within a previous primitive. This ensures the primitives overlap and form coherent structure rather than scattered components. Each primitive is also assigned a random 3D rotation. Finally, the toy assigned one of four colors: blue, red, green or yellow. By repeating this process, we generate training set of 250 diverse toys, including 27 made of two primitives, 35 of three, 38 of four, and 47 of five, as well as individual primitives such as 46 cuboids, 18 balls, cylinders, and 19 rings. All toys are both simulated and 3D printed for grasping data collection.
Collecting Grasping Data. We collect toy grasping trajectories in both simulation and real. In simulation, use ManiSkill (Tao et al. (025) with a Franka arm and gripper; in the real world, we use the same Franka arm with Robotiq gnpper, as well as a Unitree H1-2 humanoid equipped with Inspire RH56DFTP hands. We collect all data via teleoperation, except for grasping individual primitives simulation, which is performed using motion planning. During collection, we ensure diverse of grasping poses per object, since individual objects be grasped in many different ways. We collect 2,500 trajectories simulation, 1,500 on the real Franka, and 500 on the H1-2.
3 THE LEGO METHOD
To enable policy trained on "Cézanne toys" generalize to real-world objects, introduce novel object-centric approach. Our method's s key distinction from full-scene architectures is its use of detection pooling mechanism obtain an object-centric visual representation, which empir- ically show is the key to robust generalization. This section details our preliminaries (Section 3.1 full architecture (Section 3.2 and the detection pooling method (Section 3.3
3.1 PRELIMINARIES
Robotic Tasks. Robotic tasks be represented as temporal sequences of observations and actions. The observations typically consist visual observations 1 and proprioceptive states $1.7, where T' is the episode length, is Nxx* denotes the images captured by N cameras time step
3

---

Published as a conference paper at ICLR 2026
Vision Feature
Proprioception
Action
Object Patch Tokens
Non- Non-Object Patch Tokens
Mean Pooling
Transformer Policy
MLP
MLP
Attn Block
Attn
ViT w/ DetPool
VIT w/ DetPool
Detect
Object
(a) LEGO Architecture
(b) VIT with DetPool
Figure 3: The LEGO architecture with DetPool. (a) LEGO uses a VIT with DetPool to extract features of the target object and uses transformer to predict future actions based on the visual features and the proprioception. (b) The ViT extracts features that focus on the target object via DetPool which restrains the attention to the object patches using an attention mask and performs mean pooling the output object patch tokens get the final object-centric vision feature.
f, and E Rdd is the proprioception (e.g., the joint positions) of the robot at step t. The actions a1. Rdd represent how the robot commands its joints (e.g., target joint positions) step
Policy Learning. The training objective is to learn policy that maps history of the past C' steps- visual inputs ic-++:: and robot states St-C+1-- to future action sequence of length K, in order to successfully complete the task: T (14-C+1a St-++:a) + d-t+K-1.
3.2 ARCHITECTURE
Below, we describe the different components of our policy architecture as well as the training objec- tive. Figure 3 (a) illustrates the overall LEGO architecture.
Vision Encoder. Given the observations 4-C++: and St-4+1: from past C' steps, our model uses vision encoder to encode each set of visual observations i, into visual embeddings e!: where ep E Rdd the embedding of the n-th camera image at step 4, and de is the hidden dimension of the vision encoder. We a pretrained MVP (Xiao al. 2022) as the vision encoder. These resulting features are then input into transformer- based architecture for further processing.
Transformer Policy. We use transformer-basee architecture as our main policy network. first concatenates the visual embeddings e!:N and the proprioception 9; along the channel dimension into a single token, and then projects with an MLP. The transformer backbone then takes the projected tokens from all past C' steps and predicts the concatenated actions 0c:+K-1 for the next K steps from the last token. The transformer's designed to have same size as a ViT-B (Dosovitskiy al.) 2 to get the best performance (see the ablation in Section 4.5).
Training Objective. Following the regular behavior cloning algorithm, the training loss is the mean l loss between the predicted actions 0r.4K-1 and the ground-truth actions 0c:4K-1. i.e.,
Kd. [0tt+ a=t4k-1ll1.
To enable generalization to novel objects, we design the vision encoder to object-centric through detection pooling, introduced in the next section.
3.3 DETECTION POOLING
We design a detection pooling mechanism in the vision encoder such that the extracted visual fea- ture is focused the object be grasped, as shown in Figure (b). Specifically, we first obtain the
4

---

Published as a conference paper at ICLR 2026
object segmentation mask for each frame using SAM 2 (Ravi al. 2024b). We then use the object mask to set the attention mask in the vision encoder such that there is no attention between object patch tokens and non-object patch tokens. In this way. ensure that the object patch tokens only contain features from the object itself while ignoring features from non-object patch tokens. Note that this method still allows the vision encoder to understand where the object is in the scene due to the use of positional embeddings. At the end of the vision encoder, we obtain the object-centric visual feature by applying mean pooling on the object patch tokens, which is the final visual embed- ding we use for the policy model. We empirically find that DetPool is crucial for achieving strong zero-shot generalization compared other pooling methods such as mean and attention pooling that do not restrict the attention mask within the VIT and only pool the final output tokens (Section 4.2
EXPERIMENTS
We evaluate LEGO on the YCB object benchmark (Calli et al. 2015) using the ManiSkill simu- lator (Tao et al. 2025). For comparison, we include vision-language-action (VLA) models such as FASTT and OpenVLA-OOE LF- which aim to generalize through large-scale pretraining. We fur- ther analyze how performance scales with the number of unique toys and demonstrations. Beyond simulation, we test LEGO on two real-world setups: a 7-DoF Franka Emika Panda with a I-DoF Robotiq 2F-85 adaptive gripper, where evaluation is done on the YCB benchmark, and Unitree H1-2 humanoid with Inspire dexterous hands, evaluated on a 13-object set of everyday items.
4.1 IMPLEMENTATION DETAILS
Model and Training Setup. LEGO is implemented using PyTorch (Paszke et al. 2019). Its ar- chitecture consists of a VIT-L encoder from MVP (Xiao et al., 2022) for feature extraction and a ViT-Base transformer backbone. The policy is conditioned on history of C = 16 timesteps to predict K 16 future actions. For our DetPool mechanism, we use SAM 2 (Ravi et al. 2024a to obtain object masks for walld images and use ground-truth masks in simulation. The model is trained on eight NVIDIA A6000 GPUs using the Adam optimizer (Loshchilov & Hutter 2019). and evaluated
single A6000.
State and Action Parameterization. We parameterize the proprioceptive space using the joint angles of the robot arm used and a continuous gripper state (when applicable). This yields an 8- dimensional vector for the Franka setup, and a 40-dimensional vector for our H1-2 setup (which includes feedforward torques and finger joints). The model then conditions on state vectors from past timesteps and predicts action vectors for future timesteps, where state and action vectors are represented using absolute joint angles, rather than relative (delta) angles.
4.2 SIMULATION EVALUATION
Experimental Setup. Our training set contains 2,500 demonstrations, comprising 10 successful grasps for each of our 250 unique toys. To analyze scaling laws, all models are also trained on subsets of this data. For evaluation, we use set of 65 graspable objects from the YCB benchmark, selecting only those feasibly graspable by the Franka robot; each object is tested 16 times on predefined grid, and we report the mean success rate across all trials.
Baselines. We compare LEGO (86M parameters) against two significantly larger, state-of-the-art VLAs that rely on large-scale pretraining: FAST (3B) (Black et al. (024) and Open VLA-OFT (7B) (Kim et al. 2025). -o-FAST is a ofate-of-theear VLA model trained on large-scale robotics dataset. Open OFA-OFT is 7B-parameter VIA model pretrained on the Open-X Embodiment (OXE) dataset (Collaboration et al. 2023). Both models are fine-tuned on the same data as ours. To validate the contribution of our core DetPool mechanism, we also conduct ablation studies, replacing DetPool with standard alternatives like attention pooling, CLS pooling, and mean pooling.
Results. Our simulation results, summarized in Table 1 highlight the superior generalization and scalability of LEGO compared to baselines. While LEGO's performance scales reliably with more data achievinn top success rate of 80% with 2.500 demonstrations, the state-of-the-at VLA baselines falter. We find that Th FAST is too data hata-h for the small dataset and struggles with a real-to-sim domain gap from its pretraining. Similarly, OPeFVLA-OOF shows initial promise with
5

---

Published as a conference paper at ICLR 2026
Table Results of zero-shot grasping simulation. We compare our model with state-of-the-ae models (OpenVLA-OR) and To- To-FAST) finetuned our dataset in simulation, as well as different pooling baselines. Our model outperforms the finetuned baselines simulation, with our DetPool proving key to generalization by boosting performance 22-48% over other pooling baselines.
# Demos
Method
500
1000
1500
2000
2500
Open LLA-OFT Kim et al.
2025
30.10
36.35
22.31
15.38
14.71
12.79
T--FAST Black et al. 2024
8.85
7.60
7.69
8.56
4.23
4.13
Ours Attn Pooling
34.71
40.10
44.23
48.27
49.81
51.63
Ours CLS Pooling
24.71
20.29
36.92
41.44
42.40
49.81
Ours Mean Pooling
32.98
30.38
36.15
39.90
40.29
40.58
Ours Det Pooling
56.63
68.17
71.15
74.62
76.83
80.00
250-500 demonstrations but quickly overfits more data added, causing its performance to de- teriorate. On the other hand, while attention pooling is the strongest baseline, it is still significantly outperformed by our DetPool mechanism. In contrast, DetPool enables robust and scalable general- ization, underscoring the effectiveness of object-centric visual representation for generalizability.
4.3 FRANKA ROBOT EVALUATION
Experimental Setup. For real-world experiments, we use 7-DoF Franka Emika Panda arm with Robotiq 2F-85 gripper, consistent with the DROID benchmark. We 3D-print the 250 toys with the highest simulated success rates in simulation and collect 1,500 successful grasp demonstrations. All models are then evaluated on test set of 64 YCB objects. Following the simulation protocol, each object is tested 16 times predefined grid, and report the mean success rate.
and ShapeGrasp (Li et al. 2024b) which is training-free LLM-based approach that uses pretrained language models to decompose objects geometrically before selecting a graspable part.
Baselines. We compare LEGO with strong baselines including To FAST,
A common theme across these methods is reliance large-scale pretraining: either extensive in- domain robot trajectories (for T0-FAST) or internet-scale multimodal data (for ShapeGrasp). In contrast, LEGO is trained from scratch only 1,500 demonstrations, yet achieves competitive performance despite being orders of magnitude smaller in both dataset size and model scale.
Results. As shown in Table 2 LEGO achieves the second-best performance among all models tested, highlighting the effectiveness our approach. outperforms OpenVLA-OFF and Shape- Grasp, well as -FST in its zero-shot setting, despite these being much larger models and To-FAST using in-domain DROID data. This also shows the strength of our object-centric represen- tation, as LEGO attains superior performance using far less data and a smaller model architecture.
The finetuned version of -AST achieves the best overall performance, which we hypothesize because finetuning on additional demonstrations from our DROID setup allows to utilize its pretrained knowledge and adapt to the specific lighting and physical environment, improving per- formance. In addition, finetuning on the randomized toys dataset likely provides an additional per- formance boost. In contrast, OLA is less effective. We observed that minor inaccuracies frequently caused grasp failures, indicating difficulty generalizing to novel objects and settings.
4.4 H1-2 DEXTEROUS HANDS EVALUATION
Experimental Setup. We perform real-world experiments with the Unitree H1-2, a humanoid robot. Each 7-DoF arm is equipped with 6-DoF Inspire RH56DFTP hand, which has 12 total joints. The 6 DoFs capture the independent motions of the fingers, while the 12 joints result from each DoF being implemented design mimics human-like dexterity more closely than traditional gripper end-effectors, making well-suited for experiments that require fine-grained grasping.
pair of mechanically linked joints driven by single linear servo. This hand

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot grasping results on the real Franka robot. We compare our model against ShapeGrasp. OpenVLA-OFT (finetuned), and state-of-the-ant To-FAST (zero-shot and finetuned). Our model achieves 66.67% success rate, outperforming all baselines except finetuned Ty- TAST.
Demos
Method
Pretraining
Tuned on Toys
# Parameters
1500
Open (Kim ct al.
2025
OXE
V
7B
9,47
No- FAST Black
2024
π Dataset + 75K DROID
3B
61.82
Tu FA-AAST Black al.
2024
π Dataset 75K DROID
V
3B
76.56
ShapeGrasp
Li et
2024b
GPT4o
X
26.56
Ours
86M
66.67
Table 3: Results on H1-2 humanoid robot. We compare our model with state-of sfate-of-the-ar the-art models
(To-Fast and OpenVLA-OFF that are finetuned on our data. We show our model achieves superior performance without any pretraining on real objects.
Bell Pepper
Pink Cube
Baton Cookies
Solder Coil
Pink
Piggles
Method
Tomato
Ribbed Ball Stuffed Toy
VLeNVLA
40
40
Fo-FAST
20
20
0
20
20
20
40
Ours
60
40
60
40
60
60
60
Mike Wazowski
Red
Red
Paper
Hand
Yo-Yo
Average
Stuffed Toy
Tape Dispenser
Solo Cup
Towel Roll Sanitizer
OLenVLA-OFT
60
0
60
0
18.46
Fo-FAST
60
40
40
40
20
0
26.15
Ours
60
60
20
60
60
20
50.77
We evaluate our model and baselines 13 everyday objects using the left arm and hand. Each object tested times across predefined grid, with trials scored identically to the Franka experiments.
Baselines. We compare LEGO with OLenVLA-OFT and To -FST (see Section 4.3 for details).
Results. As shown Table LEGO achieves the highest success rate 50.77% in a more chal- lenging setting than the Franka DROID experiments, despite being trained from scratch with only 500 demonstrations. In contrast, -o FAST struggles due to limited demonstrations and the likely ab- sence of this embodiment in its pretraining data. Open VLA-OFF also underperforms, having been pretrained on robot arm and gripper data without humanoid or dexterous- examples. These re- sults underscore LEGO 's data efficiency and the role of DetPool in enabling robust generalization.
4.5 ABLATION STUDIES
We present our ablation studies below, conducted within the ManiSkill simulation environment, which offers a stable and reproducible setting for rigorously evaluating the impact of various factors.
Effect of Number and Diversity of Demonstrations. We perform an ablation study examine how both the number of unique toys in the training set and the number of grasping demonstrations construct eight object sets containing 1, 5, 15, 25, 125, 250, 500, and 1000 unique toys, respectively. For each set, we collect 2,500 total grasping demon- strations and train our model using varying numbers demonstrations per set. The results (left panel, Figure 4 indicate that increasing the number of unique objects improves performance. but with diminishing returns. In contrast, the number of demonstrations has stronger impact on learn- ing generalizable grasping, result consistent with findings from cognitive science literature (Smith) & Slone 2017).
influence performance. Specifically,
Effect of Model Size. To investigate how the size of the policy's transformer backbone affeets performance, we conduct an ablation study. Using the 250-object set wet-which yields the best overall performance-we vary the transformer's size and evaluate the policy across different numbers of

---

Published as a conference paper at ICLR 2026
Performance Different Object Sets
Performance Sizes
0.8
0.7
0.75
0.70
Statons Reta
0.5
Sucuasa Rata
0.65
0.3
0.60
0.2
0.55
Mumberd npn
0.50
Unian 00
0.0
500
1000
1500 2000
2500
2000
2500
Figure 4: Scaling studies. Left: The zero-shot success rate scales with both the number of demos and the number of unique toys. We also find that once the number of demos sufficient, 15 toys is already enough achieve robust zero-shot transfer. Right: The performances scales with the size of the policy transformer until it saturates at the size of 86M.
demonstrations. The results, shown in the right panel of Figure indicate that ViT-Base (86M) the best overall choice: it matches slightly surpasses ViT-Large (307M) in performance while being significantly smaller and thus allowing for faster inference.
Importance of Individual Primitives. To assess the relative importance of each of the four primi- tives, we conduct an ablation study which the training set excludes toys containing a given prim- itive. For each case, the model is trained with varying numbers of demonstrations. The results, presented in Table show that the sphere the most critical primitive, as its exclusion results the largest performance degradation. In contrast, the ring and cylinder appear to be less important, with a relatively small performance drop when they are omitted.
Effect of Toy Complexity. We perform ablation study to complexity, quantified by the number of constituent primitives, and model performance. The model trained demonstrations containing toys with 2-5 primitives. As shown in Table toys with two primitives contribute the most to performance, while toys with five primitives are still beneficial but less influential. This is likely due to the evaluation set's distribution of sizes, which contains more toys with two or three distinct parts; highly complex toys with five parts are relatively rare.
the relationship between toy
Table 4: Ablation of primitive types. We study the importance of each primitive type by remov- ing each one from the primitive set, and report the mean success ferent numbers of total demonstrations.
Table 5: Ablation of toy complexity. We
study the importance of each toy complexity level by training polices only toys com- posed of certain number of primitives, and report the mean success rate the YCB ob- ject set different numbers of total demon- strations.
the YCB object set for dif-
Primitive Removed
200
500
1000
Toy Complexity
25
125
250
Cuboid
37.88
56.35
65.38
72.12
Two Primitives
9.04
32.6
44.42
Sphere 44.13
47.31
61.83
63.08
Three Primitives
7.31
15.77
23.17
Ring
44.23
67.5
68.56
72.6
Four Primitives
7.69
12.4
23.36
Cylinder
45.29
57.6
69.52
72.31
Five Primitives
4.32
10.87
10.19
Geometric Similarity Between Training Toys and Target Objects. We analyze the geometric similarity between the training toys and target objects by computing the Chamfer distance (CD)
8

---

Published as a conference paper at ICLR 2026
Table 6: Geometric similarity analysis between training toys and target objects. Objects are grouped into difficulty buckets based on the 33rd and 66th percentiles of Chamfer distance (CD) to the closest training object. We report average success rate (mean + 95% confidence interval) and summary statistics of Chamfer distance within each bucket.
Difficulty
# Objects
Success
CDMeun
CD
CD
CDMas
Easy
22
75.6 + 13.8
0.0568
0.0123
0.0392
0.0756
Medium
21 22
83.0 + 12.0
0.0990
0.0070
0.0871
0,1080
Hard
81.5 1 12.7
0.1300
0.0283
0.1082
0.2164
between each target object and its closest counterpart in the training set. Target objects are grouped into easy, medium, and hard categories based on the 33rd and 66th percentiles of Chamfer distance.
As expected, the average Chamfer distance increases monotonically across difficulty buckets. How- ever, as shown in 6 the average success rate remains relatively consistent across groups. This indicates that grasp performance does not strongly depend on geometric similarity to the training toys. Overall, the results suggest that our approach is robust to geometric variation and exhibits strong object-level generalization.
5 RELATED WORK
Cognitive Approaches for Manipulation. Developmental psychology shows that infants acquire manipulation skills through exploration and practice (Thelen et al.
1993 Schneiberg et al.) 2002
Needham et al., 2002). Several studies (Ruff 1984 Rochat 1989 Yoshida & Smith 2008 have
revealed a gradual progression in which they learn to manipulate objects by focusing on increasingly
diverse features such as shape and color. [Rakison Butterworth (1998) further demonstrate that infants generalize to unseen objects by applying learned actions to familiar object parts. Motivated by this literature, we investigate achieving similar generalization robotic manipulation.
Existing object modeling, inspired by infant learning (Farhadi et al. 2009), uses descriptive at- tributes (Cohen et al.] 22199: Sun al., (203)), explicit segmentation (Liu et al.) [024] Li et al.]
2024a by Vahrenkamp et al. 2016; Aleotti & Caselli, (011). or represents objects
3D primi- tives (Tulsiani et al. 2016 Monnier et al. 2023] Lin et al. (025). Our work builds on these ideas and explores whether generalized object representations can emerge from just a few primitives.
Generalization in Robotic Manipulation. While several robotic manipulation models have shown capability of mastering various real-world tasks (Zhao et al. 20233 Fu et al. 2024 Barreiros et al.) 2025). they often generalize poorly to novel objects and scenes. One common approach to address this is scaling up the training data (Brohan et al. 2022 Zitkovich et al. 2023 Intelligence et al., 2025| Eppnner et al. 2021 Fang et al. 2020 Ye et al. 2025). In contrast, we show that strong gen- still achievable even with few hours of training data. Another line of works improves the generalizability of robotic models through heavy data augmentation (Hansen & Wang] [021] Tobin al.) 2017 Saddghi & Levine (0166). while we achieve strong generalization without any augmentation. Other works enhance transferability by learning better visual representa- tions (Burns et al. 2023 Srirama al. 2024). Compared these approaches, which usually require costly visual pretraining, we improve generalizability by obtaining object-centric visual representa- tions via DetPool, which is light- weight and can be applied to any pretrained vision model.
eralization capabilities
Object-Centrre Vision Models. Object-centric models represent a scene as a collection of discrete object-level entities. They are shown improve performance and robustness in computer vision. Methods like|Burgee Burgess et al. [ngeecc et al. (2019): Locatello et al. (2020) learn object-centric Locateleoo representations from 2D images, typically for scene decomposition. Extending object-centric learn- ing to the temporal domain, Herzig et al. (2021) introduce object-region transformer that learns temporally coherent object-level features across video frames for tasks such as action recognition and dynamics prediction. Other approaches extend this idea into the 3D domain via world mod- els (Ferraro et al.) 2023) Jeong et al. (202)). Unlike these works, our approach focuses on the effect of object-centric representations generalizable robotic manipulation.

---

Published as a conference paper at ICLR 2026
When it comes to manipulation, variety of object-centric grasping approaches have been
plored (Chen et al. 2024, Zurbrügg et al. 2024 Mandikal & Grauman 2020 Emukpere et al.) 2025), including with large VLA models (Zhong et al. (025a) pretrained with internet-scale infor- mation. In contrast, our model is orders of magnitude smaller, focusing specifically on grasping to demonstrate that the visual representation induced by detection pooling drives object-level gener- alization. Papers such as Devin et al. (2017) address generalizable robot learning through object- centric methods by using attention mechanisms. However, they only evaluate the generalization be- tween similar objects and do not explore the limit of generalization between vastly different objects such as real objects. The most similar approach to ours OTTER (Huang et al. 2025). which uses the vision-language attention map in CLIP to obtain object-centric visual representations. However, is only limited to CLIP, while our method can be applied to any vision transformer.
CONCLUSION
In this work, we demonstrate that robots acquire robust ceneral-purooss grasping skills by learn- ing from a simple set of objects composed from just four basic shape primitives: spheres, cuboids, cylinders, and rings. We show that training on these toys enables a policy to generalize range of real-world objects. Our method learns an object-centric visual representation using a de- tection pooling and transformer architecture, and is trained on dataset of 250 toys with 1,500 demonstrations in the real Franka setting, This policy achieves a 67% zero-shot success rate on the YCB dataset, outperforming state-of-the-ar models such as ---FAST and Open VLA-OFF de- spite them being trained on more diverse and larger datasets. Our findings on grasping scaling laws highlight how we can efficiently optimize performance with limited data. Ultimately, this work demonstrates a scalable path to robotic manipulation by showing that real-world grasping general- ization can emerge from learning on object composites of few primitive shapes. We believe this work offers a promising path to scalable and generalizable learning in robotic manipulation.
wide
7 LIMITATIONS AND FUTURE WORK
While we show that our method offers a promising path toward generalized grasping, it is important to acknowledge its limitations to guide future research. One key limitation is the diversity of the training domain: the model's performance may degrade on objects with different physical properties. Furthermore, our current work focuses on simple, single-step grasping. Future work could extend this approach to complex, long-horizon tasks such as cloth folding and manipulation in dynamic scenes. Finally, the computational cost of the model's architecture presents challenge for real- world deployment on eessource-consrrainne hardware, pointing toward need for future optimization.
ACKNOWLEDGMENTS
We would like to thank Caitlin Regan for setting up teleoperation on the Unitree HI-2, Hunter Tsai for his efforts in building a 3D printing farm and supplying the robot toys for real experiments, and Raj Saravanan for helping with initial toy design and data collection with telcoperation. We would like to thank Ilija Radosavovic for helpful discussions and feedback. We also thank Nicole Walters for creating our lovely logo. Authors, as part of their affiliation with UC Berkeley, were supported in part by the Defense Advanced Research Projects Agency (DARPA) under the TIA- Design of Robustly Implementable Autonomous and Intelligent
MAT program Machines, and/or the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program, as well as the Humanoid Intelligence Center program. Matteo Gioia conducted this work while enrolled in the Data Science Doctorate at Sapienza University of Rome, funded by the European CiioonNNetGeeerratiinn
Mission 4. Component I (CUP B53C22003870006). Matteo Gioia also acknowledges partial support from Sapienza grant RM1241910501F557 (V3LI).
REFERENCES
Jacopo Aleotti and Stefano Caselli. Manipulation planning of similar objects by part correspon- dence. 2011 15th International Conference on Advanced Robotics (ICAR), pp. 247-252, 2011. 1. URLhttppe www.pp....aa....mmm
18127179
10

---

Published as a conference paper at ICLR 2026
Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching- Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. care- ful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.0533/ 2025.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.106:3 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman. Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. TO: A vision-languuage-attion flow model for general robot control. ArXiv, abs/2410.24164, 2024. URL https://api. http:/ppp.semmmttcccccccoooooooe http:/chhh...a.ooooorrrrrrpa.eee :2731111:2
James J. Bonaiuto and Michael A. Arbib. Learning to grasp and extract affordances: the integrated learning of grasps and affordances (ilga) model. Biological Cybermetics, 109:639 669, 2015.
URL https::////mm //api. semanticschollatt org/ wwtp:o//oooo.i1113333330
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22 2022.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and repre-
sentation. ArXiv, abs/1901.11390 2019. URL sema::/ ott
CorPusID::9523722
Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? arXiv preprint arXiv:2312. /2444, 2023.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abheel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pp. 510-517. IEEE, 2015.
Hanzhi Chen, Binbin Xu, and Stefan Leutenegger. Funegrasp: Learning object-centric neural grasp functions from single annotated example object. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 1900-1906, 2024. URL https:: api. www.nticscooola.......o.ooo..
Ttrg/Corussiii ID:267447822
Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, and Song-Chun Zhu. To- wards human-level bimanual dexterous manipulation with reinforcement learning. ArXiv, abs/2206.08el 08666, 2022. 249848184]
URL https: senaantisscooo...rrg//oooo
Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jiechuan Jiang, Zongqing Lu, Hao Dong, and Yaodong Yang. Bi-dexhands: Towards human-level bimanual dexterous manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:2804-288 2818, 2023. URLhttps.
//api. semant i icscoolaa ooccoola...rr///oooossS 265801144
Naomi Chukwurah, Abiodun Sunday Adebayo, and Olanrewaju Oluwaseun Ajayi. Sim-to-real
transfer in robotics: Addressing the gap between simulation and real-world performance. Journal of Frontiers in Multidisciplinary Research, 2024. URL /ttttt:://aaat http:////aa..smmmiiiiiicccccce www.nti
Dxc/CorpusID:227698..
Vanya Cohen, Benjamin Burchfiel, Thao Nguyen, Nakul Gopalan, Stefanie Tellex, and George Dim- itri Konidaris. Grounding language attributes to objects using bayesian eigenobjects. 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1187-1194,
2019. URL httpt::/p .ema:////....aaat.......mm.
semant www.iicc.coocccccccooooooco htt.aatt...ccc.......ooooooooaaa

---

Published as a conference paper at ICLR 2026
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Mad- dukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- atsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf. Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le. Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang, Hao Su. Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homanga Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovie, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan Peters. Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao. Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Jun- hyek Han, Kaiyuan Wang, Kanishka Rao. Karl Pertsch, Karol Hausman, Keegan Go. Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krish- nan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto. Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang. Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Muhammad Zubair Irshad, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart'in in-Mart' in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kir- mani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subrama- nian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchan- dani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Dar- rell, Trinity Chung, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Vitor Guizilini, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Ja- son Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang. Yonatan Bisk, Yongqiang Dou. Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Robotic learning datasets and RT-X mod- cls. https:: // arxiv. org/ abs 2310. 08864 2023.
Coline Devin, P. Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representations for generalizable robot learning. 2018 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 7111-7118, 2017. URL https: http://pp.piac http:ppp...ntattccccooo...aaeaaa org/ CorpusID:3375e50
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICI.R, 2021.
12

---

Published as a conference paper at ICLR 2026
David Emukpere, Romain Deffayet, Bingbing Wu, Romain Brégier. Michael Niemaz, Jean-Luc Meunier, Denys Proux, Jean-Michel Renders, and Seungsu Kim. Disentangled object-centric im- age representation for robotic manipulation, 2025. URL http:://aa.....o//////////////// 11565
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. ArXiv, abs/1907.13052 2019. URL UtLhttss:: api semant www.:maanncccccccr..oommm wtt.aaattccccaaa....oooooaaaaaaa
Clemens Eppner, Arsalan Mousavian, and Dieter Fox. Acronym: A large-scale grasp dataset based on simulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp- 6222 6227. IEEE, 2021.
Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-lbilionn A large-scale bench- mark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11444-11453 2020.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Alexander Forsyth. Describing objects by their attributes. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 1778-1785, 2009. URL http::ts:
Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, and B. Dhoedt. Focus: Object-centric world URL https://app.
models for robotics manipulation. ArXiv, abs/2307.02433 002277, 2023.
Semanticschol wwtps/ccch...o.ooooooooooeeeeeee wrwpCorpus ID: 259342267
Zipeng Fu, Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.00 /0 17, 2024.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation, In 2021 IEEE International Conference Robotics and Automation (ICRA), pp- 13611-13617. IEEE, 2021.
Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 3138-3149, 2021. URL
https://api. www.seeenttieeee wes.aniicccooaaa..... org/ wttg/CorppssD:::0m : 23874400
Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai. Retinagan: object-aware approach to sim-to-real transfer. 202/ IEFE International Conference Robotics and Automation (ICRA), pp. 10920-10926, 2020. URLhttps: /api..p...// aemanticscholar:
Per//CorpsID::: 226278453
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Huang Huang. Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Gold- berg. and Pieter Abbeel. Otter: A laaio--lnggage-acttion model with text-aware visual feature extraction. arXiv preprint arXiv:2503.03734, 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi vision-language-uttoon model with open-world generalization. arXiv preprint arXiv:2504 16054, 2025.
Youngjoon Jeong, Junha Chun, Soonwoo Cha, and Taesup Kim. Object-centric world model for Language-guider manipulation. ArXiv, abs//2533.0617,, 2025. URL htttps://ppi.
Semanticschollee wwtp:/ooors.D:2201.......
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, P Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Ye Ma, Patrick Tree Miller, Jimmy Wu, Suncel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sung Yul Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng
13

---

Published as a conference paper at ICLR 2026
Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean-Pierre Mereat, Abdul Rehman, Pannag R. Sanketi, Archit Sharma, C. Blake Simpson, Quang Uyen Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang. Arefeh Yavary, Tony Zhao, Christopher Agia, Ro- han Baijal, Mateo Guaman Castro, Da Ling Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosa Maria Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J. Lim, Jitendra Malik, Roberto Mart'in-Mart'i in, Subramanian Ramamoorthy, Dorsa Sadigh, Shu- ran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale will robot manipulation dataset. ArXiv, abs/2403. 12945, 2024. URL https: /atpp//ppp.p
org/ Cwrpus 2665333131
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. ArXiv, abs/2400.09926, 2024. URL https: www.seeetninnccoieeeee wwtp/orrpssaaooooooo.oo.opoewrre 2 270440391
Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision lanoon-lnngagg-atttinn models: Opti- mizing speed and success. arXiv preprint arXiv:2502. 2025.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with leamed local models: Application to dexterous manipulation. 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383, 2016. URL https:/ /api...... semanticscholar org/
Corpus D:: 7586242
Haosheng Li, Weixin Mao, Weipeng Deng, Chenyu Meng, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Hongan Wang, and Xiaoming Deng. Seggrasp: Zero-shot task-oriented grasping via semantic and geometric guided segmentation. ArXiv, abs/2410.08901, 2024a. URL https:
7/api. semant ccscholllaaeoooo hrtpecooo....orgoooootttt//:///0
Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi, and Leonidas J Guibas. Supervised fitting of geometric primitives to 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2652-2660, 2019.
Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia P. Sycara, and Simon Stepputtis. Shapegrasp: Zero-shot task-oriented grasping with large language models through geometric decomposition. 2024 IEEE/RSJ International Conference Intelligent Robots and
Systems (IROS), pp. 10527-10534, 2024b. URL http:hhtt//aaaa www.pp.ssemmtttcsschlaaa..ooo
Corpus1D:2687231
Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learn- ing for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20336, 2025.
Weiyu Liu, Jiayuan Mao, Joy Hsu, Tucker Hermans, Animesh Garg, and Jiajun Wu. Com- posable bast- manipulation. ArXiv, abs/2405.05876. 2024. URL htttps://api.
Commantcsccoll. lar, wwwo/or//ooo..eoooooooooo Ct:
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot tention. ArXiv, abs/2006. 15055, 2020. URL https:: //api. semant wwschoaatccccllaaa org/
CorpusID:220127922
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representations (ICLR), 2019. URL
Forum??1==Rkg6Ri.com
14

---

Published as a conference paper at ICLR 2026
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. 2021 JEEE International Conference on Robotics and Automation (ICRA), pp. 6169- 6176, 2020. URL https: apii semant :tt.:oaaa....ooooouuuuoo::a:.aa.
David Marr and Herbert Keith Nishihara. Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society of London. Series Biological Sciences, 200(1 (00(1140):269-294 1978.
Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference Robot Learning, 2023.
URL htttt::t/app.. wew.eteiscaancooooooooooe wttp/Corruu1112222688888888
Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros, and Mathieu Aubry. Differentiable blocks world: Qualitative 3d decomposition by rendering primitives. ArXiv, abs/2307.05477. 2023. httppp/eeeeppp...mmaaaccccoooeaa wtt.aa/ww....cl.o.....ooooooaaaa wttp:oo///oooo:::::10:1r
Amy Work Needham, Tracy M. Barrett, and Karen Peterman. A pick-me-up for infants' exploratory skills: Early simulated experiences reaching for objects using 'sticky mittens' enhances young infants' object exploration skills. Infant Behavior & Development. 25:279-299 295, 2002. URL
https: aapp.p....// www.nticsccoooo.o.......
:6:2279970
Erhan Oztop, Nina S. Bradley, and Michael A. Arbib. Infant grasp learning: computa- tional model. Experimental Brain Research, 158:480-503, 2004. URL https://api.
semanticscholat htt.:olaar..rr//ooossss::1::/.1
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. ArXiv, abs/2006. 13205, 2020.
URL https: http:p/s//app.eeenncccccccccccoo
semanticscholatt wtt.:ooolla...oooooooosoliil
219981151
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-languagg-action models. arXiv preprint arXiv:2501.09747, 2025.
Ricardo Gareia Pinel, Robin Strudel, Shizhe Chen, Etienne Arlaud, Ivan Laptev, and Cordelia Schmid. Robust visual sim-to-real transfer for robotic manipulation. 2023 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 992 992-999, 2023. URL
https://apl. semant www.nticscho..ooo...o... iescholar.. wtt.oorg//ooooooooooo htte/Crrpussi. ID: 20:::6603320 60315942
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang. Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In Eu- ropean Conference on Computer Vision, 2021. URL htttt:://apppi http//ppp...nannisschhhle.l..
Drc/CorpuuDD::338813
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik, Robot learning with sensorimotor pre-training. In Conference Robot Learning, pp. 683-693. PMLR, 2023.
David H. Rakison and George Butterworth. Infants' use of object parts in early categorization. De- velopmental Psychology, 34:49 62, 1998. URL htt.hhtts///// http:/ppp....mmm....ccccco...a.. org/
CorpusID:2103933
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/1 2024a. U //axxive www./a/a.s.... http///awww..../a/.eeesoee
00714
15

---

Published as a conference paper at ICLR 2026
Nikhila Ravi, Valentin Gabeur, Yuan- Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/1 00714, 2024b.
Philippe Rochat. Object manipulation and exploration in 2- to 5-month-old infants. Develop- mental Psychology, 25:871-8844 1989. URL https: api semanticscholar org
1expusID:1976595
Holly A. Ruff. Infants' manipulative exploration of objects: Effects of age and object characteris- tics. Developmental Psychology, 20:9-20, 1984. URL https::/aat semant. cacholar.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image.
arXiv preprint arXiv: 16/ 1.6611..00 ..4201, 2016.
Sheila Schneiberg, Heidi Sveistrup, Bradford J. McFadyen, Mckinley, and Mindy Levin. The development of coordination for reach-to-grasp movements in children. Experimental Brain Re- search, 146: 142-154, 2002. URL / 15714879
semanticscholeeeeooo.e /arg//orpusss:
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Robert Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, and Alberto Rodriguez. A long horizon planning framework for manipulat- ing rigid pointcloud objects. In Conference on Robot Learning, 2020. URL https://appi
semanticscholazz www.:aarrrrroooooosriiiee http:/cpuuuS.:2264714444
Linda B. Smith and Lauren K Slone. A developmental approach to machine learning? Fron- fiers in Psychology, 8, 2017. URL https: http:///.pp....ntt....mmm escho lar. wrt./orrrs/rrr.. 22527330
Mohan Kumar Srirama. Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta. Hrp: Human affordances for robotic pre-training. arXiv preprint arXiv:2407. /89/1 2024.
Yuyin Sun, Liefeng Bo, and Dieter Fox. Attribute based object identification. 2013 IEEE ternational Conference on Roboties and Automation, pp. 2096-2103, 2013.
URL https:
//api. ses...senanticccoolaa org wwwp/oorp/roo.......o :
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Ca- landra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025.
Esther Thelen, Daniela Corbetta, Kathi Kamm, John P. Spencer, Klaus Schneider, and Ronald F. Zernicke. The transition to reaching: mapping intention and intrinsic dynamies. Child develop- ment, 64 4:1058-98, 1993. URL https:::............. //api semant i cscho ar www./org//oooooe prg/CorpusTD: 7142337
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei Efros, and Jitendra Malik. Learn ing shape abstractions by assembling volumetric primitives. 2017 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 1466 1466-1474, 2016. URL hhtpps://api.
semant ccccoo ar. org/ Corpus wtt.oooo...o.oooootiio.o 2380406
Nikolaus Vahrenkamp, Leonard Westkamp, Natsuki Yamanobe, Eren Erdal Aksoy, and Tamim
Asfour. Part-based grasp planning for familiar objects. 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp. 919-925, 2016. URL https: /
semant cscho oraccholla..ooog/ooorrrttto1111
16

---

Published as a conference paper at ICLR 2026
Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Fei-Fei Li, and Karen Liu. Dex- cap: Scalable and portable mocap data collection system for dexterous manipulation. ArXiv, abs/2403.07788, 2024. URL
268363547
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173: 2022.
Jianglong Ye, Keyi Wang, Chengjing Yuan, Ruihan Yang. Yiquan Li, Jiyue Zhu, Yuzhe Qin, Xueyan Zou, and Xiaolong Wang. Dex lb: Learning with lb demonstrations for dexterous manipulation. arXiv preprint arXiv:2506. 17198, 2025.
Hanako Yoshida and Linda B. Smith. What's view for toddlers? using a head camera to study visual experience. Infancy: the jf of the International Society Infant Studies, 13 3: 229-248, 2008. URL htttts:///aatt www.atppieeeniiiiteieeieii...eeo
semant wcsccccchoooooooooooo ar org :t:p://opppss:::11111111:
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Zhang Chen, Tianrui Guan, Fanlian Zeng. Ka Num Lui, Yuyao Ye, Yitao Liang, Yaodong Yang, and Yuanpei Chen. Dexgraspvla: A vision-language-action framework towards general dexterous grasping, 2025a. URL https:
//arxiv.orq/a// abs// 2502. 20900
Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Zhang Chen, Tianrui Guan, Fanlian Zeng, Ka Num Lui, Yuyao Ye. Yitao Liang, et al. Dexgraspvla: A vision-language-attion frame- work towards general dexterous grasping. arXiv preprint arXiv:2502 209000 2025b.
Yifeng Zhu, Zhenyu Jiang, Peter Stone, and Yuke Zhu. Learning generalizable manipulation policies with object-centric 3d representations. arXiv preprint arXiv: 14386, 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-acttio acttin models transfer web knowledge to robotic control. In Conference Robot Learning, pp. 2165-2183 PMLR, 2023.
René Zurbrügg, Yifan Liu, Francis Engelmann, Suryansh Kumar, Marco Hutter, Vaishakh Patil, and Fisher Yu. legnet: A unified approach for instance-centric grasping. 2024 IEEE International Conference on Roboties and Automation (ICRA), pp. semant. i cschol l ar. orgg/oorpust::2573338533
2024. URL https://app.
17

---

Published as a conference paper at ICLR 2026
A TOYS DESIGN
A.1 REAL 3D TOYS DESIGN AND MANUFACTURING
Primitive Design. To design the toys, wrote Python script that uses the SAPIEN physics engine to generate random dimensions for a set of primitives in the amount desired, such as a cuboid and cylinder for two primitive toy. These primitives random offsets between them that ensure the primitives are still physically connected to each other. Finally, we export the toy mesh into STL file using the Trimesh library. We list the dimension ranges of the primitives in Table 7
assembled into a toy by placing them at
Table Dimension ranges for primitive shapes.
Shape
Diameter/Width (cm)
Height (cm)
Length (em)
Cuboid
2-7.2
1-20
2-28
Sphere
1-8
N/A
N/A
Cylinder
4-7
4-12
N/A
Ring
6-20
2-6
0.6-1.8 (wall thickness)
Toy Manufacturing. We printed a total of 250 toys in PLA filament, in addition to multiple test prints to validate the toy geometry and print quality. This was done using fleet of eight Bambu PIP printers over span of four weeks, enabling a maximum throughput of 200 toys per week by printing multiple toys on a single print bed (excluding FivePrimitive toys, whose size meant that they took up the entire print bed and took significantly longer to print). The fleet was managed using the Bambu Farm Manager platform.
The biggest challenge with printing the toys was the delicate geometry of the rings. The original designs had very thin ring walls that would snap during removal from the print bed. To compensate, redesigned the toys have thicker ring walls strengthen the print. In addition, the intersection of shape primitives often resulted in large overhanging bodies, which required large amounts of tree supports to be modelled and printed. Toys with larger primitive counts had significantly higher print times due to their increased volume and complexity. Certain FivePrimitive toys had to be scaled down in size by 20% fit in the 256mm 256mm 256mm print volume.
We have provided the full Bambu printer settings used for our prints for ease reproducibility in Tables 17 the toys into boxes and using label printer to label them with their names is important for keeping track of all the toys, such as if reprint needed.
99 and 20 Any omitted settings are assumed take the default value. Organizing
B ROBUSTNESS ANALYSIS
We systematically study the robustmess of LEGO under challenging visual conditions, including lighting variation, scene clutter, and distractor objects the ManiSkill environment.
B. LIGITING VARIATION
To evaluate robustness to illumination changes, we reduce the lighting intensity to 70% and 40% of the original environment lighting during evaluation. As shown in Table g LEGO maintains nearly identical performance across lighting conditions, with only minor degradation even under severe illumination reduction. This suggests that the learned representations are largely invariant to moderate lighting variations.
B.2 SCENE CLUTTER
To study robustness to clutter, we add 2-5 randomly selected YCB objects to the tabletop scene in addition to the target object. Results in Table Table[]show performance drop in cluttered environments; however, LEGO remains effective despite the presence of multiple irrelevant objects. This indicates that the model can selectively attend to task-relevant regions even in visually crowded scenes.
18

---

Published as a conference paper at ICLR 2026
Table Performance under different lighting conditions.
Lighting
250 demos
2500 demos
100% Lighting
56.63
80.00
70% Lighting
56.44
79.33
40% Lighting
55.00
78.65
Table 9: Performance under cluttered environments.
Condition
250 demos
2500 demos
No Clutter 56.63
80.00
Clutter
46.73
67.21
B.3 DISTRACTOR OBJECTS
We further evaluate robustness to distractors by placing either a random object or an additional in- stance of the target object (lookalike distractor) the table in the ManiSkill environment. As shown in Table both distractor types reduce performance, with visually similar distractors presenting a particularly challenging scenario. Nevertheless, LEGO retains strong performance, suggesting that can distinguish target objects even under visual ambiguity.
Table 10: Performance under distractor settings.
Condition
250 demos
2500 demos
No Distractor
56.63
80.00
Single Random Distractor
50.19
71.35
Single Lookalike Distractor 49.23
74.13
Overall, LEGO demonstrates strong robustness to lighting variation, clutter, and visually similar distractors. We attribute this robustness the detection pooling mechanism, which encourages the model to focus on target-object regions rather than irrelevant parts of the scene.
B.4 MASK NOISE
We additionally evaluate the robustness of our method to noise in the segmentation mask by per- turbing the bounding box provided to SAM during tracking. Specifically, we introduce randomized offsets to the ground-truti truth bounding box and quantify the noise level using the Intersection over Union (IoU) between the original and perturbed boxes. Lower loU corresponds to higher noise.
Table 11 reports success rates under varying IoU levels for both 250 and 2500 demonstration set- tings. Performance degrades only marginally as mask noise increases. Even at 60% loU, the drop in success rate is small relative the 100% loU setting.
This robustness stems from detection pooling operating at the patch level. Since attention masks are applied patches, small spatial perturbations often result in the same set of object patches being selected, leading to inherent tolerance to moderate mask inaccuracies.
Table 11: Effect of mask noise on performance. Noise is quantified by the IoU between the ground-truth and perturbed bounding boxes.
IoU
250 demos
2500 demos
100%
56.63
80.00
80%
79.29 53.19
55.83
60%
77.98
19

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
C.I MORE COMPARISONS OF LEGO TO OBJECT- OBJECT-CENTRES CENTRIC APPROACHES
We further compare our method to related object-centric approaches in robotic manipulation. Specif- ically, we add GROOT Zhu al. (2023) and DexGrasp VLA Zhong et al. (2025b) experimental baselines in ManiSkill, with the results presented in Table
Comparison between LEGO and object-centric approaches. We further include ad- ditional comparisons with object-centric approaches on a subset of our data. These methods are fine-tuned on our 250-toy and 2500-toy demonstration datasets and then evaluated in zero-shot manner on downstream tasks.
Table 12:
Method
250 demos
2500 demos
GROOT Zhu et al. (2023
0.87
1.54
VLenVLA-OFF Kim (2025)
30.10
12.79
To-FAST Black et al. (2024
8.85
4.13
DexGraspVLA Zhong al. (025b)
20.77
48.85
Ours
56.63
80.00
The first baseline, GROOT, object-centric point clouds predict robot actions. This approach can be difficult to scale and deploy in the real world, and as seen by the results, also does not generalize well from toys to real-world objects. Even with the use of ground-truth depth data from the simulator, this approach unable to adapt when encountering novel objects. The other object-centric baseline DexGraspVL.A achieves decent performance, getting higher success rates than Open VLA-OFT and -AST.. However, it still does not reach the performance levels unlocked by detection pooling, and its performance does scale with increasing amounts data as well.
C.2 EFFECT OF ToY COLOR
We measure the impact of toy color on performance by conducting an ablation study comparing policy trained on set of only red toys to our original set where toys were randomly assigned one of four colors (red, green, blue, yellow). As shown Table [3] color diversity improves performance. This is likely because exposure to toys with varying colors during training helps the model learn more robust visual features so can generalize better to weal-world objects.
Table 13: Effect of toy colors on zero-shot generalization. We compare the zero-shot performance of model trained single-color toys with multi-color ones. Training on toys with multiple colors boost performance by about 1%-4% although training on single-color toys still yields a strong gen- eralization to real objects.
Toy Colors
250
1000
1500
2000
2500
Red
50.1
66.44
68.94
72.6
75.48
76.35
Red Green + Yellow
56.63
68.17
71.15
74.62
76.82
80
C.3 PUSH-TO-TARGET MANIPULATION
We evaluate our method on an additional ManiSkill manipulation task, push object to target, which requires sustained contact and precise force control to slide object into designated region, and differs substantially from grasping. As in the grasping setup, we train on randomized set of 250 unique toys and evaluate zero-shot generalization the YCB object benchmark.
The quantitative results are reported Table 14 with PLA-NLLA-OFI Kim et al. (2025) included as baseline. Detection pooling continues to deliver superior performance in this distinct manipulation setting. Although Open VLA-OFT shows improved results compared to its grasping performance, its success rate remains substantially lower than that of method. Moreover, performance improves
20

---

Published as a conference paper at ICLR 2026
Table 14: Performance comparison for "push object to targef" task.
Method
250 Demos
2500 Demos
OpenVLA
58.46
75.77
Ours
67.31
81.83
with additional demonstrations for both approaches, consistent with the scaling trend observed in the grasping task. Overall, these findings indicate that the benefits of detection pooling extend beyond grasp-centric behaviors more general contact-rich manipulation tasks.
C.4 ABLATIONS ON CONDITION AND PREDICTION LENGTH
We study the effect of varying the condition length and prediction length & on model performance. Results are reported in Table 15
Table 15:
function of condition length and prediction length k. The num- bered columns denote the number demonstrations used during training. All experiments described in the main paper use 16 and & 16.
Performance
k
250
500
1000
1500
2000
2500
16
4
63.65
71.92
63.94
69.85
69.42
70.67
16
8
59.04
69.90
68.85
77.31
73.27
78.65
16
16
56.63
68.17
71.15
74.62
76.83
80.00
38.94
54.71
57.60
62.31
70.00
71.15
16
22.12
30.38
38.46
52.07
52.88
54.30
From the bottom three rows in Table 15 we observe that reducing the condition length leads to substantial drop in performance across all training steps. This indicates that sufficient conditioning context critical for stable and effective prediction. In contrast, comparing the top three rows shows that increasing prediction length & has a comparatively smaller effect. While longer prediction horizons generally improve final performance, the gains are more gradual and less dramatic than those obtained from increasing the condition length.
C.5 DELTA vs. ABSOLUTE END-EFFECTOR CONTROL
We chose absolute joint control rather than delta control in all our experiments since we empirically found it to perform better. We present a comparison the two control modes across a varying number of demonstrations in Table 16
Table 16: Delta vs. absolute joint control across different data scales. Absolute control consis- tently and substantially outperforms delta control at every demonstration count.
Method
250
500
1000
1500
2000
2500
Delta Control
29.71
30.19
31.63
34.90
36.44
37.02
Absolute Control
56.63
68.17
71.15
74.62
76.83
80.00
As shown in To6 absolute control significantly outperforms delta control across all data scales. Delta control plateaus around 30-37% success rate regardless of the number demonstrations, suggesting that it struggles to benefit from additional training data. In contrast, absolute control not only achieves much higher performance at every scale but also exhibits a clear upward trend as the number of demonstrations increases, reaching 80.00% at 2500 demos. These results confirm that absolute joint control is the more effective control mode for our setting, likely because it provides a more stable learning target that avoids the compounding errors inherent delta predictions.
21

---

Published as a conference paper at ICLR 2026
D REAL ROBOT HARDWARE CONFIGURATION
D.1 FRANKA EMIKA PANDA
We deploy our policy Franka Panda Robot with DoF' equipped with RobotiQ gripper and a ZED 2i wrist camera. The DoFs allow for precise and dexterous manipulation of the gripper to grasp various types of objects from every part. Two additional ZED 2i cameras are positioned the left and right sides of the robot. Each camera provides an RGB stream at 720p and 30 FPS, without depth information. The hardware configuration is shown in Figure
Left Camera
Right Camera
Wrist Camera
Manipulation Region
Figure 5: Hardware Configuration for Franka Emika Panda with Robotiq Gripper.
D.2 H1-2 HUMANOID DEXTEROUS HANDS
We also deploy our policy to Unitree H1-2 humanoid robot. The robot is equipped with two Inspire RH56DFTP dexterous hands, each with DoF', motors and linear drive design with six miniature linear servo drives and six pressure sensors integrated inside. Given these characteristics, the hands are good fit to emulate real dexterous operations by human. The robot is also equipped with ZED 2i head camera mounted below the original head camera improve the quality of the egocentric data captured. Two ZED 2i cameras are positioned to the side the robot, creating similar setup the one used for the Franka Each camera provides an RGB stream 720p resolution and 30 FPS, without depth information. The hardware configuration is shown in Figure
E ROBOT DEMONSTRATIONS COLLECTION
E. MANISKILL SIMULATION MOTION PLANNING
ManiSkill (Tao et al. 2025) is simulation environment built the SAPIEN framework. We generated dataset of demonstrations for a Franka arm grasping and lifting single primitive objects using scripted planners.
22

---

Published as a conference paper at ICLR 2026
Right
Left Camera
Head Camera
Left Hand
Manipulation Region
Figure 6: Hardware Configuration for H1-2 Humanoid with Inspire Dexterous Hands.
E.2 MANISKILL SIMULATION TELEOPERATION
Using ManiSkill, we also designed a simulation environment collect data via human teleopera- tion. The teleoperation data collection process was then standardized as follows: the arm was first positioned slightly above the target grasp pose. then moved down to the grasp position, its grip- per was closed to secure the object, and the object was then lifted upward. While it is possible fully automate the data generation pipeline using grasping planners for more complex toys, we encountered engineering challenges that ultimately led us to rely on teleoperation.
E.3 FRANKA REAL ROBOT TELEOPERATION
We teleoperated the Franka robot using Meta Quest 3 headset, with only the right-hand controller mapped to arm control. Each upck-up demonstration was executed in one smooth motion on foam- covered table to protect the objects. We recorded videos from the left and right ZED cameras as well as the wrist camera, and additionally logged the robot's proprioceptive states. We adopt the Tranka-DROID robot settings provided by the DROID dataset (Khazatsky al. 2024).
E.4
H1-2 WITH DEXTEROUS HANDS TELEOPERATION
To collect real-world data for the H1-2 humanoid robot, we used teleoperation setup with the Apple Vision Pro (AVP) VR headset, built on Unitree's XR Teleoperate platform. The headset provides an RGB 2D view from the head camera, giving the operator human-like perspective via Vuer visualization toolkit. Our tracking script controls both dexterous hands and monitors arms' poses; however, due to hardware limitations, restricted data collection to the left arm and hand, Each recorded episode corresponds to a single toy grasspinngggg demonstration.
F DETPOOLING
Creating Attention Masks. To pool visual features, we first extract the target object's segmentation mask from camera views. In the ManiSkill Franka simulation, ground truth object masks are directly available and used identify vision encoder patches overlapping with the object. For real Franka and H1-2 dexterous hand setups, we manually annotated 200 toy images with bounding boxes to train a Faster R-CNN detector with a Re1 backbone [] The detector's bounding boxes are
23

---

Published as a conference paper at ICLR 2026
Table 17: Print Quality Settings
Table 18: Print Speed Settings
Setting
Value
Setting
Value
Layer Height
0.3 mm
Initial Layer Speed
mm/s
Initial Layer Height
0.3 mm
Initial Layer Infill
55 mm/s
Line Width (All)
0.62 mm Aligned
Outer Wall Speed
120 mm/s
Seam Position
Inner Wall Speed
150 mm/s
Smart Scarf Seam Application
On
Top Surface Speed
150 mm/s
Scarf Application Angle
155
Sparse Infill Speed
100 mm/s
Scarf Steps
10
Travel Speed
500 mm/s
Scarf Joint for Inner Walls
On
Normal Printing Accel.
10000 mm/s
Role-based Wipe Speed
On
Travel Acceleration
10000 mm/s /s
Slice Gap Closing Radius Resolution
0.049 mm
Initial Layer Travel Accel.
6000 mm/s
0.012 mm
Initial Layer Accel.
500 mm/s?
Are Fitting
On
Inner Wall Accel.
0 mm/s²
Elephant Foot Comp.
0.15 mm
Outer Wall Accel.
5000 mm/s²
Ironing Type
No Ironing
Top Surface Accel.
2000 mm/s²
Initial Layer Density
90%
Sparse Infill Accel
100%
Table 19: Print Strength Settings
Table 20: Print Support Settings
Setting
Value
Setting
Value
Wall Generator
Classic
Enable Support
On Tree(auto)
Order of Walls
Inner/Outer
Type
Bridge Flow
Style
Default
Wall Loops
2
Threshold Angle
30
Top/Bottom Shell Pattern
Monotonic
Remove Small Overhangs
Top Shell Layers
3
Raft Layers
0
Top Shell Thickness
0.8 mm
Top Distance
0.2 mm
Bottom Shell Layers
3
Bottom Z Distance
0.2 mm
Bottom Shell Thickness
mm
Top Interface Layers
Internal Infill Pattern
Rectilinear
Top Interface Spacing
0.5 mm
Sparse Infill Density
10%
Support/Object XY Distance
0.35 mm
Sparse Infill Pattern
Triangles
Support/Object First Layer Gap
0.2 mm
Infill/Wall Overlap
15%
Tree Support Branch Distance
mm
Infill Direction
45
Tree Support Branch Diameter
Ensure Vertical Shell
Enabled
Tree Support Branch Angle 45
then used input to SAM 2 obtain segmentation masks, from which the attention masks constructed in the same manner as in simulation.
Pooling Visual Features. For detector-based pooling, follow standard vision processing pipeline. The image is first patchified and passed through Transformer blocks. From the final block, obtain spatial feature maps, and then apply the attention mask obtained above pool the cor- responding spatial features, yielding the final pooled features. For the visual encoder, we adopt the off-the-shelf ViT-L MVP model, which was pre-trained with masked autoencoder objective and has been demonstrated to be effective for robotic control in prior work (Radosavovic el al. 2023).
G ROBOTIC POLICY TRAINING DETAILS
Observation. For the simulated Franka robot setting, we use three camera views visual inputs: two fixed cameras mounted on the tabletop and one wrist-mounted camera. For the real Franka robot, the hardware configuration follows the standard DROID setup, with two mounted cameras and one wrist- mounted camera. For LEGO policy training, we use only the two tabletop- mounted cameras visual inputs.
24

---

Published as a conference paper at ICLR 2026
Action Space. The LEGO policy is conditioned on the previous and current states, represented by the 7-DoF arm joint positions and the I-DoF gripper state. The policy is trained to predict future action chunks, consisting of joint poses and gripper states.
Training Details. We adopt a learning rate of 5 x 10 with a weight decay of 0.01. Training is con- ducted for 900 epochs with 30-epoch warm-up and global batch size of 512. In comparison to foundation VLA models such as T0-FAST (Black et al. 2023) and OpenVLA-OFT (Kim et al.) 2024), our approach demonstrates substantially lower GPU memory requirements and achieves faster convergence, highlighting the efficiency of the proposed architecture.
H BASELINES IMPLEMENTATION DETAILS
H.1
TOSSA
We adopt To To-FAST (Black et al. 2024) as a baseline for our simulated Franka, real-world Franka, and real-world H1-2 Dexterous Hands experiments, following the official code and instructionss-
Simulated Franka Robot. On the ManiSkill simulation platform, we fully finetuned the released simulated toy dataset. We use joint position control,
base autoregressive my- AT-FATT model adapting the pretrained model to predict the absolute 7-DoF joint pose and 1-DoF gripper status. We left camera view and wrist camera view as visual inputs, and use "pick the toy" as the language instruction. We follow the default learning rate in the original implementation and finetune the model for 10K steps with batch size of 32 for each setting reported in Table
Real Franka Robot. For the real-world Franka robot, we use the DROID setting. Instead of velocity control, we adopt joint position control and finetune the released base autoregressive FAST on our teleoperated toy dataset. The pretrained model is adapted to predict the absolute 7-DoF joint pose and 1-DoF gripper status. We use left camera view and wrist camera view as visual inputs, and use "pick the toy" as the language instruction. Following the default learning rate, we train for 10K steps under both the 500-demonstration and 1500-demonstration settings shown Table 2
Real H1-2 Robot with Dexterous Hands. We further extend our setting to a real humanoid platform equipped with dexterous hands. Specifically, we fine-tune the released base autoregressive -ASF model on our 500-demonstration teleoperated toy dataset using delta joint control. The action space consists of a 20-dimensional vector, including 7-DoF right arm joints, 6-DoF wrist torque, and 6- DoF finger joint angles. We experiment with both absolute joint control and delta joint control. Visual observations are obtained from the left camera view and the head camera view, and the language instruction is "pick the toy with dual arms." " Our results show that delta control consistently outperforms absolute control.
In this embodiment setting (in contrast the DROID setting, which is covered during pretraining), we observe that TD-FAST tends to overfit when fine-tuned on limited data, likely due to its large model capacity. To mitigate overfitting, we select an early checkpoint where the cross-entropy loss remains at reasonable value above 1.0. In comparison, for the DROID setting, the model does not exhibit overfitting even when the loss decreases to around 1 x 10-2 likely because is pretrained on large-scale DROID dataset. For all results reported in this paper, we use the default learning rate and train for 1K steps on the 500 demonstrations, as summarized in Table
H.2 OPENVLA-OFT
We use Open VLA-OFT (Kim et al. 2025) as a baseline for both simulation and real-world experi- ments, following the official implementation and finetuning instructions] We use LoRA (Hu al.) 2021) finetuning with a rank of 32 for all experiments.
Simulated Franka Robot. On the ManiSkill simulation platform, we use delta position control and input images from the front, base, and wrist cameras. The model is trained with a global batch size of 16 and an initial learning rate of 1.25 10-4, decayed 1.25 10-5 after 100,000 steps.
25

---

Published as a conference paper at ICLR 2026
Figure 7: Real-world Evaluation Settings. We have DROID Franka setting with YCB dataset on the left and H1-2 robot with dexterous hands and 13 everyday objects.
Training runs for total of 150,000 steps, with checkpoints at every 20,000 steps evaluated to select the pest-performing model for each experiment.
delta joint position control, consis- tent with simulation experiments. The model receives images from the left, right, and wrist cam- eras. The model is trained with a global batch size of 16 and an initial learning rate of 1.25 x 10-4 decayed to 1.25 x 10-5 after 100,000 steps, for a total 150,000 steps. We used the last checkpoint for evaluation.
Real Franka Robot. In the real Franka DROID setting,
Real H1-2 Robot with Dexterous Hands. The model is conditioned on images from the left, right, and head cameras. It receives 26-dimensional state vector-corree correspdonni DoF per hand-and and predicts a 40-dimensional output, which includes absolute joint targets for all joints as well as feedforward torques for both arms. Training uses global batch size of 16 and an initial learning rate of 1.25 x 10-4, decayed to 1.25 x 10-3 after 100,000 steps, for a total of 150,000 steps. We used the last checkpoint for evaluation.
DoF per and
H.3 SHAPEGRASP
We evaluate ShapeGrasp on our real Franka setup using the official implementatiorf: ShapeGrasp uses GPT-4o identify graspable part from a decomposition graph, where nodes represent ob- ject parts (modeled convex shapes) and their spatial relationships. It outputs pixel location along with z-axis rotation for a top-down grasp. Using calibrated Intel RealSense D435 camera, project the pixel-level grasp prediction into 3D space. An executable grasp trajectory is then generated by interpolating between the robot's current pose and the predicted grasp pose.
EVALUATION DETAILS
For the simulated Franka robot, we use the default task environment "PickClutterCCB-rTl" for eval- uation, with details available in the official documentation. For the real-world experiments, consider two settings, as shown Figure 7 The left panel illustrates the standard DROID setup with the YCB dataset used for evaluation, while the right panel shows the H1-2 robot equipped with Inspired dexterous hands and the 13 everyday objects used for evaluation.
26

---

Published as a conference paper at ICLR 2026
1.1 MANISKILL SIMULATION EVALUATION
To evaluate policies in simulation, we defined a 0.15 x 0.15 m square workspace, subdi- vided into x grid. The grid was constructed from the Cartesian product of the sets {-0.075, -7.055, 0.025, 0.075) along both the I and axes, resulting in 16 evenly spaced place- ments. For each trial, the object was placed at one grid location with its z-rotation initialized using a random seed. Each object was tested across all 16 placements, and success rates were averaged across objects and placements. above height threshold of 0.3 m. For Open VLA-OFF policies, we reduced the success threshold to 0.15 m, as the gripper would often prematurely open after grasping the object for these policies. Trials in which the object was not lifted above the threshold were marked unsuccessful (0).
trial was considered successful (1) if the robot lifted the object
1.2 FRANKA ROBOT EVALUATION
To evaluate our policy on the Franka Panda arm, we defined a 0.5 x 0.28 m rectangular workspace on the table, subdivided into grid cells, with its z-axis orientation randomized. We evaluated policies by testing each object across all 16 placements and averaged the results compute the final success rate. A trial was considered successful (1) if the robot securely lifted the object above height threshold of 0.2 m, and unsuccessful (0) otherwise.
grid. For each trial, the object was placed in one of the 16
1.3 H1-2 HUMANOID DEXTEROUS HANDS EVALUATION
To evaluate our policy the H1-2, we defined grasping workspace by taping off 40 cm X 36 em rectangular zone on the table, positioned within the head-mounted ZED camera's field of view and centered between the two Inspire hands. This rectangle subdivided into six equally sized 3 in x 3 in squares. For each object tested, we conducted five grasping trials, placing the object different square for each trial. Performance was scored if the robot successfully picked up the object and 0 otherwise. All trials were executed using the left arm and hand. During evaluation, we encountered technical issues with the Inspire hands most notably unresponsive thumb joints on both sides which limited the scope of humanoid grasping experiments we were able to carry out.
27


Reviews:
[{"review_id": "BmC7rPj6UG", "reviewer": "Reviewer_CYqi", "summary": "This work introduces LEGO, demonstrating that robots can acquire robust general-purpose grasping skills by learning from a simple set of objects composed of just four basic shape primitives: spheres, cuboids, cylinders, and rings. \n\nA key contribution of this paper is the use of a detection pooling mechanism to learn a critical object-centric visual representation, enabling the policy to generalize to a wide range of real-world objects in a zero-shot manner.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper executes a significant number of real-world robot experiments, which is excellent.\n\n- The idea of using randomly assembled objects that are composed from just four shape primitives—spheres, cuboids, cylinders, and rings is insightful.", "weaknesses": "- The detection pooling proposed in this paper shares similar ideas with those widely applied in object-centric manipulation. The authors should discuss these related works and clearly articulate the differences between them and their own approach. For instance, the following works:\n\n[1] Learning Generalizable Manipulation Policies with Object-Centric 3D Representations\n\n[2] Transferring foundation models for generalizable robotic manipulation\n\n[3] DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping\n\n- Another major shortcoming lies in the paper's evaluation. First, the paper only evaluates the grasp task. Evaluating other contact-rich manipulation tasks is necessary, as this is central to distinguishing the proposed method from pure grasping approaches like Anygrasp. Second, the comparison made against models such as FAST and OpenVLA-OFT is unfair, as the evaluation task is a single-task setup that does not require language understanding or reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VYzb54OHgs", "reviewer": "Reviewer_oAEF", "summary": "The paper proposes training robotic grasping policies exclusively on randomly assembled “toys” made from four shape primitives, then evaluating zero-shot on real objects. A detection pooling mechanism (DetPool) is introduced to induce object-centric visual representations. Experiments in simulation and on two real robot platforms (a Franka with a parallel gripper and a humanoid with dexterous hands) show strong transfer, including notable gains over large pretrained vision-language-action models under several settings. Ablations study the role of toy diversity, number of demonstrations, model size, and primitive/complexity composition.", "questions": "1. Why only consider Transformer architectures here? It seems to me this could be applicable to CNNs as well?\n2. Can you clarify precisely which variant of DetPool is used in your main experiments — masked self-attention plus pooling, or masked pooling only. Were they both compared head-to-head?\n3. Since LEGO uses explicit object masks while other baselines do not, did you evaluate variants of the baselines with comparable object-localization inputs (e.g., crops or masks)? If not, how do you justify the fairness of these comparisons?\n4. Is the ViT-L MVP encoder frozen or fine-tuned in each experiment? If partially fine-tuned, could you specify which layers receive gradients?\n5. The model is described as having 86M parameters. Does this exclude the ViT-L encoder? Could you report both total and trainable parameter counts for transparency?\n6. Why is the success threshold for OpenVLA relaxed relative to other methods? Did you perform sensitivity analyses to confirm that this does not alter comparative trends?\n7. Some baselines use different control modes and camera setups. Can you quantify or comment on how much these differences might affect performance comparisons?\n8. Have you tested how performance degrades under realistic mask noise, detection failures, or segmentation errors? Since DetPool relies on accurate masks, this seems crucial for generalization. \n9. Could you include analyses under more challenging conditions, such as heavy clutter, occlusion, lighting variation, or distractor object, to better support claims of robust object generalization?\n10. Have you compared DetPool against simpler alternatives such as crop-and-encode or mask-as-input-channel baselines (e.g., Shariar et al. 2025) to isolate the contribution of attention masking itself? \n11. The abstract claims outperforming state-of-the-art approaches, though π0-FAST surpasses LEGO in one setup. Could you clarify or qualify which baselines and settings support this claim?\n\n\n**References**\n1. Shahriar, F., Wang, C., Azimi, A., Vasan, G., Elanwar, H. H., Mahmood, A. R., & Bellinger, C. (2025). General and Efficient Visual Goal-Conditioned Reinforcement Learning using Object-Agnostic Masks. arXiv preprint arXiv:2510.06277.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear and compelling motivation with an interesting set of experiments\n- Training only on random compositions of four primitives is a nice way to study object-generalization. \n- The procedure for generating and 3D-printing toys is well specified and reproducible.\n- Simple, modular representation idea with strong empirical effect:\n- Results in simulation and a real Franka setup on 64 YCB objects, and a humanoid with dexterous hands\n- The study disentangles the effects of number of unique toys versus demonstrations per toy, showing stronger gains from more demonstrations once a small toy set is present (Figure 4, left). This is a useful, actionable finding for data collection planning.\n- Detailed reporting of implementation and experiment setup in the appendix", "weaknesses": "- Inconsistency/ambiguity in DetPool’s implementation and mechanism:\n  - Section 4.3 states that the object mask is used to enforce no attention between object and non-object tokens inside the vision transformer and then mean-pool object tokens. Appendix E, however, describes patchifying, passing through transformer blocks, and then using the mask only at pooling time to select spatial features. This distinction matters: masked self-attention changes the encoder’s computation; masked pooling does not. The paper should explicitly reconcile which variant is used in each experiment (simulation vs. real) and in Table 1’s ablations, and whether both variants were tried and compared head-to-head.\n- Mask generation pipeline is split across sections, creating avoidable ambiguity:\n  - Section 5.1 states SAM2 for real and ground-truth masks for simulation, while Appendix E explains the two-stage real pipeline (Faster R-CNN boxes → SAM2 masks). The pipeline is present, but unifying its description in the main text would reduce confusion and make the experimental setup easier to follow.\n- Unfair evaluation of baselines:\n  - LEGO has access to object masks (ground-truth in simulation, detector→SAM2 pipeline in real), whereas competing VLAs (OpenVLA-OFT, π0-FAST) are not given any object cues (e.g., crop/box/mask). Since the paper’s main contribution is about the value of object-centric features, this mismatch likely affects the performance gap. A fair comparison should provide comparable object-centric inputs (or report variants with/without such cues for all methods).\n- Mixed messaging on “outperforming state-of-the-art”:\n  - The abstract claims outperforming state-of-the-art approaches trained with much more in-domain data. In Table 2, a fine-tuned π0-FAST achieves higher real-world success than LEGO on the Franka setting. The paper discusses this, but the abstract/overall positioning would benefit from qualifying the claim or toning down these claims.\n- Evaluation protocol mismatches and thresholds:\n  - In simulation, the success threshold for OpenVLA is relaxed to 0.15 m (vs. 0.3 m for others) due to gripper reopening (Appendix H.1). Although documented, thresholds should be unified or the relaxation carefully justified and accompanied by sensitivity analysis.\n  - Sensor and control mismatches exist across methods (e.g., camera views, absolute vs. delta joint control). While some of these are inherited from baseline codebases, the paper should quantify or mitigate their impact.\n- Vision encoder training status unclear and parameter counts potentially misleading:\n  - The paper uses a ViT-L MVP encoder (Section 5.1), but it is not explicitly stated whether it is frozen or fine-tuned in each setting. Reporting whether gradients flow to the encoder is important for reproducibility and for fair comparison to large VLAs that train all vision stacks.\n- Missing or limited analyses that would strengthen the generalization claim:\n  - Failure-mode and robustness analyses (lighting changes, heavy clutter, occlusion, distractors with similar colors/shapes, heavy texture) are limited. Given the object-centric focus, stress tests around segmentation failures (mask noise, false positives/negatives) would be highly informative.\n  - An oracle-cropping or oracle-bounding-box control for baselines would isolate the contribution of precise object localization versus the transformer architecture itself.\n\n**To the authors:** I find the motivation behind this work compelling and see potential for a strong contribution. I’d be glad to reconsider my evaluation if you can effectively address the issues highlighted in the review and outline a concrete plan to strengthen the experimental validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PjC6Q5W2DM", "reviewer": "Reviewer_oJDK", "summary": "This paper presents a method LEGO (LEarning to Grasp from tOys), to grasp diverse objects by learning to grasp random, synthetically generated toys composed of primitive 3D geometries (spheres, cuboids, cylinders, and rings). For each toy, a random number of primitives of random sizes are fused in random positions and orientations. In total, 250 such toys are generated in simulation as well as 3D printed in the real world. Grasping data is primarily collected via teleoperation except for single primitives, which are collected with motion planning.\n\nA neural network is trained with behavior cloning on the grasp demonstrations. The architecture consists of a Vision Transformer (ViT) for extracting visual features. Authors use a pretrained MVP. SAM 2 is used to get a segmentation mask of the object of interest. Attention is masked between object and background patches using the segmentation mask. Finally, features in object patches are mean-pooled to get a vision feature. This masking mechanism and pooling is coined Detection Pooling (DetPool).\n\nThe vision feature is concatenated with proprioception to get a single token at a time step. Multiple tokens from the history of timesteps are passed to a transformer policy to predict the next action token.\n\nResults, both in simulation and real robot, show competitive performance, even when compared to the state-of-the-art VLAs.", "questions": "1. What was the reason for choosing absolute instead of delta joint prediction?\n2. Line 307: \"our unique 250 toys\". Line 341: \"We 3D print the 250 toys with the highest\". If 250 is the total no. of toys, how many of them were selected for Franka experiment?\n3. What is the performance trend when trained between 1 and 25 toys?\n4. What were the primary failure modes? What's preventing higher success rates, say 90%, 99% or even 100%?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Positive result**\nThe authors show results in simulation as well as the real world on 2 robot platforms (Franka robot and Unitree H1-2), comparing their method against state-of-the-art VLAs (both scratch and finetuned). In simulation and Unitree H1-2, the proposed method outperforms all baselines. In real Franka, the proposed method outperforms all baselines except finetuned $\\pi_0-FAST$\n\n**Extensive ablations**\n1. As they increase no. of toys, performance increases initially but quickly saturates with no/minila difference beyond 25 toys\n2. As they increase no. of demonstrations, performance increases\n3. As they increase model size, performance improves and then saturates. 86M is the sweet spot for their setting\n4. Sphere is the most important primitive shape as removing it leads to significant drop in performance. Ring has the least impact\n5. Toy complexity (defined by no. of primitives): 2 primitives contributes the most. Likely due to the level of complexity of toys in the test set", "weaknesses": "**Doesn't capture desired grasp location**\nMany objects have desired grasp location, e.g. a cup should be grapsed by it's handle & a vertical cylindrical object is more stably grasped sideways instead of top-down. The proposed method finds a grasp location but doesn't guarantee grasp at desired grasp location since grasping is learnt on random toys.\n\n**Doesn't capture physical properties of objects**\nSome objects may have delicate parts necessitating that the robot doesn't apply too much force which grasping. As such, it's a stretch to claim that the method can grasp any object as per the title. A better wording would be - diverse real world objects or something similar.\n\n**Constraints imposed by masking**\nThis makes it more difficult to extend to complex manipulation tasks involving multiple objects or occlusion\n\n**Minor comments**\n1. Line 191: Grammar - \"details our including preliminaries\"\n2. Line 197: Definining visual observations as $i_{1:T}$ is incorrect since at $t=1$, visual observation is $i_{1-C+1:1}$ which includes time $t<1$ which is outside the $1:T$ range. Suggestion - We denote visual observation at time $t$ as $i_t$ and proprioceptive state as $s_t$. Observation at time $t$ consists of $i_{t-C+1:t}, s_{t-C+1:t}$.\n3. Line 425: \"Effect of Number and Diversity of Demonstrations\" - Diversity can come from grasp style variation and not just object variation. Suggestion - Effect of Number Demonstration and Number of unique toys\n4. Table 4 and 5: Label the entries. No. of demonstration and success rate.\n5. In conclusion, it's not entirely correct to claim that the method outperforms baseline VLAs since for Franka finetuned $\\pi_0 FAST$ significantly outperforms.\n6. Line 403: Missing citation of ManiSkill\n7. Line 425: Missing details on test objects, no. of evals, etc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dIkWTbzhZC", "reviewer": "Reviewer_7VRd", "summary": "This paper studies “cross-object generalization in grasp learning.” Inspired by developmental psychology, the authors train a grasping policy using only four geometric primitives (sphere, cube/rectangular prism, cylinder, and ring) that are randomly combined into “toys,” and achieve zero-shot generalization to real, unseen objects. The core idea is a “detection pooling” method called **DetPool** to build a **target-centric** visual representation: during visual encoding, a target segmentation mask constrains attention so object patches do not attend to background patches; the model then pools over the object patches to obtain a compact representation. A policy network takes this representation together with proprioception and predicts an action sequence (the overall framework is dubbed **LEGO**). Experiments show that training on only a few hundred random “toys” with limited demonstrations attains strong zero-shot grasp success on the real YCB object set. Compared to common pooling baselines (CLS token, mean pooling, attention pooling), DetPool yields markedly better generalization and scales well as data increases. Against several large vision-language-action baselines, LEGO is more robust under small-data and domain-shift settings; it also achieves leading or competitive success rates on a real Franka platform and a humanoid dexterous-hand platform, supporting the “target-centric representation + few demonstrations” path as a data-efficient solution.\n\nMain contributions:\n\n1. Proposes a “toy-driven” grasp learning paradigm and the **LEGO** framework, showing that random combinations of a few geometric primitives suffice to support generalization to complex real-world objects.\n2. Introduces **DetPool**, which explicitly constructs a target-centric representation during visual encoding, significantly improving zero-shot and out-of-distribution generalization.\n3. Provides systematic evaluation and scaling analyses in both simulation and real robots (including Franka and a dexterous-hand platform), demonstrating consistent gains across data scale, object diversity, and model size.", "questions": "1. Authors use GT masks in simulation and SAM2 masks in the real setup. Could this amplify DetPool’s gains? Please unify to automatic (noisy) masks in both sim and real, report performance as a function of mask quality (e.g., IoU corruption, under/over-segmentation, occlusion levels), and include “box-only” and “no-mask” variants to quantify DetPool’s pure benefit under weak or absent prompts.\n2. How is DetPool materially different from existing mask/ROI-guided object-centric representations (e.g., ROI pooling, mask-guided token selection, segmentation feature fusion)? Please compare against these closest alternatives under a fair hyperparameter grid and explain where DetPool’s advantage comes from (attention constraint vs pooling choice).\n3. Your policy uses history length C=16 and predicts K=16 future actions. Please provide modular ablations where you hold the controller fixed and swap the visual representation (and vice versa). Also include latency/frame-rate vs success curves to show the gains are due to DetPool rather than increased temporal capacity or compute.\n4.On Franka you use ~250 toys and ~1,500 successful demos; on the dexterous hand ~500 demos are needed. Please provide real-world scaling plots (toys count × diversity × demos → success), identify the “minimal viable toy set” and demo thresholds.\n5. Quantify geometric similarity between training toys and target objects (e.g., Chamfer/SDF/voxel features), bucket by similarity, and report success with error bars. When targets diverge strongly from the primitives, does DetPool still generalize? Diagnose failure sources (grasp candidate generation, gripper aperture, approach strategy, etc.).\n6.When mask boundaries are biased, have holes, or stick to background, how distorted is the representation? Do you need boundary refinement or multi-scale token aggregation to stabilize features? Please provide sensitivity analyses and any corrective post-processing that helps.\n\nIf the authors can address these questions convincingly during rebuttal, I am inclined to raise my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality\n1.Proposes a “toy-driven” paradigm that composes a few geometric primitives to approximate real-world shape complexity, avoiding reliance on large annotated object corpora.\n2.Introduces DetPool to build a target-centric representation during visual encoding (mask-constrained attention + pooling), offering a simple, effective object-centric alternative to CLS/mean/attention pooling.\nQuality\nComprehensive evaluation: sim-to-real comparisons, strong VLA baselines, and ablations/scaling on pooling, data scale/diversity, and model size.\nClarity\n Coherent structure from motivation to analysis, clearly explaining why target-centric encoding and primitive-based training support generalization.\nSignificance\n1. Demonstrates zero-shot grasping on complex real objects without heavy real-data/annotation costs, improving practical feasibility.\n2.Provides concrete evidence for object-centric representations in embodied manipulation, nudging the field toward “structural priors + target-centric encoding.”\n3. Cross-embodiment results suggest portability across manipulators/dexterous hands/mobile settings, indicating broad impact.", "weaknesses": "1. **Mask dependence and insufficient robustness**\n   *Current setup:* GT masks in simulation, SAM2 in the real system; this may inflate DetPool’s gains.\n   *Specific gaps:* No sensitivity curves of “mask quality → performance,” and no weak/no-prompt controls (box-only, no-mask).\n   **Suggestion:** Use automatic (noisy) masks in both sim and real; plot performance vs IoU corruption, under/over-segmentation, and occlusion levels; add box-only and no-mask variants to quantify DetPool’s intrinsic benefit.\n\n2. **Evaluation too idealized; extrapolation radius unclear**\n   *Current setup:* Single-object, regular poses; the dexterous-hand setting uses only 13 objects with ~50% average success and high variance.\n   *Specific gaps:* Missing stratified stress tests for clutter, multi-object scenes, heavy occlusion, look-alike distractors, with attribute-bucketed results.\n   **Suggestion:** Build layered benchmarks and report attribute-bucketed success and failure attributions; provide “shift strength (viewpoint/material/scale) → success” curves to characterize the effective operating radius.\n\n3. **Insufficient ablations; key claims are questionable**\n   *Claim:* DetPool is the key to generalization and outperforms mean/attn/CLS pooling by 22–48%.\n   *Specific gaps:*\n\n   * All pooling variants share the same architecture/training recipe, but there’s no evidence they aren’t simply overfitting under limited data.\n   * Missing **no-pooling** baseline using full-image features (full-image ViT).\n   * No analysis of **SAM2** mask quality on performance (robustness to mask noise unclear).\n   * SOTA baselines (e.g., OpenVLA-OFT 7B) overfit in small-data regimes; the paper doesn’t show LEGO remains superior at larger scales.\n     **Suggestion:** Add data-regime sweeps (few→many), include a full-image no-pooling baseline, conduct mask-quality sensitivity analyses, and report matched-compute/parameter learning curves to verify whether DetPool’s advantage persists with more data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kdll6bKGnk", "reviewer": "Reviewer_V8CY", "summary": "The main contribution of this paper consists of the introducing a data-efficient, object-centric learning framework—called LEGO (LEarning to Grasp from tOys)— that enables robots to generalize grasping skills to unseen real-world objects by training only on randomly assembled synthetic toys, built from only 4 primitive shapes. It gets its inspiration from an analogy between human cognitive development and robotic skill acquisition.\nAs many other papers in the field, it emphasizes the critical role of an object-centric perception model, through a detection pooling (DetPool) mechanism.", "questions": "In the experimental section, you are often referring to a pre-defined grid when testing objects multiple times. Could you give more information about that pre-defined grid?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Paper clearly written, well structured ans easy to understand, with good intuitive explanations\nElegant, data-efficient learning setup leveraging primitive-based object construction.\n* The proposed method shows strong empirical performance with minimal data (the data efficiency clain is well demonstrated empirically).\n* I appreciate the experimental coverage including some ablations and, more importantly, multiple robot platforms and real-world experiments!", "weaknesses": "* The approach is overall heuristic (based on an analogy with child development) and does not formally support that toy-based training will yield to generalizable features.\nIn robotic manipulation, it is well known that geometry is not enough: mass distribution, friction/contact forces, affordances, texture are also very important factors to deal with when we want to generalize to a wide class of objects. All these aspects are not sufficiently covered in the paper and it is likely to explain the drop in performance on complex or non-rigid objects.\n* Dependence on accurate segmentation, as it is based on reliable object masks (obtained via SAM2). In real world settings, due to lightning conditions and other occlusion factors, this could not be the case.\n* The Object-centric representation (DetPool) is not really new. It could be useful to add (and compare with) two recent papers: https://arxiv.org/abs/2505.11563 and https://arxiv.org/abs/2503.11565.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)