Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
Authors: Alex X. Lee
Coline Manon Devin
Yuxiang Zhou
Thomas Lampe
Konstantinos Bousmalis
Jost Tobias Springenberg
Arunkumar Byravan
Abbas Abdolmaleki
Nimrod Gileadi
David Khosid
Claudio Fantacci
Jose Enrique Chen
Akhil Raju
Rae Jeong
Michael Neunert
Antoine Laurens
Stefano Saliceti
Federico Casarini
Martin Riedmiller
raia hadsell
Francesco Nori
Venue: CoRL2021 Poster (2021)
Keywords: sim-to-real
offline RL
manipulation
stacking
robot learning

Abstract: We study the problem of robotic stacking with objects of complex geometry. We propose a challenging and diverse set of such objects that was carefully designed to require strategies beyond a simple “pick-and-place” solution. Our method is a reinforcement learning (RL) approach combined with vision-based interactive policy distillation and simulation-to-reality transfer. Our learned policies can efficiently handle multiple object combinations in the real world and exhibit a large variety of stacking skills. In a large experimental study, we investigate what choices matter for learning such general vision-based agents in simulation, and what affects optimal transfer to the real robot. We then leverage data collected by such policies and improve upon them with offline RL. A video and a blog post of our work are provided as supplementary material.

Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes
Alex X. Lee', Coline Devin", Yuxiang Zhou", Thomas Lampe", Konstantinos Bousmalis"i Jost Tobias Springenberg", Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fantacci, Jose Enrique Chen, Akhil Raju, Rac Jeong, Michael Neunert, Antoine Laurens, Stefano Saliceti, Federico Casarini, Martin Riedmiller, Raia Hadsell, Francesco Nori
axXiv:2110.0619222 [cs.RO] 3 Nov 2021
DeepMind, London, UK
Abstract: We study the problem of robotic stacking with objects of complex geometry. We propose challenging and diverse set of such objects that was care- fully designed to require strategies beyond a simple "pick-and-place" solution. Our method is reinforcement learning (RL) approach combined with vision- based interactive policy distillation and simulation-to to-reality transfer. Our learned policies can efficiently handle multiple object combinations in the real world and exhibit a large variety of stacking skills. In a large experimental study, we in- vestigate what choices matter for learning such general vision-based agents simulation, and what affects optimal transfer to the real robot. We then leverage data collected by such policies and improve upon them with offline RL. video and a blog post of our work are provided as supplementary material.
Keywords: sim-to-real, offline RL, manipulation, stacking, robot learning
Introduction
There has been plethora of work on applying learning algorithms solving difficult real robot manipulation problems in a general way with a large and diverse set of objects. However, the focus of existing work has primarily been on tasks like grasping [1, 2, 3,,,, which do not typically require complex inter-object contact dynamics. Manipulation tasks which involve interactions between di- verse objects, c.g. construction of simple structures, require skills and control strategies that are more complex than and qualitatively different from simply grasping and placing such objects.
As step in this direction, we study whether, and how, we can tackle stacking of diverse objects in the real world via a learned policy, using only information from RGB cameras and proprioception. Several works have recently considered vision-based real-robot stacking [5, 6, 7, 8, 9, 10, 11]. How- ever, the focus has almost exclusively been on stacking cube-like objects (which does not require reasoning about orientation, shape, stack stability), with limited diversity [5]. Additionally, prior work reported low task success rates, pointing towards the difficulty of stacking even cuboids in the real world. In order to study stacking in principled way, we propose a new benchmark, RGB-Stacking, see Figure 1, which consists of a set of 152 poceduraall--eenraate and 3D-printed rigid geometric objects. These were carefully designed to pose different degrees of grasping and stacking difficulty for a parallel gripper on a 4-DoF arm. The benchmark is standardized and we release all relevant information for replicating it as supplemen- tary material. 4 What differentiates our work are two primary characteristics: our large variety of objects and our extensive real world evaluations.
limited set of known irregular objects
To highlight the challenges in stacking these objects, we instantiated two RGB-stacking tasks (with corresponding versions in simulation and real world) designed to investigate set of research ques- tions. In the first task, we consider mastering stacking for a set of 5 specific combinations of objects.
"Equal contribution.
Corresponning authors: konstantinonn int.rraemniinnddeemiiiiiid. com and apringenbergätteepinnd com.
Videeo dpmd. airrobotics-stacking YT. Blog post: apamdlairobotic--ttaiinngg *Simllated environment and RGB-objects: https: http::///iiiee /ggtttt.. com http//ind/nnggggiiiinnomm
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(a) Simulation enviroument
(b) Real robol enviromment
(e) Successful stack example
(d) RGB-RRBooeee objects
The RGB-stacking tasks involve three objects colored Red, Green, and Blue signal to vision- based agent which one should stacked on top which. We present tasks stacking red objects on top of blue ones, with the green ones as distractors. The variations in shape require successful agents to reason about contact dynamics and object orientation to form stable stacks. The benchmark defines a parametric family of RGB- objeetjects, which allows designating held-out objects that are quantifiably different from the training objects.
Figure 1:
These, shown in Figure 3, were chosen in terms of the challenges they present: precision in grasping and stacking, balancing, and using the top object as a tool to flip bottom object that has slanted face pointing up. For this task, we investigated whether it is possible to learn a single vision-based policy that can reliably stack all 5 combinations in the real world. For the second task, we consid- ered the challenge of learning general stacking strategies from large set of training objects and test how they transfer to held-out objects- assess generalization. In both cases, we find that using an agent trained in simulation is the most efficient way to bootstrap data collection for offline RL.
Our contributions are as follows. (a) We present and release a benchmark for stacking that features
a diverse set of geometric objects and tens of thousands of possible stacks. (b) We show that is possible to learn a vision-based policy that can stack multiple combinations of objects and can demonstrate a variety of stacking strategies for non-cuboid objects, emergent from RL training. (c) We present the framework we used to obtain our results without the need for human demonstrations. Specifically, we train iision-based agents with domain randomization via an interactive distillation decoupling learning the required stacking skills with RL training from mapping the skills to perception with imitation. We also ablate individual components of this pipeline, requiring more than 54 000 stacking attempts
real robots. (d) Finally, show that an offline RL step using data gathered on the real robots can boost performance when the data is collected with sim-to-real policies; but doing so using real episodes from scripted agent was worse than zero-shot sim-to-real.
2 Related Work
Several works have recently dealt with vision-based real-robot stacking either by learning curriculum of the different stages of the task [6]. by combining human demonstrations and RL [7, 8]. or by using some flavor of sim-to-real transfer [7, 10]. In most prior work. the focus has almost exclusively been on cube stacking. Furrer et al [5] deals with stacking 6 known stones of difffrrnt shapes. These stones however all have fairly wide support, high friction, require similar pick-and-place strategies, and only 11 cpisodes were used for evaluation of the suggested method. In contrast, RGB-staae -tacking provides a general, reproducible, and significantly more difficult benchmark for robotic manipulation. It involves thousands of stacking combinations and our evaluations are significantly more extensive than in prior work.
Robotic Stacking.
Large-Scale Deep RL for Robot Manipulation. Deep learning for robotic manipulation was in part popularized by large-scale data collection for grasping in the real world [1, 2, 3, 4], task that allows for automated evaluation and resetting. However, these efforts considered problems with short time horizons and limited dynamical effects (e.g. position-controlled robots with actions taking up to a second). As result they often used simple Q-lcarning from collected data with direct optimization for action selection [3]. In contrast we consider velocity-based control at rate of 20 Hz leading to long episodes (400 timesteps). This is more challenging scenario in which learning with, e.g., QT-Opt [3] from recorded data alone would exhibit problems due to wrongly- eestimaed Q-values [12]. Additionally, large-scale RL-based learning in the real world can be difficult to reproduce as the entire real-world process (which can last months [2, 3)) a function of the initial conditions. In contrast, our simulation-basee based training is inherently reproducible. In-hand object manipulation provides an alternative route for a difficult challenge, but automating episodic resets become more challenging as the objects easily fall out of the hand [13, 14].

---

Polygon Axis
Curved Support
Trapazold Axis
Limttd Support
Parallslogram Axis
Slanted Support
Rectangle Axis
Wide Support
Illustration of the deformations applied for each of the 4 major of the RGB- RGB-Objects parametric family. Each deformation changes the stacking affordance of RGB (GB-objects: for bottom objects the stacking support is illustrated with red-green lines showing where a stack is possible (green).
Figure 2:
Imitation Learning and (Offtine) Reinforcement Learning. For training in simulation, and sub- sequent training of sim-to-real policies, we use techniques from off-policy RL. followed by an in- teractive imitation learning approach. For training in simulation we consider RI with full state- information, in order to avoid problems with partial observability. In this setting we use the MPO algorithm [15]: a sample efficient, sate--of-the-ars off-policy actor-critic method. Initial train- ing is then followed by our pipeline for obtaining a (general) vision policy via DAgger [16] style interactive imitation learning approach to distill the MPO policy. We found this to signifi- cantly outperform Behavior Cloning [17] policy distillation [18] due to the fact that DAgger-style training provides corrections for mistakes of the rision-based policy. We Domain Random- ization [19, 20, 21, 22, 23, 24] to obtain good sim-to-real transfer in the absence real-world data; more details this are given in the supplementary. Finally, when improving policies with offline RL from real data we consider: i) a filtered behavior cloning loss that equivalent to CRR- exp [25] but with data from single policy source (analogous to work combined offline and online RL [26, 27]); ii) Behavior Cloning based on only successful trajectories. Both allow for stable offline learning without the problems standard RL algorithms exhibit this setting [12].
3 The RGB-Stacking Benchmark
Despite stacking being addressed robotic learning task in prior work, previous approaches have been limited to a reduced set of objects, typically cubical in shape. We focus on the problem of stacking variety of objects characterized by different shapes. This is obtained by defining para- metric family RGB-objects. The design principle is to vary the grasp and stack affordances of these objects for parallel gripper. Our choices significantly change the difficulty of the stacking task by requiring an agent to exhibit behaviors that go beyond a simple pick-and-plact [28] strat- egy. We do while keeping the benefits of automated learning and evaluation as in [29, 2]. In Appendix A, we analyze the difficulty of the task in various ways. We qualitatively evaluate the grasping affordance based general principles on force closure and object funnels [30], and quan- titatively based on the Ferrari-Canny grasp metric [31, 32). Similarly, we qualitatively depict the stacking affordance (Figure 2), which is varied by having bottom objects, whose top flat surfaces differ in area, shape, and orientation. We also quantitatively evaluate both affordances by evaluating the stacking performance of human teleoperators in simulation, and of carefully scripted agent.
3.1 The RGB-Objects Family
Our objects are all obtained by applying a deformation to a seed cube, a 2D vertical extrusion of a planar square. We defined major axes of deformation, illustrated in Figure 2, resulting in different shapes. These shapes can be thought of as the vertical extrusion of 2D shape. The Polygon Axis transforms the planar square into regular polygon. The Trapezoid Axis morphs the planar square to an isosceles trapezoid. The Parallelogram Axis changes the orientation of the extrusion axis, from vertical progressively more slanted axes. Lastly, the Rectangle Axis uniformly scales the object along the x, y or z-axis.
These deformations and their combinations form parametric family of objects. We obtain the final set of objects by uniformly sampling, for both the major axes and all pairwise combinations, objects between the seed object and maximally deformed object. Figure (d) shows subset of these in the real world, and Appendix A contains a complete depiction.

---

3.2 RGB-Stacking Tasks
The RGB-stacking tasks we tackle in this paper involve three objects in basket in front of the robot. Those are colored Red, Green, and Blue to signal to a vision- agent which one should be stacked on top of which, and which one is just distractor. In all our experiments, red is assigned to the top object, blue to the bottom object, and green to the distractor. We note that the role of the latter is not just to distract visually but also to serve as an obstacle during stacking. A successful stack is achieved when the red object is above the blue one and their centroids are vertically aligned within some thresholds. A detailed description can be found in Appendix B.
3.2.1 Skill Mastery on Specific Triplets
The first task involves 5 RGB-object triplets, with each object in a triplet being from a major axis. The triplets were chosen to have different degrees of difficulty for a stacking agent. The task is to achieve skill mastery on these 5 fixed combinations, shown in Figure 3, with a single vision-based agent. We explain the challenges for each triplet in loosely descending order of difficulty:
Triplet 1.
The main challenge is grasp- ing the top object, a grasp that involves the gripper closing on the slanted sides will fail. The bottom object also pro- vides difficulty, as it has a limited stack- ing surface and can easily roll.
Triplet Triplet
Triplet 2. In this triplet, the bottom ob- ject has sloped sides and may need to be reoriented by using the the top ob- ject as a tool before stacking.
Trppet4 Triplet
Triplet 3. The challenge with this triplet is to have a secure central grasp for the elongated object and balance it on top of the slanted bottom object.
Triplet 5
Triplet 4. An easy triplet, it has rectan- gular prisms for both red and blue ob- jects; the challenge is primarily to align their centroids, required for a stack to be considered successful.
Figure 3: Stacking Skill Mastery for our 5 Specific Triplets. All the stacks shown here are performed by single agent trained solely in simulation with domain randomization and transferred to the real robot a zero-shot fashion. Each triplet poses its own unique challenges to the agent: Triplet requires a precise grasp of the top object; Triplet 2 often requires the top object to used tool flip the bottom object before stacking; Triplet 3 requires balancing: Triplet requires preci- sion stacking (the object centroids need to align); and the top object of Triplet 5 can casily roll off if not stacked gently.
Triplet 5. In this triplet, the top object can easily roll off the bottom object as has a large number (10) of faces and is nearly cylindrical.
3.2.2 Skill Generalization
The second RGB-stacking task we are tackling a transfer task. For this we designate the axes of all pairwise combined deformations (see Appendix A for details) as the training axes and the ones of single deformation- the major axes above as the held-out axes. Based on these, we created a training object set, which consists of 103 different shapes, and a held-out set (containing 4 axes of deformation), containing 40 shapes. The 5 fixed triplets chosen for the previous task belong to the held-out axes and final performance is evaluated based on them.
4 Stacking Diverse Set of Objects in the Real World with a General Policy
Since we have access to a simulator with a reasonably accurate model of our real robot and its cell, our approach is to solve the RGB-stacking tasks in three stages: 1) RL training of an expert in simulation using state information, 2) imitation of such an RL expert with a vision-based policy in simulation with domain randomization for both visual appearance and dynamics, and 3) policy improvement step using data gathered on the real robots. Explicitly decoupling these steps allows for faster and cheaper iteration by enabling parallel experimentation and tuning for all three stages.

---

Notation. We consider the Markov Decision Process (MDP) in which policy π [[a]s)- prob- ability distribution over actions a E A that is conditioned on states S acts maximize the discounted sum of rewards, with sparse rewards r(s, a) and discount factor 7 E (0. 1]. We use Q* EB,. [Σ== 7'((8,, at)]s0 = 8, Ao = a] to denote the Q-function, where Pr the trajec- tory distribution induced by and A (s, a) Q" (s, a) Eg'- ~aa [Q(s, a')] for the advantage when executing я. In the real world, we cannot assume access to full state information (e.g. object poses and labels), so both the policy and advantage are defined based on observations o(s) O (actuator positions of the robot and two camera images) as Taa)o(s) and A (o(s), a), respectively.
4.1 Training Expert Policies from State Features in Simulation
We train expert policies via off-policy RL in simulation with shaped reward. We used the MPO algorithm [15] for this purpose (details in Appendix D), but any RL algorithm could have been used in this stage. We found training directly from state s to be significantly faster than training from vision and thus trained expert policies Tee Tala,s, whwer y denotes a triplet number in {l.....5) .5} for the Skill Mastery task, or the parameters of object deformation for the Skill Generalization task.
4.2 Interactive Imitation Learning of a Visien-Based Policy for Sim-to-Real Transfer
We use imitation in simulation to distill the state-based expert(s) into single "student" vision-based policy (a)o((( that can be executed in the real world, where accurate object state is unavailable, and that is designed for maximizing sim-to to-rlll transfer using components described in Section 5.2. We could attempt to train this policy via supervised Behavior Cloning (BC) [17] trajectories from the teacher, but we find this suffers from covariate shift, where the student visits different states than the teacher (in which it fails). We instead perform DAgger-style [16] interac- tive imitation learning (IIL) of teacher actions on states from trajectories obtained by executing the student policy. We refer to this as III., in which we perform optimization alongside data-collection (rather than alternating collection and supervised learning as in [16]). Specifically, our student policy To is continuously executed in heavily domain randomized simulation environment (see appendix for details), collecting trajectories stored in a replay buffer Dvis. The student is trained to maximize:
dataset of
LuL-22 =
E (y.T)~Dais
Σ
log पe (ag (o())) ar ~ Tee (-(())1.))
where (y, (U) is a pair of object information y and trajectory r (so, do; .1,...,,77 8T). Note that if we replace the collected data Dvis with expert data De from executing the teacher Te: then we recover standard BC but with infinite data. The form of the teacher and the data above depends on the task.
Skill Mastery. We have access to 5 teachers indexed by the object triplet number y E {I....,5) 5). Each teacher {Te (a|s, s, y = .).....
..... Te (a y = 5)} is independently trained only on the respective triplet y. The student collects trajectories on all triplets, chosen at random for every episode.
Skill Generalization. We use the same technique for the generalization task, except that now have access to a single general teacher Te (-(ss.)) y). This generalist is conditioned on parameters y of object deformation (see Figure 2) and can specialize to different objects. During IIL, trajectories in Deis are from the training objects, as the test objects will not be seen until final evaluation.
4.3 One-Step Offline Policy Improvement for Sim-to-Real Policies
Once we obtained sim-to-real policy, we consider improving its behavior on the real robots. We perform a single step of policy improvement in which we: 1) deploy н. on the robots to collect dataset Dreat 2) train new policy "e imp offline, using filtering function [(s, a, T)) and 3) de- ploy the improved policy on the robot again for evaluation (1-3 could be iterated to obtain a policy improvement loop). The offline training step aims to maximize the following objective:
LiMP 0;; f) (yeT)~DDee
Ee.acer f(e,, a,, 7) log T (a (arlo(s)))
imp
(2)
We use the filter fs, a, T) exp(AA" a)/a) with temperature a (a hyperparameter) and learned Q-function (to calculate advantage), referred to as CRR-IMP, which recovers the CRR-
exp [25]/WWAC [26] objectives, but with data from the deployed policy only. Unlike in Section 4.1, we here use a binary, sparse, reward for success per step. We also consider f(s,a, a, r) = r(87), referred to as BC-IMP, which corresponds to BC on successful trajectories only (stack in last frame).
5

---

5 Experiments: Solving RGB-Stacking
We outline our efforts to solve the two stacking tasks we proposed in Section 3.2. In both tasks we evaluate our agents 5 fixed object triplets, and assume access unlimited simulated data but only to fixed budget (< 100 000) of real episodes. In Skill Mastery we have access to the fixed object triplets during all stages of training. In Skill Generalization our agents only have access to the training object set; the fixed triplets are now held out both in simulation and the real world.
In all experiments we use standardized evaluation protocol: the positions of the objects are ran- domized at the beginning of each episode, and the agent has 20 seconds at 20 Hz to stack the red object on the blue one. We define task success as having a stack the end of the episode. This definition was chosen to exclude cases in which a stack is briefly achieved incidentally in the middle of episode when the red object is above the blue one but subsequently falls off. In simulation, we evaluate on the training object set by running each policy on 5000 random object triplets for 2 episodes each. For the 5 fixed test triplets, we evaluate each policy for 1000 episodes per triplet. In the real world, we only evaluate on the test triplets, as evaluating on a sufficiently large sample of training objects to obtain statistics is impractical. We use one robot per triplet (evaluating for 200 episodes), for total of 1000 episodes per policy. Mean performance is calculated by averaging the success rate across the triplets. For state-based policies and vision- -bsed policies trained on real data we report the average across 2 training runs (i.e. random seeds), and for the aision-based policies distilled from a teacher we report the average across 4 runs: distillation runs for each of the teacher seeds. Results for all runs individually are in Appendix E.
5.1 Evaluation on the Simulation Stacking Benchmark
We first evaluate several baselines on our simulated benchmark task. We are interested in the follow- ing questions: (a) How difficult is the task in simulation? (b) Would it be preferable to train directly from vision? (c) How does interactive imitation and the design choices for it affect performance compared to the state-based teacher?
To assess the task difficulty, we tuned a scripted agent with access to the object positions and eval- uated it on both our training set and our test triplets. We also hired 4 individuals with no relation to the research team attempt the sim task via teleoperation and game pad for 846 episodes. For details on these indicative baselines, see Appendix C. These demonstrate the difficulty of the task as in both cases the success was lower than 50% (see Table 1).
Next, we evaluate our benchmark tasks in the dense reward setting with MPO from state. A first finding was that the state agents, which are given the full pose of each of the three objects, learn the task 10 times faster than the vision agent; as illustrated in Appendix D. using a vision agent to train from scratch on the robot would be impractical for these tasks. The MPO state-based agents obtain 79.3% on the test triples in the skill mastery setting and 68.8% on the training objects in the skill generalization setting, significantly outperforming the scripted agent. However, generalization from training to test objects is only slightly higher than the scripted agent's 's performance (47.8%). This is because these base- agents are conditioned on object parameters, which out of distribution for the test triplets. After obtaining vision agents via IIL-s2r we observe (Table 1, bottom) that although the training set performance drops, performance improves for the test triplets, the agents can now generalize from visual cues to make inferences about the shape.
We next compare the interactive imitation learning setting (IIL) to learning from teacher trajecto- ries with behavior cloning loss (BC). Rather than explicitly constructing a fixed size dataset, we generate the data on-the-fly by executing the teacher continuously during training. As shown in Table 1, learning from data generated by the student (interactive imitation) is crucial: IIL performed significantly better than the alternative in both task settings.
5.2 Evaluation on the Kerl-World Stacking Benchmark
We then investigate the following questions related to RGB-stacking in the real world: (a) Is possible to solve the challenging tasks on real robots with a single vision policy, and is that better than a scripted baseline tuned for the test triplets?
(b) How well does zero-shot transfer of sim- trained agents solve the tasks, and which of our decisions were important for zero-shot performance? (c) Do we get a single-step improvement with BC-IMP and CRR-IMP from using data collected in the real world? (d) And how does the data distribution affect the performance of these algorithms?

---

Simulation Success
Sim Success
Real Success
Method
Method
Train Set
Test Triplets
Test Triplets
Test Triplets
Human teleoperator
46.6%
Skill Mastery
Scripted agent
45.0%
43.1%
I1L- (deterministic)
71.7%
67.9%
Skill Mastery
I1L- (stochastic)
74.2%
67.4%
State teacher
N/A
79.3%
No Transformer
73.3%
69.3%
BC
N/A
52.4%
No image augmentation
72.8%
65.7%
11L-s2r
N/A
74.2%
No action delay
73.5%
68.7%
No binary gripper
66.1%
20.7%
Skill Generalization
MSE & no binary gripper
65.7%
25.7%
State teacher
68.8%
47.8%
Skill Generalization
BC
49.4%
41.2%
IIL-s2r
56.0%
51.9%
IlI-s2r
64.7%
56.0%
No Transformer
51.3%
45.4%
Table 1: Simulation Success. Comparison of distillation methods for solving. in simulation, our CGGB-stacking tasks from vision and propri- oception. These methods assume we have ci- ther a large dataset generated by a teacher policy (BC) or the teacher policy itself to supervise the student's interaction (III.. Interactive learning leads to much higher performance than learning only from teacher's behavior.
No object parameters
53.7%
41.1%
Table 2: Sim-to-Real Transfer Success. Ablations of the
components of the sim-to-real policy. The simulation and real-world evaluations use 20000 and 4000 episodes for the triplet average, respectively. The choice of using bi- nary gripper actions for the policy seemed to be the most important factor for sim-to-real transfer success, followed by the choice of executing the deterministic policy on the robots.
As described in Section 4.2, we distill a vision- policy from a single or multiple state-based experts in simulation. These are specialists for the Skill Mastery task and a single generalist for the Skill Generalization task. We can then directly execute this distilled vision policy on the real robots. As discussed previously, this decoupling allowed us to quickly iterate and investigate what aids sim-to-real transfer for these tasks. We compare, in Table 2, the simulation performance and the zero-shot sim-to-real transfer performance when ablating these various choices for our method. Policies were trained up to million learner steps, and the sim-to-real policy was selected to be the one with the highest performance in the training setting for each task. As our policies are stochastic, for inference we have the option to execute random sample from the action distribution or its mode (i.e. the action with highest probability). We execute the stochastic policy in simulation and the deterministic mode of the policy on the real robots, unless otherwise specified. We are able to achieve high performance of 67.9% on the Skill Mastery task and 51.9% on the Skill Generalization task in the real world. Qualitatively, we also see in Figure 3 that we can, with a single vision-based agent, show a diverse set of skills to address the challenges needed to solve the task for the 5 specific triplets we have chosen. Videos of the behavior are in the supplementary.
A Ablations. We ablate different algorithmic choices Table (extended table in the supple- mentary). One of our decisions was on the BC loss for distillation, which can either be the negative log-likelihood or the mean squared error (MSE), which respectively correspond to log T (a, (({))) or Ba-ssullla
"²", where Og are actions from the teacher (the two are not equivalent even for the Gaussian case as the policy variance is state dependent). We also decided to use hybrid action space, where the 3D Cartesian and angular velocities are modeled as continuous actions (Gaussian), but we have a binary gripper action (Bernoulli). This applies to both state- based and vision-based policies. This choice seemed be the most important factor for sim-to-real transfer success. However, note that the effects are only visible when transferring (corresponding simulation performance does not change as much). For the Skill Generalization task, the base- expert is conditioned on the object parameters However, the object parameters are not passed to the agent needs to infer physical object properties from image observations. We also chose to use a Transformer model [33] after the ResNet vision encoder for the policy's 's network architecture, which gives the agent access to temporal information with an attention mechanism. The Transformer does not have a significant effect on Skill Mastery. However, it seems to be able to utilize the skill variety of state-based expert conditioned on the object parameters a more successfully, when com- pared to an MLP model with the same number of weights. This is clear both in simulation and the real world (and we hence used it in all experiments). Random action execution delays of 0, or timesteps, and standard image augmentation (random color perturbations and image translations) did not seem to affect transfer but we decided to include them in our training procedure to increase our agents' robustness to natural perturbations.
the
7

---

Real-Robot Success
Method
Triplet Avg.
Triplet 1 Triplet 2 Triplet 3
Triplet 4
Triplet 5
Scripted agent _gent
51.2%
36.3%
23.0%
34.4%
84.9%
77.6%.
Skill Mastery
BC (scripted agent data)
53.4%
34.5%
26.8%
38.3%
84.5%
82.8%
CRR (scripted agent data)
43.4%
20.5%
28.0%
38.3%
64.0%
66.3%
11L-s2r (sim-to-real)
67.9%
72.8%
48.8%
61.0%
85.1%
71.9%
Sim-to-real agent for test triplets data
69.6%
76.4%
52.7%
60.4%
86.5%
72.0%
BC-IMP (sim-to-rcal agent data)
74.6%
75.5%
60.8%
70.8%
87.8%
78.3%
CRR-IMP (sim-to-real agent data)
81.6%
87.3%
68.3%
75.3%
88.3%
88.8%
Skill Generalization
slL- (sim-to-real)
51.9%
25.0%
39.6%
36.5%
91.1%
67.0%
Suboptimal agent for training set data
32.6%
21.5%
16.5%
17.0%
60.0%
48.0%
BC-IMP (suboptimal agent data)
49.0%
23.0%
39.3%
39.3%
77.5%
66.0%
CRR-IMP (suboptimal agent data)
55.6%
31.3%
41.3%
42.0%
81.5%
81.8%
Different approaches for solving our RGB- RGB-stacking tasks in the real world. The kind of data used matters significantly for BC and CRR: using scripted agent data does not result in improved performance, but using data from a sim-to-real agent does. Note that the sim-to-real data used were collected by a single agent per task. For Skill Generalization, as data collection with multiple triplets from the training set was exceptionally time-consuming, we used a suboptimal agent trained earlier in our investigation. This agent was not trained with the best settings we now have for sim-to-real. See text for more details.
Table 3: Real-Robot Success.
Our sim-to-real approach also outperforms scripted agent tuned on the test triplets, as shown on Table 3. For the Skill Mastery task, we further improve results via offline RL with BC (BC- IMP) on 32 651 successful real episodes collected by the scripted agent and with CRR (CRR-IMP) on total of 67 446 episodes. which included the unsuccessful ones collected by the same agent. Hyperparameters and architecture choices were selected after such an investigation in simulation. As evident in the table, learning from scripted agent data is not better than sim-to- to-ll. Qualitatively, the policies learned seem to exhibit the same "robotic" movements as the scripted policy that generated the data they were trained on. However, single policy improvement step based on data collected by a sim-to-real agent with an average success of 69.6%: 85 213 episodes (58 979 successful) yields 74.6% (BC-IMP) and 31.6% (CRR-IMP), resulting in policies with remarkable stacking consistency.
As data collection for the training object set is particularly time-consuming, we only did a single collection (38 446 episodes, 37.4% of which were successful) with an earlier iteration of a sim- to-real agent for the Skill Generalization task (performance of 32.6% when evaluated on the test triplets). As before we trained BC-IMP and CRR-IMP on this data and again observed improved performance on the test triplets, this time using only the episodes collected from the training object set. Note that even though the suboptimal sim-to-real agent trained on the training set of objects was performing worse than the seripted agent, and the data in this case was more limited and of different objects, BC-IMP and CRR-IMP trained on this more diverse data lead to a significant improvement. Finally, Table 3 showcases the variation in difficulty of the 5 triplets: all methods perform best on triplets 4 and 5, these can sometimes be solved with a stereotypical "pick and place" behavior. In contrast, triplets 1, 2, and each require specialized, advanced, strategies as indicated by the large gap between sim-to-real performance and scripted performance in the Skill Mastery task.
Conclusion
We studied the problem of vision-based stacking with large variety of geometric objects that re- quire a diverse set of stacking strategies. We propose a benchmark for studying this problem in a principled way, and make significant progress in solving two tasks involving both skill mastery on specific objects and generalization across them. We use simulation-trained agents to collect data in the real world, which in turn can be used for further performance improvement with offline RL. Our best agent a single vision-based policy that is capable of variety of stacking strategies and achieves high performance. Finally, we provide thorough real-world evaluations and discuss what important for solving our tasks. Despite this success, our benchmark still poses many open chal- lenges as, for example, shown by the gap between Skill Mastery and Skill Generalization results and we hope that it can help development of new methods for learning general policies (e.g. by further adaptation at robot deployment time).
8

---

Acknowledgments
The authors would like to thank Christopher Schuster, Nathan Batchelor, Serkan Cabi, Dave Barker, Jean-Baptiste Regli, Yusuf Aytar, Dushyant Rao and many others of the DeepMind team for their support and feedback during our project and the preparation of this manuscript.
References
[1] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. CoRR, htss/1509006825 2015. URL http:://arr.v..org/aaaaa 06825.
[2] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. CoRR, abs/1603.02199, 2016. URL httt::tttt/sstte org/at ltggaa///000 02199.
[3] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang. D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learn ing for vision-based robotic manipulation. CoRR, abs/1806. 10293, 2018. URL http: httlv/.org/bb///80/00 // //arxiv.org/aatt aaaaaaaa//100 /106. 10293.
[4] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, Pastor, K. Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 4243-4250. IEEE, 2018
[5] Furrer, M. Wermelinger, H. Yoshida, Gramazio, M. Kohler, R. Siegwart, and M. Hutter. Autonomous robotic stone stacking with online next best object target pose planning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2350-2356, May 2017. doi:10.1 109.1109/ 201.CRR..117.7992722
[6] M. A. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, V. de Wiele, V. Mnih, N. Heess, and J. 'T Springenberg. Learning by playing solving sparse reward tasks from scratch. CoRR, 1802. 2018. URL http: http: //ax:vv./.vivvv// wttp:ivv.rr//b////222222 180208802200 10567.
[7] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, Cabi, Tunyasuvunakool, J. Kramár, R. Had- sell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018. doi: 10.1 w0..1560///...111.....
[8] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zoina, Y. Ay- tar, D. Budden, M. Vecerik, al. A framework for data-driven robotics. arXiv preprint arXiv: Ttt.//999........ 2019.
[9] R. Jeong, Y. Aytar, D. Khosid, Y. Zhou, J. Kay, T. Lampe, K. Bousmalis, and F. Nori. Self-supervised sim-to-real adaptation for visual robotic manipulation. arXiv preprint arXiv: 2019.
[10] L. Hermann, M. Argus, A. Eitel, A. Amiranashvili, Burgard, and T. Brox. Adaptive cur- riculum generation from demonstrations for sim-to-real visuomotor control. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1405- IEEE, 2020.
[11] A. Hundt, B. Killeen, N. Greene, H. Wu, H. Kwon, C. Paxton, and G. D. Hager. "Good Robot!": Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer. IEEE Robotics and Automation Letters, 5(4):6724-6711, 2020.
[12] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, 09-15 Jun 2019. URL ttte:://proccedings..o httpp//cddnn.......r..es/sss/// press/v97/ fujimoto19a.htm.
[13] O. M. Andrychowicz. B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.

---

[14] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[15] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller. Maxi- mum posteriori policy optimisation. 2018.
[16] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 627-635, 2011.
[17] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129 129. Oxford University Press, 1996.
[18] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, Kirkpatrick, R. Pas- canu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv:1511.06295 2015.
arXiv preprint
[19] F. Sadeghi and S. Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv: /.661.1.401111111 2016.
[20] J. Tobin, R. Fong, A. Ray. J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. CoRR, 0bss/1703.669 2017. URL http:://araaaa
org/ abs/1b03/ 06907.
[21] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1710.06537, 2017. URL http://arxiv. org/abs/ /1700. 06630..
[22] S. James, A. J. Davison, and Johns. Transferring end-to-end visuomotor control from simulation to real world for multi-stage task. CoRR, abs/1707.00267, 2017. URL http: //arxiv. warg/abs/1700 02267.
[23] I.. Pinto, M. Andrychowicz, Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. CoRR, abs/1710.06542, 2017. URL http://arxiv.orr abs/1710, 12.414...0652.
06542.
[24] J. Tobin, L. Biewald, R. Duan, M. Andrychowicz, J. Schneider, P. Welinder, al. Domain randomization and generative models for robotic grasp- ing. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3482-3489, IEEE, 2018.
A. Handa, V. Kumar, B. McGrew, A. Ray,
[25] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, and N. de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[26] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09.: 09359, 2020.
[27] R. Jeong, J. T. Springenberg. J. Kay, D. Zheng, A. Galashov, N. Heess, and F. Nori. Learning dexterous manipulation from suboptimal experts. In Conference on Robot Learning (CoRL), 2020.
[28] N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman. Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 18()):172-188 Jan 2018. ISSN 1558-3783. 10 100.111909ASE22262660555
[29] M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar. Robel: Robotics benchmarks for learning with low-cost robots, 2019.
[30] M. Mason. The mechanics of manipulation. In Proceedings. 1985 IEEE International Conference on Robotics and Automation, volume 2, pages 544-548, March 1985. doi: 11110990800T119851887777
10

---

[31] C. Ferrari and J. Canny. Planning optimal grasps. In Proceedings 1992 IEEE International Conference on Robotics and Automation, pages 2290-22999 2295 vol.3, 1992. 1ot:10.1109/RBBO 1992.219918.
[32] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. CoRR, abs/1703.00 /001170039112, 2017. URL http: http://axxvvvorro org/abs/ /wts//10.... 0003009.........
[33] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xi: Attentive language models beyond a fixed-length context. arXiv preprint WrXiv:191..02800 2019.
[34] D. Wang, D. Tseng, P. Li, Y. Jiang, M. Guo, M. Danielczuk, J. Mahler, J. Ichnowski, and K. Goldberg. Adversarial grasp objects. In Proc. IEEE Conf. on Automation Science and Engineering (CASE), pages 241-248. IEEE, 2019.
[35] R. M. Murray, S. S. Sastry, and L. Zexiang, A Mathematical Introduction Robotic Manipu- lation. CRC Press, Inc., USA, 1st edition, 1994. ISBN 0849379814 14.
[36] B. Çalli, A. Walsman, A. Singh, S. S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in manipulation research: The YCB object and model set and benchmarking protocols. CoRR, abs/1502.03143, 2015. URL http: / arxiv. org/ abss/502 11111101211......1
[37] A. Zeng, S. Song, K. Yu, E. Donlon, F. R. Hogan, M. Bauzá, D. Ma, O. Taylor, M. Liu, E. Romo, N. Fazeli, F. Alet, N. C. Dafle, R. Holladay, I. Morona, P. Q. Nair, D. Green, I. Taylor, W. Liu, A. Funkhouser, and A. Rodriguez. Robotic pick-and-pee unddepace novel objects in clutter with aulti-afffordance grasping and cross-domain image matching. CoRR, abs/1710.01130, 2017. URL httt:///rxix org/ aaaaabas//11// 1aths//11...000 01330.
[38] N. Mansard, O. Stasse, P. Evrard, and A. Kheddar. A versatile generalized inverted kinematics implementation for collaborative working humanoid robots: The stack of tasks. In Interna- tional Conference on Advanced Robotics, Munich, Germany, 2009.
[39] A. Rocchi, E. M. Hoffman, D. G. Caldwell, and N. G. Tsagarakis. Opensot: a whole-body control library for the compliant humanoid robot coman. In IEEE International Conference in Robotics and Automation, 2015.
[40] Kanehiro, F. Lamiraux, O. Kanoun, E. Yoshida, and J. Laumond. A local collision avoidance method for non-strictly convex polyhedra. In Robotics: Science and Systems, 2008
[41] C. Fang, A. Rocchi, E. M. Hoffman, N. G. Tsagarakis, and D. G. Caldwell. Efficient self- collision avoidance based on focus of interest for humanoid robots. In International Confer- ence on Humanoid Robots, 2015.
[42] B. Stellato, G. Banjac, Goulart, A. Bemporad, and S. Boyd. OSQP: operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4): 637-672, 2020. 100.10007/1123220000000 112532-020-001790
URL https://doi.org///
[43] R. I. Hartley and P. Sturm. Triangulation. Comput. Vis. Image Underst. 6862):146-157, Nov. 1997. ISSN 1077-3142, 100:10..006/ciiuu1997.44777 URL https://doi.org// cviu. 1997. 0547.
[44] M.-K. Hu. Visual pattern recognition by moment invariants. IRE transactions on information theory, 8(2):179-188) 1962.
[45] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[46] V. Mnih, K. Kavukeuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 51 518(7540):5295333 2015.

---

[47] R. Munos, T. Stepleton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off off-policy reinforcement learning. In Proceedings of the 30th International Conference on Neural Infor- mation Processing Systems, NeurlPS 16, page 1054-1062, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
[48] A. Abdolmalcki, J. T. Springenberg. J. Degrave, S. Bohez, Y. Tassa, D. Belov, N. Heess, and M. A. Riedmiller. Relative entropy regularized policy iteration. CoRR, abs/1812.02256, 2018. URL http://arxiv. http:/awwwi..raaaa/////aa
org/ /bs///b/////// 1812. 02256..
[49] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Confer- ence on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 149-458, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:://proceedding. .aagggssmmm.... mlr. prwss/ ceeess//70/bbelemarrr
html.
[50] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25:1097-1105, 2012.
[51] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to-real In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11157-11166 2020.
[52] D. Ho, K. Rao, Z. Xu, Jang, M. Khansari, and Y. Bai. Retinagan: An object-aware approach to sim-to-real transfer. arXiv preprint arXiv:201 1.00011000 0.1448,, 2020.
[53] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Donain-adversariil training of neural networks. The journal of machine learning research, 17(1):099-20300, 2016.
[54] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adapta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[55] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via coltomiizd-to-ccnooii adaptation networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
[56] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasp- ing via randomized -oaneedd------cccaall adaptation networks. CoRR, abs/1812.07252., 2018. URL http://arxiz wwwpsiiwwwrr/aa///eeee/ee /tt///1812.222555
[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. arXiv preprint arXiv:/106.037722 2017.
[58] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[59] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
[60] Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc., 2017. URL https: .df.
[61] R. Li, A. Jabri, T. Darrell, and P. Agrawal. Towards practical multi-object manipulation using relational reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4051-4053 IEEE, 2020.
12

---

[62] Y. Liu, S. M. Shamsi, L. Fang. C. Chen, and N. Napp. Deep q-learning for dry stacking irregular objects. pages 1569-1576, 10 2018.
[63] A. Menezes, Vicente, A. Bernardino, and R. Ventura. From rocks to walls: A model- free reinforcement learning approach to dry stacking with irregular rocks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2057-2065, June 2021.
[64] M. Noseworthy, C. Moses, I. Brand, S. Castro, L. Kaelbling, T. Lozano-Pérez, and N. Roy. Active learning of abstract plan feasibility. In RSS, 2021.
[65] N. Macias and J. Wen. Vision guided robotic block stacking. In 2014 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 779-784, 2014. doi:10. 109/ 6942647.
[66] 7. Jia, A. C. Gallagher, A. Saxena, and T. Chen. 3d reasoning from blocks to stability. IEEE Transactions on Puttern Analysis and Machine Intelligence, 37(5):905 -18, 2015. doi:10. (U09/TPAML2014.255444
[67] A. Lerer, S. Gross, and R. Fergus. Learning physical intuition of block towers by example. In M. F. Balcan and K. Q Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings Machine Learning Research, pages 430- 438, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: /ptpss////cceecnss.... mlr. pressrressseeee http//////eeee..e.ee htmm...trm.........
[68] O. Groth, F. B. Fuchs, Posner, and A. Vedaldi. Shapestacks: Learning vision-based phys- ical intuition for generalised object stacking. In Proceedings of the European Conference on Computer Vision (ECCV), pages 702-717, 2018.
[69] Hamrick, R. Allen, Bapst, Zhu, K. McKee, J. Tenenbaum, and P. Battaglia. Relational inductive bias for physical construction in humans and machines. In C. Kalish, M. A. Rau, Zhu, and T. 'T. Rogers, editors, Proceedings of the 40th Annual Meeting of the Cognitive Science Society, CogSci 2018, Madison, WI, USA, July 25-28, 2018. cog- antivesciencesociey.org. 2018. URL https: ptttts::/minnen /t/m//mmmmdddiin.n.m.mm crg/cogc1i//18/////// papers/ 0341/index html.
13

---

Appendix
Table of Contents
A The RGB-Stacking Benchmark
14
The RGB-objects family
Benchmark Analysis..
A.3 Publicly Released Objects
B Environment Details
18
B.I Real-World Environment
18
B.2 Environment Observations
20
B.3 Object Position Estimation
21
B.4 Task Evaluation
22
B.5 Simulation
23
Baselines
26
C.I Human performance
26
C.2 Scripted Agent
26
D Methods
28
D.1 Details on Training Expert policies from State Features in Simulation
28
D.2 Details on Interactive Imitation Learning for Rim-to-Rcal Transfer
30
D.3 Details on Training Improved Policies from Real Data
31
E Experimental Details
31
E.1 Domain Randomization and Image Augmentation
31
E.2
Additional Network Architecture Details
33
E.3
Additional Training Details
36
E.4
Detailed Real-Robot Results
36
E.S
Qualitative Analysis
37
F
Additional Related Work
39
Additional Supplementary Material
40
A
The RGB-Stacking Benchmark
In this section, we provide more details on proposed benchmark and analyze its difficulty in var- ious ways. We qualitatively evaluate the grasping affordance based on general principles on force closure and object funnels [30], and quantitatively based on the Ferrari-Canny grasp metric [31, 32]. Similarly, we qualitatively depict the stacking affordance (Figure S5). which is varied by having bottom objects, whose top flat surfaces differ in area, shape, and orientation. We also quantita- tively evaluate both affordances by evaluating the stacking performance of human telcoperators in simulation, and of carefully scripted agent.
A.1 The RGB-objects family
The RGB-objects are all obtained by applying a certain deformation eube, which our seed object. We have defined four major axes of deformation (see Figure 2) of the seed object, which
14

---

Polygon Trapanidd Auk
Trapasiid pilcstngg Ask
&fectongle & A Asls
Podygae Pardkrngraa Auk
Toppasod Resiigg
Pedygas
Polygon Roctanglo
Parsteinggoee Axik
Paralllogprr A Traporoid
Panilsiograr & Ractangle Asis
Tropandd Ace
Re
Illustration of the deformations applied for each of the 15 axes of the RGB-Ohjects parametric family (major axes and their unique pairwise combinations). Each deformation changes the stacking affordance of RGB-oojects.
Figure S1:
result in different shapes. These shapes can also be thought of as the vertical extrusion of a 2D shape, which is also the name of each axis. As already described in the main paper but reproduced here for completion, these are:
Polygon-Axis [N] deformation obtained by transforming the extruded planar shape (i.e. the square) into regular polygon.
Trapezoid-Axis [R+ J: deformation obtained by progressively morphing the planar square to a isosceles trapezoid.
Parallelogram-Akki [R.,:: I: deformation obtained by changing the orientation of the extru- sion axis, from vertical (i.e. orthogonal to the plane of the planar shape) to progressively more slanted axes.
Rectangle-Axis
[ }: deformation by uniformly scaling the object along the y or z-axis.
These deformations and their combinations define a parametric family of objects. Figure SI shows representative sampling of this family. For our Skill Generalization task designate the axes of all pairwise combined deformations as the training axes and the ones of single deformation- the major axes above-as the held-out axes. The training axes are Polygon & Trapezoid, Polygon & Parallelogram, Polygon & x-Rectangle, Trapezoid & Parallelogram, Trapezoid & x-Rectangle, Trapezoid & Reeetangle, Trapezoid & z-Rectangle, Parallelogram & x- x-Rectangle, Parallelogram & y-Rectangle, Parallelogram & z- Rectangle, and x-Rectangle & y- y-Rectangle. Pairwise mixing of the major axes leads objects that are duplicates and therefore we omitted certain axes and objects (e.g. the X-Rectangle& & y-Rectangle and ReRetaagle & z-Rectangle axes). Also note that the x-, y-, refer to these as single major Rectangle axis.
Rectangle axes are the same
Based on these 15 axes illustrated in Figure SI, we created a training object set, which consists of 103 different shapes, and a held-out set, containing 40 shapes. Figure S2 shows a depiction of all the objects that are included in the benchmark and were used in our experiments. The 5 specific triplets chosen for the Skill Mastery task belong to the held-out axes with each object in triplet being the seed or from a different a While final performance is evaluated on these 5 fixed test triplets for both tasks, during training for Skill Generalization we hold out not just these objects but the entire 4 axes of deformation they belong to. That is, during training we can use the 103 objects from the training object set, while performance is evaluated still the 5 specific triplet combinations from the Skill Mastery task.
Techhically and for legacy reasons, although the objects for the 5 triplets are sampled from the held-out axes, not all of them are actually depicted in the held-out object set illustrated in Figure $2. As seen in their figure in the main text, are the seed cube object (itself held out), and out of the 15 are actually in the held- out object set. These are the 4 top objects that are not the seed cube. The rest 8 objects are almost identical to existing objects in the held-our out object set shown in Figure S2, In the released set of object models, these will be in their sub-directory.
15

---

Parallelogram
Trapezoid
Polygon
y-, Rectangle
x- x-Rectangle y-Rectangle
Polygon & Trapezoid
Parallelogram z- z-Rectangle
Polygon & Parallelogram
Parallelogram y-Rectangle
Polygon & x-Rectangle
Parallelogram X-Rectangle
Trapezoid & Parallelogram
Trapezoid & z-Rectangle
Trapozoid & Reetangle Trapezoid & y-Rectangle
Figure S2: The RGB-objects that are included in the benchmark grouped according chosen axes deformation. seed ohject the center; all the other objects are result deformations of this cube. These deformations change the grasping and stacking affordances of the objects. The held-out objects (major axes) are enclosed in the teal sector; the training objects (pairwise mixing of two major axes) enclosed in the blue sector. Some objects cannot be grasped with parallel gripper with 85 aperture (i.e. Robotiq 2F-85); these objects are transparent and were omitted in our experiments.
the 15
Rectangle Axis object funnel
Trapezoid Axis object funnel
Figure S3: A visualization of parallel-gripper grasp-affordance. Left: visualization of how the rectangle- object grasp-funnel (beam with square section) varies with object-gripper relative orientation: successful grasps are invariant to significant orientation differences. Right: visualization of how the trapezoid-object (trapezoid section) grasp-funnel varies with the object-gripper relative position: small differences in the relative pose hamper successful grasp.
A.2 Benchmark Analysis
The design principles outlined above aim at varying the resulting objects' grasp and stack affordance for parallel gripper. this section we qualitatively and quantitatively discuss these variations. The qualitative evaluations follow from some general principles on force closure and object's funnel (see Mason [30] for a definition). Figure S3 shows the graphical notation which we will use to qualitatively visualize the grasp affordance with respect to the gripper-object relative pose (left) and rotation (right).
A two-contact stable grasp is achieved if and only if Murray et al. [35, Theorem 5.6] the line connecting the contact points lies inside both friction cones (Figure $4).
16

---

Axes of Deformation
Sced
Polygon
Trapezoid
Parallelogram
Rectangle
Q = 0.132
Q 0.110
Q - 0.041
Q 0.128
Q 0.045
Figure S4: A sketch of the design principles adopted to vary RGB-objects' grasp affordance as a function of the applied deformation. Top row: keeping in mind object funnels for the parallel gripper. different objects tolerate different 4-DoF displacements (3D Cartesian and vertical rotation) with respect the grasping pose visualized picture; in green: displacements leading to successful grasps, in red: displacements leading to unsuccessful grasps. Bottom row: a visualization of the grasp metric used in Mahler et al. [32], Wang et al. [34], Each grasp is visualized a line segment penetrating the object at the grasping locations; the color of the segment corresponds to the corresponding value of the Ferrari- Ferrari-Canny [31] epsilon grasp metrie (robust grasps green, weak grasps in red). The displayed value the average of the epsilon metric for all visualized grasps.
Figure $4 shows this qualitative visualization for some representative RGB-objects: the seed object, and 4 maximally deformed objects of the 4 major axes. The visualization aims at showing that the seed cube relatively easy grasp and forgiving of significant errors in the gripper positioning. The Polygon axis requires precise positioning but it's relatively robust towards errors in gripper orientation. The Trapezoid axis offers two non-parallel faces which require accuracy in both posi- tioning and orientation. The Parallelogram axis is designed to offer the same grasp affordance of cube but quite different stack affordance. Finally, the Rectangle axis elongates one dimension above the gripper maximum aperture thus preventing qualitative considerations are supported by a quantitative metric, based on the Ferrari Ferrari-Canny Canny [31] epsilon grasp metric. This evaluation is shown for each of the objects considered, using the code provided in Mahler et al. [32], at the bottom row Figure S4. A high Q, used to symbolize the average metric for all grasps sampled the object, signifies that the object evaluated is easy to grasp. A low Q, as the case e.g. with the Trapezoid, means that the object is harder for grasping.
grasps given orientations. Interestingly, these
Qualitative considerations similar to the ones done above for the grasp-affordance hold for the stack- affordance, as shown in Figure S5. this case, the affordance varied by having bottom objects the top flat surfaces of which differ in area, shape and orientation. The Polygon axis introduces a deformation which reduces the top surface and increases the likelihood of the object to roll; the Trapezoid axis just reduces the top surface and in some configuration doesn't 't afford grasp at all requiring a re-orientation to another configuration which affords a stack; the Parallelogram axis offers stacking surface which has off-set with respect to the center of mass and therefore requires the top object to be carefully placed to avoid objects from tipping over; the Rectangle axis offers an augmented stacking surface along axis and reduced
another axis.
We have discussed how the shapes of objects influence their grasp and stack affordances. Other interesting affordances are needed to effectively solve these tasks: (1) the clutter affordance and (2) task-oriented affordance. Clutter influences affordance since only a subset of grasps is feasible in presence of obstacles (right panel in Figure S6). Additionally, for some objects valid grasps are
17

---

Axes of Deformation
Sced
Polygon
Trapezoid
Parallelogram
Rectangle
occcecot.o
Figure S5: A sketch of the design principles adopted to vary RGB-objects' stacking affordance. Bottom objects offer support different shapes. From left to right: curved support, small support, slanted support and wide support.
Figure S6: Left: visualization of good stacking-oriented gripper orientations (in green) and good only- for pick&plccppee gripper orientations (in yellow) the slanted cylinder. Right: sketch to visualize how affordances vary the clutter; only a subset of grasps are possible the clutter.
suitable for stacking and this requires our agent perform task-oriented grasp (left panel in Figure S6).
A.3 Publicly Released Objects
Instead of standardizing the objects purchase as e.g. in Çalli al. [36], we standardize their man- ufacturing procedure. We will be releasing the RGB-Objects described above and used in our ex- periments, depicted in their entirety in Figure S2. We decided choose the objects to be uniform color and eventually chose only three colors: red (top objects), green (distractor objects) and blue (bottom objects). This facilitates manufacturing since each object can manufactured with stan- dard 3D printer, single filament and no additional assembly steps. Additionally, we can provide, to interested researchers, instructions on how to have such objects 3D-printed by an external vendor.
B Environment Details
In the following, we are going describe the components of the robot setup that was used conduct experiments, as well as technical considerations such the procedure for automated evaluations, specifics of the actions and observations exposed the agent, and reward computation.
B.1 Real-World Environment
The environment in the real world consists of a robot arm with gripper, set of sensors and basket (Figure S7), chosen both for their durability to allow continuous and autonomous operation and for safe interaction with human operators.
18

---

Figure S7: Overview of the physical system. Left: Front view the basket. Top right: Neutral pose towards which the Cartesian controller's 's null-space is biased. Bottom right: Detail view of the endpoint tooling: force- torque sensor and gripper with custom fingertips.
Figure S8: Example image observations provided to the agent. These captured from the basket cameras, then cropped and sub-sampled to 128 x 128. From left to right: front left view, front right view, and back left view. this work, only used the front views.
Robot arm: assuming that exploration and manipulation require a certain level of physical inter- action, we have chosen robot capable of sensing, controlling and enduring the forces exchanged with the surroundings and the manipulated objects. We eventually selected the Sawyer from Rethink Robotics", both for its force and torque sensing capabilities and its use of series elastic actuators that provide passive compliance.
Gripper: the gripper chosen is a Robotiq 2F-85 which guarantees industrial robustness while allow- ing additional interaction through passive-spring retracting degree of freedom. It is outfitted with custom fingertips printed from nylon, as the stock fingertips' rubber coating tends to wear off onto the objects and interfere with tracking.
Sensors: besides the torque and position sensing offered by the Sawyer, we equipped the robot with Robotiq FT 300 force- tore-torque sensor at the wrist. Additional perception is guaranteed by surrounding the robot with three Basler ace RGB cameras which give complete view over the robot playground (Figure S8).
the basket in front of the robot, also referred to as "playground", is laser-cut basket with slanted sides to delimit the robot's working area and to help with objects confinement. It has a cm x 25 em bottom surface; the robot constrained moving its TCP inside a cm-high virtual cube on top of this surface to ensure safe operation.
Playground:
TTe Sawyer is developed and retailed by the Hahn Group
19

---

Component
Degrees of Freedom
Range
Unit
Cartesian translation
3
0.07, 0.07]
m/s
Cartesian rotation (z-axis) Gripper
1
[-1, 1]
rad/s
[-255, 255]
ticks"
Table S1: Ranges and units of the different components of the agent action.
B.1.1
Control Actions
While the robot's 7 DoFs natively controlled in joint space, implement Cartesian controller to reduce the action space. We restrict the gripper to be oriented vertically, thus allowing only 4- DoF motions (3D Cartesian and ID rotation). This restriction is used in a number of prior works studying vision-based manipulation [2. I, 37]. A control action is fully specified by a 3D Cartesian velocity Uz: Vy. 2/z. and an angular velocity w' around vertical axis parallel to gravity. We use P-controller to compute the horizontal angular velocity components We and wy that keep the gripper oriented vertically. and combine with the agent's 's actions to create the command for our Cartesian 6D velocity controller. Finally, directional velocity action for the grippers single degree of freedom added, yielding the actions summarized Table $1.
At every environment step, after choosing an action, we then solve a constrained least-squares problem to compute the joint velocities that best realize a target Cartesian 6D velocity of the TCP [38, 39]. The null-space is controlled to bias the robot's joint positions towards a nominal configuration in the center of the joint limits. Constraints are specified prevent the robot from violating the joint position and velocity limits, well as avoiding collisions between the robot and the play ground [40, 41]. The constrained least-squares problem solved using an off-the-shelf QP solver [42], and the computed joint velocities are forwarded to the robot's proprietary joint velocity controller.
B.2 Environment Observations
The robot, sensors and cameras provide various readings, which are collected at fixed rate of 20 Hz and merged into single observation. Not all possible observation elements used in all stages of the system; most notably, the Cartesian object positions provided by the tracking system described in Section B.3 are only used for computing rewards, for reset, and for the scripted baselines-the the learned agent cannot access this information.
We distinguish between two observation sets.
Full: contains all values provided by the real robot. Parts are used for environment resets, reward computation, and scripted baselines.
Evaluation: a subset of the full set, with tracker information removed.
In addition, each observation is stacked over several time steps. Observation stacking was chosen since the physical system subject to actuation delays, and thus would fulfil the requirements of an MDP without stacking. Camera observations are excluded from the observation stacking due real-time and memory constraints.
The available observations, their units, and the places where each is used, are listed in Table S2. Note that all of these sets denote available observations, and agents can choose omit entries; for instance, the image observation from the back camera is omitted by our agent architecture to reduce inference time.
'The gripper does not allow setting velocities in natural units, but byte value that is mapped to a corre- sponding percentage of the maximum speed, which is nominally 150 mm/s.
'The actual values provided by the sensor are for no grasp. 2 for an inward grasp, and for an outward grasp that is not possible with non-hollow objects.
10 Measured relative to the base the robot.
This contains the previous 7-DoF joint action sent to the robot, which is distinct from the 4 4-DoF Cartesian action selected by the agent, and reduces ambiguity in the state.
20

---

Observation Set
Observation
Unit
Size
Obs. History
Full
Evaluation
Joint angles
rad
7
Joint velocities
rad/s
7
3
Joint torques
N m
7
Wrist pose
m. quat.
Pinch pose
m. quat.
7
3
Finger angle
ticks
3
V
Finger velocity
ticks
3
V
Grasp
discrete"
3
Wrist force
3
3
Wrist torque
Nm
3
3
Wrist velocity
rad/s
3
Front left camera
RGB values
Front right camera
RGB values
128 x
128
x 3
Back left camera
RGB values
128
128
3
Object positions"
Joint action"
rad/s
7
2
Table S2: Observations provided by the real-world robot setup.
(a) Maximum opening.
(b) 30: Minimum required open- generate task reward.
(e) Minimum opening for random initial episode states.
(d) 255: Fully closed.
Figure S9: Overview of various relevant gripper positions.
Furthermore, the Robotiq gripper used in setup has parallel mechanism that causes non-linear relation between the motor encoder ticks used throughout this work, and the Cartesian distance between the fingertips. Since this relation be hard to visualize intuitively, present number of poses that were used Figure $9.
B.3 Object Position Estimation
Several components of our setup rely on availability of a tracking system to determine the position of the objects, relative to the robot. Specifically, this is required for the scripted baseline used this work and described in detail in Section C.2, for computing per-timestep rewards (Sec- tion B.4), and for automatically resetting the environment in automated evaluations (Section B.4.2). therefore implemented color-based object position estimation algorithm that provides an esti- mate of the 3D centroids of the red, green and blue objects. Given the critical role of this component, we calibrate (both intrinsics and extrinsics [43]) and use all three cameras available in our robot setup. The position estimation algorithm works follows:
1. Convert the RGB into YUV images: this conversion allows finer control over colors us- ing the chrominance components UV and robustness over brightness variations trough the luminance component Y.
2. Apply red, green and blue color masking using UV components. It is worth noting that we used the same ranges across all robot cells used in this work, while regularly applying white balancing.
21

---

Figure S10: Various success conditions. From left to right: 1. Successful stack with gripper open and the top object in the cylinder region. 2. The objects centered but the gripper is closed. 3. The top object is off-center, with its centroid outside the admissible cylinder. 4. The top object below the cylinder region.
3. For each color, find the largest contour and evaluate its centroid using image moments [44].
4. Estimate the 3D centroids {[ Me. ze. (e of the objects in the robot reference frame through triangulation [43].
B.4 Task Evaluation
The performance of agent is tested on number of trials 200, unless specified otherwise. The initial state of the objects is randomized between episodes in the following way:
Using modified variant of the scripted controller (Section C.2), all objects are moved to random positions inside the working area.
The robot's TCP is moved to random position inside the working volume, excluding the lowest 8 em to avoid collisions with objects.
The wrist joint is rotated to random position within 1- g1
The fingers moved random opening angle within [0, 100) (i.e. open half-open; see Figure $9 for visualization).
Each trial lasts 20 seconds, or 400 steps at a control rate of 20 Hz. Trials are prematurely terminated when the wrist force sensor senses horizontal forces of greater than 2 2 or vertical force of greater than 2.5 N. In this case, an RL agent receives a discount of zero.
B.4.1 Reward Definition
We all can understand what a stacked pair of objects should look like, however it is surprisingly difficult to define for large set of geometric shapes, for the purposes of automated evaluations. Here we formalize the definition "stacking" used in the main paper.
For objects to be considered "stacked", the top object's position tem) must be inside a cylindrical volume of cm radius, starting 2.5 em or higher above the bottom one, and the gripper must be fully opened. Specifically, for object centroids [Ttop: Vtop: Stopl and [Toottom:: Ybottom; }bottom] and for finger-opening angle f, we define the sparse, binary, stack reward:
(as estimated by the tracking sys-
1, if (Ztop Zbottom 0.025) A (Z)pp Ytop) (Ibottom Bbottom )l < 0.03) A (f < 30)
0, otherwise.
(S1)
The open-ended cylinder was chosen to accommodate objects of arbitrary sizes. For instance, top (red) object of Triplet 3 has a length of 15 cm, the centroid can be considerable distance from the bottom object when standing on its long side. A visual depiction of different stacked pairs being considered successful (green) or failures (red) is given in Figure S10.
22

---

B.4.2 Automation for Unattended Learning and Evaluation
The training and evaluation process used in this work is largely automated. In fact, the experiments were continued throughout multiple COVID-19 lockdowns. In particular, the randomization proce- dure described above is fully automatic, with modified version of the PID controller developed for the scripted baseline (Section C.2) being used to move objects. Note that the lack of object orien- tation provided by the tracker means that pose randomization is not deliberate. Instead, rely on incidental rotation when objects are dropped, and run evaluations for at least 200 trials to reduce variance.
pool of identical robot cells is used for data collection. To guarantee comparability of results, evaluations are always performed on the same five cells, each of which is associated with one specific test triplet. Furthermore, tare performed on the wrist force-torque sensor between episodes, in order to prevent drift. All cameras are white-balanced and their brightness adjusted to the same level, to counteract both daytime fluctuations in lighting, and differences between individual robot cells.
Evaluation requests are enqueued for each cell, and processed in order of arrival; thus, there is no closed training- triniig-evaluution loop, but training from any robot data always has to be offline to some degree. In the absence of new evaluation requests, old ones are automatically repeated in order to gather additional data and reduce variance. A number of consistency checks between episodes ensure that all sensors report data at the expected rates, and that actuators are operational -failing these checks would trigger the only required human interaction.
B.5 Simulation
Our simulation environment was implemented in the MuJoCo [45] physies simulator. Like the equivalent real robot environment, it contains Sawyer arm with Robotiq 2F-85 gripper mounted behind the playground, with three cameras attached to the basket.
The simulation was designed to provide the same observations as the real robot, with the same ranges and shapes. A small number of observations were too dissimilar to of use, notably the torques, and are thus omitted. For the full list of available observations in our simulated environment at different stages of training, see Table S5. Like the observations, the simulation exposes the same 4-DoF Cartesian actions as the real robot's given in Table $1. It uses the same QP controller as described in Section 1. to compute joint velocities from Cartesian velocities at 20 Hz. It was also designed to have similar appearance and dynamics to the real environment. However, as Figure $11 illustrates, the low-level appearance is noticeably different. Likewise, the physies differ in the way objects interact and slide off each other.
The evaluation protocol in simulation follows that of the real robot, with a few key differences.
1. We perform 1000 evaluation episodes per policy per Triplet rather than 200.
2. The entire 6-DoF pose of the objects is randomized, rather than only the position.
3. The sparse evaluation reward makes use of privileged information from the simulator, which isn't available on the real system, specifically whether objects are directly in contact with each other. This allows us to have a wider admissible cylinder of 5 cm in which the top object may be placed, and eliminates the need to check the gripper's 's opening angle.
if [3rop: Mtop) (Ibottom Wbottom)) ||
0.05
if (Ztop Zbottom) < 0.02
r
if top object is not in contact with bottom object if top object is in contact with robot or basket 1. otherwise.
(S2)
B.5.1 Shaped Reward
Our shaped reward, which we designed to use for training state-based agent in simulation only, forms a curriculum leading to a successful stack. It is divided into five progressive stages: reaching
When evaluating the training object set evaluate episodes for 5000 triplets.
23

---

Canonical Simulation
Real Robot
Camera Observation
Camera Observation
Front Left
Front Right
Front Left
Front Right
Figure $11: Real and simulated environments. Equivalent real and simulated environments with the camera
observations used during training.
0.9
6
06-
0.4
2
0.3
0.1
60 80 100 120 1.40
1.00
1.00
200 200 240 200
200
300
Figure S12: Reward trace for 300 steps of an episode. From left to right: 1. Approach during Rreach part
of Ryrasp stage. 2. crwpoeegrrpppecooooeee of Ryrusp stage becomes active as the gripper is closed while realigning the graspable object faces. Transition Rust stage the object is slightly lifted. 4. Rhover increases the object moved closer the target. Objects precisely enough placed he considered stacked as per Retock- 6. Gripper has moved far enough away to enter final Rieave stage.
and grasping the top object, lifting it more than 10 em above the basket, hovering the top object over the bottom object, stacking it, and leaving the objects stacked by moving the gripper away from them. Each stage generates reward in [0, 1], and the highest-level stage to produce reward of 0.1 or more is considered the "active" one.
4+ Rirana
if Rieave > 0.1
5
3+ Ratonk
if (Rstack 0.1) A (Rieave $ 0.11)
2 RRuver
if (Rhover > 0.1) (Rstack < 0.1) (Rieave $ 0..1)
(S3)
if (Rise > 0.1) A (Bhover < 0.1) A (Retack & 0.1) A (Rieave < 0.1) otherwise.
Ryraep
Intuitively this amounts to an agent being rewarded incrementally for each of stages that are required to complete stable stack. A detailed description of each of the stages and the definition of the equivalent rewards can found below. example reward trace illustrating the different stages is also shown in Figure $12.

---

1
Dilja-bll. 0.15, 0.0) Dijja-bll, 0.01)
0.8
0.6
0.4
0.2
0.05
0.1
0.15
0.25
Figure S13: Examples distance function used in several reward terms, with the x-axis showing distance between two entities a and b. Note how the value always decays to 0.05 (dashed line) as the distance reaches the shaping tolerance
We now describe the distinct stages of the shaped reward described above, all of which produce rewards in [0, 1]. As laid in Equation (S3), the stages are combined into total reward that consists of the "highest" of these stages that is currently generating reward over 0.1, plus a fixed amount for each "lower" stage.
Several of the stage rewards make use of a distance function D(a, b, 8, t), which is defined as the tanh over the distance between and b, which decays to 0.05 the distance reaches If the distance is below a tolerance of f, the maximum value of is returned. This distance function is illustrated in Figure $13.
if if bl
D(a, s,t,)
1 tanh (la 61| tanh "00.5
2
(S4)
otherwise.
Reaching and Grasping The first stage Ryrasp provides reward when the tool center point (TCP) moved close to the top object, with an additional bonus for closing the parallel gripper that is given only when already very close to the object.
0.5 + Retosegripne 0.5.
if Rreach > 0.9 otherwise.
Rgrasp = Rreach
(S5)
shaped distance between the TCP position POSTCP (UTCP, WTCP, ZTCP) and that of the top object poStop (Ntops Utops Stop), de- caying within em and with no tolerance. The positions are provided meters with respect to robot frame of reference, centered around the arm's 's base.
The reaching component Rreach
Rreach D(postcP, postop: 0.15,0). 0).
(S6)
The component Rolose gripper in turn is maximal when the grasp sensor is triggered. If not, smaller shaped reward is given, which approaches its maximum as the gripper opening angle ff reaches its maximum closing angle of 255.
Roloes.grrpper
if grasp sensor triggered
Relose.gripper p(1.255.2550)
(S7)
otherwise
25

---

Lifting The lift stage Russ also makes use of the grasp component Relose grippers but multiplies it with shaped reward that linearly increases as the designated top object's centroid moves between a minimum height of 5.5 cm and a maximum of 10cm.
Rust Retose.grioper gripper Rmooeetop top. up
(S8)
1,
if Stop > 0.1
R Rmooe Aop.up
0,
if Stop < 0.055
(S9)
Stop 0.055
otherwise
0.1-0.055
where Stop is the height of the top object from the basket.
Hovering The hovering stage Rhover simply provides reward for the top object being close to a position 4 cm above the bottom one. Maximum reward is given with a em tolerance around this position to account for noise the tracking system. Outside this tolerance, the reward decays within 20 cm.
Rhover D (poStops pOShottom (0.0, 0.0, 0.04), 0.2, 0.01).
(S10)
Stacking The stacking stage Rstack is a sparse reward that is only non-zero when the red object's horizontal position within 3 cm of the blue one's, and its vertical position within cm of the point 4 cm above the blue one. Note that this differs from the open volume in which the red object is allowed be (which is used for the real robot's evaluation in Equation (SI)).
if (xtap: Mtop) (Ibottom, Wbottom) I! 0.03
0,
Rytack
if (Ztop Zbottom 0.04) 0.01
(S11)
1,
otherwise.
Leaving The final leaving stage Rieave is identical to the stacking stage Roteck but multiplied by a shaped term that rewards moving the TCP to a position 10 cm above the red object, thus forcing the agent to let go of the object. Since it is not important whether that position is precisely reached, maximum reward is given with a tolerance of cm.
Rieave
Rotack D(2TOP, Stop 0.1, 0.05, 0.03).
(S12)
C
Baselines
C.1 Human performance
As rough indication of task difficulty, we collected few demonstrations of the task in simulation from human teleoperators. The demonstrations were collected by 4 individuals who were not part of the research team. They used game pads to control the robot arm, and faced the same time limit as was used for evaluation of learned or scripted agents. Unlike agents, teleoperators were given single camera view at a high resolution. Teleoperators recorded a total of 846 episodes (the number varied from 141 to 331 per participant), with the object set randomly replaced every 10 episodes.
These demonstrations were not used to train any of the agents mentioned in the paper. Results are summarized in Table S3.
C.2 Scripted Agent
The scripted baseline is classical robotic control approach using lot of prior knowledge, coded in a lmite-state-machine It uses the same observations available to the agent, well as the 3D positions of the blue and red objects' centroids. In the real environment, the 3D positions of the objects are obtained from centroid estimation algorithm (Section B.3); in the simulated environment, the
26

---

Objects
Success Rate
Reward
Episode Count
(Sim) Triplet I
37%
23
204
(Sim) Triplet 2
36%
16
160
(Sim) Triplet 3
35%
24
159
(Sim) Triplet 4
59%
39
133
(Sim) Triplet 5
66%
47
190
Table S3: Average success rate and cumulative sparse reward for each of the test object sets, from human leleoperators.
Ini
odgec arocaro
abbob art suadeu
open object
corooppltion
2
G sng tog. oujectt
choomppttionl
Move closed grppeer So tofte Fright
Move closed of liper la botton object hovel posittin cocomppetion
Move closed groper to bettom object stack pesition cnoompinea
eleecte
on falore
Open gripper bottion object stack podition
abjectr stecsed
<uech Sporeed
dejects stecked
End
Figure S14: (Left) Examplc scripted baseline run for the test object set Each figure shows the state ID on the top right, and the current step counter the bottom left. (Right) State diagram for the finite-state machine used in the scripted baseline.
positions are obtained directly from the simulator. The performance of our scripted behaviour is meant be used data point understand how far task-solver can get when ignoring relevant information such as the objects' orientation and shapes.
The scripted baseline states and 12 unique transition functions. A state diagram representation of the FSM shown in Figure $14. During normal execution, the FSM starts the Oth state and continues through states 1-6 until the objects are stacked. the objects are found to be stacked at any point during the execution of the FSM, a transition to the final state ("End") is made. any of the states fail, the FSM immediately transitions to the 8th state, which re-positions the tool-center-point (TCP) of the robot and loops back to state 1. Note that states 0, 1, 5, and do not implement transitions or failure detection and are always executed until completion.
implemented through the use of a finite-state machine (FSM) with
The description of each of the states in the FSM is given below:
0. Init: No-op state that initializes the FSM and immediately transitions to the next state. This state always executes until completion;
1. Move open gripper to top object centroid: Opens the gripper and moves the TCP towards the position of the red object. Executes a non-zero angular velocity if slep > 100, zero otherwise. Completes the TCP is within a pre-defined threshold the red object. This state always executes until completion;
2. Grasp top object: Closes the gripper while maintaining the TCP position close to the red object. Executes a non-zero angular velocity if slep > 100, zero otherwise. Completes if grasp is detected based on readings from force reading. Fails the gripper closes and no grasp is detected;
27

---

3. Move closed gripper to safe height: Moves the TCP up while maintaining the gripper closed. Completes if the TCP is above 20 em. Fails if grasp not detected, or the distance between the TCP and the red object becomes too large;
4. Move closed gripper to bottom object hover position: Moves the TCP to a point at an ab- solute height of 20 em directly above the blue object while maintaining the gripper closed. Completes if the TCP is above the blue object. Fails if a grasp is not detected, or the distance between the TCP and the red object becomes too large;
5. Move closed gripper to bottom object stack position: Moves the TCP to point 3 cm above the bottom object while maintaining the gripper closed. Completes if the TCP is within a pre-defined threshold of this point. This state always executes until completion;
6. Open gripper at bottom object stack position: Opens the gripper while maintaining the TCP position cm above the bottom object. Fails if the objects are not stacked after open- ing the gripper;
7. End: Opens and lifts the gripper to an absolute height of 30cm. Final state. Fails the objects are not stacked;
8. Move open gripper to top object hover position: Opens the gripper and moves the TCP to a point at an absolute height 20 em directly above the red object. Executes non-zero angular velocity if step 100, zero otherwise. Completes if the TCP is above the red object at a pre-defined height. This state always executes until completion and will result in different "random" grasp orientations.
Position control of the TCP is achieved through a low-gain P-controller on the error between the desired 3D Cartesian position and the current position of the TCP. The desired position of the TCP computed on each state individually based on the predefined behaviour of each state and the measured position of the objects through perception pipeline. The orientation of the wrist is only actively controlled during the execution of the 1, 2, and 8th states, which execute a random angular velocity about the vertical axis after the 100th step,
zero otherwise. The outputs of the P-controller are passed directly to the first 3-DoF of the action space exposed by the environment, while the angular velocity commands (4th DoF) are set to zero during states that do not actively control the orientation.
Objects
Success Rate
Reward
(Sim) Triplet 1
35%
48
(Sim) Triplet 2
30%
58
(Sim) Triplet 3
27%
40
(Sim) Triplet 4
66%
128
(Sim) Triplet 5
67%
128
(Real) Triplet
36%
54
(Real) Triplet 2
23%
39
(Real) Triplet 3
34%
49
(Real) Triplet
8.%%
152
(Real) Triplet
77%
143
Scripted baseline performance success rate and average cumulative reward for each of the object sets the simulated and real environment.
Table S4:
Table S4 summarizes the average performance of the scripted approach on the test sets. Each test set was evaluated for 1000 episodes in simulation, and for at least 800 episodes in the real setup. The agent achieved a success rate of 43% on the training set over 10 000 episodes in simulation.
D Methods
D.1 Details on Training Expert policies from State Features in Simulation
As outlined in the Section 4.1, the first step in our approach is to train policy, in simulation, either specializing on each of the 5 fixed triplets for the Skill Mastery task, or a general one on the 092 727 triplets that possible with the 103 training objects for the Skill Generalization task. As discussed,
28

---

Observation Reward
State-Based (Simulation)
Vision Based (Simulation)
Vision-Based (Real)
Joint angles
Joint velocities
Joint torques
Wrist pose
Pinch pose
Finger angle
Finger velocity
Grasp
Wrist force
Wrist torque
Wrist velocity
Front left camera
Front right camera
Back left camera
Object positions
Object pose
Joint action
Shaped (Simulation)
Sparse (Simulation) Equation (S2)
Sparse (Real) Equation (S1)
Reward
Equation (S3)
Table S5: Observations and rewards used at different training stages. The state-based teacher trained with privileged information simulation, and then distilled to a vision- based policy that has access to images and only proprioception observations that are realistic in simulation and can also be later used for zero- shot tim-to-rcal transfer. For this reason, exclude velocity, force, and torque observations for distillation in simulation. For the one-step offline policy improvement, dataset. This improved policy now includes velocity observations, excludes force torque observations since they too noisy in the real system. We did not use the back camera in our experiments.
vision-based policy is trained from real-world
300
State-based Agent
Vision-based Agent
Average Epicodo Roward
0.6
Average
0.4
50
State-based Agent
Vision- Agent
00
10
20
30
50
60
0.0
Estimated
Estimated Hours
Figure S15: State-based vs Vision-based MPO training. Comparison of average reward (left) and average task success on the training set (right) for the Skill Generalization task when training from all available state information (State-based Agent) training from vision and proprioception (Vision- based Agent)
29

---

we found training directly from state features s to be significantly faster than training from vision in this step and thus exposed the full simulation state proprioccptiiv information from the robot and 6-DoF pose information about the objects- to the agent. The complete list of observations available to the state-based policy can be found on Table S5. At this stage of the learning pipeline, we are mainly concerned with obtaining high-performing experts in a fast manner in simulation. Thus on top of access to full state information, we also use a shaped reward which is only available in simulation and enables fast learning. The shaped reward is described in Section B.5.1 and visualized in Figure S12. We provide a comparison between training with state features vs, training directly from vision in Figure S15, As is evident from the comparison training from vision results in large slow-down in terms of training time.
As mentioned, any off-the-shel RL algorithm could have been used for training our sate-based policies. We opted to use MPO [15], which we found lead to fast policy improvement while allowing for stable learning. MPO does not directly optimize the RL objective, but instead considers a KL regularized objective that optimized with a policy iteration approach. Concretely, in iteration k we first learn a corresponding Q-function Q0 20-'(( (s, a, y) for the policy from the last iteration (starting from random policy No at ko), which can be learned from replay buffer 'D by finding the function that minimizes the squared temporal difference error:
(r(ss) +7
[0 (44+1,,d, Q% "St-" Ct, y)
arg min (S,,0e:
E
E d' (1
(S13)
where ø' are the parameters of target network [46] that are replaced with the current parame- ters e for the Q-function every 200 optimization steps, and we use 20 samples from the policy to estimate the inner expectation. Instead of the single transition temporal difference error above, Ab- dolmaleki et al. [15] also considered a n-step temporal difference target calculated via the Retrace this target in our MPO implementation. This Q-function is then used to define the following KL constrained objective for policy optimization:
algorithm [47] and
Lgro(g) = E
E and
IQTH (s,
(S14)
s.t. E [DKI. v), nu. (-1s,w)) < EF; PTH
where DKL denotes the KL divergence to the last policy, which restricts changes in the policy and induces stable learning. A solution to this problem can be found in closed form as q(ass, s y) x Tk-1 (a]ss, y) exp(Q ((s, a, v)/a) which can be projected back to parametric policy by finding the expert policy no, as the maximizer
Tee (als, v) arg max Tee
E a~it
[exp(Q=* as, o)) log ne (als,
AND
(S15)
s.t. E (TKL(TT-I(H) (-1s, v), ne, (-1s. v))] EM, Prgg
which corresponds to minimizing the KL between q and Ta, and where cm specifies an additional trust-region constraint placed on the policy (we set Ty. for each iteration to ne, after 200 optimization steps). We use trust-region constraint that splits the influence on the mean and covariance for Gaussian policies as in Abdolmaleki et al. [48]. When using the hybrid space of continuous and discrete actions, we have separate third trust-region constraint for the Bernoulli distribution, though we found that the discrete component didn't 't require a trust-region for stable learning. Optimization can be carried out via Monte-Carlo estimation of the objective using samples from the policy T to estimate the inner expectation, and samples from the replay buffer for the outer expectation. For a full description of the algorithmic details of solving this optimization problem we refer to Abdolmaleki et al. [15, 48].
D.2 Details on Interactive Imitation Learning for Sim-to-Real Transfer
After obtaining the experts via MPO, we distill the state based experts into single vision-based policy T via interactive imitation learning, as described Section 4.2. In this step the Th uses only a subset of the available observations (vision and proprioceptive readings but no information about object positions; see Table S5 for a complete list). The two key decisions made for distillation are: 1) We collect data using TT while it is being trained. For this purpose run large number of
30

---

"actor" processes (1000) in simulation with the domain randomized environment. These fetch the parameters from the learner process at the beginning of every episode and send data replay buffer Daa- 2) We train z0 based on feedback from the expert's on data sampled from the replay (this DAgger style training resulted in best performance as outlined in the experiments). We note that this is purely supervised learning problem on changing dataset; as is standard the influence of ts on the dataset collection process is only implicit (i.c. we do not calculate the gradient of the sampling process for data-collection.
D.3 Details Training Improved Policies from Real Data
When training improved policies from real data collected by executing "8" on the real robots, we use slightly different subset of observations for the improved vision- policy To
imp (now including
velocity information; see Table S5 for a complete list). As described in Section 4.3, we use a filtered cloning loss of the data for this purpose, with filtering function f(s;, (22 7) where corresponds to the trajectory data from the executed episode. When using BC-IMP, we simply set [(8t, Gt. (t,T) = r(sT), i.e. it is 1 if the binary sparse reward of the last step in the episode (at time T) is 1, and o otherwise. This sparse reward information is readily available from the recorded episodes. For (o(8)), dt)/ in which case we need learn an estimate of the advantage alongside the policy xo We follow the implementation of CRR [25] and learn a distributional action-value function [49] from the same data that the policy is learned from by gradient descent on the objective:
the exponential advantage filter (i.e. CRR-IMP), we use [(s1, dt. exp( tm
E CRR-IMP (0) (a.:0.: )-
D r(()) +
E
(Q (o(8411) Qo(o(s)))
aloat " (-10(s++1))
(S16)
where D denotes the distributional Q-learning operator, ø' denotes the parameters of target net- work (that are swapped for & every 200 optimization steps) and where (o(s))) az) now is param- eterized as categorical distribution with 101 categories representing equally spaced bins of values from |-150-0,0 150.0]. We learn this Q-function alongside the policy, and use Qu to calculate policy improvement (i.e. the advantage used in the exponential filter is also fixed for 200 optimization steps at a time). We calculate the advantage AT imp
(o(st), ot) using a Monte-Carlo estimate of A (o(s)), oz) = Q (o(s)) at)) a' e (-((((()))))
E
1Q (o(s),
where we estimate the expectation with 20 samples from Ta (=(o)s))).
imp
E
Experimental Details
E.1
Domain Randomization and Image Augmentation
As mentioned above, our strategy for solving the RGB-stacking tasks in the real world is simulation- to-reality transfer. is therefore of paramount importance to ensure that both stages described above will result in policies that are able to bridge the reality gap and perform well in our real-worke setup. We do so by relying on (a) simulation environment that is closely aligned to the real robot envi- ronment (in terms of camera poses, robot joint limits, etc.); and (b) a sufficient amount of domain randomization [19, 24] and visual data augmentation [50]. These ensure that the simulation-trainn policies can successfully deal with the domain gap that still exists between the simulated and the real environments, and the increased stochasticity of the real world. Although we did consider learned adaptation methods as used in prior work [4, 51, 52] like using domain-adversarii losses [53, 54] and randomized-tor -iaaziiedtt--annnicall networks [55], preliminary results did not seem to provide clear ben- efits on top of domain randomization and data augmentation.
E.1.1 Domain Randomization
Domain Randomization (DR) has been shown to be a simple and powerful method to achieve gener- alization of simulation-trained policies to the real world for robotic learning problems [56, 19, 4. 13]. In our simulated environment we randomized number of physical properties (e.g. mass, friction, damping, armature) for all agents, as well as the delay of executing their actions on the environment.
31

---

Domain Randomized Simulation
Camera Observation
Front Left
Front Right
Front Left
Front Right
Front Left
Front Right
Figure S16: Visual illustration of our visual domain randomization with different sampling the properties listed Table for the same set of objects Triplet 2. Although all geometries can vary freely, the RGB- objects are restricted to a certain range "around" red, blue, and green to aid with the identification of these colors in the real world as well, as that is the way the vision-based agent knows which objects is the top, the bottom and the distractor.
We also randomized a number of visual properties (e.g. object colors, object textures, camera poses, lighting) for our vision- agents at the distillation phase. our randomized environments we uniformly sample, from pre-defined ranges, colors and textures for all geometries in our simulator, as well as lighting, and camera poses create a large visual diversity. Action execution was ran- domly delayed 0, or 2 timesteps, the equivalent of 0, 50 or 100 ms. Most physics properties did not require any particular tuning and are perturbed uniformly within 100% of their default values in the non-randomized version of our environment. The only exceptions are the ranges for the tangen- tial, torsional, and rolling friction the gripper, which were tuned carefully to prevent unrealistic grasping behaviours, e.g. grasping and lifting an object by a corner. The ranges for these were de- termined by teleoperating the simulated robot with different friction values. A list of all properties randomized, along with the range these were uniformly sampled from, can be found Table S6. A few samples illustrating our object color and texture randomization can be seen in Figure S16. Note that MuJoCo multiplies the RGBA values if both fexture and rgha properties undesirably dark appearance. Thus, for each geom, we alternate sampling textures or colors. Our RGB-objects
set, which results
treated differently in order maintain their basic color, the task is defined based on the color theme of the objects. Firstly, they are never assigned texture. Secondly, the hue range of each object is predefined way that maintains the color theme, and for each RGB-object we sample the color in HSV space.
E.1.2 Image Augmentation
In order to further increase the diversity of the data and the zero-shot real-world performance of our simulation-only trained agents, applied number of image transformations to our visual obser- vations on top of domain randomization. In addition, we applied the same image transformations when directly training from real- world data (e.g. for policy improvement). Unlike domain random- ization, image augmentation is applied directly on image observations, so it is applicable to images from both simulated data and real-world data.
The following transformations are applied for image augmentation: random brightness, random hue, random saturation, random contrast, and random translation. These transformations are applied sequentially in that order. The random translations use bilinear interpolation and "reflect" fill mode (i.e. the input is extended by reflecting about the edge of the last pixel). For temporal consistency, we sample the random augmentations and apply the same random offsets for all images within trajectory subsequence. A list of the image augmentation properties, along with the ranges these
32

---

Property
Range
RGB
[[128, 128, 128), (255, 255, 255) RGB
Texture
of 117 textures
object color
[(-35, 0.5), (35, 1)] HSV
Green object color
(95, 0.5, 0.5), 1,1) HSV
Blue object color
[200, 0.5, 0.5), (270, HSV
Ambient light color
0.3,0.3,0.33 (1=0.1) RGB
Diffuse light color
(.6.0.,,..6)-111-00000 RGB
Camera front left position
(1 m 1.000,
Camera front left Euler
0.05) rad
Camera front right position
(0.967, 0.381, (1 0.1)
Camera front right Euler
(1.08,, 0.001, 2.362) (1 0.05) rad
Camera field of view
400,40]
Gripper friction coefficient Hand friction coefficient
[0.3, 0.0,0, (0.6,,,,,
(1.0, 0.00,, (1100)
Arm friction coefficient
(0.1, 01.000..
Basket friction coefficient
0L0,0.001100001)-1111
Objects friction coefficient
7.000.006,000001---1111
Objects
0.201-
Arm joint armature
1.- L0- kg
Hand driver joint armature
0.1. kgm²
Arm joint damping
0.1 0.
Hand driver joint damping
0.2 0.
Hand spring link joint damping
0.00125 01
Arm joint friction loss
0.3 (1 0.1) kg (m/(mms"
Actuator gear
(1. 0. 0. (11
Action delay
[0, 2] timesteps
Table S6: Domain randomization properties that are randomized in simulation and their ranges. These properties were sampled uniformly at the beginning of every episode.
Property
Range
Brightness
32 /255, 32 32/255] /24, 1/24
Hue
Saturation
[0.5, 1.5]
Contrast
[0.5, 1.5]
Translation (horizontal and vertical)
4, 4] pixels
Table S7: Image augmentation properties that are randomized and their ranges. We sample random offsets from these ranges and apply the same random offscts the entire sampled trajectory subsequence. We resample the offsets for each subsequence in the batch. That the random augmentations consistent across time, but not across batch.
uniformly sampled from, be found on Table S7. We chose these ranges qualitatively without any tuning for evaluation success. For the random perturbations the hue, we chose ranges small enough that the red, green, and blue objects stay reasonably close their respective colors. few samples illustrating the effect of image augmentation can be seen in Figure S18 for images from the real robots, Figure $19 for canonical images from simulation, and Figure S20 for domain- randomized images from simulation.
E.2
Additional Network Architecture Details
The inputs to the networks are preprocessed in the same way for all the networks. The image obser- vations are normalized to 00, 1]. whereas the non-in mon-immae observations are flattened and concatenated into a single vector. The actions are normalized to [-1, 1). The networks operate with normalized actions, i.e. critic networks processes normalized actions as inputs, and actor networks output action distributions the normalized space. The agent scales back the actions the original space when executing them in the environment.
33

---

Hyperparameter
Value
actor network
input normalizer size
512 (512, 512, 256, 256) 10-4
MLP sizes
MVN distribution Jmin
activations
ELU
critic network
input normalizer size
512 (512, 512, 256) ELU
sizes
activations
Table S8: Network Architecture Hyperparameters for State-Based Agents.
In both the state-based and vision-based agents, we use an input normalization layer for the non- image observations. This input normalization layer consists of linear layer, layer norm layer, and a tanh non-linearity. The size of the input normalizer refers to the number of units of the linear layer.
We use an output distribution layer for the output of the actor networks. The output distribution layer for the actor network outputs independent joint distribution of multivariate normal (MVN) distribution with diagonal variance for the continuous action dimensions, and a Bernoulli distribu- tion for the binary action dimension. The mean of the MVN is the output of linear layer and the diagonal standard deviation is the output of a fully-connected layer with softplus non non-linearity plus a bias of mmin- This distribution is not constrained to output normalized actions in [-1, 1]; instead, we clip samples from this distribution depending on the context. The logits vector of the Bernoulli distribution is the output of a linear layer with output size 2. This distribution scaled accordingly to output normalized actions in {-1,
State-based agents. The actor network consists of input normalization layer, MLP, and output distribution layer. The critic network starts with an input normalization layer for the observations and clipping of the actions to [-1, 1]. then both streams are concatenated, and followed by an MLP and a linear layer with 1 output. These MLPs use exponential linear unit (ELU) activations. See Table S8 for a full list of network architecture hyperparameters used for the state-based agents.
based agents. The actor network consists of an observation encoder, MLP, Transformer, another MLP, and output distribution layer. When using critic (i.e. in CRR), the critic network starts with an observation encoder for the observations and clipping of the actions to i-1, 1], then both streams concatenated, and followed by an MLP and discrete-valued output distribution layer.
The actor and critic networks use the same architecture for their observation encoders, but their parameters are not shared. The observation encoder consists of two parallel streams- ResNet stack for the image observations and an MLP with a final activation for the proprioception observations- and the outputs are merged by concatenation. The ResNet stack consists of pair of ResNet encoders (one for each of the two images), activation, flattening and concatenation (of encodings from both images), and MLP with a final activation. Each ResNet encoder consists of 3 ResNet group modules. Each group module first applies convolution followed by downsampling with max- max-pooling layer, and then applies residual blocks modules twice. Each residual block consists of 2 convolution layers interleaved with non-linear activations.
The output distribution layer for the critic network outputs a discrete- distribution with sup- port (Umin: Umax] that is uniformly spaced among Natoms atoms bins. The logits vector of this distribution is the output of a linear layer with output size naaooss
See Table S9 for a full list of network architecture hyperparameters used for the vision- agents. We found preliminary experiments that having each image processed by a ResNet encoder led to better sim-to-real transfer performance compared to stacking the two images along the channel dimension and processing this stacked image with a single ResNet encoder. sharing the parameters for the ResNets of both images led to even better transfer performance. This parameter sharing also has an additional advantage of computational speedups (this can be achieved by applying the ResNet in single pass to both images concatenated along the batch dimension).
also found that
34

---

Hyperparameter
Value
image observation encoder (actor and critic)
individual ResNet per image?
yes
share parameters
ResNets both images?
yes
ResNet number channels for each group
(64, 128, 256) 2
ResNet number of blocks group
ResNet convolution kernel size
3 x
ResNet max-pooling size
ResNet max- max-pooling stride
post-ResNet MLP sizes
(2556)
activations
ReLU
proprioception observation encoder (actor and critic)
input normalizer size
256
MLP sizes
(256)
activations
ReLU
network (after the observation encoder)
pre- reeTTransfrmer MLP sizes
(512, 512)
Transformer number of heads
4
Transformer number of layers
1
Transformer value size Transformer memory size post- Transfommer MI.P sizes
64
8
(256, 256)
-MIP sizes for ablation without Transformer MVN distribution min
(1024, 512, 512, 256, 256) 10
activations
ELU
critic network (after the observation encoder)
MLP sizes
(512, 512, 256) 101
discrete-valued distribution number of atoms Tlyoos
discrete-valued distribution range of values (Umin, Vmax activations
[ 150.0, 150.0] ELU
Table S9: Network Architecture Hyperparameters for Vision-Basee Agents. The critic hyperparameters only applicable to the methods that use critic, ie. CRR. Although the actor and critic networks use the same observation encoder architecture, their parameters are not shared.
Note that although the observation encoders use rectified linear unit (ReLU) activations throughout, the subsequent MLPs use ELU activations.
Transformer Architecture. While image augmentation and domain randomization helps with bridging visual and physics domain gap, the gap of transition dynamics remains. A potential ap- proach is give agents access temporal information, which encourages them to "reason" about the transition dynamics. has been shown in prior work on simulation-to-realiit transfer [13], that doing so bridge the reality gap further, and might tem identification that can help with transfer in previously unseen environments. The Transformer architecture has been widely adopted for natural language processing [57, 33] well for com- puter vision [58]. However its application to control problems remains limited. Thereby we explore the Transformer model, which demonstrated huge power of sequence based data processing with attention mechanism, to encode temporal information. In this work, we adapted to the Transformer. XL [33]. The transformer network has stacked self-attention module that apply to the input sequence repeatedly. The transformer module consists of 1) multi- head attention submodule followed by 2) a multi-layer perceptron network.
enable the agents to be performing sys-
The transformer torso takes encoded observations as input. The multi-head attention module applies scaled dotoprodu attention for every timestep:
QKT
Attention(Q2 K, softmax
(S17)
Vdz
35

---

Hyperparameterr
Value
MPO
discount factor 7
0.99
actions sampled per state
KL constraint eg on parapmearticc policy
0.1
trust-region éar policy (MVN mean)
X 10
trust-region 6ar on policy (MVN covariance)
x 10
trust-region ear on policy (Bernoull))
none
update period for target networks
200
General
batch size
512
trajectory length
10
environment frames per gradient step
replay buffer size
2 x 10
optimizer
Adam [59]
actor initial learning rate
x 10-6
actor learning rate decay factor actor learning rate decay schedule
0.9
range(100000)) 5000000,
critic initial learning rate critic learning rate decay factor critic learning rate decay schedule dual learning
x 10
0.9
range( Tange(100000g 1000000,, 5000000, 300000)
x 10
Table S10: Hyperparameters for Training State-Basee Agents.
In addition to perform a single attention operation, is beneficial to project Q, K, V h times with learned linear projections respectively, where is number of heads:
MultiHeeaAAttentiinn((Q K, V) Concat(head),, ..., head,)) W
(S18)
head, = Attention(Q:?? .W:K, VW.)
(S19)
Following the multi-head attention module, a residual connection and layer normalization are ap- plied. To leverage the order of the sequence, positional-encoding layer is added to the input embeddings. A fixed positional encoding using sine and cosine functions various frequencies are used in this work: SelfAttention(( (, LayaaNoom((uiiiHHadAAtttttooooo V) PositionalEncoding (V))
(S20)
On top of the self-attention layer, fully-connected feed-forward network is applied to the output of each timestep separately. The MLP consists of two linear layers with a ReLU activation in between.
E.3 Additional Training Details
We use trust-region constraint for the policy update in MPO and CRR. Similarly to Abdolmaleki al. [15], the mean and standard deviation of the multivariate normal (MVN) of the actor distribu- tion have separate trust-region constraints. However, we do not use a trust-region constraint for the not necessary for stable learning.
Bernoulli distribution component
found
We use different learning rates for optimizing the actor, critic, and dual variables. We anneal the learning rates for the actor and the critic, following exponential decay schedule denoted as range(Esuuts Fend; istep), where istart and innd indicate the gradient step iteration at which the annealing starts and ends, respectively, and sep indicates the interval at which the learning rate is multiplied by the decay factor.
See Table $10 and Table S11 for full list of training hyperparameters used for state-based and vision-based agents, respectively.
E.4 Detailed Real-Robot Results
In Table and Table we provide, for each setting, averages of multiple runs. These are runs for sim-trained distilled agents: 2 seeds for each of the 2 state-based teacher policies we distilled these
36

---

Hyperparameter
Value
IIL (DAgger) and BC
loss
negative Log-likelihood mean squared enror
loss ablation using MSE loss
CRR
discount factor y
0.99
actions sampled per state
20
KL constraint eg on non-parametric policy
0.1
trust-region
policy (MVN mcan)
0.1
trust-region exr on policy (MVN covariance)
0.1
trust-region EM on policy (Bemoulli)
none
update period for target networks
200
General
batch size
64
trajectory length
10
environment frames per gradient step (online)
250
replay buffer size (online)
1 x 10
85 213 episodes, 58 979 successful (Skill Mastery)
dataset size (offline)
38 446 episodes, 14 381 successful (Skill Generalization)
optimizer
Adam [59]
actor initial learning rate actor learning rate decay factor actor learning decay schedule
1 x 10-4
0.9
range(25000, 1000000, 25000) (Skill Mastery) 2 x 10
critic initial learning rate critic learning rate decay factor critic learning rate decay schedule dual learning rate
0.9
range(25000, 1000000, 25000)
10
Table S11: Hyperparameters for Training Vision-Based Agents. The critic hyperparameters are only appli- cable to the methods that use a critic, i.c. CRR. We found that CRR doesn't need a tight trust-region for stable learning, so we chose a loose constraint of 0.1 without further tuning. The number of environment frames per gradient step and replay buffer size only applicable in the online setting, which uses the simulated environment. The dataset size is only applicable in the offline setting, and the dataset consists of real-world episodes collected on robots. The dataset Skill Mastery only has the test triplets and the dataset Skill Generalization only has the training objects.
from, and runs for vision agents trained from real data. Here, we provide the results for each of the runs in each setting, and also present the results for each specific triplet. We hope that this provides a better sense of the variance in each setting. Entries in Table S12 correspond to Table 2, whereas entries in Table S13 correspond to Table 3.
E.5 Qualitative Analysis
In the main text we described different challenges posed by the test triplets. Aside from the quan- titative success scores above, it is therefore also interesting to look at the resulting policies qualita- tively, and examine whether our agent visibly learns to overcome these challenges. Even if such an approach is naturally to be taken only anecdotal, it nevertheless provides an indication of open challenges that are worth pursuing further.
We therefore performed a number of evaluations with the best-performing agent, the Skill Mastery CRR-IMP policy trained on sim-to-real agent data (see Table S13). We adversarially moved the objects to generate challenging situations and observed the agent's behaviour.
Triplet 1.
The main challenge of this triplet is the need to precisely orient the gripper, since closing them on the slanted sides of the object will fail. The agent exhibits this behaviour, waiting to close the gripper until the wrist is properly aligned (Figure S17(a)).
The bottom object in this triplet can be oriented in such a way that its top surface slanted, making it impossible to stack without first tipping it over. The agent can be seen to perform this kind of behaviour, although not perfectly reliably; if the object is already oriented so that can be tilted by pushing it against the basket's slope, it will do so (Figure S17(b)). However, if the
Triplet 2.
37

---

Samelation Success
Redl Robot Success
Mahsal
Triniay Objezs
Tiplet Avg.
Trijles
Tripke
Trijket 3
Triplet
Trightt
Triples Ang.
Triple
Trijkes
Triplet
Trightt
Triple
Sta Monery
=0 (onrmmisnn))
Teacher 1- Seed
N/A
72.8%
14.2%
67.5%
K2.315
18.455
84.8%
68.1%
14.5%
49.198
59 19.3%
$7.0%
70.5%
11 22 (dmemmneee))
Trasher
NA
71.2%
14.4%
61.89
73.7%
79.0P%
64.99
69.55
55.0%
71.0%
(esermaniss)
Teasher
NA
71.4%
75.2%
45.2%
79.9%
72.65
34.2%
66.1%
67.5%
49.0%%
57.5%
$2.0%
22r
(deermnnsss)
Teacter
N/A
71.5%
76.05
44.1%
50.95
71.855
34.75
63.7%
71.0%
33.0%%
57.516
35.5%
22r
(soocusttc)
Teacter
N/A
75.1%
50.05
52.115
87.25
66.7%
71.0%%
47.0P5
59.515
36.0%
71.0%
(sooharttc)
N/A
12.31
Teadher
51.2%
14.55
3646
76.1%
48.5%
83.5%
74.5%
<2 (soccastic)
Teacher 2-Seed
N/A
74.2%
53.68
74.45
86.2%
69.1%
54.5%
58.0%
No Tratsfomer
Teacher
N/A
74.3%
74.9%
52.6%
87.5%
7188
12.5%
36.5%
Transfomer
Teacher
N/A
72.5%
$1.35
13.246
86.2%
66.6%
40.515
53.5%
79.5%
No Trarafomecr
Toasher 1-Soxl
XIA
73.4%
78.59
48.69%
50.9%
73.0%
$5.6%
70.2%
71.5%
$1.0%
64.59
$6.5%
77.05
No Trarafomerr
Toaherr 2- 2-Soxl 2
X/A
73.1%
76.39
48.87%
52.19
72.29
56.0%
67.6%
60.5%
38.0%
5X,09
87.5%
85.0%
No image sugrerattion
Toadier 1-Ses8
NAA
74.1%
79449
50.25
B1.R
73.39
36.1%
66.6%
77.5%
44.0%
69.59
31.0%
61.0%
No image ssgnetttionn
Teadher 2-Sood
NA
71.4%
76.5%
47.8%
$1.68
65.0%
$6.2%
68.1%
67.5%
32.0%
44.5%
$6.0%
70.5%
No iagg sugmenitiion
Traeher 1- --aad33
N/A
74.1%
14.498
$1.0%%
K2.454
72.9%
39.65
70.85
78.09
53.9A
61 5%
90.0%
7005
No image sagneatsiion
Tiaeher 2-Sax8
N/A
71.6%
76.25
45.8%
$0.65%
70.25
$5.1%
56.9%
63.03
36. 56.
48.5 48.5%
79.05
67.5%
No sntionn delay
Tiashar 1- 1-Raxl
X/A
74.5%
70.06
$0,60%
k3.9%%
72.8%
$5.3%
71.1%
74.5%
52.5%
71.3%
$3.5%
73.5%
No accionn delay
Toahar 2- Sead
X/A
72.1%
75.29
47.8%
$1.3%
70.75
$5.31
67.2%
73,09
45.5%
36.5%
$9.5%
71.5%
No action delay
Teadher 1- -Sed
N/A
74.0%
77.3%
$1.55
K3.05
73.755
86.11 14.1%
68.9%
23.0%
43-216
09 69.3%6
80.5%
75.0%
No actionn delay
Tracher Sood
N/A
73.3%
74.8%
57.95
K3.65
71 11.3%
4%
67.6%
M6.5%
57 37.5%
58.5%
81.8%
74
MSF no hinory gripper
Trasher
N/A
72.2%
74.95
$1.05
78.45
19
32.8%
38.8%
44.95
73.5%
7.0%%
20.0%
MSE no hissrs pripper
Toasher
N/A
6000%
74.8%
0.36
65.4%
$1.8%
24.2%
56.09
0.0%
24.5%
60.5%
MSE no gripper
Teacteer
N/A
72.4%
73.355
35.5%
73.07%
71.066
33.7%
24.4%
55.0%
24.0%%
13.5%
39.0%
MSU
gripper
N/A
59.1%
73.1816
0.2%
75,05
06.35
21.0%
21.2%
27.0%
26.5%
hirery gripper
77.216
51.49
370%
43.6%
61.2%
10-218
01.5%
prtpper
No binayy gulpper
Teacher
N/A
74.5%
76.3%
53.2%
36.9%
16.4%
12.5%
8.0%
No binury gripger
Teacher
NoA
57.7%
1.0%
30.0%
13.9%
29.0%
Contrmiionnss
Teattrr
65.7%
58.9%
38.19
38.5%
44.7%
53.5%
90.5%
54.8%
43.5%
42.0%
93.0%
[1]
53.8%
15.7%
40.7%
54.6%
[1]
52.1%
12.5%
48.4%
20.55
30.5%
91.0%
Trarafomerr
61.5%
52.7%
22.09
34.4%
43.6%
22.5%
82.5%
Transforee
Teacher
57.0%
45.1%
11.7%
16.71
75.7%
46.8%
24.5%
28.3%
79.0%
74.0%%
Transfomerr
Teacher
64.455
54.2%
15.3%
41.1%
40.9%
81.256
92.3%
47.8%
36,09
17.5%
88.5%
65
No Tranafomerr
Teacher Sand
63.6%
53.2%
17.8%
99.4%
18. 15
9%
46.7%
34.59 54
36. 96.99
32.5%
87.5%
63.5%
No object panumeters
Teadher 1- SSed
65.0%
50.5%
29.9%
15.0%
38.34
$2.6%
35.0%
31.1%
23.0%
18.5%
24.0%
36.0%
59.0%
No object panumeters
Teather Seed 2
65.0%
58.0%
29.9%
41.0%
49.9%
$1.9%
87.4%
29.3%
24.5%
21.0%
21.5%
21.5%
52.0%
No object panameters
Teacher 2 Seed
61.715
53.2%
24.0%
33.3%
39.05
79.5%
36.0%
$3.5%
29.55
42.0%
29.0%
87.5%
79.5%
No object parameters
Teacher 2- Need 2
62.31
53.8%
24,74
38.7%
38,045
80.616
36.9%
49.6%
30,05
33.518
30.056
86.0%
78.5%
Table S12: Reall Transfer Success. Ablations of the components of the sim-to-real policy. This table
gives a full account of all evaluations for the equivalent Table 2 in the main paper. We execute the stochastic and deterministic policies in simulation and on the robots, respectively, unless otherwise specified.
Real-Robot Success
Method
Run
Triplet Avg.
Triplet
Triplet 2
Triplet
Triplet 4
Triplet 5
Scripted agent
N/A
51.2%
36.3%
23.0%
34.4%
84.9%
77.6%
Skill Mastery
BC (scripted agent data)
Seed I
50.8%
25.5%
22.5%
35.0%
85.5%
85.5%
BC (scripted agent data)
Seed 2
55.9%
43.5%
31.0%
41.5%
83.5%
80.0%
CRR (scripted agent data)
Seed
40.4%
17.0%
11.4%
41.5%
66.0%
66.0%
CRR (scripted agent data)
Seed 2
46.4%
24.0%
44.5%
35.0%
62.0%
66.5%
Sim-tt-rcal agent for test triplcts data
N/A
69.6%
76.4%
52.7%
660.4%
86.5%
72.00%
BC-IMP (sim-to-real agent data)
Seed 1
75.1%
76.0%
59.5%
70.0%
90.5%
79.5%
BC-IMP (sim-to-rea) agent data)
Seed 2
74.1% %
75.0%
62.0%
71.5%
85.0%
77.0%
ERR-IMP (sim- to agent data)
Seed
81.0%
88.0%
66.5%
74.0%
88.0%
88.5%
CRR-IMP (sim-to-real agent data)
Seed 2
82.1 %
86.5%
70.0%
76.5%
88.5%
89.0%
Skill Generalization
Suboptimal agent for training set data
N/A
32.6%
21.5%
16.5%
17.0%
60.0%
48 48.0%
BC-IMP (suboptimal agent data)
Seed 1
48.2%
23.0%
33.0%
37.0%
82.0%
66.0%
BC-IMP (suboptimal agent data)
Seed 2
49.8%
23.0%
45.5%
41.5%
73.0%
66.0%
CRR-IMP (suboptimal agent data)
Seed I
54.6%
27.5%
42.0%
41.0%
79.5%
83.0%
CRR-IMP (suboptimal agent data)
Seed 2
56.5%
35.0%
40.5%
43.0%
83.5%
80.5%
Table S13: Real-Robot Success. Different approaches for solving our RGB-stacking tasks in the real world. This table gives a full account of all evaluations for the equivalent Table 3 in the main paper, except for I1L- which are given in Table $12.
38

---

(a) Agent aligns gripper the graspable faces of the object.
(d) Objects aligned even after forced into off-center grasps.
The bottom object tipped over during approach.
(e) No attempe to adjust off-center stack.
(c) Failed artempt to stack onto sloped bottom object.
(f) Agent supports the top objoct for the rest of the episode.
Figure S17: Agent behaviour during challenging situations.
bottom object's orientation is unfavourable, it will not rotate it to achieve this (Figure S17(c)), and instead try to naively stack the objects.
Triplet 3. The main challenge in this triplet lies the asymmetry of the top object, and needing to balance it onto the bottom in such way that their centroids are aligned. Even if disturbed into an off-center grasp, the agent still aligns both objects precisely (Figure S17(d)).
Triplet 4. Considered the easiest triplet, the main challenge is to align the centroids, even when the top object can be placed far off-center on the larger bottom object. Perhaps surprisingly, while the agent usually places the object in the center, it shows attempts recover from occasional off-center stacks (Figure S17(e)).
Triplet 5. Due to the rounded cross-section of the top object, there is high risk of it rolling off after stacking. As a testament this, the agent will often stay close the object after stacking, sometimes (but not always) even supporting with the gripper until the end of the episode when the bottom object is on a slope and a stack thus otherwise impossible (Figure S17(f)).
Additional Related Work
Our work deals with real-world vision-based stacking with diverse set of objects and learned policy. We therefore did not discuss, in the main text, prior work on e.g. stacking from extracted discuss such works here.
features in simulation. For completeness,
Furrer [5] is a very interesting paper on pick-and-place strategies with classical robotics meth- ods that we have now included in our main text discussion. We should highlight here that it deals with only 6 specific stones that offer wide support and have high friction propose in our benchmark. To the best of our understanding, their
contrast to the 152
objects with diverse geometry method would neither be able to handle Triplet 2 (flipping the bottom object needed), nor gener- alize to unseen objects, as is the case for our "Skill Generalization" task. We should also note that the evaluation on that work was done on a total of 11 episodes (or maximum of 33 possible stacks in that setup) in contrast to the more than 54,000 episodes we have evaluated our various choices with.
Duan et al [60] deals with cube stacking in simulation only with policies that have access to task demonstrations, the state information of each cube and no visual input. Later work by Li et al [61] in the same environment shows that reinforcement learning without demonstrations can learn to stack the cubes from state. Our stacking task is also related other manipulation tasks, such as dry stacking [62, 63] where rocks of irregular shapes must be stacked to form wall, but these methods
39

---

do not address. However, while these methods deal with high-level planning and goal understanding, our benchmark task requires dealing with low-level contact dynamics and perception make real object stacking possible with strategies that emerge from RI. training in simulation.
Noseworthy al [64] also deals with high-level planning for stacking cubes, this time in the real world. The challenge in this work is that each cube was created with slightly different center of mass, requiring precise stacks. However, this
vision-based task: each cube also AR- lagged and therefore privileged information about each cube are known during evaluation. Similarly, Macias et al [65] use combination of binary markers on objects as well as high level planning to perform pick and place stacking.
Some lssion-bssed methods have addressed related but distinct problem of predicting stack sta- bility [66, 67, 68, 69). Lerer et al [67] deal with intuition around physics and identifying whether a tower block will collapse, and even the trajectory the blocks will take, focusing primarily simu- lation experiments. Similarly, Hamrick et [69] deal with identifying stability of simulated block towers, with the aid of Graph Neural Networks and without using vision. They also attempt to ad- dress which parts of the tower to "glue" in order to fix an unstable tower. Groth et al [68] classify structures as stable or unstable from visual inputs and learn "stackability affordances" for objects of various shapes. However, this line of work does not address the dynamic aspects of manipulation, as there is no physical robot interacting with the objects.
G Additional Supplementary Material
In this appendix we attempted to provide as much detail we could regarding our benchmark, our environments, and our implementation and experimental details. As part the material that supplement this paper, we also include a video that shows our agents in action as well as designs and instructions for recreating the real robotic cell and the STL files for the RGB-Objects in Figure 1
Here a list of what is included in the supplementary material as part of this submission:
Video that supplements the main text.
BOM (Bill of materials, list of things and quantity) to build:
Cell
Basket
Building instructions:
Cell
Basket
Wiring diagram
We will further release, post-submission under an Apache Licence, the following:
3D Assembly drawing to complete/ / integratee assembly instruction
3D models and Manufacturing drawing for all the parts (cell and Basket) that need to be:
Machined
3D printed
Laser Cut
All STL files for the training set, held-out set, and the specific triplets, in separate folders.
version of the simulated environment.
VVideo: htttps://dpre httpp:://pad..aa//o/oe htt.://iiw....bobiiooooooooata..
MReal cell documentation: htt.mnttw.........ccccoooo/////t documentation.
objects:: https://githut http:///thhccccccooaaa/////////a props/rgb_objects.
40

---

(a) Real-world image sequences without image augmentations.
(b) Real-world image sequences with image augmentations.
Figure S18: Real-world image sequences without and image augmentations listed Table S7. Note that we randomly sample different color and translation perturbations for each sequence in a batch, but we use the same random perturbations for all the images within sequence.
41

---

(a) Image sequences from the simulation without image augmentations.
(b) Image soquences simulation with image augmentations.
Figure S19: Image sequences from simulation without and with the image augmentations listed Table Note that we randomly sample different color translation perturbations for each sequence a batch, we use the same random perturbations for all the images within sequence. Although in our experiments we always image augmentation in combination with domain randomization, here show example sequences without domain randomization for visualization clarity. See Figure S20 for examples with domain randomization.
42

---

Image soquenees from the homain--andomiian simulation without image augmentations.
(b) Image sequences from the domain randoiizadddmized simulation with image angmentations.
Figure S20: Image sequences from the domain-randomizee simulation without and with the image augmen-
tations listed in Table $7. Note that randomly sample different color and translation perturbations for each sequence a batch, but we use the same random perturbations all the images within sequence. See Figure S19 for examples without domain randomization.
43


Reviews:
[{"review_id": "RnHKouDj4ya", "reviewer": "Reviewer_gHPD", "summary": "This paper proposes a method to deal with the problem of robotic stacking with objects of different geometry by training a vision-based reinforcement learning, and a benchmark for stacking geometric objects is presented. They successfully achieve skill mastery of fixed combinations of objects and also assess skill generalization across them. To achieve sim-to-real transfer, interactive imitation learning is used to distill the state-based policy into a vision-based policy, and the final policy receives relatively high accuracy in the real world. Besides, lots of real-world evaluations including ablation study are provided and discussed, showing what matters in this task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. This paper proposes a benchmark for the interesting but challenging problem of stacking objects with complex geometry.\n2. High accuracy is achieved in real-world experiments, and a large ablation study is provided for sim-to-real transfer and imitation of a simulation-trained RL policy.\n\nWeakness\n1. The expert policies from state features in simulation are conditioned on parameter ‘y’, which means the object triplet number in skill mastery or object deformation number in skill generalization. However, I think it’s not necessary to add this parameter because the information is contained in the states in simulation or the visual observations in the real world. It might limit the capacity of generalization in this work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a benchmark for the problem of stacking objects with complex geometry, and achieves good performance in real-world experiments. However, I think the limited variety of triplets during training limits the contribution of this work because there exist endless object combinations in the real world, which might reduce the policy’s capacity for generalization.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "List issues to be addressed during the period of author response and revision. The authors have the opportunity to submit a revised manuscript during the period.\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "8tkM3bYwWE6", "reviewer": "Reviewer_JZ71", "summary": "*This work developed a novel benchmark in the context of stacking manipulation. The benchmark is novel and challenging. The setup cost is reasonable and details have been provided for the ease of reproduction which is valuable to the community.\n\n*It is shown to be possible to learn a single vision-based policy that can handle multiple combinations of objects and can demonstrate a plethora of stacking strategies, emergent from RL training. Zero-shot transfer of these strategies to the real robot has been achieved.\n\n*The ablation studies suggest what matters for sim-to-real transfer and for imitation of a simulation-trained RL policy (that uses privileged state information) with a vision-based general policy. The proposed interactive distillation approach allowed for faster experimentation and iteration.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n*The developed benchmark is novel, challenging and interesting. It involves diverse kinds of objects while requiring different manipulation strategies. The setup details have also been provided for reproduction.\n\n*The paper is well written and easy to follow.\n\n*The design choices are reasonable and has been reflected in the ablation study.\n\nWeaknesses:\n\nThere is no obvious weakness found, except for some minor issues explained below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "In the video 0:44, in the first row, isn’t the triplet5 a failure case? Triplet3’s execution is also not complete.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "5dCnv2XLs5f", "reviewer": "Reviewer_SRSs", "summary": "This paper studies robotic stacking of diverse rigid objects. It introduces a benchmark for robotic stacking and proposes a framework that learns a policy to solve the tasks in simulation and transfer the learned model to the real world. The proposed framework achieves a reasonable task success rate in both simulation and the real world. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- In this paper, the authors design and implement a simulation environment for stacking rigid objects. A parametric set of objects is proposed to create diverse challenges for the stacking problem. This simulation environment could incentivize interesting future research on table-top robotic manipulation if the authors will release the codebase.\n\n-   The proposed sim-to-real transfer algorithm seems to be effective. The model achieves a reasonable success rate of stacking diverse objects in simulation and the real world. \n\nWeaknesses: \n\n- The contributions of this paper are confusing. Although the title claims \"Beyond Pick-and-Place\", the object stacking task studied in this paper is indeed composed of picking and placing actions. In addition, while stacking diverse objects have been studied in prior work (e.g. Furrer et al ICRA 2017), the title focuses on the application without highlighting any technical novelties proposed in this paper. The authors claim that \"We show that it is possible to learn a single vision-based policy that can handle multiple combinations of objects and can demonstrate a plethora of stacking strategies, emergent from RL training\" However, this has already been widely studied in the community and demonstrated in prior work such as Duan et al. NeurIPS 2017.\n\n- The introduction section of the paper fails to explain the importance of the problem and does not mention any previous works or their limitations. The first two paragraphs mainly talk about the task design and the experiment setups. \n\n- The term and the formulation of \"sim-to-real policy\" in Sec. 4.2 and 4.3 are misleading. In Sec. 4.2, the authors train the policy in simulation and expect it to be directly executed in the real world. So there is anything related to sim-to-real transfer about this policy $\\pi_{\\theta}^{s2r}$ itself. It would be better to call it just student policy or something else. \n\n- Although it seems that the proposed sim-to-real transfer approach is one of the main contributions in this paper, the paper lacks a thorough comparison with state-of-the-art sim-to-real transfer approaches in the experiment. Without such comparisons, it is hard to justify the significance of this paper. \n\n- Many important papers are missing in the related work section, including but not limited to:\n    - Furrer et al. ICRA 2017\n    - Macias et al. ICRA 2014\n    - Noseworthy et al. RSS 2021\n    - Duan et al. NeurIPS 2017\n\n- (minor) The listed keywords include CoRL which seems to be a mistake. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper studies an interesting robotic manipulation problem and proposes a valid technical solution. However, the contributions of this paper are not clearly described or justified given its current form and the lacking of comparisons with prior work. Therefore, I believe this work requires another iteration before being ready for publication. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "I would suggest the authors re-think their contributions in this paper and make major revisions to Sec. 1 and Sec. 4. I would also encourage the authors to compare with state-of-the-art sim-to-real transfer learning approaches to justify the significance of their technical contributions."}}, {"review_id": "4FhvvXXJe_d", "reviewer": "Reviewer_6zgt", "summary": "This paper presents a new benchmark of completing pick-and-place tasks using a robot manipulator. In comparison to existing pick-and-place tasks that deal with simple-shape objects, this paper described a parametric way of generating more than a hundred of blocks of diverse shapes that were 3D printed. The agent first learned a policy from simulation using RL methods, where the domain is fully observable (object positions, etc). Then imitation learning methods were used to generate a vision-based policy, where domain randomization was enabled for alleviate the sim-to-real issue. Finally, the policy learned from simulation was improved for real-world deployment. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The 3D-printed parametric block generation is a strength of this paper, and has the potential of providing a strong benchmark to the community. So the first contribution listed on page 1 is solid: \"We present and release a benchmark for stacking that features\na diverse set of geometric objects and tens of thousands of possible stacks.\" It's also good that the paper used state of the art methods for learning policies in simulation and in the real world. It will be better if the objects shown in Figure 1 (d) can go beyond toy blocks that have been widely used in robot manipulation research. \n\nThe downside is that the paper does not make it clear if any of the used algorithms is new from the literature. As a result, the reviewer is not convinced that the second contribution is valid: \"We show that it is possible to learn a single vision-based policy that can handle multiple combinations of objects and can demonstrate a plethora of stacking strategies.\"  There are many existing methods for vision-based manipulation, and there are competitions on this robot manipulation. The reviewer respectfully believes that there is rich literature on learning vision-based policies for manipulating irregular-shape objects. The indication that no existing work support vision-based handling of multiple objects is questionable. There's a similar concern on contribution (d), which is a claim about sim-to-real policy learning. Improving sim-to-real policies using real-world data is not new. \n\nContribution (c) on ablation study makes sense. However, since those are all mature methods from the literature, one probably won't say it's a major contribution to the community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the reviewer didn't observe anything incorrect from the paper, and the paper was nicely generated. However, the paper mostly demonstrated that existing methods can be applied to vision-based policy learning for manipulation tasks on a new 3D-printed set of objects. The contribution is solid but not significant. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new benchmark for stacking manipulation. Reviewers are overall positive about the paper however there are some concerns:
1. Relationship to prior work
2. Lack of comparison to state of art

Please respond to reviewer SRSs concerns.

==
Post Rebuttal

The authors were able to address the concerns of the reviewers. AC believes this is a strong paper which should be accepted.
Decision: Accept (Poster)