RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Authors: Konstantinos Bousmalis
Giulia Vezzani
Dushyant Rao
Coline Manon Devin
Alex X. Lee
Maria Bauza Villalonga
Todor Davchev
Yuxiang Zhou
Agrim Gupta
Akhil Raju
Antoine Laurens
Claudio Fantacci
Valentin Dalibard
Martina Zambelli
Murilo Fernandes Martins
Rugile Pevceviciute
Michiel Blokzijl
Misha Denil
Nathan Batchelor
Thomas Lampe
Emilio Parisotto
Konrad Zolna
Scott Reed
Sergio Gómez Colmenarejo
Jonathan Scholz
Abbas Abdolmaleki
Oliver Groth
Jean-Baptiste Regli
Oleg Sushkov
Thomas Rothörl
Jose Enrique Chen
Yusuf Aytar
David Barker
Joy Ortiz
Martin Riedmiller
Jost Tobias Springenberg
Raia Hadsell
Francesco Nori
Nicolas Heess
Venue: Accepted by TMLR ()

Abstract: The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learning. Inspired by recent advances in foundation models for vision and language, we propose a multi-embodiment, multi-task generalist agent for robotic manipulation. This agent, named RoboCat, is a visual goal-conditioned decision transformer capable of consuming action-labelled visual experience. This data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With RoboCat, we demonstrate the ability to generalise to new tasks and robots, both zero-shot as well as through adaptation using only 100–1000 examples for the target task. We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. We investigate the agent’s capabilities, with large-scale evaluations both in simulation and on three different real robot embodiments. We find that as we grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient at adapting to new tasks.

Published in Transactions on Machine Learning Research (12/2023)
RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation
Konstantinos Bousmalis Giulia Vezzani', Dushyant Rao", Coline Devin', Alex X. Lee", Maria Bauza', Todor Davchev", Yuxiang Zhou', Agrim Gupta "1 Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo F. Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Zoina, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Mar- tin Riedmiller, Jost Tobias Springenberg, Raia Hadsell', Francesco Nori*, and Nicolas Heess"
All authors are affiliated with Google DeepMind, "Equal contributions, TEqual senior contributions, Work done during an internship
Reviewed OpenReview: http:://gpaaaressssrrrrmm/msrtts
Abstract
The ability to leverage heterogeneous robotic experience from different robots and tasks to quickly master novel skills and embodiments has the potential to transform robot learn- ing, Inspired by recent advances in foundation models for vision and language, propose multi-embodimentt, multi-task generalist agent for robotic manipulation. This agent, named RoboCot, is visual goal-conditioned decision transformer capable of consuming action- labelled visual experience. This data spans large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. With Robo- Cat, demonstrate the ability generalisc to new tasks and robots, both zero-shot well as through adaptation using only 100-1000 examples for the target task. We also show how trained model itself can be used to generate data for subsequent training iterations, thus providing basic building block for autonomous improvement loop. We investigate the agent's capabilities, with large-scale evaluations both in simulation and three different real robot embodiments. We find that grow and diversify its training data, RoboCat not only shows signs of cross-task transfer, but also becomes more efficient adapting to new tasks.
1 Introduction
Much of real-world robot. learning research has focused developing agents for one task at time. This is because, even though the cost of task design and robol experience generation very high, leveraging heterogeneous robot data at scale has remained challenging problem in the field of robotics.
The advent of high-capacity models, such the transformer model (Vaswani et al., 2017), has enabled recent successes for multi-task learning in language and vision. These developments have led to progress

---

Published in Transactions on Machine Learning Research (12/2023)
in modelling multi-modal behaviour and predicting actions with generalist agent, Gato (Reed et al., 2022), being able to play Atari, caption images, chat, and show some, albeit limited, robotic manipulation capabilities. Specifically in robotics, recent works (Brohan et al., 2022; Driess et al., 2023) have focused bridging the gap between large pretrained language models and vision-based manipulation by training language-conditionnd transformer policies to solve multiple simple, visually- tasks that have the same observation and action spaces.
In this work, we propose RoboCat, self-improving generalist agent for vision-based robotic manipulation, instantiated as large transformer sequence model. Inspired by foundation models in other domains (Bom- masani et al., 2022), we ultimately aim for foundation agent. for manipulation to be mulli-embodiment agent trained on a large set of robotic episodic experience that enables it to quickly adapt, via fine-tuning, to broad set of new downstream lasks. As step lowards this goal, we trained RoboCat on a very large dataset of diverse manipulation behaviours: precise and dexterous vision-based tasks, performed with bodiments with different degrees of freedom, various observation and action specifications, and operating different control frequencies. Our agent handles these variations natively without requiring common action or observation representations, by leveraging the transformer's ability to input and output variable-length sequences based on context. is able to successfully adapt to multiple new tasks including new robot em- bodiments, unseen behaviours, objects and perceptual variants, and sim-to-real via fine-tuning on a small dataset of new episodic experience of between 100 to 1000 demonstrations. This significantly reduces the cost of acquiring new skills and onboarding new embodiments. We further use the fine-tuned RoboCat models to gather additional data that is later added to train new iterations of our agent. This self-improvement process, illustrated in Figure 1, makes for a more capable agent, improving its cross-lask transfer and fine- tuning capabilities to even more tasks, and demonstrating better performance on existing lasks. We therefore remonstrate fine-tuning to a large range of unseen lasks at multiple stages of this self-improvement process, in addition to generalist capabilities on training lasks.
RoboCat is based on the Gato architecture with a VQ-GAN encoder (Esser et al., 2021) pretrained on a broad set of images; this choice of encoder enables fast. training and iteration. We specify tasks via visual goal-conditioning, which has the desirable property that any image in a trajectory can be labelled as a valid "hindsight goal" (Andrychowicz et al., 2017) for all time steps leading up to it This means that hindsight goals in existing data can be extracted without additional human supervision and that even suboptimal data collected by the agent can be incorporated back into the training set for self- improveroveet..... Additionally, visual goals provide an intuitive interface to indicate to the robot which task it should perform.
Our main contributions in this work are outlined below: (1) we demonstrate, for the first time, that a large transformer sequence model can solve a large set of dexterous tasks on multiple real robotic embodiments with differing observation and action specifications; (2) we investigate RoboCat's capabilities adapting to unseen tasks, with just a small dataset of expert demonstrations, lowering the bar of learning a new skill, compared to baselines; (3) we show that it is possible to incorporate these skills back to the generalist with a simple but effective self-improvement process; and (4) we show that by scaling and broadening the training data, RoboCat performs better on training tasks and is more efficient at fine-tuning.
The rest of the paper is structured as follows. We first describe RoboCat and the self-improvement loop in Scction 2. We introduce the embodiments, tasks, and object sets that we have used in this work in Section 3.
2

---

Published in Transactions on Machine Learning Research (12/2023)
Train RoboCat: multi task, multi embodiment, visual goul-conditoned agent
Start with large training dataset diverse set of tasks and robots
Collect demonstrations for new task/rohot rrooot
RoboCat
Expand this dataset with swwly-geeeerttd
Fine-tune RoboCat to specialise the new task/ rrobot
Fine-tuned agent self- self-generates much more training data
Figure 1: The self-improvement process. RoboCat is multi-task, multi-embodiment visual goal-conditioedd agent that can iteratively self-improve. diverse training agent, which can be fine-tumed to new tasks with 100-1000 demonstrations and then deployed on real robots to generate much more data for these tasks. The resulting trajectories are then added the training dataset for the next iteration of RoboCat, increasing the generalist's repertoire of skills and improving performance across tasks, Our experiments demonstrate one successful iteration of this process.
used to train initial version of this generalist.
We describe experimental setup for both training and evaluation in Section 4, before present extensive experiments to support our claims in Section 5. We finally discuss our work in the context of related work in Section 6, and discuss RoboCat's potential avenues for future work in Section 7.
2 RoboCat
We introduce RoboCat, a self-improving generalist agent for robotic manipulation that can perform multiple tasks and control multiple embodiments in simulation and the real world. In this section, we describe each phase of the RoboCat training process, summarised in Figure 1. In the training phase, the VQ-GAN tokeniser is pre-trained, and then the RoboCat generalist agent is trained on wide dataset covering multiple domains and embodiments, specifying tasks via visual goals. The generalist is then finetuned on a small set of human-telcoperated demonstrations to specialise to new task, and deployed to collect on-policy data autonomously. This data is finally added to the original data to train the next, self-improved RoboCat.
2.1 Training and task specification
We consider lision-based tabletop object manipulation tasks. Each task is defined by its (uncountably infinite) set of valid start and end states, and an episode is evaluated for task success by checking if the last state is in the set of valid end states. For example, for the task "Insert the apple into the bowl", the set of valid start states is all states with apple outside bowl, and the set of valid end states is all states with the apple inside the bowl. We exclusively consider tasks where success can be determined from only the end state.
We want to train agent that performs task when conditioned that task. Our goal-conditioned agent is represented by policy n ((a)lo..e)) where a denotes the action vector, ot (#t, Tt) are the proprioceptive observation (e.g. robot joint positions and velocities) and image observation, respectively, and gt is an image of the desired task. Note that the goal image is an example of
image of valid end state of
3

---

Published in Transactions on Machine Learning Research (12/2023)
the task being solved and it does not indicate specific state that the agent should reach. The goal image effectively indicates the task that the agent should do and the agent is only evaluated for task success.
We model ((aroor 96) via an autoregressive transformer model (Vaswani et al., 2017),
T(arlor, 9r) = Po(as Pet
(1)
where the subscript t denotes observations and goal images prior to time step t. Note that the dimension- ality the actions and proprioception observations vary across embodiments. Internally, the autoregressive model operates with tokenised inputs and outputs.
For training, we assume access to a dataset D = [- of trajectories that are transformed into a dataset of tokenised trajectories D = [ addition, during tokenisation, the trajectories are augmented with "at
goal images. Concretely, a tokenised trajectory i E D represented as f f= (x+ A.M g
1:Q . 7:41 v (++107+41 where L, M, N, Q denote the number of tokens required to encode proprioceptive inputs, images, goals, and actions, respectively, and T is the number of transitions in the
at', 17-M .Txh , aT
trajectory. Note that L and Q vary by embodiment. The goal observations It. are fixed within a trajectory and repeated for each lime step.
A natural choice for a goal image is a hindsight goul. Since, by definition, a trajectory always "succeeds" reaching own last image, we can use the last image of the same episode as the goal image, of = 1 for any trajectory 7f. Alternatively, we can also consider goal selection using sementicully-eqqiiveeee goal. That is, for any successful episode 7", we can select the last image of a different episode that succeeded at the same lask, of = Fr+1 where 7 another successful episode from the dataset D, as measured by a success detector or reward function for a given task. We train with both sources of goals for successful episodes, and use only hindsight goals for unsuccessful episodes. Details on how we weight the different tasks and goal sources are available in Appendix E.2.
2.1.1 Architecture and pretraining
Our model is based on the transformer archilecture described in Gato (Reed et al., 2022). For tokenisation of proprioceplive observations and agent actions, we follow the same procedure as in Reed et al. (2022). For image tokenisation, however, we instead use pretrained and frozen VQ-GAN (Esser et al., 2021), which allows for faster training of the generalist, the image be tokenised once in advance. The VQ-GAN, similarly to VQ-VAE (van den Oord et al., 2017), consists of encoder that encodes input image into series of latent vectors and decoder (which we do not after training). The encoded vectors are discretised via nearest neighbour lookup in codebook of quantised embeddings. Each image is tokenised into an 8 8 grid of tokens.
We pretrain our VQ-GAN encoder diverse collection of images we find this improves generalisation. Specifically, the encoder is trained on dataset that consists of images from ImageNet (Deng et al., 2009), images from the control tasks in Reed et al. (2022) including Atari and MuJoCo (Todorov et. al., 2012) locomotion tasks, as well images from our visual robotic manipulation dataset. These datasets, training details, well extensive ablations that informed our design choices be found in Appendix D.
4

---

Published in Transactions on Machine Learning Research (12/2023)
To train the agent model use dataset D containing the joint collection of data from all training tasks (see Section 3) and utilise standard token prediction loss. While Gato only predicted actions, find that, when VQ-GAN is used, performance is improved by additionally training for predicting future image tokens produced by the VQ-GAN encoder (Appendix D.3). Specifically, predict image tokens k 5 time steps into the future images step apart can look very similar.
Combining the action and observation prediction losses, at the token level, we obtain the following objective to train the model Po: L(0;D) =B Σt= E i log [a I .g
+ - Mf log Po x 1 g(:)) Note that, in practice, instead of conditioning on the full history of observations (as indicated by the subscript < t), we use fixed total token length of 1024 for the model (which corresponds to roughly 3 time steps of history).
2.1.2 Fine-tuning and self-improvement
A key contribution of our work is our study into how RoboCat agents can be fine-tuned and self-improved given a relatively small number of demonstrations, This capability especially crucial in a real robotics context unlike in simulation, data is bottlenecked by real-time operation per robot, and high-quality su- pervision scarce.
Fine-tuning To perform fine-tuning and celf-improvement we first collect 100-1000 demonstrations per task via teleoperation. The generalist RoboCat agent is fine-tuned on these demonstrations, which are tokenised and augmented with goal images the same way as for the generalist. training. Formally, we perform the optimisation 0% = arg maxg C(0; D P)emo) where DV Demmo is the demonstration data for the task y that we want to fine-tune on, and is initialised with the weights from pretraining (Section 2.1.1). At the end of this fine-tuning step, we obtain an agent that specialised to the new task but that may lose performance on the original training tasks.
Self-improvement In order to integrate new tasks into a new generalist, we deploy the fine-tuned policies Pog to autonomously collect a large dataset additional trajectories for each of the self-inprovement tasks y E y. After data collection, we perform hindsight goal relabelling as described in Section 2.1. Note that, when using semantically--qquivelenn goals, we require a reward function to determine the successful trajectories for a given task. For this purpose, we employ learned reward models as described the next section. The resulting relabelled trajectorics form a self-improvement dataset D for the task we want to improve. Finally, using this data, we can construct a new training dataset for training the next iteration of our generalist RoboCat agent. We combine all trajectories with the previons data to form the next dataset,
D Dnext = D U U (DY demo U D.V. U.D..p),
yEy
which is then used to train new VQ-GAN model, after which we continue with the next iteration of training generalist Ouext arg maxg L(:; D Duext).
5

---

Published in Transactions on Machine Learning Research (12/2023)
Training Embodiments
Held-Out Embodiment
Sim Sawyer 7-DoF
Sim Panda 7-DoF
Real Sawyer 5-DoF
Real Panda 7-DoF
Real KUKA 14-DoF
Figure RoboCat supports multiple robotic embodiments and control modes. These are all the different embodiments RoboCat is tested on, and the dimensionality of the action it needs to output for each. All robot arms have a Robotiq parallel gripper attached them, with the exception of KUKA arm which has proprietary Chree-finge hand. Unlike the Panda and Sawyer embodinents, the KUKA embodinent was not seen during training and only used during fine-tuning.
2.2 Real-world deployment
In order to integrate the new task into a new generalist, we deploy the fine-tuned policy on a real robot to collect a large dataset on the new task using images from the demonstrations as goal images. Collecting real-world data autonomously presents two challenges: success classification and task resets.
Success detection via reward models While final evaluation numbers are counted mamually for accuracy, automated success detection is necessary for the hindsight, goal relabelling of semanticall--qqivvaeent goals described above during training. In addition, success detection is necessary for determining when a reset is needed. To this end, we train vision-based reward models to detect when a task has succeeded. We first collect human demonstrations and data from policies trained to perform the task (e.g. evaluation episodes of a RoboCat policy). These episodes are annotated via crowd-sourcing interface, where annotators mark the time step after which the task is solved in each episode (if at all), resulting in binary annotations. These are then used to train a binary classifier that can be used to detect task success from image observations at any given time step.
Autonomous resets with policy pools Resetting the environment for single task requires bringing the state from the end state back into the set of valid start states for that task. However, manually programming such reset routines is highly non-trivial endeavour (in many cases performing reset is almost as complicated solving the task itself) leaving us with problem for autonomous data collection. We solve this issue by observing that the set of end states for some tasks overlap with the set of start states of other tasks. Thus we can "re-use" tasks trained for given task as reset mechanisms for tasks whose end states overlap with the valid start states for another task. We implement autonomous reset mechanism based this observation that refer to policy pool. A policy pool is simply collection of policies (or policies implicitly defindd by a pool of goal images) with overlapping start and end states. In each episode, then pick policy from this pool to be next and record its trajectory and success. By pooling multiple policies in this way, we get automated resets, increase the robot utilisation (by reducing the need for explicit human resets) and increase the diversity of initial conditions for evaluation and data collection. We itilise two types of policy

---

Published in Transactions on Machine Learning Research (12/2023)
pools in evaluations: stateless policy pools, in which the policies are executed in some order regardless of the state of the environment (e.g. for lifting tasks); and a state-based policy pool, which samples the next policy to execute based on the state of the environment (e.g. performing remove task when the initial state corresponds to successful insertion). In the latter case, the trained reward models used to evaluate the state of the tabletop and determine which policies eligible for next exccution. More details are provided in Appendix F.2.
3 Tasks and Data
(a) RGB objects
(b) NIST-i and base
(c) YCB fruit, YCB-i vegetables, bowl
(d) Shape-matching objects
Figure 3: The real-world object sets used by RoboCat.
The first two object sets are used to systematically study structure- srructur-builddng and insertion affordances, respectively. The other object sets are store-bought objects that add visual diversity and challenge the agent with various lifting, insertion, and removal tasks.
One of the main contributions of this work is demonstrate that RoboCat can learn diverse and dexterous bchaviours to solve a large set of tasks. The tasks we use require fine motor skills and hand-oye coordination, and the understanding of complex affordances and multi-object intoractions. Additional diversity in the data is obtained through the use of multiple simulated and real embodiments and different approaches to data generation: RL-trained expert trajectories, human-teleoperated demonstrations, as well as self-generated data from RoboCat (see Section 2.1.2). In this section, we provide an overview of the embodiments, object sets, tasks, and datasets that we refer to in this paper.
3.1 Embodiments
The embodiments used in this work, shown in Figure 2, all in standardised cage (see Lee et al. (2021)), which contains a "basket" that defines the robot arm's workspace. RoboCat trained with data from Rethink Sawyer arms controlled with 7-DoF (simulation) and 5-DoF (real), and Franka Panda robot controlled with 7-DoF (simulation and real), all fitted with a Robotiq parallel gripper. These action spaces comprise 6-DoF and 4-DoF Cartesian end-effector control with additional dimension for the gripper action. The proprioception observations for Panda and Sawyer have different dimensionalities, and even for the common 7-DoF case, the physical and kinematic characteristics between the embodiments that the action distributions are different. RoboCat also able control KUKA 14-DoF arms, which are fitted with a new, custom-made, three-finger robot hand an embodiment that was only seen during the fine-tuning phase. In total, we used 36 real robots in this work: 15 Panda, Sawyer, and 4 KUKA arms.
The simulated Panda and Sawyer embodiments are analogous the real ones, though they are only coarsely aligned. We did perform any careful system identification and the images rendered from the simulation
Details of this robot hand will be released the near future.

---

Published in Transactions on Machine Learning Research (12/2023)
RGB stacking (Sawyer 7-DoF)
Gear insertion (Pauda 7-DoF)
RGB tower (Sawyer 5-DoF)
Vegetable lifting (Sawyer 5-DoF)
Gear insertion
Fruit insertion (Panda 7-DoE)
Shape insertion (Panda 7-DoF)
Gear lifting (KUKA 14- Dof)
(Panda 7-DoE)
Figure 4: Example goal images. These images correspond to a subset of the embodiments, task families, and object sets used by RoboCat. The first two images correspond simulated embodiments and the remaining images to real-world embodiments. See Figure 12 for more examples.
were not visually realistic. We randomised the physics paramcters in simulation but we did not randomise the visual appearance of the scenc. More details about embodiments are available in Appendix C.
3.2
Object sets
We use different object sets and a total of 134 real objects to enable a varicty of complex behaviours and affor- dances (sce Figure 3). The first two sets of objects are 3D-printed and have been designed to systematically study types of robotic manipulation that involve multi-object interaction, specifically, structure-bulding (RGB objects) and insertion (NIST-i gears). We use subset of these (123 objects) in simulation. The other real-world sets include store-bought objects.
RGB objects These 116 objects with parametrically defined shapes (only subset shown in Figure 3(a)) were introduced as a benchmark (Tee et al., 2021) to systematically study the physical understanding of multi-object interactions in the context of stacking: To solve the benchmark an agent needs to understand which shapes which poses can be reliably stacked on top of each other. We use them here to additionally study related structure-buiddnng tasks. The basket always contains a triplet of these objects, with respective colours red, green, and blue.
NIST-i gears and 3-peg base This set of objects first introduced in this work to aid a systematic study of the insertion affordance. Inspired by the NIST beuchmark for robotic manipulation (Kimble et al., 2020), we designed three gears, different in sizes (small, medium, large), which are to used in conjunction with a 3-peg base. The pegs are spaced such that successful meshing requires a specific allocation of the gears to the pegs (see Figure 3(b)). In the real world, the shafts are metallic and the base is not fixed to the basket, which significantly increases the difficulty of the task. In simulation, the base is fixed. In both cases, there is a mm tolerance when inserting a gear. See Appendix B.1.4 for more details.
YCB fruits, YCB-i vegetables, and bowl In this work, we use subset of the YCB object set (Calli et al., 2017), namely the fruit (apple, banana, peach, lemon, strawberry), shown in Figure 3(c). The YCB-i vegotables (carrot, cucumber, pepper, potato) and bowl, also shown in Figure 3(c), are inspired by, but not part of, the official YCB benchmark. This collection of textured and geometrically different objects introduces additional visual diversity and allows us to benchmark RoboCat on tasks with everyday objects.
Shape-matching objects and base These wooden objects parts of real shape-matching cube, used by toddlers to practice fine-motor control skills and 3D shape understanding. We used three shapes (circle, pentagon, square) and the shape-matching eube lid, shown in Figure 3(d). The lid is used as base with matching holes that be used for insertion and removal tasks. These objects used to further study
8

---

Published in Transactions on Machine Learning Research (12/2023)
the insertion affordance. Unlike the NIST-i gears, this toy often requires difficult reorientations to get the objects in the correct orientation for insertion.
3.3 Task families
We consider total of 253 different task variations which we group into task families. We define a task family be a group of tasks that utilise the same skill or sequence of skills. For example, lifting the large NIST-i gear and lifting the YCB apple are two different task variations from the same task family. We provide a complete list of the task families in Table
The lask families stacking, tower building, pyramid building, and inverted pyramid building consist of building structures with either RGB objects or gears. They differ in difficulty, but all cases require dexterous and precise movements ensure that the structure remains stable after completion. The lifting task family consists of picking up specific object in a basket with multiple objects. The objects are either Iruits, vegelables, or gears. The motivation behind the lifting lasks to study goal understanding and generalisation new embodiments and objects. The insertion and removal task families come three flavours, either involving fruits and a bowl, gears and a 3-peg base, or shape-matching objects and a base. We treat them as separate task families since they require different skills. The latter two require precise positioning into low-tolerance pegs or base, and shape-matching requires shape understanding and often reorientation. The bowl and bases can freely move in the real world, which substantially increases the complexity those tasks. For all insertion and removal tasks, we use no resets other than the learnt respective removal and insertion tasks.
Each task variation refers to the combination of a specific embodiment (e.g. simulated Sawyer vs real-world Panda), task family, object set (e.g. RGB triplet vs NIST-i gears), and perceptual variation (e.g. stacking red on blue vs green on red objects). Example goal images corresponding to specific task variations are shown in Figure 4.
3.4
Data sources
RoboCat is trained on both expert and non-expert data. Different subsets of the data are collected in different ways. We use three types of data generation: (i) data produced by specialist RL agents, particularly employed in simulation; (ii) human teleoperated expert data, mostly used for the physical world tasks; and (iii) self- gefereteer data. The primary difference between the two expert types of trajectories is that agent data provides fairly smooth and efficient trajectories due to the way the RL agent acts in the world, while teleoperated data often includes pauses teleoperators employ behaviours similar to bang-bang controller. The self-generated data is obtained by running extensive evaluations whenever new version of RoboCat is available: the data collected this way saved and then reused for the next RoboCat training. This data is collected from RoboCat, agents fine-tuned on teleoperated expert data. Therefore, the self-generated data resemble the teleoperation behaviours. See Appendix B.2 for further details about the nature of the data.

---

Published in Transactions on Machine Learning Research (12/2023)
Training Task Variations 28 & 5
Evaluation Task
Average
Embodlment
Task Family
Object Set
Task Success
Variations
Sawyer 7-DoF
Stacking
RGB objects & NIST-i gears
28 & 5
82%
Simulation
Stacking
RGB objects & NIST-1 gears
30 &
30 & 6
80%
Tower building
ohjects & NIST-i gears
8 3
&
60%
Panda 7-DoF
Pyramid building
RGB objects
30
65%
Lifting
NIST-i gears
88%
Training
Stacking (red on blue)
Insortion-pog
NIST-i gears
RGB objects
92
3 75%
80%
Self- Self-impie imprroveennnt tasks
Stacking (bluc on green)
RGB objects
45%
Real World
Sawyor 5- 3-D
Tower huilding
ohjents
23%
Inverted pyramid building
RGB objects
17%
Listing
YCB-i vegetables
54%
Lifting
YCB fruits
16
54%
Lifting
NIST- gears
3
9.1%
Panda 7-DoF
Insertion-per peg
NIST-i gears
77%
Removal-peg
NIST-i gears
97%
Fine-t tuming
Real World
Insertion-houg
YCB
YCB-i bowl
84%
Removal-bowl
YCB fruits and YCB-i bowl
64%
72%
Panda 7-DoF
Insertion-bast base
Shape-natching objerts
6%
13%
Removal-hase
Shopo-matching objects
70%
82%
KUKA 14-DoF
Lilting
NIST-i gears
56%
86%
Table 1: Final RoboCat performance on evaluation tasks. This table lists the tasks used for training and fine-tuning of the final RoboCat agent, and highlights the set of tasks used the self-imprrvement process. The success rates are averaged across all the respective evaluation task variations. For fine-tuming experiments, report success rates when fine-tuning on 500 and 1000 demonstrations, respectively. Note that data from the fine-tuning tasks unseen during generalist training and the fine-tuned agent only uses up to 1000 demonstrations alone for these new tasks.
4 Experimental Setup
4.1 RoboCat training tasks
The full RoboCat agent is trained on 240 tasks and fine-tuned on a further 13 tasks, for total of 253 tasks. This includes data from 2 simulated and 3 real-world embodiments, 5 simulated and 11 real task families, and 123 simulated and 134 real objects. Table summarises the tasks, organised separately for training and fine-tuning tasks. An important contribution is the fact that the RoboCat generalist. agent can self- improve, by fine-tuning the previous iteration of the generalist to new tasks, and self-generating additional data for the next round of generalist training. These self-improvement tasks (unseen in the previous RoboCat iteration) are indicated in Table 1, and more detailed experiments presented in Section 5.3.
4.2 Training and fne-tunnng
We train generalists following the procedure outlined in Reed et al. (2022) except for differences in the encoder where applicable. The majority of the experimental results are based models with 1.18B- parameter decoder-only transformer (Vaswani al., 2017) with 24 layers, an embedding size of 2048, and post-aliention feedforward hidden size of 8196. To allow for more extensive experinentation, we use smaller models with 364M parameters for a few ablations (Figure 5(a), Figure 10, Appendix G.2, and Appendix G.4).
We fine-tune our generalists set of diverse real tasks using a limited number of human teleoperation demonstrations, between 100 and 1000 demonstrations for each task.
10

---

Published in Transactions on Machine Learning Research (12/2023)
4.3 Evaluation
For cach of the simulated and real tasks, we evaluate cach model by averaging over 100 episodes (or more, if specified), using different goal image for each episode as well as randomised initial states of the environment. The episode length and control frequency varies from task to task, always matching the length of the expert data used for training. The control frequency of training data is not provided to the agent during training, since it may not be known readily available. Table 14 in Appendix F report the episode length and control frequency used for each task family in simulation and real.
When fine-tuning generalist to a specific real-world task, it can be difficult to determine the optimal number of fine-tuning steps, since there is reliable offline measure of success. To address this in a systematic and reproducible way, we employ the following evaluation protocol for each task: we first evaluate the checkpoint every 5000 steps for 25 episodes each assess the best performing chockpoint, and then evaluate that checkpoint for 100 episodes to measure the final performance.
4.4 Baselines
In order to contextualise the difficulty of the tasks, we compare RoboCat to high capacity, pretrained vision foundation models (VFMs). These present alternative approach to training robot policies: instead of training a single agent diverse set of roboties tasks, we can take readily-available powerful vision model and fine-tune on each task separately. This comparison also demonstrates the utility of robotics data in the case of RoboCat, versus vision datasets for the VFM baselines, when adapting to robotics tasks.
We trained and evaluated 59 different VFM baselines (see Appendix G.3 for the complete list) on a subset of tasks in simulation and selected the best two as the main baselines for these experiments: the 438M parameter NFNet-f6 model (Brock al., 2021) and the 197M parameter Swin-L model (Liu al., 2021), both pretrained with CLIP (Radford et al., 2021). These models are smaller in size than the main RoboCat models because (i) they only need deal with single task (versus hundreds); and (ii) they were obtained by fine-tuning existing VFM architectures, limiting flexibility in size. For each comparison, the VFM models are trained with the same behavioural cloning loss and the same successful episodes that the RoboCat model uses for given task variant.
We also utilise other baselines for subset of the tasks. To isolate the impact of diverse, robotics dataset, we use a Gato baseline (Reed et al., 2022), which employs a similar transformer architecture but with the majority of data from diverse non-roboties domains. We compare RoboCat with Gato on the robotics tasks used in their work, namely the RGB-Stacking Benchmark (Lee et al., 2021), and fine-tuning to bluc-on-groen stacking (Reed al., 2022). For the former, we also use the BC-IMP specialist agents from (Lee et al., 2021).
5 Experiments
The evaluations and comparisons present in this section investigate the following questions:
1. Can RoboCat learn from heterogeneous data and solve a large set of tasks, specified with visual goals and requiring dexterity on multiple physical and simulated embodiments? (Section 5.1)
11

---

Published in Transactions on Machine Learning Research (12/2023)
RohoCCtt
RoboCat
NFNNc
Swin
NTNet
Swin
RoboCat
100%5
10095
54%
97%
705%
77%
71%
68%
755
54%
54%
50%
20%
26%
22%
850
16
6%
12%
9%
4%
0%
075
Stuser y-Dool
Pands pulbe RGll stacking (eingh tnak)
Pands g-DoP RGB tower (single task)
Pandi p-Dool RGB peramod Gringle taak)
Sunyer Dof
Savrer 5 Dof inveried garamii
Sawget 5 DoF Vigetable Eiding
Panda 7. Dol
Panda Dof'
Panda 7
Fanda DoF
RGB Stackeng (single lonk)
luwerr building
Frait Bilig
Geer lifling
Geur
Gear runord
ineerikon
Simulation training tasks (single variants)
(b) Real-world training tasks (averages per task family grouping)
Figure 5: RoboCat compared to VFM baselines on training tasks. RoboCat performs better on the vast majority training lasks, compared to single-lask baseline agents trained on the same data for each lask. We compare here with the best-pert best-performing visual foundation model-based agenls, chosen out of 59 strong baselines (see Section 4.4 for details). advantage of multi-task joint training on robotics data to perform significantly better than the baselines. A much smaller version of the generalist also evaluated on the sim tasks, and demonstrates similar performance for stacking but much lower success for the more challenging cases.
the real world, where we have limited data compared to simulation, RoboCat can take
Can RoboCat adapt, with small number of demonstrations, to challenging new scenarios such as unseen lasks, new objects, and new embodiments with unseen morphology and action spaces? (Sections 5.1, 5.2, and 5.3)
3. Does RoboCat exhibit cross-task transfer and generalisation to held-out tasks? (Section 5.2)
4. Can RoboCat self-improve by autonomously collecting data and integrating that new data into the next RoboCat iteration? (Section 5.3)
5.1 Overall RoboCat performance
We evaluated RoboCat over all the training tasks and we report task success rates averaged within each embodiment, task family, and object set, in Table (see Appendix G.1 for per-task success rates). The tasks are broadly categorised into training (which include the tasks from the self-improvement process) and fine-tuning tasks. The RoboCat generalist agent trained on all of these training tasks and then evaluated total of 141 training task variations. We demonstrate that single RoboCat agent is able to perform all of these tasks, which involve multiple embodiments-in simulation and the real world-and multiple task families and objects sets.
For the fine-tuning tasks, the RoboCat generalist agent fine-tuned to individual task variations and then each fine-tuned agent was evaluated its respective task. We fine-tuned either 500 1000 demonstrations and report results for both cases also in Table 1. RoboCat is able to fine-tune tasks that not only include previously unseen task families (e.g. fruit insertion into bowl), but also new object sets (e.g. shape-matching set) and previously unseen embodiment (real KUKA 14-DoF robot).
In Figure 5, we compare RoboCat to visual foundation model (VFM) baselines trained each task indepen- dently. In simulation, only ran these baselines for task from each task family due to the large number of task variations in simulation, whereas for the real-world tasks,
them on all of the task variations. The simulation results in Figure 5(a) show that the VFM agents competitive the Panda stacking fask, but outperformed by RoboCat on the other simulated building tasks. As shown in Figure 5(b).
12

---

Published in Transactions on Machine Learning Research (12/2023)
NFNet (1000)
Swin (1000)
RoboCat (500)
RoboCat (1000)
100%
84384%
72%
75%
64%
56%
25%
16%
5%
13.
1%
2%
0%
Real Panda 7-DoF Fruit insertion into bowl
Real Panda 7- 7-Dolf Fruit removal from bowl
Real KUKA 14-Dol' Gear lifting
Figure 6: RoboCat fine-tuning compared to VFM baselines. RoboCat efficiently adapts to each of these pre- viously unseen lasks which include unseen object sets and new 14-DoF embodinennt, whereas the visual foundation model-based baselines agents perform very poorly. The number of fine-tuning episodes are shown in parentheses for each method.
Method
T1
T2
T3
T4
T5
Average
RoboCat
87%
70%
82%
93%
08%
80%
BC-IMP
75%
61%
74%
95%
88%
79%
Gato
81%
98%
Table RGB Stacking Mastery Benchmark. RoboCat performs, on average, similarly to prior works BC- IMP (Lee et al., 2021) and Gato (Reed et al., 2022) on this stacking benchmark, despite also being able to solve many other manipulation tasks. All three methods were evaluated on the same Sawyer robots with identical conditions, evaluation protocol, and successful episodes visually counted.
this is even more apparent in the real-world lifting, insertion, and removal tasks, where the VFM baselines are significantly outperformed by RoboCat. We also evaluate a much smaller (364M) RoboCat generalist agent on the simulation tasks; this model is slightly smaller than the NFNet baseline but still has to jointly learn all training tasks. We see from Figure 5(a) that the performance of this smaller model is comparable to RoboCat on the stacking tasks, but significantly lower for the harder cases: the success rate at least matches single-task baselines for pyramid building, but is poorer for tower building. Thus, the generalist does require sufficient capacity to perform well in the multi-task regime, least for the harder sim tasks. Indeed, the full 1.18B RoboCat model can outperform the single-task basclines with only 3-6 times the capacity, despite being trained 250 tasks.
For fine-tuning experiments, where only up to 1000 demonstrations are available per task, compare fine-tuned RoboCat agents to VFM agents that trained with only 1000 demonstrations. The results in Figure 6 show that the VFM baselines perform very poorly whereas the fine-tuned RoboCat agents perform well even when only using 500 demonstrations. Since the VFM agents are trained for single tasks, they are unable lo leverage the large amounts of existing training data done by RoboCat.
Lastly, in Table 2, we compare to previously reported performance the real Sawyer 5-DoF stacking tasks, which part of the RGB-Stacking Benchmark (Lee et. al., 2021). This allows us to compare RoboCat performance these tasks with vision-based BC-IMP specialists (Lee et al., 2021), as well the Gato generalist (Reed et al., 2022). The latter allows compare the benefit of training diverse robotic
13

---

Published in Transactions on Machine Learning Research (12/2023)
Training Task Varlations
Held-out Task
Embodiment
Task Family
Object Sets
Varlations
Sawyer 7-DoP
Starking
RGB ohjects, NIST-i gears
Stacking
RGB objects, NIST-i gears
Simulation
Panda 7-DoF
Tower building
RGB objects, NIST-- gears
11
Pyramid building
RGB objects
Real World
Sawyer 5-DoF
Stacking
RGB objects
(a) Training tasks used by RoboCat-lim,, with specific objects and task variations explicitly held out
Evaluation Task Variations
Generalisation Axis
Embodiment
Task Family
Object Set
Perceptual variation
Sim Sawyer 7-DoF"
Stacking
NIST-i
(Stacking blue on green)
Sim Panda 7-Dof'
Objects
Sawyer 7-DoF
Stacking
RGB triplet 5
(Sawyer Stacking triplet 5)
Real Sawyer 5-DoF
1
Behaviour source
Real Sawyer 5-DoF
Starking
RGB triplet 1
DDemonstration dataa
Sim-to-real
Stacking
RGB triplet 5
Real Panda 7-DoF
seen simulation)
building
triplet
Task family
Real Sawyer 5-Dol'
Inverted pyramid building
RGB custom triplet
Embodiment
Real KUKA 14-DoF
Lifting
NIST-j large gear
(b) Unseen fine-tuning tasks used by RoboCat-lim, grouped by generalisation axis
RoboCat-lim (o)
RoboCat-lim (100)
RoboCat-lim (500)
RoboCat-lim (1000)
100%
75X2699
82%
81%
79%
71%
73%
75%
69%
62562%
60%
50%
441
25%
7%
0%
Pereeptial varaatin:: Sim Sawyer Dolf RGB susking (Nin on green)
Peroeptual vortation: Sin Panda I-l RGB stacking (bine gowen)
Objects
Demonstrabin data: Real Stwyer Doll RGB studing (Nunoon green)
Vint teat Real Panda Uol RGB stacking
Stm- kmm-to-nel:: Real Panda Dear
Task family: Real Panda 7-Dotr
Embodhment:
Sim Sanyer p-Dol RGB stucing (triplet 5)
Real KUKA KL se- Gear lifting
BGB lowize building
RGB zaverted
pyramié buiiding
(c) RoboCat-lim 0-shot and k-shoo fine-tuning performance generalisation
Figure 7: Generalisation and adaptation study for RoboCat-lim. RoboCat-lim can be effectively fine-tuned, given limited number of demonstrations, tasks that are novel terms of objects task variants, and even completely new robot embodiment.
nipulation data rather than training on tasks from vastly different domains such as Atari or VQA. Although the relative success rates vary per object triplet, RoboCat is comparable to prior methods on average on this benchmark, despite being able to solve many other manipulation tasks.
We demonstrate that RoboCat, by using visual goals, is able to learn from hetcrogencous data and per- form a large set of tasks on multiple embodiments, and can quickly adapt-using a limited mimber of demonstrations to unseen tasks, new object sets, and new embodiments.
5.2 Generalisation and adaptation
In order to analyse how RoboCat agents generalise and adapt, trained separate model of the same size but only subset of structure-building tasks (stacking, tower, and pyramid) with specific objects and task variations explicitly held out. The training and held-out tasks listed in Figure 7(a). We refer to this limited-dataset model RoboCat-lim. This enables to investigate generalisation and adaptation of
14

---

Published in Transactions on Machine Learning Research (12/2023)
Task Success
Data Source
100 episodes
500 episodes
Expert agent data
63%
84%
Demonstration data
82%
88%
Table 3: RoboCat-lim fine-tuning using different sources of data. Despite RoboCat-lim only being trained agent data originally, the model can be fine-tuned with either agent or human demonstration data. The 0-shot success rate for this task is 0%. This task is the held-out real-world perceptual variant of blue-on-green stacking.
this agent along specific axes (see Figure 7(b)). Furthermore, since the training tasks for RoboCat-ling are a subset of those used by the final RoboCat model, we can evaluate the effect of training on more tasks.
First, we measure how RoboCat-lim generalises to the 23 held-out tasks, both 0-shot and with k-shot finetuning (Figure 7(c)). In simulation, RoboCat-lim generalises 0-shot to a held-out object set on the Sawyer (third plot from the left) and the blue-on-green stacking task variant on the Panda (second plot), but does not generalise to that same task variant on the Sawyer embodiment (first plot). However, the model is effective at fine-tuning to this task variant with as little as 100 demonstrations. On the real-world blue- on-green stacking variant (fourth plot), RoboCat-lim achieves 88% when fine-tuning on 500 demonstrations, compared to the 60% success reported for Gato on the same data?. The remaining cases show RoboCat's ability to adapt to real-world variants of previously seen simulation tasks (both stacking and tower building), the challenging inverted pyramid building task family (for which even teleoperator success is only 52%), and to the real-world dexterous KUKA embodiment with nearly 80% success.
Overall, we show that RoboCat-lim adapts with only 100-500 episodes to a broad set of downstream tasks, including unseen variations and objects, different data sources (agent vs demonstrations; see Table 3), and an unseen task on an entirely unseen embodiment with twice as many degrees of freedom than seen in training. In addition, the results demonstrate the importance and potential of multi-embodinent training, The zero-shot performance on the unseen KUKA embodiment is zero (given it has entirely different action and observation spaces), but fine-tuning to relatively small amount of data from this embodiment yields 69% success. The sim Panda and sim Sawyer also have different proprioception observations, but given that some Sawyer stacking data is used during RoboCat-lim training, the agent can generalise zero-shot to held-out Sawyer tasks that have been trained only for the Panda (object triplet 5).
We also compare RoboCat-lim to the VFM-based agents for few-shot fine-tuning on couple of individual tasks in simulation. The results in Figure 8 show that our model can learn the tasks with significantly less data than the basclines.
Finally, we measure how much RoboCat benefits from its diverse training set, which includes all the tasks used for RoboCat-lim, the simulated structure building tasks held out from RoboCat-lim, all of the additional real-world data for the self- iaff-improvetnntt tasks, and both simulated and real-world NIST-i gears tasks (lifting, insertion, and removal). In Figure 9(a), we compare RoboCat with RoboCat-lim specifically on the tasks that the limited model was trained on. Rather than its performance being negatively impacted due to the additional training tasks, RoboCat exhibits positive transfer across its training tasks and outperforms the more specialised RoboCat-lim. This trend of positive transfer also holds when adapting to new real-world
2 The Gato madel was fine-tuned with additional simulation episodes of the task, but was not originally trained with the object set. this task.
15

---

Published in Transactions on Machine Learning Research (12/2023)
NPNet (500)
NFNet (1000)
Swin (500)
Swin (1000)
RoboCat-lim (0)
RoboCat-lim (500)
RoboCat-lim (1000)
87%8%
81% 78%
76%
75%
56%
58%
60%
50%
37%
36%
(00%)
25%
0%
0%
Percepiual variation: Sawyer 7-DoF RGB stacking (blue green)
Perceptual variation: Sim Panda 7-DoF RGB stacking (blue on green)
Figure 8: RoboCat-lim 0-shot and k-shot fine-tuning compared to VFM baselines. RoboCat-lim performs better than the baselines given the same number of episodes on a new task, even for task in which RoboCat-lin gets O-shot 0% success. This shows that the model can quickly adapt by reusing information from the tasks and embodiments seen during training. The number fine-tuning episodes are shown in parentheses for each method. The results here are for single task variants, unlike the results in Figure 7(c).
RoboCat-limn (500)
RoboCat-Ili (1000)
RoboCat-lim
RoboCat
RoboCat (500)
RoboCat (1000)
100%
100%
84184%
78%
78% 79%
72%
75%
60%
61%
6.%%
75%
64%
58%
51%
50%
50%
44%
25%
Sim Sawyer
Sim Panda
Sim Panda
Sim Panda
Real Sawyer 5-DoF RGB
Real Panda 7-DoF insertion into bowl
Real Panda 7-DoF Fruit removal from bowl
7-DoF RCB
7-DoF RCB
7-DoF RGB
7-DoF RGB
stacking
stacking
tower
pyramid
stacking
(a) Training tasks used by RoboCat-lim (subset of tasks used by RoboCat)
(b) Real-world fine-tuning taskss
Figure 9: Positive transfer across tasks: RoboCat-lim vs RoboCat. Training on more tasks (RoboCat)
improves performance the limited training tasks compared to only training these limited tasks (RoboCat-lim) In addition, RoboCat is better when fine-tuming to the insertion and removal tasks. The reported numbers are averages of task variants within each grouping.
tasks, c.g. RoboCat was trained on real-world fruit and vegetable lifting data, it adapts better to the insertion and removal tasks with the fruits and bowl (Figure 9(b)).
5.3 Self-improvement via RoboCat fine-tuning and self-generation of data
In this section, we demonstrate the key ability of RoboCat to perform self-improvement. That fine-tune to task with limited number of demonstrations, self-generate larger amount of experience, and
16

---

Published in Transactions on Machine Learning Research (12/2023)
RoboCat-lim (demos only)
RoboCat-lim (demos & gelf-generated data)
Generated data
Generated data demonstrations
RohoCat
50%
60%
52%
54%
54%
54%
45%
45%
40%
42%
42%
37%
40%
30%
22%
32%
23%
80%
18%;7%
13%
11%
10%
1%
0%
0%
Real Sawyer
Real Panda 7-DoF Fruit lifling
Real Sawyer 5-DoF RGB stacking (blue on green)
Real Sawyer 5-DoF RGB tower
Real Sawyer 5-DoF RGB inverted
Real Sawyer 5 5-DoF Vegetable lifting
Real Punda
5-Doff RGB slacking (blue on green)
7-DoF
Fruit lifting
building
pyramid
Figure 10: RoboCat-lim trained with additional demonstrations with additional demonstrations and self- generated data.
Figure 11:
RoboCat compared with the performance of the data-
generating agents the combined performance of these and the demon- strations, the latter of which is used for training RoboCat.
retrain a more capable generalist with this additional data. This represents first step towards a foundation agent which can iteratively learn new tasks.
To perform self-improvement, fine-tuned older RoboCat-lim equivalent models to number of unseen real-world tasks using human-teleoperateed data. These included the real blue-on-green stacking and inverted pyramid building task already mentioned in Section 5.2, well as tower-building task and 8 vegetable and fruit lifting tasks). We then used these policies generate large amounts of data autonomously, All of this data was part of the dataset used to train the main generalist shown in Section 5.1.
We first perform smaller experiment with subset of the tasks, to provide proof-of-concept of self- improvement, and carefully isolate and evaluate the contribution of self-generated data alone. We train a smaller 364M model on the structure-building tasks (i.e. those used for RoboCat-lim) and 500 deronstratiins from only few self-im ief--improvement tasks: fruit lifting (apple, banana, and peach) and blue-on-green Sawyer stacking. This represents baseline of directly incorporating the few available demonstrations into the training data for the generalist. We also train 364M "self-improved" model that additionally the self-generated data for these tasks. The results in Figure 10 show that the self-improved agent outperforms the baseline agent in all four of these tasks. In other words, given demonstrations of task, the self- improvement procedure (fine-tuning and self-generating additional data) is significantly better than using the demonstrations directly in training the generalist.
Next, demonstrate seff-improvement at scale: we incorporate self-generated data from numerous task- specific RoboCat-lim fine-tuned agents to yield stronger generalist. This is the process by which we obtained the main RoboCat generalist, presented in Section 5.1.
Figure 11 shows the performance of these self-data-generrtinng agents, compared with the performance of the full RoboCat generalist. For most cases, the RoboCat generalist performance is similar to even better than that of the agents generating the data. These results highlight. the potential for RoboCat to self-improve
17

---

Published in Transactions on Machine Learning Research (12/2023)
and grow its multi-task capabilities, we have also seen from other experiments. By self-generating data and incorporating additional data from diverse set of tasks, the resulting agent has better generalisation and fine-tuning capabilities broader set of real-world tasks.
5.4 Further ablations
We report number of additional ablations and evaluations in the appendix. These include ablating the choices for VQ-GAN tokeniser and observation prediction (Appendix D.3), comparison of different RoboCat model sizes (Appendix G.2), evaluations over many different vision model baselines (Appendix G.3), and ablations of the NIST-i environment (Appendix G.4).
6
Related Work
6.1 Transformers for decision making
Transformers (Vaswani et al., 2017) have shown impressive results at scale in domains like vision (Doso- vitskiy et al., 2020; He et al., 2022), language (Vaswani et al., 2017; Devlin et al., 2018; Brown el al., 2020) and speech (Radford et al., 2022), and can be fine-tuned to many different downstream tasks and modalities (Lu et al., 2021). Inspired by these successes, earlier efforts to leverage transformers for decision making focused on improving their training stability for RL (Parisotto et al., 2020), using self-attention for improving relational reasoning (Zambaldi et al., 2018), one-shot imitation learning (Dasari and Gupta, 2021), fast adaptation to novel tasks (Ritter et al., 2020), on the fly adaptation in 3D environments (Team et al., 2023), 3D reasoning (Shridhar et al., 2023). and multi-embodiment continuous control (Kurin et al., 2020; Gupta et al., 2022). However, these works leverage the transformer architecture within the frame- work of standard RL and imitation algorithms. Recently, generative pretraining for sequence modeling has been exlended to oflline RL (Janner et al., 2021; Chen et al., 2021), where transformer model is trained to autoregressively maximise the likelihood of trajectories in the offline dataset for specialist. agents with low-dinensional states. Building on this, Reed et al. (2022); Lee et al. (2022); Jiang et. al. (2022) train multi-task generalist agents with high-dimensional image observations. VIMA (Jiang et al., 2022) shows the power of multi-modal prompting for accomplishing many lasks with single model. Unlike our work, VIMA uses high level observation and action spaces, by assuming accurate object detection and pre-defined action primitives. Our work is closely related to Gato (Reed et al., 2022) but differs in the variety and scale of robotic tasks mastered by single agent. We show positive transfer between tasks and fast adaptations to many real-world robot tasks.
6.2 Visual pretraining for control
The use of pretrained visual representations presents promising approach for efficient robot policy learning, requiring less robot-specific data. Early efforts focused on using supervised pretraining for navigation (Zhou et al., 2019; Chen al., 2020a; Sax et al., 2018) and manipulation (Zhou et al., 2019; Chen et al., 2020a; Yen-Chen et al., 2020) domains. Building the progress in self-supervised representation learning, multiple recent works have shown that frozen visual encoders, trained through self-supervision on internet-scale datasets, can enable sample-efficient behaviour cloning (Nair et al., 2022; Parisi et al., 2022; Radosavovic al., 2023; Majumdar et al., 2023; Sharma et al., 2023), on-policy reinforcement learning (Xiao et al.,
18

---

Published in Transactions on Machine Learning Research (12/2023)
2022; Khandelwal et al., 2022; Majumdar et al., 2023). Robot-agnostic visual dynamics models also show skill transfer between robots when deployed with visual model-predictive control (MPC) policy (Hu et al., 2022). Our work differs in that we directly learn the action prediction for all embodiments jointly rather than using video prediction for planning. In this work we frozen pretrained VQ-GAN (van den Oord et al., 2017: Esser et al., 2021) trained on diverse collection of images to speed up training time significantly, and combine the VQ-GAN tokens with future frame prediction (Gupta et al., 2023) for sample-cfficient transfer learning- Concurrently, Kotar et al. (2023) also find similar generalisation benefits of using the combination of VQ-GAN tokens and future frame prediction during policy learning for the navigation domain.
6.3 Goal-conditioned policies
Goal-conditioned agents have long been of interest policy learning (Kaelbling, 1993; Schaul al., 2015). Hindsight goal relabelling a popular method for annotating arbitrary trajectorics with goals (Andrychowiez et al., 2017). Learning from visual goals is challenging as images contain a lot of information that may be unrelated to the desired goal-conditioned behaviour, such as lighting or positions of distractors (Pinto et al., 2018). As we are primarily concerned with goal images as task specification in a behaviour cloning setting, this work does not address the question of goal distance, goal generation, or exploration. Unlike Nair et al. (2018), we assume a dataset of goal images available during evaluation and data collection, as we only deploy our goal-conditioned agent for data collections on tasks for which we had telcoperated episodes to learn from. Davehev et al. (2022) also utilised a dataset of goals, bootstrapped from demonstrations. However, they do not work with images. Similar to RoboCat, Groth et al. (2021) also instruct a behaviour-cloned policy with goal images but rely on explicit. inductive biases in the network architecture to infer the task. Ghosh et al. (2019) propose iterated goal-conditioned learning as a form of reinforcement learning, which is similar to our self-improvement step.
6.4 Generalist robotic agents
Recent works have looked at the problem of training generalist robot agents. RT-1 takes language instructions to perform variely of object manipulation tasks (Brohan et al., 2022). While RT-1 trains data from two different robols, they have the same action specification. PaLM-E demonstrates that large visual-question- answering can serve planners for robotics tasks. Rather than directly controlling different robots, PaLM-E outputs language instructions (such "Pick the green rice chip bag from the drawer.") ") to pretrained lower- level controllers (Driess et al., 2023). Dasari et al. (2020) introduce a large-scale dataset of robotic interactions produced by pre-trained random policies acting on range of pick-and-place-bant placee-aaa-- manipulation tasks. They show that learned robotic agents with shared observation and action space can operate across range of environments and hardware, and also demonstrate fine-tuning capabilities.
In this work, we look to solve tasks in both simulation and the roaldworld, covering a wide set of behaviours and affordances, incorporating precision and dexterity, and embracing high digh-dimensional low-level control over multiple simulated and real embodiments. To our knowledge, RoboCat is the first work to natively support multiple real-world robotic embodiments with different observation and action specifications. We also demonstrate the ability to self-improve by fine-tuning to new tasks and self-generating data for use in retraining unique capability over all of the methods we surveyed. Finally, focus visual goal-
19

---

Published in Transactions on Machine Learning Research (12/2023)
conditioning in this work, but could also enable more flexible task specification in the future, such language conditioning full demonstrations; this is already facilitated by some of the other methods.
7 Summary and Future Work
In this report, we have presented RoboCat, generalist agent capable of solving a large and diverse set of tasks specified via goal images; across different task families, embodiments, control modes, and objects; in both simulation and the real world, and from different sources of data. RoboCat is additionally able to quickly adapt, via fino-tuning on 100-1000 demonstrations, to wide set of downstream tasks and across many different axes of generalisation. More importantly, that be added to RoboCat's training datasct for future iterations of the agent, a process call self- improvement. We have thoroughly investigated world with tens of thousands of real evaluations 36 real robots of different types. We have shown that the cost of acquisition of new skills is dramatically lower compared to single-task baselines, even when those are based on visual foundation models. Finally, we have observed that by scaling and diversifying its training data we get a RoboCat agent that performs better on training tasks and adapts better to unseen ones. Throughout our experiments, we demonstrate that RoboCat can be adapted to broad set of unseen downstream tasks (13 with the final agent, 22 more during a thorough generalisation study, and further 9 tasks in the self-improvement process). These tasks include unseen embodiments (KUKA with a dexterous hand), new task familics (eg. lifting, insertion/removal inverted pyramid building), held-out perceptual variations, real versions of previously-soen sim tasks, and many unscen objects (eg. printed fruits and vegetables, shapo-matching objects).
such adapted agents to generate data
agent's capabilities both in simulation and the real
Future work could look into enabling flexible and multi-modal task specification. Incorporating relevant existing, freely-available datasets with language annotations would be a first good step. Task specification via language offers complementary benefits to visual goals, and different tasks may be better specified by either modality. In addition, while this work focused on visual goal-conditioning and VFM baselines, which may be able to reason well over images; language-conditionnng and LLM/VLM baselines may offer better temporal reasoning capabilities.
Another research avenue could explore improving both training and fine-tuning capabilities of such model with reinforcement learning (RL), since RoboCat in its current form only employs behaviour cloning. While visual goal specification already allows the agent to learn from failures and sub-optimal data, incorporating RL would enable both learning with rewards and learning online with real-world interaction. Finally, while RoboCat aims to lackle behavioural diversity in manipulation tasks, the different embodiments are all in a controlled lab setting with visually-similar backgrounds. We hope that next-generation foundation agents will demonstrate robustness to different basket textures and operate in more visually-diverse environments in the wild.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Broader Impact
This work presents progress on training generalist agents for robolic manipulation. Our work presents a recipe, and first steps, in an emerging area, with experiments in controlled lab environment demonstrating promising but imperfect performance. Nonetheless, the potential impact on society from generalist robotic agents calls for increased interdisciplinaryy research into their risks and benefits. Thus, we discuss the broader impact of this line of research, beyond the specific contributions this paper. To provide an easily accessible reference for RoboCat's intended use-case and potential shortcomings we include a model card in Appendix A. We emphasise that the model is for research use only and not currently deployed in any production scenario to any users, and thus expect no immediate societal impact.
In general, RoboCat inherits many of the safety concerns discussed in Gato (Reed et al., 2022); on which it is based. In addition, since RoboCat takes actions in the physical world-and and on multiple embodiments may pose new challenges with respect to safety. For example, physical embodiments and imitation from human data can cause humans to anthropomorphise the agent; leading to a potentially misplaced trust and underappreciation for inherent dangers that come from interacting with robots Additionally, cross- embodiment transfer from one robot to another can lead to undesired movements (such as high gain motor actuation). Considerations with respect to general AGT safety (Bostrom, 2014) may also require updating when considering agents with multiple embodiments.
We consider that value alignment (Russell, 2019) with human preferences (as e.g. expressed via reward labelling in this work) is crucial for safe evolution of this technology. While our reward labelling process to determine successful and desired behaviours is a starting point for this, future work should consider adapting alignment techniques successfully used for language models to our setting (Ouyang et al., 2022; Kenton et al., 2021; Bai et al., 2022).
Finally, the self-im self-improvement loop we designed for RoboCat allows us to improve the model over time by retraining on data collected from deploying a previous version to our robots. Such a self-improvement loop poses additional challenges with respect to AGI safety since it, partially, implements a reinforcement learning loop; which comes with its own safety concerns (see e.g. Omohundro (2008); Turner et al. (2021)). While further work is needed into AGI safety for reinforcement learning robotic systems, it is important to note that unlike in reinforcement learning scenario, the solf-improvement capabilities of RoboCat are not autonomous and no learning takes place while interacting with the physical world. That is, data collection is started and stopped by humans and uses frozen versions of RoboCat. Learning an improved version is implemented as supervised learning problem from fixed data source and is entirely decoupled from data collection.
Acknowledgements
We would like to thank Jackie Kay for contributions to the VQ-GAN codebase; Federico Casarini for help with the robotic lab operations; Markus Wulfmeier for initial exploration into alternative fine-tuning methods; Nando de Freitas for general advice; Yixin Lin, Vincent Vanhoucke, Shakir Mohamed, and Michael Neunert for paper feedback; and Jonathan Hutchinson for the graphics of Figure 1.
We that utilise force-torque compliant controller with built safety mechanisms.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Author Contributions
RoboCat generalist training
Konstantinos Bousmalis, Giulia Vezzani, Coline Devin, Dushyant Rao, Alex X. Lee, Maria Bauza, Todor
Davchev, Yuxiang Zhou, Agrim Gupta
RoboCat fine-tuning
Giulia Vezzani, Dushyant Rao, Alex X. Lee, Coline Devin, Maria Bauza, Todor Davehev, Valentin Dalibard,
Martina Zambelli, Agrim Gupta
Core infrastructure for experiments at scale
Michiel Blokzijl, Claudio Fantacci, Akhil Raju, Antoine Laurens, Dave Barker
Data and tasks
KUKA: Murilo F. Martins, Martina Zambelli, Rugile Pevceviciute, Antoine Laurens, José Enrique Chen
NIST-i Todor Davehev, Maria Bauza, Akhil Raju, Jost Tobias Springenberg, Jon Scholz, Misha Denil, Oleg
Sushkov, Jean-Raptiste Regli, Tom Rothörl
RGB: Giulia Vewzani, Konstantinos Bousmalis, Dushyant Rao, Coline Devin, Alex X. Lee, Thomas Lampe,
Abbas Abdolmaleki, Francesco Nori, Antoine Laurens
Non-NIST-i insertion/removal:: Akhil Raju, Antoine Laurens, Alex X. Lee
Evaluation infrastructure: success detectors, no-reset evaluation, annotations
Akhil Raju, Claudio Fantacci, Misha Denil, Michiel Blokzijl, Todor Davchev, Thomas Lampe, Dave Barker,
Maria Bauza, Alex X. Lee, Jon Scholz, Tom Rothörl
VQ-GAN tokenisation
Agrim Gupta, Coline Devin, Scott Reed
Gato architecture and infrastructure
Emilio Parisotto, Konrad Žolna, Scott Reed, Sergio Gómez Colmenarejo, Jost Tobias Springenberg, Oliver Groth
Single-task VFM baselines Yuxiang Zhou, Todor Davchev, Alex X. Lee
Teleoperated data collection
Akhil Raju, Antoine Laurens, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Claudio Fantacci, Joy Ortiz
Paper and blog post content
Coline Devin, Alex X. Lee, Dushyant Rao, Konstantinos Bousmalis, Giulia Vezzani, Todor Davchev, Maria Bauza, Agrim Gupta, Akhil Raju, Antoine Laurens, Jost Tobias Springenberg, Misha Denil, Nicolas Heess
Project leadership and coordination
Konstantinos Bousmalis, Giulia Vezzani, Joy Ortiz
Advisors
Nicolas Heess, Francesco Nori, Raia Hadsell, Jost Tobias Springenberg, Martin Riedmiller, Yusuf Aytar
22

---

Published in Transactions on Machine Learning Research (12/2023)
References
Abdolmaleki, T. Springenberg, Tassa, R. Munos, N. Heess, and M. Riedmiller. Maximun a posteriori policy optimisation. In International Conference on Learning Representations (ICLR), 2018.
J. Abramson, A. Ahuja, I. Barr, A. Brussee, F. Carnevale, M. Cassin, R. Chhaparia, S. Clark, B. Damoc, A. Dudzik, P. Georgiev, A. Guy, T. Harley, F. Hill, A. Hung, Z. Kenton, J. Landon, T. Lillicrap, K. Math- ewson, S. Mokrá, A. Muldal, A. Santoro, N. Savinov, V. Varma, G. Wayne, D. Williams, N. Wong, C. Yan, and R. Zhu. Imitating interactive intelligence. arXiv preprint or arXiv:2012.05672 2021.
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, Hindsight experience replay. In Newral Information Processing Systems (NeurIPS), 2017.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan,
N. Joseph, S. Kadavath, J. Kernion, T. Concrly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravoc, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. Mc- Candlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0580 2022.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosso- lut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, Hong, K. Hsu, Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamchoti, Keeling, F. Khani, O. Khattab, W. Koh, M. Krass, Krishna, Kuditipudi,
A. Kumar, F. Ladhak, M. Lee, T. Lee, Leskovec, Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D.
Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, 1. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, Tramèr, R. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the opportunitics and risks of foundation models. arXiv preprint or Xiw:2108.07256 2022.
N. Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Inc., 2014.
A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning (ICMT), 2021.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Her- zog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov,
Y. Kuang, I. Leal, K.-H. Tee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, F. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q Vuong, F. Xia.
23

---

Published in Transactions on Machine Learning Research (12/2023)
T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, RT-1: Robotics transformer for real-world control scale. arXiv preprint arXim:2912.06811 2022.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry,
A. Askell, S. Agarwal, A. Herbert-Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Neural Information
Processing Systems (Ne,rIPS), 2020.
B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. Berkeley dataset for robotic manipulation research. International Journal of Robolics Research (IJRR), 2017.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervisdd vision transformers. In International Conference on Computer Vision (ICCV), 2021.
Chen, A. Sax, G. Lewis, I. Armeni, Savarese, Zamir, J. Malik, and L. Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. arXiv preprinl arXiv:2011.06698, 2020a.
L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and L. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Newral Information Processing Systems (NeurIPS), 2021.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning visual representations. In International Conference on Machine Learning (ICML), 2020b.
S. Dasari and A. Gupta. Transformers for onc-shot visual imitation. In Conference Robot Learning (CoRL), 2021.
S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 885-897. PMLR, 30 Oct-01 Nov 2020. URL https://proceddins www.mmlrrr......... wwwppeee////000/aarr200000 html.
T. Davehev, O. O. Sushkov, J.BB. Regli, S. Schaal, Y. Aytar, M. Wulfmeier, and J. Scholz. Wish you were here: Hindsight goal selection for long-horizon dexterous manipulation. In International Conference on Learning Representations (ICLR), 2022.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Compuler Vision and Pallern Recognation (CVPR), 2009.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
A. Dosovitskiy, T.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer,
G. Heigold, S. Gelly, et al. An image is worth 16x16 words: arXiv preprint or Xic:2010. 1929, 2020.
Transformers for image recognition at scale.
24

---

Published in Transactions on Machine Learning Research (12/2023)
D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303. 03378, 2023.
P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Computer Vision and Pattern Recognation (CVPR), 2021.
D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. Xrx preprint 1XXi:1912.06088 2019.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukeuoglu, R. Munos, and M. Valko. Bootstrap your own latent new approach self-supervised learning. In Neural Information Processing Systems (NeurIPS), 2020.
O. Groth, C.-M. Hung. A. Vedaldi, and I. Posner. Goal-conditioned end-to-end visuomotor control for versatile skill primitives. In International Conference on Robotics and Automation (ICRA), 2021.
A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: Learning universal controllers with transformers.
In International Conference on Learning Representations (ICLR), 2022.
Gupta, S. Tian, Y. Zhang, J. Wu, R. Martin-Martin, and Fei-Fei. MaskViT: Masked visual pre-training for video prediction. International Conference on Learning Representations (ICLR), 2023.
K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners.
In Computer Vision and Pattern Recognition (CVPR), 2022.
O. J. Hénaff, S. Koppula, J.-B. Alayrac, A. Van den Oord, O. Vinyals, and J. Carreira. Efficient visual
pretraining with contrastive detection. In International Conference on Computer Vision (ICCV), 2021.
O. J. Hénaff, S. Koppula, E. Shelhamer, D. Zoran, A. Jaegle, A. Zisserman, J. Carreira, and R. Arandjelovié. Object discovery and representation networks. In Europeon Conference Computer Vision (ECCV), 2022.
E. Hu, K. Huang, O. Rybkin, and D. Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representations, 2022. URL https:
/ //openreview ioapenreriee.nnt//fouutttaaeeeee
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In European Conference on Compuler Vision (ECCV), 2016.
M. Janner, Q Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Neural Information Processing Systems (NewrIPS), 2021.
Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. VIMA: General robot manipulation with multimodal prompts. arXiv preprint wxXiv:2210.03099 2022.
L. P. Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAT), 1993.
25

---

Published in Transactions on Machine Learning Research (12/2023)
Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of language agents. arXiv preprint 0rXtc:2108.14662 2021.
A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embeddings for embodied AI. In Computer Vision and Pattern Recogrition (CVPR), 2022.
K. Kimble, K. Van Wyk, J. Falco, E. Messina, Y. Sun, M. Shibata, W. Uemura, and Y. Yokokohji. Bench- marking protocols for evaluating small parts robotic assembly systems. Robotics and Automation Letters (RA-L), 2020.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
K. Kotar, A. Walsman, and R. Mottaghi. ENTL: Embodied navigation trajectory learner. arXiv preprint arXiv:2304.022 02669, 2023.
V. Kurin, M. Igl, T. Rocktäschel, W. Boehmer, and S. Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. arviv preprint 0et:::201100188500 2020.
A. X. Lee, C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, Casarini, M. Riedmiller, R. Hadsell, and Nori. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. In Conference on Rohot Learning (CoRL), 2021.
K.-H. Lee, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fischer, E. Jang, H. Michalewski, and I. Mordatch. Multi-game decision transformers. In Neural Information Process- ing Systems (NewrIPS), 2022.
T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV), 2021.
Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arxiv: 1.05101, 2017.
K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXim:2103.052: 052477, 2021.
A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, P. Abbeel, J. Malik,
D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.182) 2023.
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In Neural Information Processing Systems (NeurIPS), 2018.
S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3M: A universal visual representation for robot manipulation. In Conference Robot Learning (CoRL), 2022.
26

---

Published in Transactions on Machine Learning Research (12/2023)
S. Omohundro. The basic AI drives. In Conference Artificial General Intelligence, 2008.
T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective imitation learning. Foundations and Trends& Robotics, 2018.
L. Ouyang, Wu, X. Jiang, D. Almeida, Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In Neural Information Processing Systems (NeurIPS), 2022.
S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.0355 033500 2022.
E. Parisotto, H. F. Song, J. W. Rae, R. Pascanu, C. Gulcehre, S. M. Jayakumar, M. Jaderberg, R. L. Kaulman, A. Clark, S. Noury, M. M. Botvinick, N. Heess, and R. Hadsell. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image- based robot learning. In Robotics: Science and Systems (RSS), 2018.
A. Radford, W. Kim, C. Hallacy, A. Ramosh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krucger, and I. Sutskover. Learning transferable visual models from natural language super- vision. In International Conference on Machine Learning (ICML), 2021.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04355 2022.
I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training, In Conference on Robot Learning (CoRL), 2023.
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-maron, M. Giménez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell,
O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions Machine Learning Research (TMLR), 2022.
T. Ridnik, E. Ben- Nen-Baruch, A. Noy, and L. Manor. ImageNet-211K pretraining for the masses, anXiv preprint arXie:2104.1097 10972, 2021.
C. Riquelme, Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. Susano Pinto, D. Keysers, and
N. Houlsby. Scaling vision with sparse mixture experts. In Neural Information Processing Systems (NeurIPS), 2021.
S. Ritter, R. Faulkner, L. Sartran, A. Santoro, M. Botvinick, and D. Raposo. Rapid task-solving in novel environments. arXiv preprint arXic:2006. 03662, 2020.
S. Russell. Humon compalible: Artificial intelligence and the problem of control. Penguin, 2019.
A. Sax, B. Emi, A. R. Zamir, I. Guibas, S. Savarese, and J. Malik. Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies. arXio preprint 181 11971. 2018.
27

---

Published in Transactions on Machine Learning Research (12/2023)
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International Conference Machine Learning (ICML), 2015.
M. Sharma, C. Fantacci, Y. Zhou, S. Koppula, N. Heess, J. Scholz, and Y. Aytar. Lossless adaptation of pretrained vision models for robotic manipulation. orXiv preprint arXiv:2304.06600 2023.
M. Shridhar, L. Manuclli, and D. Fox. Porceiver-actor: A multi-task transformer for robotic manipulation.
In Conference on Robot Leorning (CoRL), 2023.
Srivastava, G. Hinton, A. Krizhevsky, 1. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research (JMLR), 2014.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. A. Riedmiller. Deepmind control suite. orXiv preprint xxXi:1801000699 2018.
A. A. Team, J. Bauer, K. Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, E. Hughes, S. Kashem, M. Loks- moks-Thomsson,
H. Openshaw, J. Parker-Holder, S. Pathak, N. Perez-Nieves, N. Rakicevie, T. Rocktäschel, Y. Schroecker, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, and T.. Zhang. Human-timescale adaptation in an open-ended task space. Tn International Conference on Machine Learning (ICMT.), 2023.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In International Conference Intelligent Robots and Systems (IROS), 2012.
A. M. Turner, L. R. Smith, R. Shah, A. Critch, and P. Tadepalli. Optimal policies tend to seek power. In Newral Information Processing Systems (NeurIPS), 2021.
A. van den Oord, O. Vinyals, and K. Kavukeuoglu. Neural discrete representation learning. In Neural Information Processing Systems (NeurIPS), 2017.
Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention you need. Neural Information Processing Systems (NeurIPS), 2017.
T. Xiao, Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint 0at::220300272 2022.
L. Yen-Chen, A. Zeng, S. Song, P. Isola, and T.-Y Lin. Learning to see before learning to act: Visual pre-training for manipulation. In International Conference on Robotics and Automation (ICRA), 2020.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, T. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap. F. Lockhart, et al. Deep reinforcement learning with relational inductive biases. In International Confer- Learning Representations (TCL.), 2018.
28

---

Published in Transactions on Machine Learning Research (12/2023)
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In Computer Vision and Pattern Recognition (CVPR), 2018
B. Zhou, P. Krähenbühl, and V. Koltun. Does computer vision matter for action? Science Robotics, 2019.
29

---

Published in Transactions on Machine Learning Research (12/2023)
Appendix
Model details
Organisation
Google DeepMind
Model date
June 2023
Model type
Transformer with VQ-GAN encoder for multi-task, multi-embodinment behaviour cloning from human, agent and self-generated data.
Madel version
Initial releuse.
Feedback on the model
konstantinos Ggoogle.cm sialiveezanni@google..co
Intended uses
Primary intended uses
Research into learning to acromplish a wide variety of tasks from expert demonstrations or multiple real robot embodiments for manipulation.
Primary intended
Google DeepMind Researchers.
users
Out-nf-scope
Not. intended for commercial production use. Military strictly prohibited.
Factors
Relevant factors
Salient factors that may alter model performance are: agent embodiment in control data, training data token amount and diversity, performance of experts in training data and goal couditioniug. Quality of policy used for self- generated data collection. Quality of the VQ-GAN encoder. Zero-shot evaluation held-out robots.
Evaluation factors
Reported factors are: number of input tokena, importance of mance.
different tokenisation schemes, agent perfor-
Metrics
Model performance measures
We chose report success at the task (incasured having solved task end of an episode) an episodic evaluation setting. Held-out tasks are used to assess generalisation, ablations show importance of different components.
Decision thresholds
N/A
Approaches to uncertainty and variability
The reported values do not take into consideration model uncertainty as they are evaluations of a single model and its ablations. It prohibitive us to evaluate models from multiple training rums would involve constantly training and evaluating robots. We account for noise in the evaluation by averaging success across multiple episodes.
Evaluation
Tasks
RoboCat evaluated on in-distribution and cutt---diitriiuutin tasks,, on both simulated and real-world robot environments. See section 3 for details on our tasks.
Training Data
Datasets
We use diverse and large number of datasets for training RoboCat. These include data from agent experi- ence, human demonstrations and self-generated data, on both simulated and real-world robot environments. See subsection 3.4 for details on our datasets.
Motivation
Create multi-modal, multi-task, multi-embodiment generalist policy by collecting much, diverse, data possible. Joint training on all the datasets has produced a single network, RoboCat, capable of performing these tasks.
Pre-processing
The multi-modal training data is tokenised into a stream of discrete embeddings. See subsection 2.1.
Quantitative Analyses
Unitary results
We present several evaluations of RoboCat on a variety of manipulation tasks, See section for an analysis of the model capabilitirs and ablations.
Ethical Considerations
RoboCat is an early research model that has not yet been evaluated for deployment and safety of use outside pure research setting.
Caveats and Rocommendation
Future work
The interaction of diverse training data domains and the different affordances faced in evaluation is poorly understood, and potential ethical and safety risks arisc the generalist's capabilities grow.
Table 4: RoboCat model card.
30

---

Published in Transactions on Machine Learning Research (12/2023)
A Model Card
We present a model card for RoboCat in Table
B Tasks and Data
In this section, we provide an extensive description of the tasks and data RoboCat has been trained and fine-tuned on.
B.1 Task families
In total, consider 253 different task variations, each of which can have an infinite number of state configurations describing it (see Table 5) and can be grouped in different task families. Figure 12 collects examples of the goal images used for each task family,
B.1.1 Lifting objects
We include handful of lifting tasks performed in the physical world and three in simulation. A lifting task is defined grasping and lifting object off the basket surface until the natural end of the episode. Our aim with this task family is primarily study goal understanding and generalisation
embodiments and tasks. This task family includes four variants for the YCB fruits (see Figure 12(n) for example of goal image), vegetable objects (see Figure 12(m)) and seven variants for NTST-i gears. For the three NIST-i gears we include lifting in simulation (see Figure 12(h)) and in real with the Panda 7DoF (see Figure 12(o).
We also include lifting the large gear with the KUKA 14DoF (see Figure 12(v)).
B.1.2 Building structures
For the RGB-objects and NIST-i objects that are coloured in red, green, and blue*, we use a set of structure- building lasks. These are difficult due to the non-cubic shapes of the objects: without orienting the objects correctly, they can easily slide off of each other.
Stacking: A top and bottom object are specified and must be stacked stably. This task family has 6 variations for triplet. Although stacking cuboids is relatively easy, RGB objects stacking requires an agent to come up with strategies beyond simple pick-and-place. These strategies differ for each task variation, well. We consider this task both in simulation (see Figure 12(a) (d) for examples of goal images) and in the real-world (see Figure 12())).
Tower Building: This requires two stacks to build a tower out of 3 objects. For this task family, we consider only subset all the variations we could have per triplet. Some variations have been particularly challenging to solve with reinforcement learning agents and therefore we did not collect training data for them. We consider this task in simulation (see Figure 12(e)) and, for one variant, also in the real-world (see Figure 12(k))-
Pyramid Building: Two objects must be placed close each other and third on top of both.
We consider this task family only in simulation (sce Figure 12(g)).
NIST-i gears coloured according their sizes: red, green, and blue for small, medium, and large, respectively.
31

---

Published in Transactions on Machine Learning Research (12/2023)
Training Task Variations
Number Training Episodes
Example Goal Image
Source of
Embodiment
Task Family
Object Set
Stacking
RGB objects
RL
022
Sawyer 7-Dol
Stacking
NIST-I
RL
182
Stacking
RGB objects
30
RL
1 142
Simulation
Stacking
NIST-i gears
RL
129 664
Tower building
RGB objects
RLL
99 725
Panda
7-DoF
Tower building
NIST-i gears
RL
42 805
Pyramid building
RGB objects
30
RL
603 133
Lifting
NIST-i geurs
Teleop
421 638
Insertion-peg
NIST-i gears
Teloop
Stacking
RGB objects
RL, Teleop, RoboCat
Sawyer 5-DoF
Tower building
RGB objects
Teleop, RoboCat
Inverted pyramid building
RGB objects
Telcop, RoboCat
15 155
Lifting
YCB-i vegetables
Teleop, RoboCat
55 325
Real World
Lifting
YCB fruits
16
Teleop, RoboCat
19 957
Lifting
NIST-i gears
Telcop
25 566
Insertion-per
NIST-i gears
Teleop
21 933
Panda 7-DoF
Removal-peg
NIST-i gears
Teleop
24 803
YCB fruits YCB.i bowl
Insertion- boow
Teloop
1000
Removal-bowl
YCB fruits and YCB.i bowl
Teleop
1000
Insertion- base
Shape-matching objects
Teleop
1000
Removal-base
Shape-matching objecta
Teleop
KUKA 14-Dof
Lifting
NIST-i gears
Teleop
Table 5: Training and fine-tuning tasks used by the final RoboCat agent. See Figure 12 for images.
32

---

Published in Transactions on Machine Learning Research (12/2023)
(x) RGB stacking (Sawyer 7-DoF)
(b) Gear stacking (Sawyer 7-DoF)
(c) RGB stacking (Panda 7-DoF)
(d) Gear stacking (Panda 7-DoF)
(e) RGB tower building (Panda 7-DoP)
Gear tower
building
(Panda 7-DoP)
(8) RGB pyramid (Panda 7-DuF)
(h) Gear lifting (Panda 7-DoF)
(i) Gear insertion (Panda 7-DoF)
(i) RGB stacking (Sawyer 5-DoF)
(k) RGB tower building
(I) RGB inverted pyramid building (Sawyer 5-Dof")
(Sawyer 5-DoF)
(m) Vegetable lifting (Sawyer 5-DoF)
(n) Fruit lifting (Panda 7-DoF)
(o) Gear lifting (Panda 7-DoF)
(p) Gear insortion (Panda 7-DoF)
(q) Gear removal (Panda 7-DoF)
(r) Fruit insertion bow)
(Panda 7-DoF)
Fruit removal from bowl (Panda T-DoF)
(t) insertion (Panda 7-DoF)
Shape removal (Panda 7-DoF)
(v) Gear lifting (KUKA 14-DoF)
Figure 12: Example goal images. These are larger versions of the goal images shown in Table 5 (in order) corresponding each embodiment, lask family, and object set combination. Images (a)-(i) correspond to tasks in simulation and images (v) correspond to tasks in the real world.
33

---

Published in Transactions on Machine Learning Research (12/2023)
Inverted Pyramid Building: This task family requires to place two objects on top of third one. We consider only one task variation where the bottom object is wider while the other two objects can barely fit on top of the bottom object. The difficulty of this task comes from the fact that the robot needs to be very precise in placing the top objects such that stacking them becomes successful and stable. We only consider this task in the real world (see Figure 12(()).
B.1.3 Inserting and removing objects
We introduce pairs of insertion and removal tasks of quite different flavours using the YCB, NIST-i, and the Shape Matching Objects.
Inserting and removing fruit in /f/from a bowl: For this task family we use the YCB strawberry, lemon, and peach, and the YCB-i bowl. When inserting the different fruits, the bowl can start either empty or with other fruits it This task family has 6 variants: inserting/removingg (strawberry, lemon, peach} in/from the bowl (see Figure 12(r) and Figure 12(s) for examples of insertion and removal respectively).
Inserting and removing shapes in/fron board: This task requires a given object from the Shape Matching Objects be inserted into the correct shape in the board. This task poses several challenges: 1) the agent has to understand the right shape for the desired object; 2) it has reorient it, order properly fit into its shape; 3) the board can move, making the task even more challenging (see Figure 12(t) and Figure 12(u) for examples of insertion and removal respectively).
Inserting and removing NIST-i gears on/from shafts (NIST-i): In this task, specific gear must be precisely inserted in particular shaft. This task poses several changes the gears are hard to manipulate, the tolerance for the shafts is low and, in the real-world version of this task, the base of the shafts is not anchored and therefore it freely move. See Figure 12(p) Figure 12(q) for examples of insertion and removal respectively in real, and Figure 12(1) for example of insertion in simulation.
B.1.4 NIST-i tasks
In this paragraph we provide some extra details on the NIST-i tasks.
Comparison of NIST-i to NIST The primary differences between our NIST-i task and the official NIST benchmark (Kimble al., 2020) are three-fold: i) unlike the NIST board, our base is not fixed allowing it to freely move around the robot cell; ii) the NIST-i gears are red opposed to white with peg board that is black as opposed to metallic; and iii) the tolerance between the shafts and gear holes is smaller for the original NIST board. The insertion tolerance of the physical NIST-i gears is 1 mm opposed to the sub-millimeter tolerance for the actual NIST board which ranges between 0.029 mm 0.005 mm. In the simulated NIST-i environment, we used green board with blue pegs and 1 mm insertion tolerance. Tn the KUKA 14DoF enviromment the NIST-i object set also includes two green pegs and additional base. These extra objects serve as distractions for the task of lifting the large gear.
34

---

Published in Transactions on Machine Learning Research (12/2023)
Comparison of simulated and physical tasks In this work the simulated and physical version of the NIST-i tasks have different colours (sce Figure 27). For instance, when stacking with NIST-i gears in simu- lation we found that telsoperation using only red gears was quite challenging for the human demonstrators mostly due the reduced depth perception. The red-green- setup for stacking tasks is nicer for contrast and therefore for people to collect data. In addition to this, for /remmvvl tasks we chose to fix the base in simulation, primarily for simplicity. Early this project, used the simulated environment as testing ground, so we wanted to simplify the problem. Specifically, by fixing the gear base could guaranlee two things: i) this makes the task easier for raters and for agents (see Appendix G.4.1); and ii) it also makes the simulation run much faster while having freely moving base makes the sim slower and overall more complex.
B.2 Data sources
This section provides further details on how RoboCat training data has been generated.
RL-agent data For all our structure building tasks in simulation we trained expert policies, per lask variation, via off-policy RL with a shaped reward for each. We used the MPO algorithm (Abdolmaleki et al., 2018) for this purpose, but any RL algorithm could have been used. We found training directly from state to be significantly faster than training from vision and thus generated all our simulation structure building datasets from state-based agents. It important to note that when training RoboCat on this data, we discard any privileged information, such as the object poses, that would not be readily available in our real-world tasks. The data for the real Sawyer 5-DoF stacking tasks was also generated by RL agents using simulation-to-rrality transfer and offline RL, as discussed in Lee et al. (2021). Table 6 shows the number of successful and failed episodes obtained from RL agents for these tasks.
Human teleoperation The rest of the data were either collected by human teleoperation or by a RoboCat agent finc-tuned on such data.
Human leleoperators collected demonstrations by controlling the robot arm and gripper with a 3DComeexio SpaceMouse compact and a web-based UI. The web UI similar to the one used by Abramson et al. (2021) and enabled us to work with participants from around the globe. We worked with over 100 participants from different countries to collect over 4000 hours human demonstrations across many of our tasks. Remote telcoperators could use the basket and wrist cameras to operate the arm while on-site teleoperators would sit next to the robots while controlling them. We noticed that on-site telcoperators achieve higher success rates than their remote counterparts, but the human baselines reported in this paper are aggregated across all locales.
During each episode, we showed a different language instruction to the teleoperator in the web UI, like "lift the apple". Some episodes were used for training or calculating human baselines, while other episodes helped ensure we had diverse starting configurations for the following episodes. Empirically, we found that these "shuffle" episodes made our agents more robust.
We also worked with the same pool of participants and used the same setup to collect data in simulation for human baselines.
httpe://batommeeiinoooo////////p /oppa///oe//m////////ccmsosse/
35

---

Published in Transactions on Machine Learning Research (12/2023)
Table 7 reports the numbers of successful and failed human teleoperations collected for the NIST-i tasks, both in simulation and with the real robots. Table 8 shows the number of successful and failed human teleoperations collected on the real robot for the tasks used for self-improving RoboCat. Table 9 shows instead the number of successful episodes collected and used for fine-tuning our final RoboCat agent. Finally, Table 10 reports the telcoperators success rates for each real robot task to contextualize the quality of the data and the difficulty of telcoperating each task.
Self-generated data Once human teleoperation had generated minimum number of successful episodes for given task variation, fine-tuned a RoboCat agent 500 1000 successful episodes, explained in subsubsection 2.1.2. We then used this specialised agent gather, autonomously, more data for the same task variation. This self-generated RoboCat data, combined with all demonstrations that ended up being collected for that task variation, used to train final RoboCat agent. The total number of episodes collected via the self-generation process can be found in Table
C Embodiments and Environments
In this section we describe the physical embodiments used, shown in Figure 2 in the paper, and the envi- ronments they are in, both in simulation and in the real world. Fach embodiment consists of a arm and an end-effector, described in the following paragraphs. Table 11 shows the subset of proprioception observations used for each embodiment both during RoboCat training and evaluation.
We use 3 different arms for our experiments: the Sawyer from Rethink Robotics, the Panda Franka Emika, and the KUKA LBR IIWA14 arm. Across these setups, we use different controllers and end-effectors. And though all arms have the same degrees of freedom, their dynamics and state distributions are significantly different given differences between arm link lengths and their controllers. In addition, sometimes constrain the Sawyer's degrees of freedom (see subsection 3.1).
For each arm, did not align the simulation and real environments. For instance, we have different observations and the controllers are often different between simulation and reality. We also did not perform system identification for any embodiment to try align the control dynamics of the simulated system with the real world. In simulation, we physics randomisation but visual randomisation.
Although the combination of different arms, end-effectors and discrepancies between simulation and real- world provides total of 5 different embodiments, in the next sections organise information by grouping it per robot arm.
All the enviromments for the arms written with the open-source MoMa library unless otherwise specified below. MoMa uses MuJoCo (Todorov et al., 2012) underneath physics engine for the simulation environments and forward kinematics library for the real ones.
The Sawyer is developed retailed by the Hahn Group. htttps///githbb httpthhw..oomdmeeeeeeeeeeeee com/deepmind/dtt httpr//oiicc/teeeeaaaaaa///////e
36

---

Published in Transactions on Machine Learning Research (12/2023)
Table 6: Quantities of RL agent data for training simulated and real tasks. Note that no real data has been collected with RL agents for the NIST-i tasks, lifting, insortion and removal,
Falluree
20.00
22
20
51.00
1901
4760
2090
3422
gears
NIST.
S Steccasee
22007
10368
18006
24086
19777
41.00
42044
1400
2780
2029
4124
51.4
4465
5807
4388
51.08
20-488
3466
2977
20442
03394
2003
1500
Triplet
303N06
27048
30018
16420
14723
15738
15097
177811
17.00
300033
35101
552836
10006
Faliure
2/808
1630
1708
6588
4299
2180
1000
4993
64400
2866
1800
1834
1015
2400
2301
717
2403
Triples 4
331.33
933088
3180
32477
14215
16030
18099
18937
1.0092
14469
10058
6312
330300
35637
920677
30000
34077
1.0222
50.0
6008
2070
6220
5013
02232
7007
7088
4403
$1.50
3183
5224
3034
7H0M
Triples
Successe
01.000
24002
01.000
32437
18694
115008
16107
12184
11470
16707
32200
30834
S0:200
07078
33003
31.000
9A70
1200
4407
2122
4402
6022
5008
6200
0152
7923
7370
3971
2089
2000
2000
8018
Triplet 2
Stice Cac
92426
27016
20709
4008
15124
19071
11000
12893
93092
30007
950000
33577
3000-9
9055
Paliure
10723
9454
1122
3212
#007
8001
4491
#300
6188
14238
2207
1974
4600
9448
3966
Triplet 1
22116
91023
32370
17263
17604
17400
10000
11826
13787
28356
32217
308002
31011
125000
Colour permutation
Ied green and bloe
Bire ot sesreddaann aan sgreen
Gremn -o red can--an-aaa -nn---iuee buiu
Crean on blue-aa and sed xed
Wed--n www.ooooooooooo wwaeeooogggeee trom
Green-on- rrcom-on-zed www.oono-dddnn-oood bl-o
Green-on on bloe bww--oo-ooet on-red
Red Red-on--ne 00--7mm
Blee ob mue-oo-greno reto
Bluwomteed
Grmmnonn-blo blne
Amb-on-blae
Ite Red-on-grann greenn
Blae-co ca cue-co--gent grenn
Blue- on-3oo
Hedi-mm-hie
Gremn-mmene
on
Bhe
Bludhine
Sawayne D-Dilt
37

---

Published in Transactions on Machine Learning Research (12/2023)
Embodiment
Task Family
Object Set
Variant
Human teleop demos
Successes
Pailures
Small
120701
18385
Lifting
Medium
122409
16517
7-DoF (Simulation)
Large
Small
G4813
30513
Insertion-peg
Medium
70638
30895
Large
84859
13972
Small
8282
210
Lifting
NIST-i gears
Medium
Large
8791
240
Panda 7-DoF
(Real World)
Insertion-peg
Medium
6715
601
Large
6558
552
Small
6699
1993
Removal-peg
Medium
6631
1642
Large
Table 7: Quantities of human demonstrations collected for the NIST-i tasks in simulation and with
real robots.
Embodiment
Task Family
Object Set
Variant
teleop
Agent trajectorics
Successes
Fallures
Successes
Fallures
Carrot
2819
1389
3195
6438
Lifting
vegetables
Cucumber
3162
1007
4052
5676
Sawyer 5-DoF
Pepper
1938
2317
2217
7122
(Real World)
Stacking
RGB objects
BG, triplet
424
U
4159
3853
Tower building
RGB objects
RBG. triplet 5
926
1821
4718
7776
Inverted pyramid
RGB objects
RBG. blocks
1409
1510
1081
11155
Apple
1956
121
5117
6038
Panda 7-Dof' (Real World)
Banana
Lifting
YCB fruits
Peach
2200
98
2409
11125
Strawberry berry
2052
137
195
2171
Table 8: Quantities of human demonstrations and self-generated data.
Emhodiment
Task Family
Object Set
Variant
Human telcop
Successes
Failures
YCB fruita and YCB-i bowl
Lemon
4039
357
Insertion-bowl
Peach
2807
355
Strawberry
1958
394
YCB fruits and YCB-J bowl
2630
Removal-bowl
Peach
2661
417
Panda 7-DoF
Strawberry
1889
385
(Real World)
Pentagon
2531
469
Insertion-hase
Shapo-matching objerts
Cirele
2526
376
Square
Pontagon
2684
313
Removal-hase
Shapo-matching objects
Cirele
2534
Square
1688
284
KUKA 14-DoF (Real World)
Lifting
NIST-i gears
Large
1936
121
Table 9: Quantities of human demonstrations collected for final fine-tuning tasks. Note that for fine-tuning we only used a subset, either 500 or 1000, successful episodes and no failed episodes.
Embodiment
Control Frequency
Task Family
Episode Length
Number Steps
Rate
Stacking
96.5%
Tower Building
81.6%
Fruit Lifting
95.4%
Fruit Inserting
30s
300
90.0%
Panda 7-DoF (Heal World)
Fruit Removing
89.1%
Gear Lifting
93.8%
Inserting
Gear Removing
84.3%
Shape Insorting
30%
300
84%
Shape Removing
30s
300
88%
10 Hz
Starking
40s
400
74.3%
Sawyer (Real World)
Vegetable lifting
20s
63.2%
20 Hz
Tower Building
60s
1200
50.0%
Inverted Pyrauid
120s
2400
52.6%
KUKA 14-DoF (Real World)
Hz
Goar lifting
60%
600
88.4%
Table 10: Human teleoperator success rates on real robot tasks. This table also reports the control frequency, episode length and number of steps used during human teleoperation for each task.
38

---

Published in Transactions on Machine Learning Research (12/2023)
Figure 13: Example view of the wrist camera. This image comes from one the Basler dart cameras attached to the wrist of the Franka Panda arms, with resolution 296 222.
C.1 Robot arms
C.1.1 Sawyer
The Rethink Sawyer arm is fitted with a Robotiq 2F-85 gripper. In simulation, the environment exposes a 7-DoF action space: 6-DoF end-effector Cartesian velocity control and a 1-DoF velocity command to control the aperture of the parallel gripper. The 6-DoF Cartesian control builds on top of 7-DoF joint integration actuators whose dynamics are not aligned with their real-world counterparts.
For the real robots, we follow the RGB Stacking benchmark (Lee et al., 2021). The environment exposes a simplified 5DoF action space consisting of a 4-DoF' end-effector Cartesian velocity control and the same 1- DoF' velocity control for the gripper. The action is reduced by restricting the gripper to be oriented vertically (3D translation and 1D rotation along the vertical axis). This Cartesian end-effector controller builds on top of the Sawyer's pure joint velocity controller. The real Sawyer environment is the only one which does not use the MoMa library.
In both simulation and the real world, besides the basket cameras (described in Appendix C.2), the full set of available observations comes from the robot's proprioception and a Robotiq FT 300 force-torque sensor affixed to the Sawyer's wrist, again matching the RGB Stacking benchmark. See (Lee et al., 2021) for more details on the setup.
C.1.2 Panda
Like for the Sawyer, the Panda Franka Emika arm is fitted with a Robotiq 2F-85 gripper. In both simulation and the real world, the environment exposes 7-DoF action space: 6-DoF Cartesian velocity control that is integrated into reference pose that the end-effector tracks, and 1-DoF velocity command to control the aperture of the parallel gripper. In simulation the 6-DoF integrated velocity Cartesian controller builds top of 7-DoF integrated joint velocity controller. Whereas in the real world we use joint torque controller (hat tracks the Largel velocity. In addition, 2 Basler dart cameras are mounted the gripper providing the view shown in Figure 13. The cameras mounted on the gripper have not been used to train or evaluate RoboCat.
In both simulation and the real world, the full set of proprioception observations includes:
39

---

Published in Transactions on Machine Learning Research (12/2023)
Joint positions, velocities and torques of the arm.
Pose and velocity of the tool center point located between the 2 fingers of the gripper.
Pose of the integrated reference pose that the TCP tracks.
Position and velocity of the fingers of the gripper.
See Table 11 for the subset of proprioception observations used for training RoboCat.
C.1.3 KUKA
We use the KUKA LBR IIWA14 in combination with 3 fingered robotic hand. We only use this environment in the real world, not in simulation. The environment exposes a 14-DoF action space: 6-DoF Cartesian velocity control that is integrated into reference pose that the end-effector tracks, and 8-DoF abstraction command to control the fingers of the hand. The 6-DoF integrated velocity Cartesian controller builds on top of a 7-DoF joint torque controller that tracks target joint velocity. Similarly to the panda, 2 cameras are mounted on the gripper and, also in this case, they not used for training evaluating RoboCat.
The full set of proprioception observations inchude:
Joint positions, velocities and torques of the arm.
Estimated force and torque at the wrist of the robot used computing the joint torque sensors of the
robot.
Pose of the integrated reference pose that the TCP tracks.
Proprioception of the 3 fingered hand.
Note that we only use subset of proprioception observations for training RoboCat which we describe in Table 11.
C.2 Basket and setup
All embodiments, in both simulation and reality, are mounted in standardised cage in front of standardised basket which defines the robot arm's workspace (Lee et al., 2021). The basket's base is 25 by 25 and it has Basler cameras fixed at its front corners. In the Sawyer environment, these cameras provide 128 128 image observations. In the Panda and KUKA environments, they provide 296 x 222 images.
C.3 Environment resets
In our real-world environments, for most object sets, do not reset the objects between episodes. Rather, rely the next episode's task to achieve the "reset." In between episodes, only verify that the physical system still functions and reset the arm into new random pose above the basket. See Appendix F.2 for more details.
However, for RGB objects, we reuse the scripted resets from the RGB Stacking benchmark (Lee et al., 2021). The reset. uses blob-based 3D position tracker for the RGB objects to locate objects and then uses simple scripted pick-and-place controller to move the objects to new locations in between episodes.
In simulation, the environment simply places the objects in new randomised poses at the start of each episode automatically.
40

---

Published in Transactions on Machine Learning Research (12/2023)
D VQ-GAN Training Details
D.1 Datasets
RoboCat-lim VQ-GAN We train the RoboCat-lim VQ-GAN on several sources of data: ImageNet (Deng et al., 2009), images from the DeepMind Control Suite (Tassa et al., 2018), images from Meta- World (Yu et al., 2020), domain randomised sim sawyer red-on-blue-stacking data from Lee et al. (2021), and images from the tasks used to train the RoboCat-lim agent: the sim panda data and the real sawyer red-on-blue stacking data. Images and reconstructions from these datasets are shown in Figure 14. Reconstructions of tasks not inchuded in the VQ-GAN training are shown in Figure 15.
RoboCat VQ-GAN The RoboCat VQ-GAN was additionally trained on data from a simulated version of the YCB fruit lifting lask, the NIST-i gear task images (real and sim), and real sawyer data of a red- on-blue agent being run with random objects in the bin, including YCB fruits and the vegetables. We didn't include the YCB fruit litting/inser//remonn and vegetable lifting data from human teleoperators in the training, Images and reconstructions from these datasets are shown in Figure 16. Reconstructions of tasks not included in the VQ-GAN training are shown in Figure 17.
D.2 Model architecture and loss
The model architecture is derived from Esser et al. (2021), which combines convolutional layers with Trans- former attention layers to encode the image. The encoder is ResNet with 2 groups of 2 blocks, followed by 3 attention layers. The decoder has 3 attention layers followed by ResNet with 2 groups of 3 blocks. The vector quantiser (van den Oord et al., 2017) uses 64 embeddings of dimension 128. The loss is weighted as (0.25 discretisation_ loss + re_roosstructoon loss + 0.1 * log Japlace_loss) We train with batch size of 64 for 1 million training steps.
D.3 Ablations for our VQ-GAN design choices
In this subsection we discuss the ablation experiments that informed certain design choices that made regarding the VQ-GAN encoder and the observation prediction loss that we used. For these comparisons, we trained smaller models of 62-million parameter subsets of the 30 Sim Panda Stacking tasks with RGB Objects. For these experiments, only trained the models with hindsight goals. Also, during evaluation always reset our simulation the first state of the trajectory that corresponds the goal image, which is much easier task compared to the tasks described in the rest of this paper. We considered 3 generalisation problems within this context: red_on ond_o____ blue where red-on-blue variations are held out from each triplet; red_on_green where the 5 red-on-green variations are the ones held out; and triplet_1 where the 6 stacking task variations of the RGB Triplet 1 are held out across all variations. In each case, we trained and ablated separate generalists the remainder tasks.
As shown in Figure 19, in the context of these generalisation problems, RoboCat model with the VQ-GAN tokeniser performs much better than the patch ResNet tokeniser, especially the held-out test tasks, but requires both training on a diverse dataset that includes ImageNet, and the observation token prediction auxiliary loss.
41

---

Published in Transactions on Machine Learning Research (12/2023)
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Figure 14: Reconstructions from our RoboCat-lim VQ-GAN on the training datasets. From right to left, panda sim, sawyer real red-on-blue, blue, sawyer sim (with visual domain randomisation), MetaWorld, DM control, and ImageNet.
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Figure 15: Reconstructions of tasks not included in the RoboCat-lim VQ-GAN training: YCB fruit lifting and vegetable lifting. Although the reconstructions are inaccurate, they contain enough information for the agent to learn the task.
42

---

Published in Transactions on Machine Learning Research (12/2023)
Original
Original
Reconstruction
Reconstructio
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Original
Original
Reconstruction
Reconstruction
Figure 16: Reconstructions from our RoboCat VQ-GAN on the training datasets. From right left, panda sim, sawyer real red-on-blue, sawyer sim (with visual domain randomisation), Meta-World, World, DM control, and ImageNet.
43

---

Published in Transactions on Machine Learning Research (12/2023)
Embodiment
Object set
Observation
Observation dimensions
Joint angles
Sawyer
TCP position
Simulation
7-DoP
objects
Gripper joint angle
Gripper grasp status
Joint angles
TCP position
RGB objects
Pose of the integrated reference pose the TCP tracks
Gripper position
Panda
Gripper graep status
7-DnP
Joint angles
TCP pose
NIST-I gears and base
Gripper position
Gripper grasp status
Joint angles
Sawyer G-DoF
RGB objects, YCB-i objects
Gripper joint angle
Pinch pose
TCP pose
Joint angles
TCP position
YCB, YCB-i objects
of the integrated reference pose the TCP tracks
Gripper joint angle
Panda 7-DoF
Real World
Gripper grasp status
Joint angles
TCP pose
NIST-i gears and base
Gripper joint angle
Gripper grasp status
angles
TCP position
TCP orientation (rotation matrix)
KUKA 14-DoF
NIST-i gears and base
Pose of the integrated reference pose the TCP tracks
Finger joint augles
12 (4 per finger) (3per finger)
Fingertip positions
Fingertip orientations (zotation matrix)
27 (9 per finger) per finger)
Fingers slip
Table Proprioception observations used different embodiments object sets while training RoboCat.
Reconstruction Original
Original
Reconstruction
Figure 17: Reconstructions of tasks included in the VQ-GAN training: YCB fruit insert/remore
and the 3-fingere gripper. While the vegetables and YCB fruit were seen in the agent play data, the bowl was not seen in the training data.
44

---

Published in Transactions on Machine Learning Research (12/2023)
BabooCas Ian
RoboCar-ár (susd))
10005
84%
55%
79%
63%
51/6
S2N
29
Nevepted beveer
Proernal dee Pedda
Pannlh 102t peninal laakking
Rordy Hit
High Fific High difi-it suitstullingg
0%
Panda DuD RGB
Hus dacking
Brglenia
R01 swe
7/0u BGB
D.De RGB
Pilu XGB
(bik grown)
staddng
pyram
stadding
atceng
(b) Simulation held out tasks.
(a) Simulation training tasks.
Figure 18: Ablating the VQ-GAN tokeniser vs the patch ResNet used in Gato, The patch ResNet tokenisur performs better on the training lasks, but worse on the held out tasks.
Token (train)
Token (test)
Pixel (train)
Pixel (test)
Ours (train)
Ours (test)
80
76.4
80
63.2
60
rate
60
70
40
20
20
0
0
40
Train tasks
Test tasks
4096
8192
16384
32768
10
VQ-GAN vocab
Shift size (k)
Figure 19: Tokenisation and obs prediction ablations the red_on on_green generalisation problem. Unless otherwise specified, we use vocabulary size 4096 and image prediction with a time step shift of = We compare the success rate (%) for train and test tasks to ablate important design decisions. Left: Predicting future tokens is significantly better than predicting future pizels. Middle: Increasing VQ-GAN vocabulary size deteriorates test performance. Right: Predicting the next token in the current image (k 0) less beneficial than predicting tokens farther in the future. We find predicting = 5 time steps in the future is optimal.
At the larger 1.18B parameter scale, we compare RoboCat-lim to the patch ResNet lokeniser used in Gato. Both models are trained on the same data. We evaluate training task performance, well zero-shot performance for various groups of held-out tasks. These groups include held-out perceptual variations (blue- on-green stacking), objects (triplet 5 Sawyer stacking), task family (Sawyer pyramid and tower building). and high difficulty tasks (tasks within the training families that did not have sufficiently good expert for inclusion in RoboCat training). As shown in Figure 18, the patch ResNet tokeniser performs better most training task families, but generally worse for the different types of held out tasks. This mirrors findings with smaller models, which showed VQ-GAN to be better for generalisation and adaptation.
Predicting future image pixels does not give us similar advantage. We compared different reconstruction targets in Figure 19. Like the decoder in MAE (He et al., 2022), added extra linear layer whose number of output channels equals the number of pixel values in patch. Output tokens corresponding to observation tokens were processed via this linear layer to produce vector of pixel values in patch. We used squared error ween the predicted (k: 5) and original frame in pixel space. The choice of prediction
45

---

Published in Transactions on Machine Learning Research (12/2023)
target does not seem to affect the performance training tasks. However, there is a huge difference in the generalisation capabilities of the two methods. Qualitatively found that the policy learnt by predicting pixels fails (zero success rate) as it seems to ignore the goal and performs different task.
Next, we investigated predicting future VQ-GAN tokens for the red on_ereen generalisation task, with different & values, where k represents the number of time steps ahead which predicting the next observation token. We sweep from k = 0, i.e. predicting the next token, to 10 i.c. predicting the next token of the observation 10 time steps ahead. As k increases, the performance on the training tasks increases, but find diminishing returns as increase past 5. However, generalisation performance we increase k from 0 to Further increase in prediction horizon results in performance degradation. These results highlight. the importance of designing a challenging self-supervision lask. There huge amount of spatial and temporal redundancy between consecutive image observations especially when the agent is operating at 20 Hz. Despite using VQ-GAN tokens, there can be significant overlap of lokens among future consecutive frames. Predicting & time steps in the future alleviates this issue and results in strong generalisation capabilities in this selting.
50% relative improvement in
Finally, we study the effect of the VQ-GAN vocabulary size in Figure 19. Increasing the vocabulary size (V) has effect the performance of the training tasks. However, the generalisation ability monotonically declines as we increase V.
E RoboCat Training Details
E.1 Data processing
The training data for each task contains
a subset of embodiment proprioception observations (see Table 11);
the embodiment actions (see Appendix C.1);
and two camera images, in particular from a front left and front right view of the basket (see Appendix C.2).
Each goal image used in training comes from the front left camera and correspond to the image obtained at the end of training trajectory. All images, including the goal images, are cropped and resized to have dimensions 128 128 x 3. We apply image augmentation by randomising brightness, saturation, contrast, and translation to the images, as described in Lee et al. (2021). All actions scaled to be in [-1, We trim trajectories discarding the final steps if no significant difference is measured with respect to the final images of the scene. This is based the intuition that expert actions that do not generate any changes in the images should not be used at training they likely do not have effect on the task at hand (e.g. the robot arm is not moving or is just hovering around the objects). The metric we to evaluate similarity between images is the the LPIPS (Learned Perceptual Image Patch Similarity) (Zhang et al., 2018). The trajectories involving just lifting of objects are also trimmed in order to have the gripper (and thus the lifted object) visible in the final image used goal image.
46

---

Published in Transactions on Machine Learning Research (12/2023)
E.2 Data weighting
Based on previous experiments we use the following formula to sample the data during training. For training the RoboCat-lim generalist, by default every task variant is sampled equally, regardless of how many episodes are available. And within each task variant, failed episodes and successful episodes are also sampled equally, For the successful episodes, the hindsight goal image is used half the time and a semantically equivalent goal image (from different successful episode of the same task variant) is used the other half of the time. When fine-tuning to limited number of demonstrations, find best performance when only training successful episodes. For the final RoboCat generalist we included NIST-i data only from successful episodes and with weighting three times bigger to mitigate potential imbalance towards stacking and lifting other object sets. Similarly, after 000 000 training steps we decrcased by half the weight of all simulated data to focus the remaining training time towards the real world tasks. Note that these decisions were preventive and thus keeping all weights equal may have also been effective.
E.3 Training and fine-tuning parameters
For training all RoboCat models we use the AdamW optimiser (Loshchilov and Hutter, 2017) with a linear warm-up and cosine schedule decay. The linear warm-up lasts for 15 000 steps, starting from and ending at a different minimum and maximum learning rates depending on the model (see Table 13). This learning rate is then cosine decayed by a factor 10 over 2000 000 steps. The AdamW optimiser has paramcters 81 = 0.9, 82 = 0.95 and c= 1 We use a batch size of 256 and a sequence length of 1024 tokens for all models. We train with an AdamW weight decay parameter of 0.1. Additionally, we use stochastic depth (Huang et al., 2016) during pretraining, where each of the transformer sub-layers (i.e. each Multi-Hoad Attention and Dense Foedforward layer) is skipped with a probability of 0.1.
Hyper parameter
RoboCat
RoboCat-lim
364M le
Max learning rate
le
Se
Min learning rate
1e-5
5e
le
Table 13: Learning rates used when training our models. Note that both RoboCat-lim and RoboCat-lim (patch) used the same learning rates.
For fine-tuning we use the Adam optimiser (Kingma and Ba, 2015) with a constant learning rate of 1e-5. The Adam optimiser has parameters 31 = 0.9, 32 = 0.95 and e = 1e-8 We use a batch size of 32 and a sequence length of 1024 tokens for all models. We train for up to 50 gradient steps. As regularisation, we use dropout (Srivastava et al., 2014) with a rate of 0.1.
E.4 RoboCat-lim training data
In subsection 5.2, evaluate the properties of the RoboCat-lim agent and systematically measure its ability to generalise and fine-tune across diverse set of axes: objects, task variants, embodiments, behaviour sources, task family, and from sim to real. In this section, we provide more details on our choice of held-out tasks.
The training tasks for RoboCat-lim are primarily in sim, comprising the simulated structure building tasks: stacking, pyramid building, and tower building for the Panda 7DoF; and stacking for the Sawyer 7DoF. The only real-world data used in training is the Sawyer 5DoF RGB-stacking benchmark.
47

---

Published in Transactions on Machine Learning Research (12/2023)
From these training tasks, also hold out all blue-on-green stacking tasks and Sawyer stacking with RGB triplet 5. This provides the first two axes of generalisation: to the unseen perceptual variation (bluc-on- green task, which has not been observed before) and the held-out objects (triple which has never been seen with the Sawyer embodiment).
To measure the ability to fine-tune to different behaviour sources two versions of data for blue-on- green stacking in the real-world: from expert agent, and from human teleoperation. These have different state and action distributions, (for example) human teleoperators tend to favour quick but constant movements with panses between 'stages' of task.
We measure sim-to-real capability with two tasks the real panda 7DoF: one from the stacking task family and one from tower building.
We also evaluate the ability to fine-tune to unseen task family: inverted pyramid building. This task is particularly challenging it requires careful and precise motion to balance two objects top of third.
Finally, evaluate embodiment generalisation and fine-tuning using the KUKA 14DoF, While the KUKA 14DoF is dexterous embodiment with 3-fingered hand, it also presents considerable challenge for RoboCat training. This is due to the significantly different observation and action spaces that it poses compared to all other tasks in the RoboCat training. For instance, all training tasks rely 1DoF parallel gripper and have total of either 5DoF or 7DoF.
Embodiment
Task Family
Episode Length
Control Frequency
Number of Steps
Stacking
Simulation
Tower Building
20
20 Hx
200
Panda 7-DoF
Pyramid Building
Lifting
120
1200
Inserting
120
1200
Sawyer 7-DoF
Stacking
20
20 Hz
Stacking
Tower Building
60s
600
Pruit Lifting
Inserting
30
Pands. 7-DoF
Fruit Removing
Hz
Real World
Gear Lifting
Gear Inserting
Gear Removing
120 s
1200
Shape matchiing
30s
Stacking
408
101x
400
Sawyer 5-DoF
Vegetable litting
20
Tower Building
60s
20 Hz
1200
Inverted Pyranii
120s
2400
Gear lifting
KUKA 14-DoF
Peg lifting
60s
10 Hz
600
Table 14: Evaluation setup. Episode length, control frequency and episode number steps per task on simulated
and real robots.
F Evaluation and Real-World Deployment
In this section, we provide more details about the evaluation procedure used in the paper.
We evaluated RoboCat and the baselines with 100 (or more, if specified) episodes for each evaluation task, both in simulation and the real robot. For the fine-tuning experiments, we shorter evaluations of 25 episodes for different fine-tuned checkpoints. We then selected the best checkpoint and the 100 episode evaluation with it.
48

---

Published in Transactions on Machine Learning Research (12/2023)
In simulation, the initial state is randomised by moving the arm to randomised initial pose and by dropping the objects in the basket. In the real world, the arm position is randomised similarly. The objects initial positions are either randomised via scripted reset with alternative approaches. We detailed these processes in Appendix F.2.
For all our evaluations, we environments with the control frequencies and episode lengths that match the ones used for collecting the data for each task. Table 14 reports this information for our simulated and real robot tasks.
In the next two sections, detail the evaluation procedure used the real-world tasks for the RGB tasks and all other tasks.
F.1 Evaluation for real-world RGB tasks
To evaluate agent tasks with the RGB objects used the infrastructure implemented in Lee et al. (2021). We rely the scripted resets mentioned in Appendix C.3 and follow the same evaluation procedure, apart from the reward function. We noticed that the reward function This is because the reward function defined in Lee et al. (2021) requires the arm height to end withing some thresholds. While such criteria for success can be fair for RL agents trained to optimise such a reward, wo thought it didn't represent fairly the ability of RoboCat to stack the correct objects and move the arm away to any position. For this reason, we decided to visually count the successes during evaluation. For fairnoss we also visually recounted the evaluations of Lee et al. (2021) and Reed al. (2022).
missing some clear successes.
F.2 Evaluation for real-world tasks with no rewards or scripted resets
For all the other real lasks in this work did not implement scripted reward model nor a reset policy. This greatly simplified the implementation of the lasks, but added signifieant complexity evaluation. Our evaluation system thus consists of two components. In place of scripted rewards, train success detectors from human annotations to detect when each task has been completed. For resets have general solution that groups several policies for different tasks together into policy pool, with each policy in the pool serving as a component of the reset policy for its peers. For instance a policy pool containing the insertion and removal policies for the same object will allow the removal policy to serve as reset for the insertion policy and vice versa.
F.2.1
Success detection
We treat success detection for each task as per-time step binary classification problem. For each task we annotate several episodes with per-time step success labels using munotation-efficiene annotation procedure. Our annotation procedure takes advantage of the following simple observation: If task is solved (or not) at time t it is overwhelmingly likely that it is also solved (or not) time + 1. Put more simply, there are relatively few time steps where a task transitions from unsolved to solved (or vice versa). Rather than collecting labels for every frame directly, we ask annotators mark transition points between solved and unsolved states and extend these to per-time step labels by painting the transition label forward in time until the next. annotation. This strategy dramatically reduces the number of annotations compared to
49

---

Published in Transactions on Machine Learning Research (12/2023)
annotating every frame. Using this strategy labelling 1200 time step episode typically requires fewer than five annotations (and frequently requires only one).
Using data annotated in this way train success detectors for each task. Each success detector is a ResNet- 18 backbone with binary classification head. To handle occlusions we use the three basket cameras in the reward model in order to observe the scene from multiple perspectives. Each camera image is scaled to 132 X 132 and further cropped to 128 x 128 before being processed separately by the ResNet backbone. The resulting embeddings from each camera averaged before being fed to the binary classification head. At train time we random cropping and image augmentation, while at deployment time disable the augmentations and take central crop of each image.
Once trained the success detectors used together with policy pools for resetting environments as we describe in Appendix F.2.2. We did not use success detectors for reporting RoboCat performance, they have an accuracy around 90%. To ensure we counted all success from RoboCat visually assessed when experiment has succeeded or failed. In future work, we aim at improving accuracy of the reward models in order to automatise also this step of the evaluation.
F.2.2 Policy pools as generalised reset policies
Policy pools general solution to resets in the real world. Their role in set up is three-fold: i) policy pool provides resets for non-self-reeetting tasks, ii) generates diverse initial conditions for evaluation, and iii) makes efficient use of robot time by interleaving the evaluation of multiple policies at time.
A policy pool groups together a set of diverse tasks and allows to schedule episodes of different tasks to be in sequence. In this simplest case, can replicate the standard evaluation setting by building pool comprised of forward task and its reset, allowing the schodule to alternate between them. However, the policy pool machinery allows us to extend beyond this simple setting in several ways.
Each prop set typically affords several tasks, for example the NIST-i gears set affords nine distinct tasks: lift, insert, and remove for cach of the three gears. A policy pool allows to group all nine of these tasks together and interleave their exccution in arbitrary orders. This is desirable for two reasons which discuss next.
From global perspective this procedure makes efficient of robot time, because every episode is evaluation of some policy some task. There is no time wasted by running a specialised behaviour that resets the environment; instead the evaluation episodes for each task simultancously serves the reset behaviour for other tasks in the pool.
From local perspective this procedure generates wider variety of initial conditions than a single fixed reset policy. From the perspective of each task its reset is provided by composition of all the other tasks in the pool. This has the nice effect that larger pools (with more diverse tasks) tend to generate more diverse initial conditions within each task.
Another extension of the basic setting is that we control the ordering of the schedule of tasks within a policy pool. In practice, this is how we handle non-self-resetting tasks. In the general eligible to be run from all reachable states of the environment. Returning to the gears example, an insert task is eligible to be run only if the corresponding gear is not already inserted on the target shaft, (similarly
not all tasks are
50

---

Published in Transactions on Machine Learning Research (12/2023)
remove task is only eligible when the corresponding gear is inserted). At the beginning of each episode scheduling system for the policy pool uses the success detectors (Appendix F.2.1) to determine which tasks are eligible to be run given the state of the environment, and then randomly chooses one of these tasks to actually run.
The mapping between tasks and policies within policy pool is arbitrary, and need not be 1-1. The same machinery can be used to implement single task evaluation (by providing different policy for each task in the pool) or multi-task evaluation (by defining pool where every task is executed by the same policy). Given RoboCat is generalist, when using policy pool we only use the policy from RoboCat to evaluate any task.
G Additional Experiments
G.1 Generalist capabilities and expert performance
In this section, we report more details on the performance of RoboCat on training and fine-tuning tasks, and we compare it to the success rate of the training data, defined as percentage of the successful episodes among all the training episodes. Table 15 shows the overall RoboCat performance per each task variant.
G.1.1 Training tasks
We use RL-trained experts for all of the structure-building tasks in sim (Panda 7-DoF stacking, pyramid, and tower building; Sawyer 7-DoF stacking, Figure 20) and in real (Sawyer 5-DoF stacking, Figure 21). These experts are state-based for the sim tasks, and vasion-based for the real-world tasks. In the sim case, the vision-based RoboCat generalist has performance reduction of 10% less compared to the state-based single-task experts (Figure 20), for all families except lower building. It is important to note that these experts were trained via RL with privileged state information which included the poses of the target objects. In contrast, our vision-based generalist agent discards any privileged state information in order to perform effectively in the real-world. As such, we would not expect RoboCat to improve upon even match the success rate of the data for all tasks. On the other hand, when RoboCat has access to same information the experts that generated the data, for the real tasks Figure 21, it is able overcome the performance of the agents generating data.
Human-teleoperated demonstrations and Figure 23 compare the success rate of the human teleoperators and RoboCat separately for each task variant. In both cases,
used for the NIST-i tasks in both sim and real. Figure 22
that RoboCat average matches the human teleoperator performance.
Self-generated data were used together with limited number of human-teleoperrter demonstrations to train RoboCat additional real robot tasks. Given limited number of demonstrations for new task, first fine-tuned previous versions of RoboCat to solve this new task. We then deployed the fine-tuned RoboCat in the real world to collect more self- aef-aeeeeateed data for the specific task. The newly generated data together with the successful human teleoperations were used in the final generalist. Figure 24 and Figure 25 aim to contextualise the performance of RoboCat these tasks. The 'generated + demos' numbers refer to the percentage of successful episodes among the total number of episodes used to train RoboCat these tasks. Successful episodes include successful self-generated episodes and successful teleoperations. We use
51

---

Published in Transactions on Machine Learning Research (12/2023)
Embodiment
Family
Object
Variations
GR
objects, triplet
$10.6
objects, triplet
7-DloF
RGB
RGB
91%
80%
RGB objects,
NIS
80%
80%
62%
79%
82%
RB
RG
GR
GB
BR
BG
RGB objects, triplet
00%
57%
11%
801%
Y4%
85%
RGB objects, triplet 2
77%
54%
62%
76%
77%
77%
Susasiation
Stacking
RGB objects, triplet
RGn objects, triplet 3
88%
77%
90%
73%
933
80%
91%
55%
53%
633
50%
81%
RCB ubjocta, triplet
02%
77%
583
50%
50%
80%
N15 geazs
67%
89%
86%0
69%
BGR
objects, triplet
67%
objects, triplet triplet
Training
7-DoF
RGB objecta,
RBC
RCB
CRB
CBR
BCR
RGB objects, triplet RGB objects, triplet
02%
755
Pyramid building
RGB objects, triplet
88%
63%
81:96
50%
61.66
0236
6s%
444%
735
48%
65%
68%
RCB objecta, triplet
01%
56%
87%
6.1%
50%
78%
RGB objects, triplet
71%
18%
77%
6.5
755
77%
Small
Medium
Large
Lilting
NIST-i geors
5676
61%
727
Insertion Mooeroooppeg
N15 goors
70%
70%
100
objecta, triplet
objocta, triplet
objects,
objects,
Beal World
Cucumber
Lifting
YCBd vapetables
50%
50%
07%
building
'Towet
RGB objects, triplet
23%
luver teil prtainid building
RGB objects, custom triplet
17%
Apple 40%
Banana 10%
Lemon
Peach
Lifting
YCB frilts
00%
Panda 7-Dol
Small
10
Larggn
Lifting Iusertion-pok
geaze
8776
911%
93370
geurs
Removalspog
geurs
Flaee tning
Real World
Ineersion-bow
YCB fruisa
YOB- howl
94% 82%
7106 50%
9a%
Panda 7-DoF
Removal-bowl
YCB fruite and
YCB-i bowl
90%
62%
69%
76%
64%
77%
Cliolee
Pentagon
Square
Insertion-baze
Shape-matching objects Shape-anttching objects
199%
8%% 7 10%
1%
T17
Removal-bue
47%
72%
87% 94%
76%
70%
KUKA
Lifting
NISTSI geane
56% / 86%
14-DoF
Table 15: Per-task RoboCat performance on training and fine-tuning tasks. This table expands Table by providing the success rate for each lask variant instead of the average task families. Fine-Luning results are reported for 500 and 1000 demonnstrations respectively, separated by slash.
52

---

Published in Transactions on Machine Learning Research (12/2023)
RL Experts
RoboCat
100%
90%
82%
80%
75%
65%
60%
50%
0%
Sawyer Stacking
Panda Stacking
Panda
Panda
Pyramid
Tower
Figure 20: RoboCat experts performance: Panda 7-DoF structure-bulddngg training tasks (sim). RoboCat performance compared to the success rate of the training data for each task family.
failed self-generated episodes, but not failed demonstrations. On average RoboCat performance matches the success rate of the training data, overcoming it in some cases (see lifting banana, peach, strawberry Figure 24 and potato Figure 25).
G.1.2 Fine-tuning performance
Figure 28 compares the success rate of the human teleoperators to contextualise the fine-tuning results. In general, see how fine-tuning more demonstrations in general improves performance, sometimes reaching similar performance the human teleoperators. This does not apply to the insert tasks, which result very challenging for our agents due to the required reorientation and shape understanding.
G.2 Impact of model size
To evaluate the effect model size, we compare the --BB-mareetter RoboCat-lim model from Section 5.2 with a smaller 364M model on the simulated structure-building tasks. The results in Figure 26 demonstrate that the larger model yields an improvement 6-21% in the success rate for the different task families. particular, while the smaller model only slightly degrades performance for the easier stacking tasks, is much poorer for the more challenging pyramid and tower-building tasks.
G.3 Visual foundation baselines
We consider four different network architectures baseline methods for manipulation tasks: ResNet (He al., 2016), normaliser-Fe free networks (NFNets) (Brock et al,, 2021), vision transformers (ViTs) (Dosovitskiy al., 2020), and the ViT variant Swin transformers (Liu et al., 2021). Within each category, used the following models: ResNet-50, ResNet-101, ResNet-200, NFNet-fO, NFNet-f3, NFNet-f6, ViT-b, VIT-I, Swin- b, Swin-1, and Swin-s. Fach model was pretrained selection of datasets: ImageNet 1 (Deng et al., 2009), ImageNet 21k (Ridnik al., 2021), Microsoft COCO (Lin et al., 201 4), and the JFT Dataset (Riquelme al., 2021). We also evaluated the use of CLIP (Radford al., 2021), masked auto-encoder (MAF) (He et al.,
53

---

Published in Transactions on Machine Learning Research (12/2023)
RL Experts
RoboCat
100%
93%
87%
76%
70%
71% 68%
75%
60%
52%
25%
0%
RGB objects Triplet
RGB objects Triplet 2
RGB objects Triplet 3
RGB objects Triplet
RGB objects Triplet 5
Figure 21: RoboCat experts performance: Sawyer 5-DoF RGB stacking tasks (real). This plot compares the performance of RoboCat on the real stacking tasks with respect to the overall success rate of the training data available for each task variant.
Human teleoperator
RoboCat
100%
94%
94%944%
93% 95%
96%
97%
98%
85%
86%
77%78%
80%
73%
75%
65%
50%
25%
Small gear lifting
Medium gear lifting
Large gear Small gear lifting
Medium gear inscrtion
Large gear Small gear insertion
Medium gear removal
Large gear removal
insertion
removal
Figure 22: RoboCat vs teleoperation performance: Panda 7-DoF NIST-i tasks (sim). RoboCat compared with the success rate of the data collected by human teleoperators, for each task variant.
2022), SimCLR (Chen et al., 2020b), BYOL (Grill et al., 2020), DetCon (Hénaff et al., 2021), Odin (Hénaff al., 2022), and DINO (Caron et al., 2021).
One common way to pretrained visual models for control tasks is to add learned policy head (Nair al., 2022; Parisi et al., 2022). The image input is first embedded by the foundation model. The output embedding then concatenated with any proprioception information and fed into the policy head. The policy head consists of multi-layer perceptrons (MLPs), each with 256 parameters. The output of the top MLP is then used by linear policy head to generate the robot action. While this means that the
54

---

Published in Transactions on Machine Learning Research (12/2023)
Human teleoperator
RoboCat
100%
87%
86%
88%
88%
84%
%
79%
79%
72%
75%
64%
56%
50%
25%
Small gear lifting
Medium gear lifting
Large gear lifting
Small gear insertion
Medium gear insertion
Large gear insertion
Figure 23: RoboCat vs teleoperation performance: Panda 7-DoF NIST-i tasks (real). RoboCat compared with the success rate of the data collected by human teleoperators, for each task variant.
Generated data demonstrations
RoboCal
80%
62%
60%
60%
48%
40%
40%
90%
0%
Fruit lifting
Fruit lifting Banana
Fruit lifting Peach
Fruit lifting Strawberry
Apple
Figure 24: RoboCat training data: Panda 7-DoF YCB lifting tasks (self-improvement, real). This plot compares the performance of RoboCat on the self-improvement tasks with respect to the overall success rate of the training data available for these tasks (self-generated data and successful human teleoperations).
VFM baselines only use the current timestep (versus approximately timesteps for the transformer- based RoboCat agent), prior experiments with stacking tasks found that incorporating context, via either LSTMs observation stacking of 4 previous timesteps, did not improve performance. Finally, behaviour cloning with mean squared loss the optimisation objective.
We fine-tuned total of 59 combinations of the above methods and evaluated subset of RoboCat tasks. Then, we selected the top representatives of each of the network architecture categories and evaluated them more tasks. For the sake of clarity, only report the top two methods in the main paper. We the average final policy success rate the performance metric in experiments. Each experiment ran for 100 episodes, with each episode lasting 400 time steps.
55

---

Published in Transactions on Machine Learning Research (12/2023)
Generated demonstrations
RoboCat
67%
60%
54%
49%
45%
42%
40%
36%
23%
18%
17%
20%
056
stacking Blue-green
invected pyramid Blue-red-green
Vegetable lifting Carrol
Vegelable lifling Cucumber
Vegelable lifting Pepper
Vegelable Potato
Red-blue-green
Figure 25: RoboCat vs training data: This plot compares the performance of RoboCat the self improvement tasks with respect to the overall success rate of the training data available for these tasks (self-generated data and successful human teleoperations).
Sawyer 5-DoF RGB and YCB tasks (self-improvement, real).
RoboCat-lin (1.18B)
RoboCat-lim (364M)
80%
72%
71%
61%
60%
39%
40%
20%
0%
Sawyer 7- DoF RGB stacking
Panda 7- DoF RGB tower
7-
DoF RGB stacking
DoF RGB pyramid
Figure 26: Impact model size. This plot compares the duccess rate of full-sized 118-mmearee model with that of smaller 364M model, on the RoboCat- lim simulated structure- cencttrr-bbilding tasks.
(a) Simulated environment.
(b) Fixed environment.
(c) Inverted environment.
(d) Moving base NIST-i environment.
Figure 27: The different types of NIST- based environments we ablate performance against. Note, the main paper report performance against enviromments from (a) and (d) only.
We fine-tuned the models with different data limitations: 500 demonstrations, 1000 demonstrations, and more than 100000 demonstrations. Training took 200 000 training steps for all models. We validated the models in simulation during training and selected the best model snapshot based the validation success rate for
56

---

Published in Transactions on Machine Learning Research (12/2023)
Human teleoperators
RoboCat fine-tuned (500)
RoboCat fine-tuned (1000)
100%
882%
86%
86%
92%
84%
90%
90%
87%
764
76%
77%
60:52%
00/
Insert into bowl Lemon
Insert intoo howl Pench
Insert bowl Strawberry
Remove from howl Lemon
Remove from bowl Peach
Remove from howl Strawberry
(a) Panda 7-DoF insert and remove tasks with YCB and YCB-i objects (real).
Human teleoperators
RoboCat finc-tuned (500)
RoboCat fine-tuned (1000)
98%
98%
100%
90%
91%
92%
88878
83%
84%
75%
75%
47%
50%
23%
.1%%
11%
10%
7%
Insert Circle
Insert Square
Insert Pentagon
Remove Circle
Remove Square
Remove Pongaton
(b) Panda 7-DoF insert and remove tasks with shape matching objects (real).
telesperators RoboCat fine-tuned (500)
tuned (1000)
86%
75%
56%
50%
95%
Large lifhing
(c) KUKA 14-DoF gear lifting task (real).
Figure 28: Final performance each fine-tuning task.
57

---

Published in Transactions on Machine Learning Research (12/2023)
medel
obj
daluset
pretroin
Supervtzed
ImageNet-1le
Imagenett leattarrtend
JFT-4B
afnes-t0
CLIP
Image Net-eNet-11
afneesto
CLIP
Image Net-11
ro-elip-align
ufact-f0
CLIP
Net-eNett1k
Ed-elip- -dittactt
ufact-10
CLIP
MeiggeNe--k
EO-clip align-sstcck
ufact-[]
Supervised
Image Net-1k
im gouet beouper iaed
afnee-fi
Suparvised
JPT-455
Iff sh th---app wew.oehhp-eeeeoo pochh--epppe supprviiid
ifnec-f2
Buparvisd
ImageNet-1kk
maagenet le-asparvized
afbetst3
Supervieed
JPT44B
fit db Sepocha-so per wervlsed Fina gervet kesupee pervleedd Ift thi th--spochh--nn sappoh--erperriine imageuet keoupets iaed
afhet-13
Supervleed
Image ge Net-1kk
afhec-f3phus
Suparvized
JPT-41
afaci-14
Supervised
Image
ufact-15
Supervised
Image
imageuet heoupet laed.
Eh-elop-altee
CLIP
Net-1e
CLIP
Net-s
afnets66
CLIP
Image Net-1 le
f6-ellp-alll
afnetsf8
Supervized
ImageNet-1 Net-1
looupereteed
ufact-17
Suparvised
JFT-4B
jit4b. Til4b--epoolh depoclu-supen laed
afaci-UTplas
Suparvised
JFT-4B
ji depoclis-uuper iaed
ResNet-101
Supervised
IntgeNet-IX
Lma genet keoupervinee
RexNat-101
DetCon
IntgeNet-1l
imagenet k--dtcon
ResNat-111
BYOL
ImtgeNet-11
imagenett le- k-by
Re<Nat-200
DetCon
Image
tina penet ke-detooo
Re<Nat-220
BYOL
Image
Imagenet penet ke-hyol
ReeNat-50
ODIN
Image
imagenet k--ddtn
ResNet-50
Supervised
Image
Imagenet Reoupervised
ResNet.50
Image
BYOL
ImageNet-12
b-ollllowze
ResNet-50
ImagsNet-ll
ReaNet-50
BYOL
ImageNet-ll Nei-1k
by-by
zwin-h
Supervized
Image Net-1k
leopprrvted
zwis-e
CLIP
JFT-4B
wwin-e--lbp-nlggee hatppp--lgnn--eeee-e....
awis-h
CLIP
ImageNet-1k
swin-b-clip-atee b--lli--iigg---eeee aitgu--tooeeeeeeee hoghh-e
swin-h
CLIP
Ima ge Net- 1k
swin-h-olip-aie hh-ooppailigne wiign-stocckeooeee
swind
CLIP
bma ge Net-1 le
awin-s
Supervised
Image Nei-eNet-1k
imageuet wtc.uettt-ooppseeeeeee desupetviaed
awia-s
ODIN
Image NeNNcl-i 1-
Imageuel kk-odin
swinst
ODIN
ImageNo
Imageuel hhodin
awin-t
Suparviaed
Image Net-1k
imagenet le-uuprrviend
vit-b
Supervised
JFT-4B
b18-224-jf--- prepttrani
Supervleed
Image
224-1211--portt
Supervteed
224-1221--n
vised
Image ImageNei-ll Nct-1k
b16-384
358-1221leppee
vit-b
MAE
Net-13
21G-224-i1
vit-b
Supervized
Net-1k
b18-224-11 111-222--11k-p-ettt- k-peetttttt
vit-deit-b
MAE
Net-1k
218-224-11k-mmeee m-mat-prtttainn
vit-dcit-]
MAE
Net-1k
110-224112-mme-
o-
MAR
his ge Net-1
210-224-11 k-mae-pretrsin
vit-1
Suparvized
Ima Net-1kk
116-221-11 k-presratm
vit-
Sipervised
ImageNet-ll
116- 1254--22
vit-1
Supervised
Image ImageNet-21k
116-224-121k-p2
/21k-preteain
vit-1
Supervieed
Image Net-11
116-384-11
vit-1
ImsseNet-1k
vit-1
Superviaed
JFT-4B
110-224-jB--rstzutl
Suparvieed
Image
Met-1e
b16-224-11 210-22e-1kkmms--pretmn
Table 16: Details of the baseline methods: including the model name, training objectives and pretaining dataset.
further comprehensive evaluation. We tried both freezing and non-freezing scenarios for the parameters of the pretrained models. Since the non-frozen pretrained model performed much better, we only report the results for non-frozen models.
Heal sfset-d
strdllape's MLAF
Tectning Ofgertion
Seprrstord
FILEP
Sepreviee
CLU
Segreooed
MAN
Dimmo
Task
morre than 101000
pondl T.Duf stakking IC -
0.122
0240
0.30
0.11
11:07
T-Doy
0.71
0131
0330
41
1000
panda TDDal Jny 2yta GRID Jut
0.00
0.00
0100
0.00
0.00
0.00
1.00
pando. T-Dof stooking
GRB
0.00
0.00
6.00
8000
0.00
0.00
pands
Doy. stnking wel RR
000
0330
0.00
110
0.53
1924
1120
500
pondl Dut stonking MC ae
00
14
1109
0.10
0.05
tanger 7-0o7 stacking DG met
0.10
0.24
0.50
6.13
0.20
0.14
0.00
0.05
pandi r-Dop stakking "L
0.63
0.00
0.90
6.00
0.08
0-83
0:46
ponoh_ 7-DoP_tower towerr GRD. jet
0.37
0.40
0.55
0.57
0.42
0.40
0.22
0.31
Table 17: Evaluation of baselines on training and held-out tasks with limited number of demonstrations.
Table 17 shows the success rate of all baseline methods on a held-out task. Then we took the top represen- tative methods of each network architecture and further evaluated them other tasks with limited data:
500, 1000, and more than 10000. We also evaluated these methods on subset of the training tasks.
58

---

Published in Transactions on Machine Learning Research (12/2023)
G.4 NIST-i extended study
We evaluate the complexity and some design decisions made for the NIST-i tasks using RoboCat. We find that fixing the base in the physical world can have interesting implications, more cameras lead to improved performance and using data from largely unrelated tasks can lead to increased performance NIST-i, suggesting indications of positive skill transfer.
G.4.1 Task complexity
So far, we considered lixed base in simulation and a freely moving base in the physical world (see Figure B.1 for details). We argue that moving base leads to both harder task in the physical world and provides more diverse set of tasks for training. We challenge this argument in this section. We do this by fixing the base to the basket in the same orientation as what we do for simulation and measure if this new setting is easier to solve. Figure 27 illustrates the different environments. Note that at train time we only use demonstrations from Figure 27(a) and Figure 27(d).
Base Location
NIST- task
Insert
Remove
Fixed base
64%
100%
Inverted fixed base
40%
99%
Moving base
38%
97%
Table 18: Real-world NIST-i task ablation. Average success over all three NIST-i gear sizes. A 364M agent performs much better the fixed base environment.
Table 18 reports the performance a smaller, 364M agent, trained on the same data as RoboCat-lim. We observe that despite not having fixed base data from the physical world, the agent performs a lot better on the fixed-based task as opposed to the other two tasks. This suggests that the fixed base environment is easier possibly due to its similarity with the simulated environment, suggesting positive skill transfer from simulation cltaa
G.4.2 Data bias
Appendix G.4.1 showed that there may be some positive transfer across similar tasks on smaller 364M parameter agent. However, the performance reported that section could also be due data bias. That is, could be that the collected demonstrations from raters might be primarily centred around the middle of the basket with the peg base orientated similarly to the one the simulated data. This itself would interfere with the hypothesis for positive skill transfer in favour having data bias. To test this, we utilise specialist behaviour cloning agents (Specialist BC) that we separately train on each NIST- task using only the subset of NIST-i training data belongs to that specific task. Behaviour cloning agents are known to overfit to the training data, leading to poor generalisation (Osa et al., 2018). the reported performance in Table 18 is due to data bias, then we expect to see the same type of improved performance of a specialist BC agent when evaluated on the fixed based tasks as opposed to the moving base tasks.
Specialist BC
Base Location
NIST-i Insert Task
Fixed Moving base
13%
Table 19: Real-world NIST-i data blas. Average success over all three NIST-i gear sizes. The agents have similar perform both fixed and moving base environments. No evidence for data bias.
59

---

Published in Transactions on Machine Learning Research (12/2023)
NISTS Insert Thask
364M error
RoboCCt etror
Reduction factor
Rixeed hase
0.36
0.13
2.77
Moving buse
0.02
0.23
2.70
Table 20: Skill transfer analysis. Average accumulated error over all three NIST-i gear sizes. Moving from the 364M model to the full RoboCat agent eliminates just under two thirds of the gear insertion failures in both the fixed and the moving base settings, indicating a smaller performance gap for the RoboCat.
Table 19 shows that having fixed or moving base results in negligible difference in the performance for the specialist BC agents. In fact, the relationship between the performance on fixed and moving base is inverted (all training data is collected while having moving base). This indicates that for both the fixed and moving base tasks there is no particular data bias towards the peg base being situated in the centre and oriented in the same way in simulation.
G.4.3 Skill transfer
So far, showed that there are some indications of skill transfer for the smaller 364M generalist. In this section, study to what extent this holds for final RoboCat agent. Notably, we the same NIST-i insertion data for training both the 364M and RoboCat models. However, final version of RoboCat is trained larger number of tasks and is of bigger capacity (1.2B). In this sense, a successful skill transfer would close the gap between the achieved performance the fixed and moving base environments for the final generalist. This what observe in Table 20. Specifically, moving from the 364M model to the full RoboCat agent eliminates just under two thirds of the gear insertion failures in both the fixed base and the moving base setting, This indicates that indeed training a bigger model on more data that involves different types of bring to pose tasks can be beneficial for solving the moving base insertion tasks. Moreover, the additive performance gap for RoboCat depending on the base state is much smaller than the additive performance gap for the 364M agent suggesting again indications of positive skill transfer.
G.4.4 Additional observations
Finally, the process of finding the best performing specialist BC agent we noticed that including the wrist cameras to the observation the agent siguificantly improved the performance of our specialists. That is, instead of using just two camera observations, as we do for RoboCat, using a total four camera observations two from the basket and two from the wrist the robot, was very beneficial for a specialist BC agent (see Table 21).
NIST-i task
Performance of
Insert
Specialist BC (2 cameras)
13%
24%
Spocialist BC (4 cameras)
36%
54%
Table 21: Dependency on the number of cameras. Average performance over all three NIST-i gear sizes. More cameras have a significant effect on the specialist's 's performance.
An exciting direction for future research is to understand the effect of different camera observations for contact-rich manipulation lasks in the context of RoboCat.
60


Reviews:
[{"review_id": "Qahux4ku0Q", "reviewer": "Reviewer_PTw5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper provides a **thorough empirical evaluation** of the proposed method on different robot arms and tasks\n- Specific questions are addressed, proving that cross-task transfer and self-improvement are possible, among others\n- The method and experiments are described in great detail, both in the paper and in the appendix\n- The writing is clear and easy to follow\n\nWeaknesses\n- Moderate methodological novelty: **the method to a large extent is based on Gato**. The difference is in using a different image tokenizer, VQ-GAN, and in predicting image tokens in addition to action tokens.\n- Low variability across robots/embodiments: just from reading the abstract and the introduction, it may seem that the paper handles all kinds of robots and tasks. But in fact, the experiments are only done using Sawyer, Franka, and Kuka arms, which all have quite similar kinematics. Furthermore, the control action seems to be the 6 DoF end-effector Cartesian velocity (plus gripper command) — thus the impact of the **\"robot embodiment\" seems to be basically irrelevant for the learning**, because the tracking of the end-effector velocity is handled by the internal robot controller.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents RoboCat, a goal-conditioned decision transformer for robotic manipulation that works on visual inputs across different robot arm embodiments and tasks, both in simulation and in the real world. Zero-shot generalization and adaption via learning from demonstration are shown. Furthermore, the model is shown to be capable of generating training data for self-improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- mention the action space already in the main body of the paper\n- if possible, provide an argument or an experiment that proves that multi-embodiment plays a role. I.e., that the improved performance is not just due to having more data. Perhaps one could achieve the same results using a single robot and then just running the learned policy on a different robot.", "broader_impact_concerns": "Broader Impact concerns are addressed in sufficient detail.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3QhwumuMV0", "reviewer": "Reviewer_qZkA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simple and clear approach to the problem\n- Large number of complex, visual control tasks tested\n- First large-scale multi-embodiment agent that I'm aware of, really cool and of definite interest to the community given the trend toward large models that others can build on\n- Low expert data requirements for new tasks is ideal, though synthetic collection on real robots is still far from trivial\n- Clear prose\n\nWeaknesses:\n- Details about baselines unclear/missing comparisons:\n  - In section 4.2 the full model size is given at 1.18B and for some ablations 346M, while in 4.4 the baselines are listed at 438M and 197M parameters. Is the comparison in Figures 5, 6 and 8 with the smaller or larger RoboCat model? Could you guide me to text describing exactly which experiments use which version of RoboCat? It seems Figures 10 and 11 are the smaller and Table 20 compares between the two sizes, so the rest would be the larger, but I am not positive.\n  - The method uses ~3 timesteps of conditioning (Section 2.1.1), while in Appendix G.2 it's stated a \"common way to use... robot action,\" but this claim does not make it clear if only a single image is used for baseline methods or also a history of a few images (and whether this makes a difference).\n  - Baseline methods are built on existing models pretrained from a different data sources that do not include images close to the deployment task, unlike the proposed method. That would lead me to conclude that Figures 5, 6 and 8 are really a statement about data, which leads me to wonder about a setting with exactly the same architecture but different data source (language is a prime candidate, discussed below), as well as a different architecture with the same data source (NFNet with Section 3.4 data).\n  - There seem to be good already existing, and cited, baselines close to the tasks considered like VIMA (https://arxiv.org/pdf/2210.03094.pdf) and GATO, I'm a bit confused as to why these were not chosen. VIMA especially as it is open-source. There is only a single citation with no discussion as to why it wouldn't make sense to compare in Section 6.1.\n  - If my reading is correct that there isn't a directly comparable baseline method with the same data and different architecture, is there some particular justification for the VQGAN + autoregressive transformer? I can see some possible interest in in-context learning, but contexts seem too short to leverage that (unlike a method like VIMA that condenses observations to single tokens).\n- From above, I think this paper missed an opportunity to make a comparison to finetuned language models (using the same VQGAN tokens as the proposed method as input). All baselines are pretrained visual foundation models which may be adept at reasoning over the content in the image, but less adept at reasoning over the history. It seems pretty straightforward, and supported in the literature to use language models as a base for control (https://arxiv.org/abs/2103.05247) given that they see an enormous variety of sequences in their data.\n- The use of\" 0-shot\" and \"k-shot\" when referring to finetuning in Section 5.2 is a bit confusing. I would take this terminology to refer to in-context learning, not the size of the dataset for further training. I can understand why in-context learning wasn't explored here, given that it takes 1024 tokens to model only 3 steps of time, but on a first read I did not parse this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a large transformer model to be used as a foundation for robotic control for a variety of different manipulation tasks on multiple different arm embodiments with different control spaces. Such a foundation model is shown to require only a small number of demonstrations, to learn new tasks through finetuning. Additional synthetic data generated from the model labeled via hindsight relabeling can improve finetuning performance, which the paper calls self-improvement. Results are validated both in simulation and in substantial real-world experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please answer the first few questions in the weaknesses section, and depending on the answers add a bar for smaller-sized RoboCat model sizes for comparisons in Figures 5, 6 and 8. I don't have an issue with the larger model being important to the results, but I just don't understand if it is and that is critical.\n- Add a same data, different architecture baseline and a different data, same architecture baseline, this seems critical to understanding the different components of the method. That or the phrasing of the contribution here will need to be more careful. I see the paper as one of a handful of demonstrations of a foundation model in a control context, and one that can be used with different embodiments. If the paper is largely a statement about the fact that large, diverse data + large sequence models also works in control that is totally fine, yet I am uneasy about the presentation of clear wins over baselines without the delineation of what makes the method work in the first place.\n- Consider adding VIMA as a baseline or discuss why it is not comparable. This seems critical as it is much closer to the application domain than the chosen visual transformers, and it is open-source (https://github.com/vimalabs/VIMA).", "broader_impact_concerns": "This is well addressed in the section on Broader Impact already and I don't have additional concerns. I would like to add that this paper would be of particular significance if the model were made available openly, though I don't see any promise of that in the text.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uzdviIsixD", "reviewer": "Reviewer_68NV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method is sound and straightforward.\n- The experiments demonstrate positive transfer across robots, which has been challenging historically.\n\nWeaknesses\n- The paper claims to demonstrate a foundation agent. This is an overstatement that goes contrary to the definition of a foundation agent used in this paper. A large number of demonstrations (500 - 1000) is required to adapt to a new embodiment and there is no zero-shot transfer to new embodiments. Furthermore, all embodiments demonstrated in the paper are very similar. The proposed system will likely not generalize to new backgrounds or bin textures. Such generalization is important for a foundation agent since it is impractical to perfectly replicate the setup in this paper in another lab. \n- The writing is confusing. Figure 1 demonstrates the fine-tuning process but it's unclear what is the original agent that is being fine-tuned. Section 2.1.1 on \"pretraining\" explains how the encoder is pretrained but not the agent itself. The section 2.1.2 on fine-tuning does not talk about pretraining. A training and fine-tuning split is later mentioned in the experimental section, from which we could guess what pretraining looks like. Most experiments do not contain the cycle in Fig 1, and only in one experiment the cycle is executed once (but not more). Overall, Fig 1 is not helpful for understanding what was done in the paper.\n- The legend in Fig 5 is rather confusing. It looks like pretraining with CLIP objective significantly hurts performance. It would seem to me that what actually is being compared is multi-task vs single task training, which is a much less surprising result. \n- There are relevant papers missing: Dasari'19, Hu'22\n- The broader impact statement is confusing. There is significant discussion of AGI safety risk which is very confusing given that current system has a 14% failure rate on gear lifting in a lab environment. Perhaps it would be useful to put this discussion in context.\n\nDasari'19, RoboNet: Large-Scale Multi-Robot Learning\n\nHu'22, Know Thyself: Transferable Visual Control Policies Through Robot-Awareness", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a system for training several robots with different embodiments and action spaces that autonomously improves through interaction with the environment and benefits from shared experience across robots. Training on different embodiments is handled by tokenizing them to embed them into the same observation and action space. Autonomous improvement is done via goal-conditioned supervised learning (Ghosh'19). There are experiments on three different robots and 2 different grippers, all in the same lab setup. The experiments demonstrate training on data from different robots together is better than training on a single robot only.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It is unclear whether the paper proposes a foundation agent or a first step towards a foundation agent. The title claims the former but the intro claims the latter:\n> Inspired by foundation models in other domains (Bommasani et al., 2022), we ultimately aim for a foundation agent for manipulation to be a multi-embodiment agent trained on a large set of robotic episodic experience that enables it to quickly adapt, via fine-tuning, to\na broad set of new downstream tasks. As a step towards this, we trained RoboCat...\n\nThis needs to be clarified.\n- Fig 1 proposes a perpetually self-improving agent. As far as I can tell, only one iteration of that cycle was ever executed, which makes the figure misleading. This needs to be clarified.", "broader_impact_concerns": "\\-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes RoboCat, a large visual, goal-conditioned transformer model that enables different robotic arms to perform a variety of robotic manipulation tasks in both simulation as well as the real world. RoboCat is capable of zero-shot generalization to new tasks and robots and can be adapted based on a small number of demonstrations. Additionally, the paper shows that the model is capable of generating synthetic data with hindsight relabeling that improves the performance of fine-tuning (self-improvement). The paper investigates RoboCat's effectiveness through a large set of experiments that involve performing various tasks using several different robot arms in both simulation and the real world.

The paper was reviewed by three reviewers from the robot learning community, and there was a healthy amount of discussion between the reviewers and the authors. The topic of the paper---the proposal of a large "generalist"/"foundation" that enables different robot arms to perform different manipulation tasks---is both highly topical and of significant interest to the community. As noted by Reviewer qZkA, RoboCat is the "First large-scale multi-embodiment agent that I'm aware of, really cool and of definite interest to the community given the trend toward large models that others can build on". The reviewers appreciate the thorough empirical evaluation on various robot arms and tasks that demonstrate the capabilities of the model with regards to cross-task transfer and self-improvement. Additionally, the reviewers emphasize that the paper is well written and that the approach is simple and clearly presented, which supports reproducibility.

The reviewers initially identified a few key weaknesses with the paper as originally submitted. Among them, all three reviewers felt that the paper overstated the generalizability of the model to different robots, tasks, and environments as would be expected of a "foundation" model. In particular, the reviewers pointed out that the abstract and introduction emphasized results on a large number of robots and tasks, yet the arms exhibit very similar kinematics and employ an action space that facilitates transfer; as well as that the paper only evaluates on two new downstream tasks. After an extensive discussion with the reviewers regarding these concerns, the authors tempered the paper's claims and rephrased their labeling of RoboCat as a "generalist" model rather than a "foundation" model. The reviewers commented that these changes help to clarify the contributions, which the reviewers find to be well aligned with the experimental results. With these updates to the text, the paper provides a solid contribution to the community.
Decision: Accept (poster)