PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining
Authors: Garrett Thomas
Ching-An Cheng
Ricky Loynd
Felipe Vieira Frujeri
Vibhav Vineet
Mihai Jalobeanu
Andrey Kolobov
Venue: CoRL 2023 Poster (2023)
Keywords: Robot learning
Robotic manipulation
Visuomotor representations

Abstract: A rich representation is key to general robotic manipulation, but existing approaches to representation learning require large amounts of multimodal demonstrations. In this work we propose PLEX, a transformer-based architecture that learns from a small amount of task-agnostic visuomotor trajectories and a much larger amount of task-conditioned object manipulation videos — a type of data available in quantity. PLEX uses visuomotor trajectories to induce a latent feature space and to learn task-agnostic manipulation routines, while diverse video-only demonstrations teach PLEX how to plan in the induced latent feature space for a wide variety of tasks. Experiments showcase PLEX’s generalization on Meta-World and SOTA performance in challenging Robosuite environments. In particular, using relative positional encoding in PLEX’s transformers greatly helps in low-data regimes of learning from human-collected demonstrations.

PLEX: Making the Most of the Available Data for Robotic Manipulation Pretraining
Garrett Thomas"
Ching-An Cheng?
Ricky Loynd
Felipe Vieira Frujerit
Vibhav Vineet
Mihai Jalobeanu
Andrey Kolobov
Abstract: A rich representation key to general robotic manipulation, but ex- isting approaches to representation learning require large amounts of multimodal demonstrations. In this work we propose PLEX, a transformer-based architec- ture that learns from a small amount of task-agnostic visuomotor trajectories and much larger amount of task-conditioned object manipulation videos type of data available in quantity. PLEX uses visuomotor trajectories to induce la- tent feature space and to learn task-agnostic manipulation routines, while diverse video-only demonstrations teach PLEX how to plan in the induced latent feature space for wide variety of tasks. Experiments showcase PLEX's generalization on Meta- World and SOTA performance in challenging Robosuite environments. In particular, using relative positional encoding in PLEX's transformers greatly helps in low-data regimes of learning from human-collectee demonstrations.
Keywords: Robot learning, Robotic manipulation, Visuomotor representations
1 Introduction
Transformers [1] have lead to breakthroughs in training large-scale general representations for com- puter vision (CV) and natural language processing (NLP) [2], enabling zero-shot adaptation and fast finetuning [3]. At the same time, despite impressive progress, transformer-base representations haven't shown the same versatility for robotic manipulation. Some attribute this gap to the lack of suitable training data for robotics [3]. We argue instead that data relevant to training robotic ma- nipulation models copious but has important structure that most existing training methods ignore and fail to leverage. These insights lead to propose novel transformer-basee architecture, called PLEX, that is capable of effective learning from realistically available robotic manipulation datasets.
We observe that sobotics-relevany data falls into three major categories: (1) Video-only data, which contain high-quality and potentially description-annotie of tasks but have no explicit action information for a robot to mimic; (2) Data containing matching sequences of percepts and actions, which are less plentiful than pure videos and don't necessarily correspond to meaningful tasks [4], but capture valuable correlations between robot's actions and changes in the environment and are easy collect sensorimotor demonstrations for target task in a target environment. Thus, scalable model ar- chitecture for robotic manipulation must be able to learn primarily from videos, while being extra data-efficient on sensorimotor training sequences and the small amount target demonstrations.
demonstrations for an immense variety
given robot; (3) Small sets of high-que eigh-uuaiity
PLEX. the PLanning-EAA EXxcuttin architecture we propose, designed to take advantage of data sources of these types. A PLEX model has two major transformer- based components: (I) task- conditioned observational planner that, given task specification and estimate of the current
Stanforr University, guthomma@stanfford. edu. Work done partly while Microsoft Research. "iirosooft Research, (chinganc, fevieira, vivineet, vivinee,,, akolobov) vivineet, @mirroooft com I dixman.ai, mihai@dexman ai. Work done partly while Microsoft Research.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

world state, determines the next state to which the robot should attempt to transition, and (II) an executor that, having received the desired next state from the planner, produces an action that should lead there from the current state. The executor is trained by optimizing an inverse dynamics loss over exploratory sensorimotor data of the aforementioned category (2), while the planner is trained by minimizing a loss of its autoregressive predictions computed with respect to video-only trajectories of category (1). The target-task data of category (3) can be optionally used to efficiently finetune the planner, the executor, or both.
We make three design choices that greatly help the data efficiency of PLEX's training:
Learning to plan in the observation embedding space. Rather than generating videos of proposed task execution using, e.g., stable diffusion as in Du et al. [5], PLEX learns to plan and execute in the low-dimensional space of observation embeddings.
Asymmetric learning of the embedding space. The observation embedding space in which the ecutor and the planner operate is induced by training the observation encoder using the executor's loss only (or even by employing a frozen feature-rich encoder such as R3M [6]). The planner's gradients don't t affect the encoder, which reduces the cost of PLEX training.
Relative positional encodings. We adopt the relative positional encodings [7] in PLEX. We em- pirically show that in robotic manipulation the relative positional encodings significantly improve training efficiency from human-collected data compared with the absolute positional encodings [1] commonly used the literature on transformers.
Most approaches that use video-only demonstrations for pretraining in robotic manipulation pro- duce purely visual representations (see, e.g., [6, 8-10]). The majority of algorithms that produce sensorimotor models need most or all of the video demonstrations to be accompanied by action sequences that generated the videos, requirement that holds only for a small fraction available manipulation data [11-17]. Few approaches have a dedicated trainable planning component; e.g. [16, 18-21] plan in skill space, which PLEX can be modified do as well. Conceptually, PLEX falls under the paradigm of learning from observations (LIO), but existing LíO approaches don't tt have multitask zero-shot planning capability [22-25] or demostrate it only in low-dimensional envi- ronments across similar tasks [26]. Of the works that have used transformers for robotic manipula- tion [14, 17, 21, 27, 28), only Brohan et al. [17] have analyzed their data efficiency, and none have looked at positional embeddings as a way to improve it. Overall, the closest approach to PLEX the concurrently proposed UniPi [5]. It also has counterparts of PLEX's s planner and executor, but its planner operates using diffusion in the image space [29], which is expensive both datawise and computationally, and may fail to model manipulation-relevan 3D object structure consistently [29]. A more extensive discussion of prior work is provided in Appendix A.
We experimentally show that PLEX's planner-executor design can effectively exploit the structure of realistically available robotic manipulation data to achieve efficient learning. On the multi-task [30] benchmark, despite pretraining mostly on video data, PLEX exhibits strong zero- shot performance unseen tasks and can be further improved by finetuning small amount of video-only demonstrations. We empirically show on the challenging Robosuite/Rooominic [31, 32] benchmark that, contrary to conclusions from NLP [7], the use of relative positional encodings significantly improves the data efficiency of PLEX learning from
demonstrations.
2 Problem statement and relevant concepts
2.1 Problem statement
We consider the problem of learning a generalist task-conditioned policy for goal-directed object manipulation. Namely, we seek policy that can control a robotic manipulator to successfully accomplish tasks that the robot may not have encountered during the policy training process; such policy formally can be viewed solution a task-conditioned partially observable Markov decision process (POMDP) described in Appendix B. In practice, learning a generalist policy that performs well on broad distribution of tasks zero-shot is very challenging, the coverage and amount

---

of publicly available training data are limited. Therefore, in this work we consider two two-phased learning process: (1) pretraining, during which a generalist policy trained, and (2) finetuning, during which this policy is adapted to a target task.
2.2 Data for training robotic manipulation models
We consider three broad groups datasets relevant to training robotic manipulation systems:"
Multi-task video demonstrations (Dinva). Being the most abundant category, it comprises data collections ranging from general YouTube videos to curated benchmarks such as Ego4D [33], Epic Kitchens [34, 35], and YouTube-8M [36] showing an agent either robot or person perform- ing a meaningful object manipulation task with an end-effector. This data contains demonstration- quality sequences of video observations and descriptions of tasks they accomplish, but not the action sequences whose execution generated these videos.
Visuomotor trajectories (Dumt). These trajectories consist of paired sequences of observations and robots' actions. Although some of them may be high-quality demonstrations of specific tasks, e.g., as in the Bridge Dataset [15], many of these trajectories are generated by activities that most people will not find meaningful, e.g., grabbing random objects in a tray, as in the RoboNet [4]. Since no strong quality, quantity, or task association requirements are imposed on Dymt data, is relatively easy to collect for any target embodiment and environment.
Target-task demonstrations (Pua). This is the most scarce but also most desirable data category, since it encompasses high-quality trajectories for specific task in question, ideally collected on the target embodiment (robot). Note, however that we don't 't require that these demonstrations be visuo- motor. In fact, our experiments show that PLEX needs only video demonstrations for finetuning to learn high-quality policy for target task.
A key data assumption make in this work is that [Dus
<<
DDmm
<
[Dmivd]-
2.3 Transformers and positional encodings
A transformer-based architecture consists of several specially structured self-attention layers and, in general, maps an input set (often called context) of K elements (called tokens) to output of the same size K [1]. In most applications, such as language translation, transformers need to map ordered sets (i.e. sequences) to other ordered sets, and therefore add special vectors called positional encodings to each input element to identify its position in a sequence. These encodings can be learned as part of transformer's training or be hand-crafted.
The most common scheme is the absolute positional encoding, where each position in the trans- former's s K Kssized context gets positional vector [1]. Some transformers, e.g., Chen et al. [37]. use what we call a global positional encoding. vector to each position in the entire input sequence rather than just the K-sized context, up to some maximum length T >>> K. Finally, models based on Transformer-Xe [7, 14, 17], instead condition the attention computation on the relative positions between different pairs of input tokens within context. In this work, we argue that robotic manipulation finetuning datasets that consist of small numbers of human-gathered demonstrations, relative positional encoding is significantly more data-cfficient than absolute or global one.
similar to the absolute one, but assigns a separate
3
PLEX architecture and training
3.1
Intuition
PLEX (shown in Figure 1) separates the model into two transformer-based submodules: 1) planner that plans in the observation embedding space based on task specification, and 2) an executor that takes the embeddings of the historical and the planned future observations and outputs an action to control the robot.
'Static image datasets, e.g., ImageNet, aren't It treated by PLEX in special way and we don't 't discuss it here, but can be used to pretrain PLEX's image encoders.

---

This design is motivated by the structure of mavdd vmn- and Dnd dataset categories, which as we explain below make them suitable for three complementary learning objectives.
REr
LEx
1. Learning to execute state transitions. The visuomotor trajectories from Dvm, collected on the target robotic manipulator or similar one, show the robot how to ex- ecute a wide variety of state transitions. By sampling an observation-action tuple (O.-H.... 0t, at, L(+L), the agent can learn to infer ae from ..- Ot, and Ot+L ing inverse dynamics, where É is the cur- rent time step, H is an observation history length, and L is a lookahead parameter.
Executor transformer
f--2
...
Frex
d,.
Lp
Learning to plan for tasks. In order to
Planner transformer
recommend meaningful action at each step, inverse dynamics inference needs the (embedding of) the desired future observa- tion. Determining the desired future obser- vation given task description is something that can be learned from multi-task video- only data Divvd since this data shows what progress towards a successful completion of specified task should look like.
Rs
F
Ruy
1vT
Prrr
IR
PPEr
Improving target-task performance. While learning plan and execute diverse Drwvd and Dvmt data can result in robotic manipulation foundation model [3] with strong zero-shot performance (see Section 4.2), on many tasks it may be far from perfect. Small datasets Dna of high-quality target-task demonstrations (e.g., through teleoperation) additional grounding to the target domain to further improve pretrained model.
Figure 1: PLEX architecture. This diagram illus- trates the information flow during PLEX training, de- scribed in Section 3.2. PLEX is optimized using planner's loss LPr. (computation shown with black ar- rows 1), and the executor's loss Lex (computation shown with gray arrows 1). The symbols '=' and "=" denote stopgrads, where backpropagation is halted. Each input modality m embedded using a modality- specific encoder @m. Video demonstration embed- dings g, Iu., (optionally) Ru.T are used to train the planner over the embedding space using the pre- diction loss LPL Visuomotor trajectory embeddings I u1.r are passed to the executor compute the inverse dynamics loss LEX. Note that the image encoder Φi isn't frozen, LEX's 's gradients will update Φ1. In contrast, the planner's focts óz (see stopgrad symbol =).
provide
PI.T,
3.2 Architecture
loss Le never af-
Following the above intuitions, we train PLEX's executor using data Dym and PLEX's planner using data D Dmavd addition if available, be used to train both the planner and executor). Specifically, let 7 IT,Pr, R1, 41 (1, ..., Ry, IT, Pr, or g, Ri.r, I1:T, PI:T 41.7 denote a trajectory. Here, is task specification, 1, is a tuple of camera image observations, Pt is proprioceptive state, is an action, and R2 is return-to-go at time t, i.e. RG = =I TV, where Tę is the instantancous reward at time t'. The length T can vary across trajectories. As Figure shows, PLEX processes these input modalities using corresponding encoders dy Φ1, @ps Φas and Φn to obtain an embedded se- quence ÿ. RI.T. P..T ā1:T- When a modality is missing, it is replaced by trainable placeholder vectors during embedding. Missing modalities are common robotic manipulation datasets; e.g., few datasets have rewards. Since PLEX's s executor and planner are designed be trainable on task- agnostic visuomotor Dvmt data and task-conditioned video-only demonstrations Dmivd respectively, each of these components is specialized to operate only the (embeddings of) modalities avail- able in their prevalent training data. Per Figure 1, task description and return embeddings and
small dataset Dna of target-task trajectories (which,

---

RIrr don't 't get routed to the executor, since they are missing from Vmtt data. Similarly, the planner only receives G. 1rrr and, optionally, Rit embeddings, since they are present D Pmrdd data. This separation holds also at the deployment time, when all modalities are available.
The planner's sole purpose determine where the agent should go the observation embedding space. As shown in Figure given embeddings H, J.r of task-conditioned video-only training demonstration, the planner outputs sequence L:TT++ of embeddings corresponding to the observations the agent should ideally see L steps in the future from its current time step; L is hyperparameter. The planner's s training minimizes the prediction loss
Planner
(p RI.T, I1T)) = - 11 1e113.
where we set I, = Ir for t +, T+ Crucially, Lp1 gradients don't backpropagate into the encoders dy and Φr. This is to prevent the collapse of the image embedding space (denoted as Eo); note the stopgrad symbols on LP' 's computation paths Figure The embedding space 6, either comes from pretrained encoders or is learned with inverse dynamics during executor training.
Executor Like the planner, the executor has specific role the deployment time. Given the observation-action sequence 01:t, ditt far and the target observation embedding L++ produced by the planner, the executor infers an action å for the current step. This inference step should be done in task-agnostic way, as the task knowledge is already incorporated in the [++L prediction of the planner. For trajectory from Dymt we optimize the executor via the inverse dynamics loss
LE (11:T PIeT, LILLL
||a á
A major difference between Lx and Lr optimization is that the former's gradients can backprop- agate into the encoders Φ1, 6o. 6p, and ég: the computation path for LEx through these encoders in Figure doesn't 't have a stopgrad. This allows executor training to shape the embedding space 6g-
Relative positional encoding Like the Decision Transformer (DT) [37], PLEX's planner and ex- ecutor transformers are derived from GPT-2. However, DT's use of global positional encoding implicitly assumes that all training trajectories have the same length Т. PLEX, in contrast, relative encoding from Dai et al. [7] the default. As we show empirically, in robotic manipulation settings where tasks are usually goal-oriented and training demonstrations vary lot in length, global positional embedding performs poorly and even the fixed absolute positional encoding common in NLP [1] performs much better. Especially, for human-collen collanccllecced demonstrations where variability is significant, our experimental results show that relative encoding [7] perform significantly better.
3.3 Training PLEX
Training PLEX generally involves both pretraining and finetuning, though the experiments in Sec- tion 4.2 show that pretraining alone already gives PLEX solid zero-shot performance.
Pretraining PLEX consists of two sub-stages:
Pretraining the executor by optimizing the LEx loss (Equation (2)) over D dataset.
2. Pretraining the planner by optimizing the LPL loss (Equation (1)) over D dataset.
the observation encoders are expected to be trained or finetuned by the inverse dynamics loss LEx, rather than pretrained and frozen beforehand, it is critical for executor pretraining to be done before training the planner. Indeed, the planner is expected to make predictions the observation encoders' embedding space, which will change if the inverse dynamics loss affects the encoders. If the encoders frozen from the start, however, the pretraining stages can proceed asynchronously.
Finetuning involves adapting PLEX using target-task demonstration dataset Dus- As with any finetuning, this involves deciding which part of PLEX to adapt.
Since Dnd can be viewed both as small Dmwvd and a small Damt dataset, it can be used to train any component of used for finetuning the encoders, it is critical to complete their finetuning before finetuning the
planner, and observation encoders. As with pretraining, if Dus
5

---

PLEX's success rate on the target tasks Meta-Woories is MLS0 split
Pretr. EX only, zero-shot
80
Pretr: PLEX, zero-shot
60
demes
Success
40-
Pretr. PLEX, ft.
demos
20
PLEX+RSM, BC on 10 rull demos
Prear. PLEX+ R3M, on 10 vidco demes Learning from Play
2and-Insert-v2
door-unlock-v2
door-lock-v2
bin-picking-v2
box-close-22
Figure 2: PLEX's generalization experiments. The confidence intervals are computed with 10 seeds.
planner. In Section 4.2, we show that finetuning just the last layer of the planner's transformer, which constitutes 5% of the parameters of the PLEX instance in the experiment, sufficient for significantly boosting pretrained PLEX's performance.
Dud can also be employed for optimizing a behavior cloning loss LBC- This amounts to training the planner, executor, and encoders simultaneussly by having PLEX predict Dua trajectories's 's ac- tions from the same trajectories' observations, and allowing the action prediction loss gradients to backpropagate through the entire PLEX model, its the inputs. The experiments in Section 4.3 demonstrate the efficiency of BC-based finetuning thanks the use of relative position encoding.
4 Experiments
We conduct two sets of experiments to answer the following questions: (i) Does PLEX pretrained on task-agnostic sensorimotor data and task-annotated video data generalize well downstream tasks? (ii) How does the use relative positional encodings affect PLEX's policy quality? Ap- pendix C provides the details about our PLEX implementation."
4.1 Benchmarks and training data
Meta-WWrid: Meta-World [30] is collection of 50 tasks featuring Sawyer arm. We use Meta- World-v2 with image observations (see details in Appendix D.1). We consider the ML45 split consisting of 45 training and 5 target tasks (door-lock, door-unlock, hand-insert, bin-picking. and box-close). We use these 5 target tasks for evaluation. Meta- World comes with high-quality scripted policies for all tasks. To get video demonstration data (Dmiva), we use these scripted policies to generate 100 successful video-only demonstrations for each of the 45 training tasks, i.e., (Pavval 4500. To generate visuomotor trajectories (Dvmt), for the 5 target tasks' environments, we add zero- mean Gaussian noise with standard deviation 0.5 to the actions of the scripted policies and record the altered actions. We collect 50 trajectories per task, i.e., [Pymel = 250. Finally, for target-task demonstrations (Dua), we employ the original scripted policies to produce 75 demonstrations per target task and sample 10 of them in a finetuning experiment run, i.e., [Dnal = 10.
Robosuite: Robosuite benchmark [31], compared Meta-World, has robotic manipulation tasks with significantly more complicated dynamics and action space. We 9 of its tasks involving single robot arm (Panda) (Lift, Stack, Door, NulAAssemblyRunnd NutAsseebbyyquure, PickPlace- Bread PickPlaceCan, PickPlaceMilk, and PickPlaceeereal), Robosuite's 's details are provided in Appendix D.I. Importantly, the training data for Robosuite was collected from human demonstra- tions, not generated by scripted policies as in Meta- World. See Appendix D.4 for details.
4.2 Generalization experiments
Here we focus on pretraining PLEX with multi-task Meta-Worke data. The results are shown in Figure 2. We train 16,639,149, 44ppaaatt PLEX instance (including the Resset--8-bbsed image encoder) from scratch with random initialization. We use the success rate on the target tasks the performance metric. For baselines, experiment with PLEX with frozen ResNet-50- based R3M [6]. an observational representation pretrained the large Ego4D dataset [33]. We denote it as 1TEE++33M. in Figure 2, Preir. PLEX+R3M was first pretrained on multitask data and then finetuned on a target task, while PLEX+R3M, BC was trained only on a single target task's data from the start. In addition, we use an adapted Learning from Play (LfP) approach [11]. The
We implement PLEX using the GPT-2 of the DT codebase [37] but without return conditioning.

---

hyperparameters and details can found Appendices C and D. In summary, the experimental results show that PLEX can perform well without seeing a single sensorimotor expert demonstration.
PLEX demonstrates zero-shot generalization capabilitiess Figure 2 shows that PLEX pretrained on as few as 4500 video demonstrations (Dmiva) from the training environments and 250 dynamics trajectories (Dvm)) from the target environments (denoted as Pretr. PLEX, zero-shot in Figure 2) exhibits good downstream performance zero-shot. To demonstrate that this performance is really due to planning learned from video-only data as opposed to the executor inadvertently exploiting biases the data, we consider a PLEX variation (denoted as Pretr. EX only, zero-shot) where we only pretrain the executor (on vmm)), not the planner. 6 The results of Pretr. EX only, zero-shol reflect a level of performance one can get with knowledge contained in the dynamics data Dymi alone. Pretr: EX only, zero-shot underperforms Pretr. PLEX, zero-shot, which shows the importance of learning from Davd via PLEX's 's planner.
Our main baseline for zero-shot generalization is Learning from Play (L.PP) [11], one of the few existing methods able to generalize zero-shot from data as low-quality as Pmvvd LiP has planning capability but doesn't 't have a way to use either the video-only data Deevd or the target-task demon- strations Dua. and performs which gives PLEX large advantage.
PLEX can be finetuned effectively using only a few video-only demonstrations We further show that finetuning only 5% of PLEX's parameters (the last transformer layer of the planner) on just 10 video-only demonstrations for given task significantly boosts PLEX's success rate there. For all 5 downstream tasks, this policy outperforms Preir. EX only, zero-shot by 2 2x. The improvement is drastic especially in the case of hand-insert-v2 bin-picking--2. and box-close-v2.
Video-only demonstrations is all PLEX needs during finetuning Interestingly, we find that full demonstrations (with both video and action sequences) don't 't increase PLEX's performance beyond video-only ones. This can seen from the experimental results of Pretr. PLEX, ft. on 10 full demos, where we finetune PLEX (the action head and last transformer layer of PLEX's planner, executor, R 11% of PLEX) on 10 full (sensorimotor) demonstrations for each task. We think this is due to PLEX's image encoder being pretrained only on observations from Dmm and frozen during fine- tuning. Because of this, finetuning couldn't help the encoder learn any extra features for modeling inverse dynamics over the observation space region covered by Dud- even if such features would improve PLEX'S performance.
The issue of impoverished observation coverage in Dvm dataset can be addressed by using a frozen encoder pretrained on an independent large dataset, as the results of PLEX+R3M, BC and of pre- trained PLEX+R3M in Figure 2 suggest. There, PLEX's R3M encoder was never trained on any WorldWorld observations but enables PLEX to perform reasonably well.
The results of Pretr. PI.EX+R3M and PLEX+R3M, BC in Figure 2 illuminate two other as- pects of using observation-only representations like R3M: (1) The sensorimotor representation that PLEX learns on top of R3M clearly helps generalization pretrained PLEX+R3M performs much better than PLEX+R3M, BC, which was trained only on a single task's data, despite pretrained PLEX+R3M seeing just video-only demonstrations at finetuning. (2) Fully frozen R3M some- what limits PLEX'S performance PLEX variants that pretrained their own encoder outperform PLEX+R3M on of 5 tasks.
4.3 Positional encoding experiments
In the Worad experiments, all training data was generated by scripted policies. In real settings, most such data is generated by people teleoperating robots or performing various tasks themselves. A hallmark of humanngenerated datasets compared to script-generated ones is the demonstration variability in the former: even trajectories for the same task originating in the same state tend to be different. In this section, we show that in low-data regimes typical of finetuning on human-generated
At run time we feed the embedding of the task's goal image as the predictions that the executor conditions on (since no planner is trained).
7

---

un
Nick
Sucoess
25
50
75
25
30
75
25
50
25
50
75
25
50
75
25
50
75
25
50
75
25
50
75
50
75
Dfgglta
Number of training trajectories
Figure 3: Data efficiency of PLEX's relative positional encoding in single-task mode on Robosuite's single-arm tasks with [Dnal varying from to 75. PLEX (with relative encodings) in most cases sig- nificantly outperforms and at worst matches the performance of its version PLEX-abs with absolute positional encodings. Both versions significantly outperform DT.
demonstrations, PLEX with relative positional encoding yields superior policies for given amount of training data than using absolute encoding. The results are in Figure 3.
Baselines, training and evaluation protocol. To analyze data efficiency and compare to prior results on Robosuite, we focus on an extreme variant of finetuning training from scratch. For each of the Robosuite tasks and each of the evaluated encodings, we train a separate 36,242,769- parameter PLEX instance using only that task's Dua dataset of full sensorimotor human-generated demonstrations. We compare PLEX with relative positional encoding to PLEX with absolute one and to two flavors of the Decision Transformer (DT) [37], which use global positional embedding. Appendix D.5 and Figure 3 provide more details about model training dataset collection, and the baselines. For each task/dataset sizc/approach, we train on 10 seeds.
Results. As Figure 3 shows, PLEX learns strong policies using most 75 demonstrations, despite having to train 36M--666Mmeeeee model including randomly initialized vision models for tasks, most of which have complex dynamics and broad initial state distributions. Moreover, PLEX with relative positional encoding (denoted simply as PLEX in the legend) outperforms the alternatives by as much as 20 percentage points (pp) on Robosuite's 's human-generated demonstration data while never losing to them. In particular, DT-global(++pn +rew and, especially, DT-global perform far worse of both PLEX and PLEXXabs. Since all models share most of the implementation and are trained similarly when PLEX and PLEX-abs run in BC mode, attribute PLEX's advantage only the combined effect of using human- human-generaae generated training data and positional encodings. We have also trained PLEX and PLEX-abs for Meta- World's 5 target tasks from the previous experiment for various amounts of the available scripted demonstrations for these tasks and noticed no significant performance differ- ence between PLEX and PLEX-abs on any task. This provides additional evidence that the utility of relative positional enconding manifests itself specifically on human- -eenanggeeertedd demonstration data.
In fact, relying on relative positional encoding allows PLEX to achieve state-of-the art performance on all Robosuite tasks in this experiment, show and analyze empirically in Appendix D.4.
5 Conclusion and limitations
We have introduced PLEX, a transformer- based sensorimotor model architecture that can be pre- trained on robotic manipulation- results show that PLEX demonstrate strong zero-shot performance and can be effectively finetuned with demonstrations to further boost its performance. In particular, PLEX shows superior perfor- mance on human-collected demonstrations because of its usage of relative positional encoding.
data realistically available in quantity. Our experimental
Limitations We believe that PLEX has great potential as a model architecture for general robotic manipulation, but in most of experiments so far, the training data came from the same robot on which the trained model was ultimately deployed. In reality, most available multi-task video demonstration data Donvd is generated by other robots or even people. This can cause a mismatch between the demonstrations and the target robot's 's capabilities and setups. Planning hierarchically first in the skill space as, e.g., in Lynch et al. [38], and then in the observation embedding space may address this issue. In addition, so far we have trained PLEX on simulated data. The eventual goal, and indeed a significant motivation for this work, would be pretrain internet-scale "in- the-wild" video datasets [29, 33, 36). Also, with the rise of powerful LLMs such Ouyang et al. [39], switching PLEX to language for task specification can facilitate generalization across tasks.
8

---

References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurlPS, 2017.
[2] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, L. Sutskever, and D. Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[3] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, Bohg, A. Bosselut, E. Brunskill, E. Brynjolisson, S. Buch, D. Card, R. Castellon, N. Chat- terji, A. Chen, Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, Etchemendy. K. Ethayarajh,
Fei-Fei, Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L.. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchan- dani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, Nyarko, G. Ogut, L. Orr, Papadimitriou, S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong. Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. 7aharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, I. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models. arXiv, 2021.
[4] S. Dasari, Ebert, S. Tian, S. Nair, Bucher, K. Schmeckpeper, Singh, Levine, and C. Finn. Robonet: Large-scale multi-robot learning. In CoRL, 2019.
[5] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv, 2023.
[6] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3M: A universal visual represen- tation for robot manipulation. In CoRL, 2022.
[7] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In A. Korhonen, D. R. Traum, and
L. Märquez, editors, ACL, 2019.
[8] L. Yen-Chen, A. Zeng, S. Song, Isola, and T.-Y. Lin. Learning to see before learning to act: Visual pre-training for manipulation. In ICRA, 2020.
[9] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-w human videos. In RSS, 2021.
[10] I. Radosavovic, Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real world robot learning with masked visual pre-training. In CoRL, 2022.
[11] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. In RSS, 2021.
[12] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, pages 991-1002, 2021.
[13] Z. Mandi, F. Liu, K. Lee, and P. Abbeel. Towards more generalizable one-shot visual imitation learning. In ICRA, 2022.

---

[14] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikow, G. Barth-Maron, M. Gimenez,
Y. Sulsky, J. Kay, J. T. Springenberg. Eeeles, J. Bruce, A. Razavi, A. Edwards, N. Heess,
Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. generalist agent, 2022.
[15] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In RSS, 2022.
[16] Nasiriany, Gao, Mandlekar, and Y. Zhu. Learning and retrieval from prior data for skill-based imitation learning. arXiv preprint arXiv:2210.1 2022.
[17] A. Brohan, N. Brown, Carbajal. Y. Chebotar, J. Dabis, Finn, Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, Yu, and Zitkovich. RT-1: Roboties transformer for real-world control at scale, arXiv, 2022.
[18] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imita- tion with skill transition models. arXiv preprint arXiv:2107.0899/ 2021.
[19] A. Ren, Veer, and A. Majumdar. Generalization guarantees for imitation learning. Con- ference on Robot Learning, pages 1426-1442 PMLR, 2021.
[20] B. Xihan, O. Mendez, and Hadfield. Skill-il: Disentangling skill and knowledge in multitask imitation learning. arXiv preprint arXiv:2205.03730. 2022.
[21] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 2022.
[22] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine. Combining self- supervised learning and imitation for vision- based rope manipulation. In ICRA, 2017.
[23] I. Radosavovic, X. Wang, L.. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In IROS, 2021.
[24] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In ICLR, 2018.
[25] B. Baker, I. Akkaya, P. Zhokhov. J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. In NeurIPS, 2022.
[26] H. Xu, L. Jiang, J. Li, and X. Zhan. A policy-guides imitation approach for offline reinforce- ment learning. arXiv, 2022.
[27] S. Dasari and A. Gupta. Transformers for one-shot visual imitation. In CoRL, 2020.
[28] H. Kim, Ohmura, and Y. Kuniyoshi. Transformer-based deep imitation learning for dual- arm robot manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8965-8972. IEEE, 2021.
[29] Ho, W. Chan, C. Saharia, J. Whang, Gao, A. Gritsenko, D. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen Video: High definition video generation with diffusion models. arXiv, 2022.
10

---

[30] T. Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, C. Finn, and S. Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforce- ment learning. CoRL, 2019.
[31] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martin. Robosuite: A modular simulation framework and benchmark for robot learning, 2020.
[32] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In CoRL, 2021.
[33] K. Grauman, A. Westbury, E. Byme, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, Sharma, M. Wray, M. Xu, Z. Xu, C. Zhao, S. Bansal, D. Batra, Cartillier, Crane, Erapalli, C. Feichtenhofer,
Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, I.. Sari, K. Somasundaram, A. Southerland, Y. Sug- ano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Ki- tani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg. Y. Sato, Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022.
Do, M. Doulaty,
[34] D. Damen, H. Doughty, G. M. Farinella, Fidler, Furnari, E. Kazakos, Moltisanti, J. Munro, Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
[35] D. Damen, H. Doughty, G. M. Farinella. A. Furnari, Ma. E. Kazakos, Moltisanti, Munro, T. Perrett, W. Price, and M. Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100 100. International Journal of Computer Vision (IJCV,, 130: 33-55, 2022.
[36] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijaya- narasimhan. Youtube-8m: A large-scale video classification benchmark, 2016.
[37] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS, 2021.
[38] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In coRL, 2019.
[39] L. Ouyang, Wu, X. Jiang, D. Almeida, C. L. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welin- der, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022.
[40] E. Chane-Sane, C. Schmid, and I. Laptev. Learning video-conditioned policies for unseen manipulation tasks. arXiv preprint arXiv:2305.062888 2023.
[41] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. arXiv preprint arXiv:230..12766, 2023.
[42] T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. In RSS, 2018.
11

---

[43] A. Zhou, E. Jang, D. Kappler, A. Herzog, M. Khansari, P. Wohlhart, Y. Bai, M. Kalakrishnan, S. Levine, and C. Finn. Watch, try, learn: Meta-learning from demonstrations and reward. In ICLR, 2020.
[44] J. Li, T. Lu, X. Cao, Y. Cai, and S. Wang. Meta-imitation learning by watching video demon- strations. In International Conference on Learning Representations, 2022.
[45] C. Finn, Yu, T. Zhang, Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. In CoRL, 2017.
[46] Z. Mandi, P. Abbeel, and S. James. On the effectiveness fine-tuning versus meta- reinforcement learning. CoRL. 2022 Workshop on Pre-training Robot Learning, 2022.
[47] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. In CoRL, 2020.
[48] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. In CoRL, 2020.
[49] D. Venuto, S. Yang, Abbeel, D. Precup, Mordatch, and O. Nachum. Multi-envirrnment pretraining enables transfer action limited datasets. arXiv preprint arXiv:2211. 2022.
[50] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[51] M. Janner, Q. Li, and S. Levine. Reinforcement learning as one big sequence modeling prob- lem, 2021.
[52] Shang, K. Kahatapitiya. Li, and M. S. Ryoo. Starformer: Transformer with state-action- reward representations for visual reinforcement learning. arXiv preprint 10X0022100006006, 2021.
[53] R. Loynd, Fernandez, A. Celikyilmaz, Swaminathan, and M. Hausknecht. Working memory graphs. In ICML, 2020.
[54] E. Parisotto, F. Song. J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, M. Botvinick, N. Heess, and R. Hadsell. Stabilizing transformers for reinforcement learning. ICMI., 2020.
[55] C. R. Dance, J. Perez, and T. Cachet. Conditioned reinforcement learning for few-shot imita- tion. In International Conference on Machine Learning, pages 2376-2387 PMLR, 2021.
[56] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do I can, not as I say: Grounding language in robotic affordances. In CoRL, 2022.
[57] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[58] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
12

---

Appendix
A
Related work
Our work lies at the intersection of scalable multi-tast representation learning for robotic manipula- tion, learning from observations, and decision-makinn using transformers.
Representation learning for robotic manipulation. Most approaches of this kind focus on pre- training purely non-motor, usually visual, representation models (see, e.g., [6, 8, 9, 40, 41], and references therein). These models don't output actions; they are meant to be foundations on top of which a policy network is be learned. Thus, in contrast to PLEX, by themselves they can't enable zero-shot generalization unseen tasks even in the limit of pretraining data coverage and amount. However, they are synergistic with PLEX: PLEX can use them as frozen observation encoders, as we show in Section 4.2 on the example of R3M [6].
Techniques that train sensorimotor models - i.e., full-fledged generalist policies, like PLEX have also been rising in prominence. Some of them [42-44] are based on meta learning [45]. However, Mandi et al. [46] have shown multi-task pretraining followed by finetuning to be more effective when the task distribution broad, and several approaches [11-17] follow this training paradigm does PLEX. At the same time, most of them need pretraining data consisting of high-quality demonstrations in the form of matching videos and action sequences. While the quality requirement can be relaxed using offline RL, as, e.g., in Singh et al. [47], in order to enable generalization across broad task distributions these sensorimotor training demonstrations need correspondingly broad task coverage. This assumption is presently unrealistic and ignores the vast potential of the available video-only data the weakness PLEX aims address.
Among the sensorimotor representation learning methods that, like PLEX, try learn from both video-only and sensorimotor data are Schmeckpeper et al. [48], Lynch and Sermanet [11], and Mees et al. [21]. Schmeckpeper et al. [48] consider single-task settings only and require the video-only and sensorimotor data to provide demonstrations for the same tasks. Lynch and Sermanet [11] and Mees et al. [21] allow the sensorimotor data to come from exploratory policies rather than task demonstrations but insist that this data must be generated from meaningful skills, strong assumption that PLEX avoids.
Architecturally, most aforementioned approaches use monolithic models that don't 't have separate components for planning and execution like PLEX. Notable exceptions are methods that mine skills from pretraining data, embed them into a latent space, and use the latent skill space for accelerated policy learning of new tasks after pretraining [16, 18-21]. This akin to planning in the skill space. PLEX can accommodate this approach hierarchically by having, e.g., a CVAE-based high- level planning model [38] produce task-conditioned sequence of skill latents and feeding them into skill-conditioned planning model that will plan in the observation embedding space. However, in this work's 's experiments, for simplicity PLEX plans in the observation embedding space directly.
Learning and imitation from observations (I/LfO) I/LfO has been used in robotic manipulation both for single-task tabula-rasa policy learning [22, 23] and pretraining [24]. Pathak al. [24] is related PLEX in spirit but lacks a counterpart of PLEX's planner. As a result, can't complete an unseen task based on the task's goal description alone: it needs either sequence of subgoal images starting at the robot's 's initial state or a sequence of landmarks common to all initial states of a given task. Beyond robotics, type of LfO was also employed by Baker et al. [25] and Venuto et al. [49] to pretrain a large sensorimotor model for Minecraft and Atari, respectively. This model, like Pathak et al. [24]'s, docsn't have a condiinnit plamning capability and is meant to serve only as finetunable behavioral prior. Xu et al. [26] investigate an LfO method akin PLEX in low-dimensional environments, where it side-steps the question of choosing an appropriate repre- sentation for planning, the associated efficiency tradeoffs, and pretraining generalizable planning policy.
13

---

Overall, the closest approach to PLEX is the concurrently proposed UniPi [5]. It also has a universal planner meant to be pretrained on a large collection of available videos, as well as an executor that captures inverse dynamics. However, UniPi ignores the issue of data efficiency and plans in the space of images (observations), using diffusion [29], rather than in the latent space of their embeddings. This is expensive learn and potentially detrimental to plan quality. Latents even from statically pretrained image encoders are sufficient capture object manipulation-relevae
details from videoframes [8], whereas diffusion models can easily miss these details or model their 3D structure inconsistently [29]. Indeed, despite being conceptually capable of closed-loop control, for computational efficiency reasons UniPi generates open-loop plans, while PLEX interleaves planning and execution in a closed loop.
Transformers for decision making and their data efficiency. After emerging the dominant paradigm in NLP [2] and CV [50], transformers have been recently applied to solving general long- horizon decision-making problems by imitation and reinforcement learning [37, 51-54], including multi-task settings [55] and robotic manipulation [14, 17, 21, 27, 28]. Mees et al. [21] provide evidence that in rohotic manipulation transformers perform better than RNNs [11] while having many fewer parameters. Of all these works, only Reed et al. [14] uses relative positional encoding, and only by "inheriting" it with the overall Transformer-XX XL architecture [7], without motivating its effectiveness for decision-making
Task specification formats. Task specification modality can significantly influence the generaliza- tion power of models pretrained on mulli-task data. Common task conditioning choices are images of a task's goal [15], videos of a task demonstration by a person [12, 42] or by a robot [13, 45), and language descriptions [11, 12, 17, 21, 56]. PLEX is compatible with any of these formats; in the experiments, we use goal images.
B Problem formalization
Formally, the problem PLEX aims to solve can be described as a partially observable Markov deci- sion process (POMDP) (G, S, O, z, A, A,p,) with a special structure. Here, 9 is the space of possible manipulation tasks that we may want to earry out the tasks in G. S = P W state space consisting of space P of robots' proprioceptive states (e.g., poses, joint speeds, etc.) and space W of world states. A state s's proprioceptive part p P is known exccution time and in some of the training data, whereas the world state w E W is never observable directly. A latent state can be probabilistically inferred from its observations o E O and state-conditioned distribution : S + A(O) that describes how latent states in S manifest themselves through observations, where denotes the space of distributions. For robotic manipulation, each observation can consist of several modalities: camera images (possibly from several cameras at each time step), depth maps, tactile sensor readings, etc. The distribution is unknown and needs to be learned. A is an action space, e.g., the space of all pose changes the robotic manipulator
achieve in 1 time step, and p S x A + A(S) is a transition function describing how executing action affects current state, which potentially is stochastic. A reward function r G x S x A x S + IR can provide addi- tional detail about task execution by assigning a numeric reward to each state transition, e.g., 0 for transitions to task's goal state and -1 otherwise. Our objective is to learn a policy क G O + A that maps a history observations O1 over the previous H steps to an action so as to lead the robot to accomplish a task g e G.
C PLEX implementation details
The transformers PLEX uses as its planner and executor are derived from the GPT-2-based version of the Decision Transformer (DT) [37]. Like in DT, we feed inputs into PLEX by embedding each modality instance (e.g., an image or an action) as single unit. This is different the way, e.g. Gato [14] and Trajectory Transformer [5 do it, by splitting each input into fragments such image patches and embedding each fragment separately.
14

---

We condition PLEX's planner on embeddings of goal images. Low- www-dimensional inputs (actions and proprioceptive states) are mapped to Re, the transformer's h-dimensional input space, using a 1- layer linear neural network. High-dimensional inputs videoframes from one or several cameras at each time step as well as goal images are processed using a ResNet-18-basel [57] encoder from Robomimic [32]. It applies a random crop augmentation each camera's image, passes through separate ResNet18 instance associated with that camera, then passes the result through spatial softmax layer [58], and finally through small MLP. The resulting embedding is fed into PLEX's planner. If the robot has several cameras, the encoder has separate ResNet instance for each. For each time step, PLEX's planner outputs an /--dimeniionll latent state representing the predicted embedding of PLEX's s visual observations time steps into the future, where is a tunable parameter. These latents are then fed directly into the planner as predictions of future observation embeddings. The output latents from the planner transformer are fed through a tanh non-linearity, which outputs action vectors in the [-1,1] range. The hyperparameters can be bound in Tables and 5.
Our PLEX implementation is available at happs//miicroooffgittuubb.LLP..
D Additional details about the experiments
D.1 Meta-World and Robosuite details
World. In our from the environment's s corner camera and the Sawyer arm's 18D proprioceptive state. The agent's actions have dimensions, each scaled to the [-1,1] range. Although Meta- World also provides privileged information about the state of the environment, including the poses of all relevant objects, PLEX agent doesn't 't access it.
setup, at each time step the agent receives an 84 x 84 image
Robosuite. The observation and action space in our experiments is exactly as in the best-performing high-dimensional setup from the Robomimic paper [32]. Namely, actions are 7-dimensional: 6 di- mensions for the gripper's pose control (OSC_POSE: and for opening/closing it. Visual observa- tions are a pair of 84 x 84 images from agentview (frontal) and eye-in-hand (wrist) cameras each step. Proprioceptive states consist of 3D gripper position, 4D quaternion for its orientation, and 2D gripper fingers' position.
D.2 Details of the baselines from prior work
PLEX +R3M [6]. We experiment with two combinations of PLEX with frozen ResNet-500 b0-bsedd R3M [6], an observational representation pretrained on the large Ego41) dataset [33] In these ex- periments, R3M replaces Robomimic's s ResNet-18, and we use versions of our Meta-World Domts Dmuvd, and Dus datasets with 224x224 image observations instead of the 84x84 ones.
One combination, PLEX +33M, BC in Figure 2, learns single-task policy 10 full sensorimo- tor demonstrations for each Meta-World target task. It operates in behavior cloning (BC) mode, whereby PLEX is optimized solely w.r.t. its action predictions' MSE loss, whose gradients back- propagate though the whole network (except the frozen R3M). The other combination, pretn. PLEX +R3M in Figure 2, follows the same PLEX pretraining and finetuning process as described previ- ously, except the R3M encoder stays frozen throughout.
Learning from Play [11]. Our final baseline is an adapted Learning from Play (LfP) approach [11]. As in Lynch and Sermanet [1 []] LiP doesn't 't use video-only Diavdd data or target-task demonstrations Dna; it trains one model for all target tasks from the "play" dataset Dvmt only. Instead of using language annotations to separate "meaningful" subsequences in Dvmts we give LiP the ground-truth knowledge of where trajectories sampled from different tasks begin and end. Accordingly, we don't use language during training either. As the case of PLEX, We train Learning from Play plan conditioned only goal images and present it with goal images from successful trajectories of the target tasks during evaluation.
15

---

D.3 Success rate evaluation protocol
In the generalization experiments on Meta-Worke all success rate evaluations are done on 50 500-step rollouts starting from initial states sampled from the test distributions of Meta-World's ML45 target tasks (door-lock, door-unlock, hand-insert, bin-picking, and box-close).
To evaluate the zero-shot success rate of the pretrained EX and PLEX models, we compute the average across 50 rollouts generated by these models on each of the target tasks the end of pretraining.
To evaluate the success rate of the finetuned models, we adopt the procedure from Mandlekar et al. [32]. The finetuning lasts for N epochs (see Table 5). After each epoch, we measure the average success rate of the resulting model across 50 rollouts, and record the maximum average success rate across all finetuning epochs.
In the positional encoding experiments on Robosuite, the evaluation protocol the same in Meta-World finetuning and in Robomimic [32]: train each model for N epochs (see Table 5), after each epoch compute the success rate across 50 trajectories (with 700-step horizon), and record the best average success rate across all epochs.
D.4 Robosuite datasets and model training
Training data for Robosuite was collected from human demonstrations, not generated by scripted policies. Robosuite provides a keyboard and SpaceMouse interfaces for controlling the Panda arm its environments, and Robomimic supplies datasets of 200 expert "pprofssional-humaa" tories collected using the SpaceMouse interface for the NutAAssemblSquuae. PickPlaceCaai and Lift tasks. For each of the tasks without pre-collected Robomimic datasets, gather 75 high-quality trajectories via Robosuite's 's keyboard interface ourselves. We employ Robosuite tasks only for ex- periments that involve training single-task policies from scratch, so all of these trajectories are used as target-task demonstration data (Dua). Typical demonstration trajectory lengths vary between 50 and 300 time steps.
trajec-
Accordingly, show the difference between relative and absolute positional encodings' data ef- ficiency, train PLEX for (Dual 5, 10, 25, 50, and 75, sampling Dia's 's from the set of 75 demonstrations without replacement. The results are presented in the main paper in Figure 3. For Lift, PickPlaceCan, and NutAAssemblySuuare, Robomimic [32] similarly provides 200 high- quality human-collected demonstrations each, as well as the results of BC-RNN on subsets of these datasets with [Pnal = 40, 100, and 200. Therefore, for these problems we train PLEX for [Dual 5, 10, 25, 50, 75, as well 40, 100, and 200. The results are shown in Table 3 and Table 1.
The only difference of PLEX model instances for Robosuite from those for Meta-World is the former having two ResNet- 18s in the observation encoder, one for the eye-in-hand and one for the agentview camera. As for WorldWorld, the encoder in the Robosuite is trained from scratch, in order to make our results comparable to Robomimic's ' [32], where models use an identical encoder and also train it tabula-rasa. In this experiment, we train PLEX in behavior cloning (BC) mode, like Meta-World's single-task PLEX +R3M, whereby PLEX optimized solely w.r.t. its action predictions' MSE loss, whose gradients backpropagate though the whole network. All hyperparameters in Table 5 in Appendix E.
We compare PLEX with relative positional encoding to PLEX with absolute one and to two flavors
of the Decision Transformer (DT) [37], which use global positional embedding. One flavor (DT- global in Figure 3) is trained to condition only on task specification (i.e., goal images), like PLEX. We note, however, that Chen et al. [37] used rewards and returns when training and evaluating DT. Therefore, we also train a return-condee ceturn-conditioned version of DT (DT (DT-global( trw) in Figure 3), with returns uniformly sampled from the range of returns in Dnd during evaluation.
16

---

D.5 Additional Robosuite results
Relying on relative positional encoding allows PLEX to achieve state- of-the art performance on all Robosuite tasks in our experiments. To establish this, in addition to the baselines Figure 3, we compare to the results of a BC-RNN implementation from the work that introduced some of these Robosuite problems [32]. Interestingly, running BC-RNN the tasks for which we have collected demonstrations ourselves resulted in 0 success rate (Table 2), while running it tasks with Kobomimic-supplics 200 trajectories (Lift, PickPlace and NutAsssemllySuuar) reproduced Mandlekar et al. [32]'s 's results. PLEX's comparison to BC-RNN's results on those prob- lems are in Table 1 in Appendix D.4. PLEX and BC-RNN are at par on the easier problems but PLEX performs better on the harder NutAssemblySquare.
Comparison to BC-RNN.
Lift
PickPlaceCan
NutAssemblySquare
Daa
40
100
200
40
100
200
40
100
200
PLEX
100 + 0 100 + 0 100 + 0
82.8
+ 8.9
95.8 + 2.8 96.6
+ 1.1
40.4
6.9
69.6
4
4.1
86.0
3.1
BC-
100 + 100 + 100
+ 0
83.3
97.3 + 98.0 0.9
29.3
4.1
64.7
4.1
82.0
0.0
Table 1: Performance of PLEX and BC-RNN on three Robosuite tasks from Mandlekar et al. [32] on [Dual 40, 100, and 200 demonstrations. BC-RNN's results come from Figure 3b and Table 27 in Mandlekar et al. [32]). On the casier Lift and PickPlaceCan, PLEX and BC-RNN are at par, but on the harder NurAsssebblySuuae PLEX performs better. On the remaining 6 problems for which we have gathered the demonstration data, BC-RNN's success rate is see Table 2.
Door
Stack
PickPlaceBread PickPlaceMilk PickPlaceCereal NutAssemblyRouud
12n
75
75
75
75
75
75
PLEX
78.4 9.2
97.3
= 2.9
92.0 4.65
65.6 4.6
72.2 4.4
49.8 - 5.5
BC-RNN
0 0
0 0
040
0 0
0+:
0
Table 2: Performance of PLEX and BC-RNN on the remaining 6 Robotsuite/Roommiiic tasks from
Figure 3. PLEX's numbers are copied from that Figure.
Better data efficiency or higher performance? Given Figure 3, one may wonder: does PLEX- abs's performance plateau lower level than PLEX's with relative positional encoding, or does PLEX-abs catch up datasets with (Dnal 75? For most tasks don't 't have enough training data to determine this, but Table 3 in Appendix D.4 provides insight for the tasks with Robomimic- supplied 200 training demonstrations. Comparing the performance gaps between PLEX and PLEX- abs on 75-trajectory and 200-trajectory datasets reveals that the gap tends to become smaller. The same can be seen for Stack, PickPlaceCeredl NutAssemblyRound already at (Dual 75 in Figure 3, suggesting that with sufficient data PLEX-abs may perform as well as PLEX. However, the amount of data for which this happens may not be feasible to collect in practice.
Lift
PickPlaceCan
NutAssembly SoAAssembySSuaare
DDaal
75
200
75
200
75
200
PLEX
100 +
100 + 0
80.4
5.7
96.6
4.1
64.0 + 4.6
86.0 + 6.1
PLEX-abs
100 + 0
100
72.8
8.0
93.0
4.7
45.2 + 5.7
76.8 + 4.9
Table 3: Performance of PLEX and PLEX-abs as the amount of training data [Pna] increases from 75 to 200 trajectories. The performance gap between the two narrower on the larger dataset. For Lift and several other Robosuite tasks, this trend becomes visible for datasets smaller than 200 (see Figure
E Hyperparameters
17

---

Meta- World (PLamerrEXeecuto)
Robosuite (Plamer/EXecutor)
Parameter name
# layers
3/3
3/3
context size
30/30 time steps
30/30 time steps
hidden dimension
256/256
256/256
transformer heads
4/4
4/4
evaluation episodes
50
50
# max. evaluation episode length
500
700
Table 4: Hyperparameters of PLEX's transformer- basedd planner and executor components for the Meta-World and Robosuite benchmarks. each case, the planner and executor use the same param- eters, but for most problems the executor's context length K can be much smaller than the planner's without loss performance, e.g., Eg = 10. For the Decision Transformer on Robosuite, we use 4 transformer layers and otherwise the same hyperparameters for PLEX.
Meta-World
Robosuite
pretraining (Plamerl(Eeecton)
last-layer finetuning
behavior cloning (Plomnel(EXecutor)
Parameter name
(Plameel(EXecutor)
lookahead steps
17-
1/
1/
learning rate batch size
5 10-4
5 10
10
256
256
256
weight decay # training epochs training steps per epoch
10-5
10-5
10-5 5
10/10
10/10(?)
10
250/250
250/250(?)
500
Table 5: Hyperparameters of PLEX training for the generalization experiments Meta- World and positional encoding experiments on Robosuite. The former use PLEX in pretraining and finetuning modes; the latter only in behavior cloning mode (training the entire model from scratch for single target task). In finetuning mode, we adapt only the last transformer layer of the planner and, in one the executor's hyperparameters indicate that
Meta-World
experiment, of the executor well. The (?) they were used only in the experiment where the executor was actually finetuned. For the Decision Transformer Robosuite use the same hyperparameters for PLEX.
18


Reviews:
[{"review_id": "ofA7wo6uQ1", "reviewer": "Reviewer_aYEz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a modularized method that attempts to learn: a) visuomotor actions through task-agnostic videos of a robot (e.g. play data) and b) planning via predicting embeddings of future states in image space through task-conditioned object manipulation videos of tasks other than those used for evaluation. The two parts are trained on disjoint data. The goal is to move a step forward towards building a foundation model for robotic manipulation that can achieve zero-shot generalization at test time and can learn from arbitrary video demonstrations including human. ", "summary_of_recommendation": "The method proposed was genuinely interesting, thank you to the authors for presenting. A direction like this could be promising towards eventually building a foundation model for robotics manipulation. My recommendation is weak reject at the moment primarily due to: \n\na) the paper is framing the work more about learning from videos in-the-wild where experiments stay in the same domain for both executor and planner models. Please rework the introduction to set expectations for the body of the paper accordingly. \n\nb) the method is not being compared with other baselines that actually claim zero-shot generalization. Please add at least one baseline of a method that claims to be doing zero-shot generalization. \n\nc) SOTA seems like too strong of a claim. Please do not claim SOTA since technically on one task, the method is not doing better and/or add numbers for the remaining 6 tasks.\n\nI'm willing to increase my score if the suggested resolutions are addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Ln. 31: \"robotic\" --> \"robot\"\n2. Ln. 32: \"meaningful task\" --> \"meaningful tasks\"\n3. Ln 53-56: Asymmetric learning of the embedding space, re: I don't quite understand this. The space that the executor and planner operate in can be induced by a pretrained frozen encoder, yet you are doing learning. You mean this is just the backbone and gets fine-tuned? This space can alternatively be induced by the executor's training loss only. Why would the planner gradients increase the training cost so much and/or cause latent space collapse? Not obvious. It's possible that these may become clear later but at this point in the paper you have not yet described your training setup at all. Consider moving the design choices after you've presented your training setup. \n4. Ln. 92-92: learning process is two-fold with second stage being finetuning, re: I was surprised to read this given that thus far, I was under the impression authors claim zero-shot generalization? \n5. Ln. 106: \"Since no strong quality, quantity, or task association requirements are imposed...\", re: Why not? I would argue that if one is doing LfO, quality and quantity at least do matter very much. \n6. Ln. 166-167: \"While learning to plan and execute ... can result in a robotic manipulation foundation model with strong zero-shot performance\", re: can it? Is this something that is shown later on in results? If yes, please add a note here to refer the reader to the results section. If not, please reduce the claim. \n7. Ln. 178: \"Rt is the reward-to-go at time t\", re: since this is a term mostly appearing in RL and even there it's not seen as commonly, considering defining this with the formula to help readers.\n8. The planner's gradients not flowing into encoders of g and I to prevent representation collapse is still not obvious. Could you please explain why this would necessarily cause a collapse? \n9. Fig. 2: what does PLEX zero-shot mean for door-unlock when door-unlock in the ML50 split seems to be included in both training and testing sets according to: https://meta-world.github.io/ ? \n10. Ln. 252-253: adding noise to the actions, re: this was not mentioned earlier. Is that just meant to be standard augmentation for robustness? Or serves some other purpose? \n11. Ln. 286: \"samll\" --> \"small\"\n12. finetuning only the last layer of the planning transformer being enough, re: may this suggest you are using a larger transformer than you need? Would like to see some experiments that investigate the impact of the planning transformer size as well as how does small transformer + finetuning the full model vs a large tranformer + finetuning only the last layer compare. \n13. Ln. 306: pretrained PLEX + R3M performs much better than the single-task one, re: what does \"single-task one\" refer to here? I don't see anything single task in Fig. 2.\n14. General comment: Looking at results, it looks like (and authors do mention this as well) PLEX + R3M where R3M is pretrained on Ego4D and frozen hurts generalization as opposed to PLEX still training all its components on the same domain (e.g. Meta World robot arm doing various tasks). This suggests that still the domain gap of learning from videos in-the-wild is still not really bridged and the method does not actually learn from videos in-the-wild but rather relies on pretrained encoders on such data as backbones to finetune. The introduction leaves the reader with the expectation that the method learns based on videos in-the-wild which is not really what's happening. Suggest reworking the message early on in the paper to avoid mismatch of expectations. \n15. Appendix D5 - Comparison with BC-RNN: \"On the remaining 6 problems for which\nwe have gathered the demonstration data, BC-RNN’s success rate is 0\": Robomimic did not provide results on the other 6 robosuite tasks. Did you train Robomimic's BC-RNN as well as PLEX with the data from the remaining 6 tasks you collected? If yes, please add those as well so that we can see PLEX's performance compared to BC-RNN even if BC-RNN is all 0.\n16. State-of-the-art is referenced multiple times in the paper. In Appendix D5, quoting \"Relying on relative positional encoding allows PLEX to achieve state716 of-the art performance on all Robosuite tasks in our experiments.\". Technically speaking, it does not perform better in all tasks. BC-RNN is slightly better on PickPlaceCan. \n17. Baselines in main paper were unsatisfying. Only Decision Transformer was included which I wouldn't have considered the most suitable baseline for a one-to-one comparison. Please include comparisons with other methods that explicitly claim zero-shot generalization (e.g. https://arxiv.org/pdf/2011.05970.pdf).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n1. Very well written paper with regards to clarity and organization, easy to follow and understand. \n2. Interesting idea that positively adds to the community's aspiration of building a foundation model for robotics. \n3. Limitations are clearly stated. \n4. Positional encoding ablation was an interesting finding.\n\nWeaknesses\n1. Baselines lacking: the method is not really compared to any baselines that also claim zero-shot generalization.  \n2. State-of-the-art claim not fully standing.\n3. Mismatch of expectations: Zero-shot generalization can be of variant degrees of difficulty. The paper initially prepares the reader for learning from videos in-the-wild which didn't end up happening. In fact, the experiments that relied on encoders pretrained on human videos demonstrated worse performance which implies that the domain gap was left unaddressed. Ultimately, the method shows good results when video demonstrations and target tasks are in the same environment with the same embodiment which is a strictly easier type of zero-shot generalization to tackle.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Uah3HTiP3G", "reviewer": "Reviewer_8ioC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an architecture for robot-learning for manipulation from readily available data. The paper conducts an analysis of the variety of data available and the volume of each type, identifying that unsupervised video demonstrations are easier to collect than sensorimotor data followed by task-specific data. Motivated by this, the paper presents a novel modular architecture and training paradigm.\n\nThe architecture is composed of a planner and an executor. First, the planner is trained using unsupervised video-demonstrations and then the executor is trained using sensorimotor data. Since observation (image) encoders are shared by the planner and the executor, the executor can be trained with fewer demonstrations (on top of frozen image encoders) after the planner is trained.  The architecture can also be fine-tuned using task-specific demonstrations. \n\nExperiments provide good empirical demonstration of the efficacy and data-efficiency of the approach.", "summary_of_recommendation": "I did not give a lower rating becasue:\n\n1. The dataset analysis presented by the authors is insightful. The architecture is devised keeping in mind the nature of available datasets.\n\nI did not give a higher rating because:\n\n1. Experimental results can be richer, solving more complex long-horizon tasks. \n\n2. Hardware experiments are unavailable.\n\n\n\n\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. I encourage the authors to add a baseline similar to 'Learning from play' but trained on sufficiently many sensorimotor demonstrations. Refer to my comment in the Weakness list, point 1. \n\n2. Please perform experiments on demonstrations of long-horizon robot manipulation benchmarks. Will just fine-tuning the planner on long-horizon video demonstrations lead to good performance on such tasks? I encourage the authors to investigate this.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n1. The paper provides an insightful analysis of groups of datasets relevant to robot learning based on quality, modality and volume. Thereafter the paper proposes an architecture and training methodology motivated by the above analysis.\n\n2. Empirical results demonstrate that the proposed architecture is effective and data-efficient.\n\n3. The approach is fairly novel and the writing is clear.\n\nWeaknesses\n\n1. I encourage the authors to add a baseline similar to 'Learning from play' but trained on sufficiently many sensorimotor demonstrations. \n     1.1 In my understanding, the reason the 'Learning from play' baseline performs poorly is because of fewer sensorimotor demonstrations     available. While this setting is valid given the hypothesis presented in the dataset analysis, the results of the proposed baseline will further establish the data-efficiency of the approach. \n\n2. I encourage the authors to perform experiments on demonstrations of long-horizon robot manipulation tasks. Will just fine-tuning the planner on long-horizon video demonstrations lead to good performance on such tasks? The authors should investigate this.\n\nMinor revisions\n\n1. Correct 'samll' to 'small' in line 286.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "LeLqmflT-H-", "reviewer": "Reviewer_aM5x", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a framework for learning robotic tasks from multiple datasets of varying quantity and quality. An agent architecture is presented that consists of a planner mapping (embedded) observations onto desired future (embedded) observations and an executor that serves as an inverse model and maps observed and desired embedded states onto actions. \nThe transformer based architecture uses embeddings that are pre-trained on ImageNet and some of the high-quantity-low-quality datasets. Robotic data comprising actions is then used to further fine-tune the agent.\nExperiments demonstrate effectiveness of the agent on a set of simulated benchmark tasks. Further, the paper demonstrates that relative position encodings appear to be superior to commonly used absolute position embeddings.\n", "summary_of_recommendation": "Overall, I recommend the paper for submission, because it is well motivated, the approach is explained clearly and the experiments examine the approach mostly well. My only concern is the choice of datasets in the experiments that might be overly simplistic. I would like to hear from the authors on this topic in the rebuttal.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I think the weakest point of this paper is the fact that D_vmt and D_ttt were generated by the same policy and tasks. Could you discuss how choosing D_vmt with more ‘meaningless’ tasks would affect the study?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I think the paper does a good job motivating the problem of learning from a lot of lower quality and little high quality data. This is a relevant problem for the robot learning community. The targeted data categories and learning system are well described as are the experiments.\nI have some concerns about the design of datasets in the experiments and some of the explanations of the approach.\n\nline 105: ‘many of these trajectories are generated by activities that most people will not find meaningful’ - In the presented experiments D_vmt and D_ttd are generated from the same agent and tasks with action noise added in D_vmt. It would be interesting to study the hypothesis where D_vmt is in fact a set of tasks that are not goal-oriented and ‘meaningless’ or at least the goals do not coincide precisely with D_ttd.\n\nline 181 ‘When a modality is missing, it is replaced by trainable placeholder vectors during embedding’ - Is there an intuition as to why you use trainable placeholders? How does it compare, for example, to feeding randomised inputs or using dropout? \n\nEquation (2) and surrounding text: I might be misreading this, but should the summation on the right-hand side start at t=T+1 instead of t=1+L ?\n\nline 229: ‘[..], this involves deciding which part of PLEX to adapt’. - In the experiments, it seems that the last layers comprising 5% of the parameters are fine-tuned. Have you compared fine tuning to other different parts of the network or how did you ‘decide’ what to fine-tune in your system? \n\nline 236: ‘behaviour cloning [..] is equivalent to a stopgrad-free version of L_EX’ - Could you elaborate on this? It seems L_EX is taking as input the past and a future state and regresses on an action, while BC usually takes in only past states and a task conditioning. Do you suggest the latter and remove the planner entirely?\n\nline 254: ‘standard deviation of 0.5’ - What are the units here and how does 0.5 relate to the distribution of actions taken by the scripted policy, i.e. your D_ttd? This relation seems to be important for interpreting how close D_ttd is to D_vmt or how much D_vmt indeed represents play-like data without goals.\n\nline 291: In the experiments D_vmt is a noisy version of D_ttd generated on the exact same tasks and with the same agent (the scripted policy). One explanation as to why adding actions does not help, is that PLEX might be able to connect image and action from D_vmt and image to goal from D_ttd. It might be an artefact of your choice of D_ttd and D_vmt. It would be interesting to conduct this study with a D_vmt that does not resemble the exact same tasks from D_ttd.\n\nAs noted in the limitations, the presented approach currently uses data from a single robot whereas the approach is to leverage large amounts of data that potentially stem from other robots. Different robots may have different proprioception and action representations. How would your approach extend from a single action / proprioception representation?\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2LQwHI5YvXG", "reviewer": "Reviewer_yTsm", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces PLEX (planning-execution), a transformer architecture with a task-specific planner and a subsequent executor that outputs actions. Each component is trained on different data: the planner is pretrained on a larger video dataset without access to actions and outputs a plan in a learned observational embedding space. The executor is pretrained on a smaller visuomotor dataset with observations and actions. MetaWorld experiments show that PLEX can complete some tasks zero-shot (without finetuning on demonstrations). Robosuite experiments show that PLEX's relative positional encoding enables better performance than the DT baseline.", "summary_of_recommendation": "The idea of enabling multiple forms of pretraining/data is well-motivated. The experiments for the most part don't actually use different data sources (just the task/presence of actions is changed). Experiments with real-world pretraining data would strengthen the paper.\n\nI'm not so familiar with the related work on transformers for manipulation, so I cannot determine the novelty/impact of this particular architecture. If most of the benefit is from relative positional encodings rather than the planner-executor split, the paper should be reframed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* For any hyperparameters that differ across settings/baselines, how were the hyperparameters selected (e.g. detail the hyperparameter search)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The PLEX architecture enables pretraining on different kinds of data, e.g. a large-scale video dataset and small-scale demonstrations.\n* The experiments show strong performance for PLEX in zero-shot, finetuning, and training from scratch settings.\n\nWeaknesses:\n* The motivation for using multiple data sources (access to large-scale internet/human videos) does not apply well to most of the experiments. The experiments use pretraining data from the same simulation environment as the evaluation. PLEX+R3M does use internet data, but it is not being presented as the primary method. Further experiments using diverse, out-of-distribution pretraining data are needed to determine the benefits of this architecture.\n* Some hyperparameters are different between MetaWorld and RoboSuite and between PLEX and DT. Is there motivation for the differences in number of layers (2 for PLEX Meta-World, 3 for PLEX Robosuite, and 4 for DT Robosuite)? The training steps per epoch is also different. \n\nMinor suggestions:\n* The introduction is quite long and overlapping with some sections. The second paragraph (L27-39) is very similar to section 2.2 and could be shortened/removed from the intro. L61-75 feels like it belongs in related work.\n* Figure 1 is hard to follow. The font size on the diagram is too small to read easily. The caption is perhaps too detailed, in addition to its small font size and lack of margins.\n* Typo: L286 samll -> small", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an architecture for learning robot policies from a mix of three types of data sources: (i) video-only data, (ii) non-task-specific sensorimotor data, and (iii) task-specific sensorimotor data.  The architecture has a few components which are trained in a prescribed fashion: a latent-space planner which can be trained from (i), and a goal-conditioned / inverse-model they call an executor.  Experiments show they are able to use this combination of different data sources with some success, for example in a way they outperform the "Learning from Play" baseline which is not designed to use (i).  While there are many other prior works that can also use (i), the work is perhaps relatively unique in its combination of all (i), (ii), (iii).

The reviewers had a variety of opinions.  Common strengths pointed to include that it is nice the method can do training on these different types of data, has strong performance in some settings, and provides useful empirical analysis.  The weaknesses mentioned mostly center around what I would characterize as an opinion by the reviewers that the authors have overclaimed their results, and/or that the results don't match the claims.  As reviewer yTsm puts it (paraphrased) "the motivation does not well apply to most of the experiments, pretraining from the same env as the eval."  Reviewer aYEz also had several critiques on the claims using the term "zero-shot", which were in part addressed in the rebuttal, but remains in their final determination still offput by the inaccuracy of the "in-the-wild" claim.  Although the other 2 reviewers suggested Strong Accept, these two reviewers (yTsm, aYEz) stood by their Weak Reject suggestions, with valid and well-argued concerns, despite the rebuttal attempts by the authors.

After careful consideration I would like to recommend the following:

- First, the reviewers should see the authors' revised manuscript which removes any mention of the term "in-the-wild" (https://openreview.net/attachment?id=iJx-Ksnsz4&name=file_upload -- I think that hyperlink will still work).  I think this partially relieves the dissenting opinion of aYEz in particular, and in part yTsm.
- However, even though the revised manuscript is a step in a good direction, the authors should amend their statements *further*... specifically, although there is no "in-the-wild" term used, the authors cite "[4-8]", which are in-the-wild video datasets.  This implies that their method will use these types of data, particularly since paragraph 2 says, "PLEX achieves exactly that".
- The authors should amend the introduction so that it does not imply this.  A suggested workaround is rather than to characterize (1), (2), (3) different types of "datasets" in the introduction paragraph, the authors can characterize three different "classes of data", and simply state that it is interesting to study methods that can use video-only data, even if the dataset comprised of such data is taken from the identical environment from which the robot actions could have simply been used -- which is the case in this paper's experiments.  The authors should only refer, even implicitly through citation, to in-the-wild video datasets in the context of related or future work.
- This is important because words matter!  There are many, for example, PhD students trying to work towards the vision of actually using in-the-wild video data in a way that is broadly useful for robotics.  This paper does not accomplish that goal, and should not imply that it does.  It can be viewed as perhaps a step towards that goal, but it does not arrive there.
- Similarly, also important, the authors should remove the notion of "embodiments" plural from the Section B, Problem formalization, because there are no experiments that address the multi-embodiment case.  This is classic over-generalization in a problem formalization.
- Regardless, if we re-examine both reviewers yTsm's and aYEz's dissenting opinions with the writing modifications requested as above, I think they would feel much more comfortable with accepting the paper.  The results are still interesting as they are truthfully, and don't need to be oversold.
Decision: Accept (Poster)