RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Authors: Jiayuan Gu
Sean Kirmani
Paul Wohlhart
Yao Lu
Montserrat Gonzalez Arenas
Kanishka Rao
Wenhao Yu
Chuyuan Fu
Keerthana Gopalakrishnan
Zhuo Xu
Priya Sundaresan
Peng Xu
Hao Su
Karol Hausman
Chelsea Finn
Quan Vuong
Ted Xiao
Venue: ICLR 2024 spotlight (2024)
Keywords: robotics
robot learning
robot manipulation
task representation
behavior cloning
multitask imitation learning
goal conditioning
Primary area: applications to robotics, autonomy, planning

Abstract: Generalization remains one of the most important desiderata for robust robot learning systems. While recently proposed approaches show promise in generalization to novel objects, semantic concepts, or visual distribution shifts, generalization to new tasks remains challenging. For example, a language-conditioned policy trained on pick-and-place tasks will not be able to generalize to a folding task, even if the arm trajectory of folding is similar to pick-and-place. Our key insight is that this kind of generalization becomes feasible if we represent the task through rough trajectory sketches. We propose a policy conditioning method using such rough trajectory sketches, which we call RT-Trajectory, that is practical, easy to specify, and allows the policy to effectively perform new tasks that would otherwise be challenging to perform. We find that trajectory sketches strike a balance between being detailed enough to express low-level motion-centric guidance while being coarse enough to allow the learned policy to interpret the trajectory sketch in the context of situational visual observations. In addition, we show how trajectory sketches can provide a useful interface to communicate with robotic policies -- they can be specified through simple human inputs like drawings or videos, or through automated methods such as modern image-generating or waypoint-generating methods. We evaluate RT-Trajectory at scale on a variety of real-world robotic tasks, and find that RT-Trajectory is able to perform a wider range of tasks compared to language-conditioned and goal-conditioned policies, when provided the same training data.

RI-TRAJECTORY: ROBOTIC TASK GENERALIZATION VIA HINDSIGHT TRAJECTORY SKETCHES
Jiayuan Gu Sean Kirmani', Paul Wohlhart', Yao Lu's MMontserrat Gonzalez Arenas', Kanishka Rao', Wenhao Yu', Chuyuan Fu', Keerthana Gopalakrishnan' ZZhuo Xu', Priya Sundaresan Peng Xu', Hao Su?, Karol Hausman', Chelsea Finn Quan Vuong', Ted Xiao 'Google DeepMind, University of California San Diego, Stanford University, "Intrinsic
ABSTRACT
Generalization remains one of the most important desiderata for robust robot learning systems. While recently proposed approaches show promise in generalization to novel objects, semantic concepts, or visual distribution shifts, generalization to new tasks remains challenging. For example, a danguage-condttoons place tasks will not be able to generalize to a folding task, even if the arm trajectory of folding is similar to pick-and-place Our key insight that this kind of generalization becomes feasible if we represent the task through rough trajectory sketches. We propose a policy conditioning method using such rough trajectory sketches, which we call RT- Trajectory, that is practical, easy to specify, and allows the policy to effectively perform new tasks that would otherwise challenging to perform. We find that trajectory sketches strike balance between being detailed enough to express low-level motion- centric guidance while being coarse enough to allow the learned policy to interpret the trajectory sketch in the context of situational visual observations. In addition, we show how trajectory sketches can provide a useful interface to communicate with robotic policies they can be specified through simple human inputs like drawings or videos, through automated methods such as modern image-generating or waypoint-generatin methods. We evaluate RT-Trajectory at scale on a variety of real-world robotic tasks, and find that RT-Trajectory is able to perform a wider range of tasks compared to language- conditioned and condiconnitioned policies, when provided the same training data, Eval- uation videos can be found at ht https: ///ttt/tttt tra jectory. g thub.oo/.
policy trained on pick-and-
INTRODUCTION
The pursuit of generalist robot policies has been perennial challenge in robotics. The goal devise policies that not only perform well on known tasks but can also generalize to novel objects, scenes, and motions that are not represented in the training dataset. The generalization aspects of the policies are particularly important because of how impractical and prohibitive it is compile robotic dataset covering every conceivable object, scene, and motion. In this work we focus on the aspects of policy learning that, we later show in the experiments, can have large impact of their generalization capabilities: task specification and policy conditioning.
Traditional approaches task specification include one-hot task conditioning (Kalashnikov al., 2021), which has limited generalization abilities since one-hot vector does not capture the similarities between different tasks. Recently, language conditioning significantly improves generalization to new language commands (Brohan et al., 2023b), but it suffers from the lack of specificity, which makes it difficult to generalize to a new motion that can be hard to describe. Goal image or video conditioning (Lynch et al., 2019; Chane-Sam et al., 2023), two other alternatives, offer the promise of more robust generalization and can capture nuances hard to express verbally but easy to show visually. However, has been shown be hard to learn from (Jang et al., 2022) and requires more effort to provide at test time, making it less practical. Most importantly, policy conditioning not only impacts the practicality of task specification, but can have a large impact generalization at inference time. If the representation of the task is similar to the one of the training tasks, the underlying model more likely able to interpolate between these data points. This is often reflected with the type of generalization exhibited in different conditioning mechanisms for example, if the policy is conditioned on natural language commands, it is likely to generalize to a new phrasing of the text command, whereas that same policy when trained on pick-and-plac tasks will struggle with generalizing to

---

Training
Policy Architecture
5
Current Image and Recent History
Hindsight Trajectory
Proprioception End-Effector Trajectory
Training
Inference
Only
Concat
Inference
Trajectory Drawings
Only
EfficientNet
Graphical User Interface
Choose
Learner
Human Video
Hand
Generate
trajectory to put
Foundation Models
the chip bag in the middle
VLM
mode base
amm
Code as Poíciee LLMs or Image Generation via VILMs
Figure I: We propose RT-Trajectory, a framework for utilizing coarse trajectory sketches for policy conditioning. We train on hindsight trajectory sketches (top left) and evaluate on inference trajectories (bottom left) produced via Trajectory Drawings. Human Videos, or Foundation Models. These trajectory sketches are used task specification for RT-L (Brohan et al., 2023b) policy backbone (right). The trajectories visually describe the end-effector motions (curves) and gripper interactions (circles).
folding task, even if the arm trajectory of folding is similar to pick-and-place, because in language space, this new task is outside the previously seen data. This begs question: can we design a better conditioning modality that is expressive, practical and, the same time, leads to better generalization new tasks?
To this end, we propose to use a coarse trajectory as a middle middle-ground solution between expressiveness and ease of use. Specifically, introduce the of a 2D trajectory projected into the camera's field of view, assuming a calibrated camera setup. This approach offers several advantages. For example, given a dataset of demonstrations, we can automatically extract hindsight 2D trajectory labels without the need for manual annotation. In addition, trajectory labels allow us to explicitly reflect similarities between different motions of the robot, which, as we show the experiments, leads better utilization of the training dataset resulting in wider range of tasks compared to language- and coal-conditioned alternatives. Furthermore, humans or modern image-editing models can sketch these trajectories directly onto an image, making it simple yet expressive policy interface.
The main contribution of this paper is novel policy conditioning framework R1-Trajectory that fosters task generalization. This approach employs 2D trajectories as a human-interpretuu tioning signal for robot policies. Our experimental setup involves a variety of object manipulation tasks with both known and novel objects. Our experiments show that RT- RT-Trajectory outperforms existing policy condi- tioning techniques, particularly terms of generalization to novel motions, an open challenge in robotics.
yet richly expressive condi-
2 RELATED WORK
In this section, we discuss prior works studying generalization robot leamingg as well works proposing specific policy conditioning representations.
Trajectory Tracking in Control Theory Trajectory planning and tracking has been a studisd setting in the optimal control literature. Given reference trajectory, optimal controllers can be designed to minimize tracking errors expressed as closed-form cost functions (Aguiar & Hespanha, 2007; Borrelli al., 2017). Such methods may work well in robot systems with known linear or nonlinear dynamics (Park al., 2004), and have been demonstrated in mobile robotics with Model Predictive Control (MPC) (Kamel et al.,
2017). Sliding Mode Control ( (Yang & Kim, 1999), Adaptive Control (Bresch-Piie Pietri & Krstic, 2009). The targeted reference trajectories may be provided and fixed after an initial trajectory planning stage (Kant & Zucker, 1986; Kawato, 1999) dynamically updated with iterative online planning (Fridovich-Kei)
2

---

et al., 2018). While performance of classical trajectory tracking methods may degrade without accurate reference trajectories provided in ground truth state space (Zuo & Wang, 2014; Li et al., 2015), online re-planning methods are able to utilize unfeasible trajectory targets in dynamic environments (Williams et al., 2016; 2017) In contrast, our proposed method makes fewer assumptions on full ground-truth specification of an accurtae coarse trajectory sketch, and instead aims to leverage the benefits of end-to-end learning to generalize to uncertain or complex scenarios with coarse trajectory guidance.
Generalization in Robot Learning Recent works have studied how learning-based robot policies may generalize robustly to novel situations beyond the exact data seen during training. Empirical studies have analyzed generalization challenges in robotic imitation learning, focusing on 2D control (Toyer et al., 2020), demonstration quality (Mandlekar et al., 2021), visual distribution shifts (Xie et al., 2023), and action consistency (Belkhale et al., 2023). In addition, prior works have proposed evaluation protocols explicitly testing policy generalization; these include generalizing to novel semantic attributes (Shridhar et al., 2021). 1). holdout language templates (Jang et al., 2021), unseen object categories (Pinto & Gupta, 2016; Mahler et al., 2017; Shridhar et al., 2022; Stone et al., 2023), new backgrounds and distractors (Chen et al., 2023; Yu et al., 2023), combinations of distribution shifts (Brohan et al., 2023b; Jiang et al., 2023), open-set language instructions (Xiao al., 2023; Huang et al., 2023), and web-scale semantic concepts (Brohan et al., 2023a). While these prior works largely address semantic and visual generalization, we additionally study task generalization which include situations which require combining seen states and actions in new ways, or generalizing to wholly unseen states or motions altogether.
Policy Conditioning Representations We examine few approaches for policy conditioning. Broadly, there are 2 axes to consider: (1) over-specification and under-speciffcttion of goals, and (2) conditioning on all states in a trajectory versus only the end state. The most prolific recent body of work focuses on language-conditioned policies (Jang et al., 2021; Brohan et al., 2023b;a; Nair et al., 2021; Ahn et al., 2022: Hill et al., 2020; Lynch & Sermanet, 2021), which utilize templated or freeform language as task spec- ification. Language-oo conditiionoddiiieeeeeee policies can be thought of as under-specified on the end state (e.g. there are many possible end-states for a policy that completes pick can). There are many image-conditioned policy representations with the most popular technique being goal-image conditioning: where a final goal image defines the desired task's 's end-state (Bousmalis et al., 2023; Lynch et al., 2019). Goal image conditioned policies can be thought of over-specified on the end state (i.e. "what to do") because they define an entire configuration, some of which might not be relevant. For example, the background pixels of the goal image might not be pertinent the task, and instead contain superfluous information. There some examples of intermediate levels of specification that propose 2D and 3D object-cemm cenertccenticc representations (Stone et al., 2023; Shridhar al., 2021; Huang et al., 2023), using multimodal embedding that represents the task as a joint space of task-conditioned text and goal-conditioned image (Xiao et al., 2023; Jiang al., 2023: Shridhar et al., 2021). 1). and describing the policy as code (Liang et al., 2022) which constrains how to execute every state. An even more detailed type of state-specification would be conditioning on an entire RGB video which is equivalent to vver-ppecification over the entire trajectory of states (i.e. "how to do it") (Chane-Sane et al., 2023). However, encoding long videos in-context is challenging to scale, and learning from high-dimensional videos is a challenging learning problem (Jang et al., 2021). In contrast, approach uses a lightweight coarse level of sateespecifiiaaion, which aims to strike a balance between sufficient sate-specification capacity to capture salient state properties while still being tractable to learn from. We specifically compare against language-conma conditioning and goal-image conditioning baselines, and show the benefits of using a mid-level conditioning representation such as coarse trajectory sketches. Concurrently, similar representation of utilizing trajectory sketches is studied in diagrammatic teaching (Zhi et al., 2023), which focused on reconstructing 3D trajectories from multi-view 2D sketches while our approach focuses on learning to condition on 2D sketch directly.
3 METHOD
3.1 OVERVIEW
Our goal is to learn a robotic control policy that is able to utilize a 2D coarse trajectory sketch image as its conditioning. A system diagram for our proposed approach can be seen in Fig I. During policy training, we first perform hindsight trajectory labeling to obtain trajectory conditioning labels from the demonstration dataset (Section 3.2). This enables to re-use existing demonstration dataset and ensures the scalability of our proposed approach to new datasets. We then train a transformer-bast control policy that is conditioned on the 2D trajectory sketches using imitation learning (Section 3.3). During inference
3

---

(a) Example Trajectory
(b) Interaction Markers
(c) Temporal Progress
(d)
(e) RT-Trajectory (2D)
(f) RT-Trajectory (2.5D)
Gripper Height
Figure 2: Visualization of the two hindsight trajectory sketch representations study. Given (a) an example robot trajectory, we extract (b) gripper interaction markers, (c) temporal progress along the 2D end- effector waypoints, and (d) end-effector height. Combining (b) and results in (e) RT-Trajectory (2D), while combining (b), (c), and (d) results in (f) RT-Trajectory (2.5D).
time, the user or a high-level planner is presented an initial image observation from the robot camera, and creates rough 2D trajectory sketch that specifies the desired motion (Fig. 1 bottom left), which is then fed into the trained control policy to perform the designated manipulation task.
3.2 HINDSIGHT TRAJECTORY LABELS
In this section, we describe how we acquire training trajectory conditioning labels from the demonstration dataset. We introduce three basic elements for constructing the trajectory representation format: 2D Trajectories, Color Grading, and Interaction Markers.
2D Trajectory For each episode in the demonstration dataset, we extract 2D trajectory of robot end-effector center points. Concretely, given the proprioceptive information recorded in the episode, we obtain the 3D position of the robot end-effector center defined in the robot base frame at each time step, and project assume that the robot base and camera do not move within the episode, which is common for stationary manipulation. Given 2D trajectory (a sequence of pixel positions), we draw a curve on a blank image, by connecting 2D robot end-effector center points at adjacent time steps through straight lines.
the camera space given the known camera extrinsic and intrinsic parameters. We
Color Grading To express relative temporal motion, which encodes such as velocity and direction, we also explore using the red channel of the trajectory image specify the normalized time step " where t is the current time step and T is the total episode length. Additionally, we propose incorporating height information into the trajectory representation by utilizing the green channel of the trajectory image to encode normalized height relative the robot base he-1
Tomas -hmsn
Interaction Markers For robot manipulation tasks, time steps when the end-effector interacts with the environment are particularly important. Thus, we explore visual markers that explicitly highlight the time steps when the gripper begins grasp and release objects. Concretely, we first compute whether the gripper has contact with objects by checking the difference d. ir -pe between the sensed (p:) and target and Pt > e, where is threshold of closing action (p: increases the gripper closes), it indicates that the gripper is closing and grasping certain object. If the status change, e.g., 8, < OV Se but 6,41 > OA > e, we consider the time step as a key step for the closing action. Similarly, circles at the 2D robot end-effector center points all key time steps for closing (or opening) the gripper.
(fx) gripper joint positions. the difference
find the key time steps for the opening action. We draw green (or blue)
Trajectory Representations In this work, propose different combinations the basic elements. In the first one, RT-Trajectory (2D), we construct an RGB image containing the 2D Trajectory with temporal information and Interaction Markers indicate particular
forms of trajectory representation from
4

---

robot interactions (Fig. 2 (e)). In the second representation, introduce more detailed trajectory
representation RT-Trajectory (2.5D), which includes the height information in the 2D trajectory (Fig. 2 (f))
3.3 POLICY TRAINING
We leverage Imitation Learning due to its strong success in multitask robotic imitation learning settings (Jang et al., 2022; Bousmalis et al., 2023). More specifically, we assume access to a collection of successful robot demonstration episodes. Each episode contains sequence of pairs of observations or and actions az (= The observations include RGB images obtained from the head camera 22 and hindsight trajectory sketch Ctraj- We then learn policy π represented by a Transformer (Vaswani et al., 2017) using Behavior Cloning (Pomerleau, 1988) following the RT-1 framework (Brohan et al., 2023b), by minimizing the log-likelihood of predicted actions dg given the imput image and trajectory sketch. To support trajectory conditioning, we modify the RT-1 architecture as follows. The trajectory sketch is concatenated with each RGB image along the feature dimension in the input sequence (a history of 6 images), which processed by the image tokenizer (an ImageNet pretrained EfficientNet-BB B3). For the additional input channels to the image tokenizer, we initialize the new weights in the first convolution layer with all zeros. Since the language instruction is not used, we remove the FiLM layers used in the original RT-1.
3.4 TRAJECTORY CONDITIONING DURING INFERENCE
During inference, a trajectory sketch is required to condition RT-Trujectory. We study different methods to generate trajectory sketches: human drawings, human videos, prompting LLMs with Code Policies, and image generation models.
Human-drawe Sketches Human-drawn sketches an intuitive and practical way for generating trajectory sketches. To scalably produce these sketches, we design a simple graphical user interface (GUI) for users to draw trajectory sketches given the robot's initial camera image, shown in App. B.I.
Human Demonstration Videos with Hand-object Interaction First-person human demonstration videos are alternative input. We estimate the trajectory of human hand poses from the video, and convert it to a trajectory of robot end-effector poses, which can later be used to generate a trajectory sketch.
Prompting LLMs with Code Policies Large Language Models have demonstrated the ability to write code to perform robotics tasks (Liang et al., 2022). We follow a similar recipe as described in (Gonzalez Arenas et al., 2023) to build a prompt which contains text descriptions about the objects in the scene detected by a VLM, the robot constraints, the gripper orientations and coordinate systems, as well as the task instruction. By using this prompt, the LLM writes code generate series of 3D poses originally intended to be executed with a motion planner, which we can then re-purpose to draw the trajectory sketch on the initial image to condition R1-Trajectory.
Image Generation Models Since our trajectory conditioning is represented as an image, we can use text-guided image generation models generate a trajectory sketch provided the initial image and language instruction which describes the task. In our work, we use PaLM-E style (Driess et al., 2023) model that generates vector-quantized tokens derived from ViT-VQGAN (Yu et al., 2022) that represent trajectory image. Once detokenized, the resulting image can be used to condition RT-Trujectory.
4 EXPERIMENTS
Our real robot experiments aim to study the following questions:
1. Can RT-Trajectory generalize to tasks beyond those contained in the training dataser?
2. Can RT-Trajectory trained on hindsight trajectory sketches generalize to diverse human-specified or automated trajectory generation methods at test time?
3. Can we quantitatively measure how dissimilar evaluation trajectory motions are from training dataset motions?
4. What emergent capabilities are enabled by RT-Trajectory?
5

---

4.1 EXPERIMENTAL SETUP
We use a mobile manipulator robot from Everyday Robots in our experiments, which has a 7 degree-oo-freedom arm, a two-fingered gripper, and mobile base.
Seen Skills We use the RT-1 (Brohan et al., 2023b) demonstration dataset for training. The language instructions consist 8 different manipulation skills (c.g., Move Neaz) operating on of 17 household kitchen items; in total, the dataset consists of about 73K real robot demonstrations across 542 seen tasks, which were collected by manual teleoperation. A more detailed overview is shown in Table
Unseen Skills We propose new evaluation skills which include unseen objects and manipulation workspaces, shown Table 3 and Fig. Both Upright and Move and Move within Drawer examine whether the policy can combine different seen skills to form a new one. For example, Move within Drawer studies whether the policy able to move objects within the drawer while the seen skill Nove Near only covers those motions at height of the tabletop. Restock Drawer requires the robot to place snacks into the drawer at an empty slot. studies whether the policy is able to place objects target positions precisely. Place Fruit inspects whether the policy can place objects into unseen containers. Pick from Chair investigates whether the policy can pick objects at an unseen height in an unseen manipulation workspace. Fold Tewel and Swivel chair showcase the capability to manipulate deformable object and interact with an underactuated system.
Figure 3: Visualization of trajectory sketches overlaid on the initial image for 7 unseen skills. From left to right: Place Fruit, Upright and Move, Fold Towel, Move within Drawer, Restock Drawer, Pick from Chair, Swivel Chair. See the rollouts in Fig. 13.
Evaluation Protocol Different trajectory sketches will prompt RT-Trajectory to behave differently. To make the quantitative comparison between different methods as fair as possible, we propose the following evaluation protocol. For each skill to evaluate, collect a set of scenes. Each scene defines the initial state of the task, described by an RGB image taken by the robot head camera. During evaluation, we first align relevant objects to their original arrangements in the scene, and then run the policy. For conditioning RV-Trajectory, we use human drawn sketches for unseen tasks in Sec. 4.2. In Sec. 4.3, we evaluate other trajectory sketch generation methods described Sec. 3.4.
4.2 UNSEEN TASK GENERALIZATION
In this section, compare RI-Trajectory with other saaming-based baselines generalization the unseen task scenarios introduced in Sec 4.1.
RT-1 (Brohan al., 2023b): language-conditioned policy trained the same training data;
RT-2 (Brohan et al., 2023a): language-conditioned policy trained on a mixture of our training data and intemet-scale VQA data;
RT-1-GGalt goal-conditioned policy trained on the same training data.
For RT-Trajectory, manually generate trajectory sketches via the GUI (see Sec. B. Details about trajectory generation are described in App. B.2. For RT-I-Goal, implementation details and goal conditioning generation are presented in App. B.4. The results are shown Fig. 4 and Table The overall success rates of our methods, RT-Trajectory (2D) and RT-Trajectory (2.5D), are 50% and 67% respectively, which outperform our baselines by large margin: RT-/ (16.7%), RT-2 (11.1%). RT-I-Goal (26%). policies struggle generalize to the new tasks with semantically unseen language instructions, even if motions to achieve these tasks were seen during training (see Sec. 4.4). RT-I-Goal shows better generalization than its innguage-conditioons counterparts. However, goal conditioning much harder to acquire than trajectory sketches during inference in new scenes and is sensitive to task- inrkvinelevant factors (e.g., backgrounds). RT-Trajectory (2.5D) outperforms RT-Trajectory (2D) the tasks where height information helps reduce ambiguity. For example, with 2D trajectories only, is difficult for RT-Trajectory (2D) infer corect picking height, which is critical for Pick from Chair.
Language-condition

---

RT-Traj (2D)
RT-Traj (2.5D)
RT-1 RT-2 RT-1-goal
100
25
Swivel
Fold Towel
and
Restock
Move
Pick Chair
Overall
Move
Drawer
Drawer
Figure 4: Success rates for unseen tasks when conditioning with human drawn sketches. Scenarios contain variety of difficult settings which require combining seen motions novel ways or generalizing new motions, Each policy is evaluated for a total of 64 trials across 7 different scenarios.
Figure 5: Trajectory from human demonstration video to fold a towel. From left to right, the first images show the human demonstration, and the last image shows the derived trajectory sketch.
4.3 DIVERSE TRAJECTORY GENERATION METHODS
In this section, aim study whether RT-Trajectory is able to generalize to trajectories from more automated and general processes at inference time. Specifically, we evaluate quantitatively how RT-TTajectory performs when conditioned on coarse trajectory sketches generated by human video demonstrations, LLMs via Prompting with Code as Policies, and show qualitative results for image generating VLMs. Additionally, we compare RT-Trujectory against non-leaming baseline (IK Planner) to follow the generated trajectories: an inverse-king -inrem-iinemaiic (IK) solver is applied to convert the end-effector poses to joint positions, which are then executed by the robot.
Method
Pick
Open Drawer
Fold Towel Avg.
Method
Pick Open Drawer
Fold Towel
Avg.
IK Planner 42%
0%
25%
39%
IK Planner
83%
71% 60%
25%
60%
Ours (2D)
94%
60%
75%
76%
Ours (2D)
89%
0%
50%
Ours (2.5D)
100%
90%
75%
88%
Ours (2.SD)
89%
60%
25%
58%
(a) Trajectory from human video demonstrations.
(b) Trajectory from LLM prompting.
Table I: Success rate of different trajectory generation approaches across tasks.
Human Demonstration Videos We collect 18, 10, and 4 first-person human demonstration videos with hand-object interaction for Pick, Open Drawer and Fold Towel respectively. An example is shown in Fig. 5. Details about video collection and how trajectory sketches are derived from videos are described App. B.3. The resulting trajectory sketches are more squiggly than the ones for training. Results are shown in Table la.
Prompting with Code as Policies We prompt an LLM (OpenAI, 2023) to write code to generate trajectories given the task instructions and object labels for Pick, Open Drawer and Fold Towel. After executing the code written by the LLM, we get a sequence of target robot waypoints which can then be processed into trajectory sketch. In contrast with human-specified trajectories. LLM-ggnerated trajectories are designed be executed by an IK planner and are therefore precise and linear as seen in Fig. 19. While they are also different from the hindsight trajectories in the training data, RT: RT-Trajectory able to execute them correctly and outperform the IK planner in diverse pick tasks due to its ability adapt motion the nuances like object orientation. Results are shown in Table 1b.

---

Figure 6: Example trajectory from image generation models. From left to right, the first image shows the overlaid trajectory sketch, and the next images show the rollout conditioned on The language instruction is: pick orange can from top drawer and place on counter.
Image Generation Models We condition the VLM with a language instruction and an initial image output trajectory tokens which de-tokenized into 2D pixel coordinates for drawing the trajectory. Qualitative examples are shown in Fig 6. Although we see that generated trajectory sketches are noisy and quite different from the training hindsight trajectory sketches, we find promising signs that RT-Trajectory still performs reasonably. As image-generating VLMs rapidly improve, we expect that their trajectory sketch generating capabilities will improve naturally in the future and be usable by RT- RT-Trajectory.
4.4 MEASURING MOTION GENERALIZATION
We wish to explicitly measure motion similarity in order to better understand how RT RT-Trajectory able to generalize to unseen scenarios and how well it can tackle the challenges of novel motion generalization. Towards this, we intend to compare evaluation trajectories to the most similar trajectories seen during training. To accomplish this, we propose to measure trajectory similarity by utilizing the discrete Fréchet distance (Fréchet, 1906) (details in App. C.I). By computing the distance between a query trajectory and all trajectories in training dataset, we can retrieve the most similar trajectories our policy has been trained on. We perform this lookup for trajectories from the rollouts for the unseen task evaluations in Sec. 3.4. Fig. 7 showcases the 10 most similar training trajectories for a selection of query trajectories.
Fig. 9, 10, and 11 in Appendix furthermore show statistics of the most similar training samples, such as the distribution of skill semantics. We find that the trajectories for unseen tasks show varying levels of similarity to training trajectories. For example, the motion for place a fruit into a tall bowl may be surprisingly similar the motion for particular seen instances of the move X near Y. However, for many unseen skills, the most similar examples in the training data are still significantly more different than for examples within the training set. In addition, even for evaluation trajectories that seem close in shape to the most similar training trajectories, we find differences in precision-criee z-height of gripper interactions (picks that are just a few centimeter incorrect will not succeed) or semantic relevance (the most similar training trajectories describe different skills than the target trajectory). Thus, we expect that the proposed new skills for evaluation indeed require a mix of interpolating seen motions along with generalizing to novel motions altogether.
factors like the
4.5 EMERGENT CAPABILITIES
Prompt Engineering for Robot Policies Similar how LLMs respond differently in response to language prompt engineering, K1-Trajectory enables visual prompt engineering, where a trajectory- conditioned policy may exhibit better performance when the initial scene is fixed but the coarse trajectory prompts are improved. We find that changing trajectory sketches induces RT-Trajectory to change behavior modes in a reproducible manner, which suggests an intriguing opportunity: if -rajectory-oonditioned robot policy fails in some scenario, a practitioner may just need to "query the robot" with a different trajectory prompt, as opposed to re-training the policy or collecting more data. Qualitatively, this is quite different from standard development practices with imguage-conditione. robot policies, and may be viewed as an early exploration into zero-shot instruction tuning for robotic manipulation, similar to capabilities seen in language modeling (Brown et al., 2020). See App. E. for examples.
Generalizing to Realistic Settings Prior works studying robotic generalization often evaluate only a few distribution shifts at once, since generalizing to simultaneous physical and visual variations is challenging; however, these types of simultaneous distribution shifts are widely prevalent in real world settings. As qualitative case study, evaluate RT-Trajectory in 2 new buildings in 4 realistic novel rooms which contain entirely new backgrounds, lighting conditions, objects, layouts, and furniture geometries. With little to moderate trajectory prompt engineering, we find that R7-Thajectory is able successfully perform
8

---

Top-10 Most Similar Training Trajectories to Query Trajectories
"close top drawer" (from Training Dataset)
(a)
"place fruit"
(b)
"move within drawer"
(c)
"pick from chair"
(d)
Figure 7: Each row contains 4 instances an initial image of evaluation rollout super-imposed with the executed evaluation trajectory (red) compared with the 10 most similar trajectories (purple) in the training dataset. Row (a) shows query trajectories of the in-distribution close top drawer skill seen the training data. Rows (b,c,d) show query trajectories of unseen evaluation skills.
a variety of tasks requiring novel motion generalization and robustness to auttof-distribution visual distribution shifts. These tasks are visualized in Fig. 15 and rollouts are shown fully in Fig. 16.
CONCLUSION AND LIMITATIONS
In this work, we propose a novel yolicy-conditioning method for training robot manipulation policies capable of generalizing tasks and motions that are significantly beyond the training data. Key our proposed approach is a 2D trajectory sketch representation for specifying manipulation tasks, Our trained trajectory
policy enjoys controllability from visual trajectory sketch guidance, while retaining the flexibility of learning- based policies in handling ambiguous scenes and generalization to novel semantics. We evaluate our proposed approach on diverse manipulation skills that were never seen during training and benchmark against three baseline methods. Our proposed method achieves success rate of 67%, significantly outperforming the best prior state-of-the-an methods, which achieved 26%.
Though we demonstrate that our proposed approach achieves encouraging generalization capabilities for novel manipulation tasks, there are a few remaining limitations. First, we currently assume that the robot remains stationary, only uses the end-effector for manipulation. and that the end-effector remains visible throughout the episode (for visual servoing). Extending the idea to mobile-manipulatioo scenarios that allow the robot manipulate with whole-body control promising direction to explore. Second, our trained policy makes its best effort in following the trajectory sketch guidance. However, user may want specify spatial regions where the guidance is more strictly enforced, such when avoid fragile objects during movement. Thus, an interesting future direction is to enable systems to use trajectory sketches handle different types of constraints.

---

ACKNOWLEDGMENT
The authors would like to thank Wenxuan Zhou for help with the human hand pose tracking infrastructure. Also, we would like to thank Cheri Tran, Emily Perez, Grecia Salazar, Jaspiar Singh, and Jodilyn Peralta for their immense contributions to evaluations.
REFERENCES
A Pedro Aguiar and Joao P Hespanha. Trajectory- and path-following of underactuated autonomous vehicles with parametric modeling uncertainty. IEEE transactions on automatic control, 1228:1166-1339, 2007.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev. Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning (CoRI.), 2022.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. arXiv preprint arXiv:2306. 0224:2 02437, 2023.
Gunilla Borgefors. Distance transformations in arbitrary dimensions. Computer vision, graphics, and image processing, 27(3):321-345, 1984.
Francesco Borrelli, Alberto Bemporad, and Manfred Morari. Predictive control for linear and hybrid systems. Cambridge University Press, 2017.
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl. Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Zotna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, and Nicolas Heess. Robocat: self-improving foundation agent for robotic manipulation, 2023.
Delphine Bresch-Pietri and Miroslav Kistic. Adaptive trajectory tracking despite unknown input delay and plant parameters. Automatica, 2509):2074-0881, 2009.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang- Wei Edward Lee, Sergey Levine, Yao Lu. Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: Vision lanionllanuage-acttt models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023a.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jomell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-L: robotics transformer for real-world control at scale. Proceedings of Robotics: Science and Systems (RSS), 2023b.
10

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2001.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Learning video-conditioned policies for unseen manipulation tasks, 2023.
Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation. Proceedings of Robotics: Science and Systems (RSS), 2023.
Danny Driess, Fei Xia, Mehdi M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023.
Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. 1994.
M Maurice Fréchet. Sur quelques points du calcul fonctionnel. Rendiconti del Circolo Malematico di Palermo (1884-1940), 22(1):1-72, 1906.
David Fridovich-Keds Keil, Sylvia L Herbert, Jaime F Fisac, Sampada Deglurkar, and Claire J ''oolii. Planning, fast and slow: framework for adaptive real-time safe trajectory planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 387-394. IEEE, 2018.
Montserrat Gonzalez Arenas, Ted Xiao, Sumeet Singh, Vidhi Jain, Allen Ren, Quan Vuong, Jake Varley, Alexander Herzog, Isabel Leal, Sean Kirmani, Mario Prats, Dorsa Sadigh, Vikas Sindhwani, Kanishka Rao, Jacky Liang, and Andy Zeng. How to prompt your robot: A promptbook for manipulation skills with code as policies. 2023.
Felix Hill, Sona Mokrá, Nathaniel Wong, and Tim Harley. Human instruction-follooingg with deep reinforcement learning via transfer-leaming from text. CoRR, abs/2005.09382, 2020. URL
httpa/iwww.....aooaaaaaaaaaaa/aa wttp://////../o/////33 09382..
Rachel M Holladay and Siddhartha S Srinivasa. Distance metrics and algorithms for task space path optimization. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5533-5540. IEEE, 2016.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, 2023.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning, 2023.
Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. Conference on Robot Learning (CoRL), 2021.
Mina Kamel, Thomas Stastny, Kostas Alexis, and Roland Siegwart. Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system. Robot Operating System (ROS) The Complete Reference (Volume 2), pp. 3 39, 2017.
Kamal Kant and Steven W Zucker. Toward efficient trajectory planning: The path-velocity decomposition. The international journal of robotics research, 5(3):72-89, 1986.
Mitsuo Kawato. Internal models for motor control and trajectory planning. Current opinion in neurobioloyy, 96):718-777, 1999.
11

---

Zhijun Li, Jun Deng, Renquan Lu, Yong Xu, Jianjun Bai, and Chun-Ys Yi Su. Trajectory-trackin control of mobile robot systems incorporating neural-dynamic optimized model predictive approach. IEEE Transactions Systems, Man, and Cybernetics: Systems, 46(6):740-749) 2015.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In arXiv preprins arXfv:2209.0773 2022.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, Wan-Teb Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. Mediapipe: A framework for perceiving and processing reality. In Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR), 2019.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Piere Sermanet. Learning latent plans from play. Conference on Robot Learning (CoRI.), 2019.
Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg. Dex-net 2.0: Deep learning plan robust grasps with synthetic point clouds and analytic grasp metrics. Proceedings of Robotics; Science and Systems (RSS), 2017.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin. What matters learning from offline human demonstrations for robot manipulation. Conference on Robot Learning (CoRL), 2021.
Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation, 2021.
OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL https://api.
semant i cscholas ar. www/Corpps/aaooooooooooooooop ID: 257532815.
Sanghyuk Park, John Deyst, and Jonathan How. A new nonlinear guidance logic for trajectory tracking. In AIAA guidance, navigation, and control conference and exhibit, pp. 4900, 2004.
Lemel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In IEEE international conference on robotics and automation (ICRA), 2016.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systemx, 1988.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation, 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL., 2022.
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Huei Lee, Quan Vuong, Paul Wohlhart, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman. Open-world object manipulation using pre-trained vision-language models, 2023.
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The magical benchmark for robust imitation. Advances in Neural Information Processing Systems, 33:18284-1822 2020.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 1433-1440. IEEE, 2016.
Grady Williams, Andrew Aldrich, and Evangelos A Theodorou. Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):44-335 2017.
12

---

Ted Xiao, Harris Chan, Pierre Sermanet, Ayzaan Wahid, Anthony Brohan, Karol Hausman, Sergey Levine, and Jonathan Tompson. Robotic skill acquistion via instruction augmentation with lisoonlanguaage models. In Proceedings of Roboties: Science and Systems (RSS), 2023.
Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprini crXiv:2307.0355) 2023.
Jong-Min Yang and wheeled mobile robots. IEEE Transactions on robotics and automation, 15(3):578- -77888577777 1999.
Kim. Sliding mode control for trajectory tracking of nonholonomic
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M. Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience. In arXiv preprint 1xXiv:2302.11550 2023.
Weiming Zhi, Tianyi Zhang, and Matthew Johnson-Roberson. I Learning from demonstration via probabilistic diagrammatic teaching. arXiv preprint arXiv22309003835, 2023.
Zongyu Zuo and Chenliang Wang. Adaptive trajectory tracking control of output constrained multi-rotors systems. IET Control Theory & Applications, 8(13):1163 11741144,411, 2014.
13

---

A EXPERIMENT DETAILS
A.1 SEEN SKILLS
Skill
Count
Description
Example Instruction
Pick Objjct
17
Lift the objectt off the surface Move the finst object near second
pick cuke
Move Object Near Object
337
mave pepsi can near rxhar blucherry
Place Ubject Upright
Place an elengated object upright
place bottle upright
Krock Ooject Over
Knock an elongated ohject over
knock redball can over
Open Drawer
Open any
cabinet drawers
open top drwer
Close Drawer
Close of the cabinct drawers
close the middle drawer
Place ubject into seceptacie
84
Place an abject into receptacle
place brown chip bug into white bowl
Pick object from Receptsole
82
Pick object from location and then place
pick green jalapeno chip from paper bowl and place on counter
and Place on the Counter
on the counter
Total
542
Table 2: The list of seen training tasks with their descriptions and example language instructions. Language instructions are only used for language-conditione baselines. "Count" refers to the number of distinct tasks per skill (e.g., Pick cake can and Pick apple are two different tasks).
A.2 UNSEEN SKILLS
Skill
Count
Description
Example instraction
Place Fruit
Place fruit into container
place orunge into basket
Upright and Move
6
Place an object upright and move it near another drawer Place objects into the desited position the drawer
place green can upright near pepsi can
within
object
near 7up top drawer place coke can into the right of drawer pick apple chair
Restack Drawer
12
chair Fold the towel by moving one comnr to another
object placed
Fold Towel
fold towell from bottom right push the chair
Swivel Chair
Swivel the office chair
Table 3: The list unseen evaluation tasks with their descriptions and example language instructions. Language instructions only used for language-condttione baselines. "Count" refers to the number of scenes collected for evaluation.
A.3 QUANTITATIVE RESULTS FOR UNSEEN TASKS
Task
RT-Traj (2D)
RT-Traj (2.5D)
RT-1
RT-2
RT-I-gou
Place Fruit
75%
75%
0%%
33%
8%
Upright and Move
33%
50%
17%
0%
0%
Move within Drawer
67%
100%
33%
0%
17%
Restock Drawer
92%
67%
42%
17%
42%
Pick from Chair
0%
38%
0%
0%
17%
Fold Towel
75%
75%
0%
0%
0%
Swivel Chair
0%
70%
17%
0%
50%
Overall
50%
67%
17%
11%
26%
Table 4: Success rates for unseen tasks when conditioning with human drawn sketches.
B IMPLEMENTATION DETAILS FOR DIFFERENT INPUT MODALITIES
B. GUI FOR HUMAN-DRAWN TRAJECTORY SKETCHES
As the main trajectory generation method we study is user-specified trajectory drawings, develop graphical user interface (GUI) for users to draw trajectory sketches. See Fig. 8 for example. Given the curmnnt robot camera image, can drag and move the mouse draw curves the canvas. Then,
14

---

they click the canvas to add markers to indicate gripper closing or opening actions. Additionally, the UI interface also supports simple height annotation. Users can specify the desired height values for pixels they select the canvas. This height value will be assigned to the closest point the drawn 2D trajectory. For unannotated points on the 2D trajectory, we interpolate their height values according to annotated ones.
High Vie
01.53 Catss G93 DKT Ole
Figure 8: Left: The GUI for users to draw trajectory sketches given the robot's current camera image. The 2D trajectory directly drawn by manual input, which can then be annotated with interaction markers or waypoints corresponding to user-specified heights. Right: The resulting height-aware trajectory sketch generated according to the output of the UI.
B.2 COLLECTING HUMAN-DRAWN TRAJECTORY SKETCHES
For each scene, we use held-out RT-Trajectory (2.5D) policy to explore different trajectory "prompts" given budget of trials, and save the trajectory sketch of the first successful episode. We refer such process as "prompt engineering" (Sec. 4.5). all attempts fail, we just save the trajectory sketch from the last episode. RT R7-Trajectory policies used for evaluation trained with different random seeds and evaluated with the saved trajectory sketches as conditioning. Note that we observe that our evaluated policies can have non-zero success rates on the scenes where fail to find successful episode during "prompt engineering",
B.3 HUMAN HAND POSE ESTIMATION
We employ Mediapipe (Lugaresi et al., 2019) to detect the human hand pose represented as 21 landmarks from the 2D image each video frame. The two landmarks on the thumb and another two landmarks on the index finger are used to represent parallel gripper. The 2D landmarks are lifted to 3D given the depth map.
then interpolate the end-effector pose from these four points. We manually annotate key frames at which the hand begins to grasp and release the target object. Given estimated end-effector poses and key frames for interaction, we can generate a trajectory sketch per video.
B.4 IMPLEMENTATION DETAILS FOR RT-I-Goal
The network architecture of RT-1-Godl is the same as RT-Trajectory, except a goal image used instead of trajectory sketch. To acquire goal conditioning for training. we use the last observation of each episode as the goal image for all frames the episode. For the quantitative comparison in Sec. 4.2, the image of the last step of the episode (App. B.2) used to generate the trajectory sketch for each scene is saved as the goal conditioning for evaluation.
MOTION DIVERSITY ANALYSIS
C.I COMPUTATION TRAJECTORY SIMILARITY
To measure the distance between two end-effector motion trajectories, we employ the Fréchet dis- tance (Fréchet, 1906; Eiter & Mannila, 1994), measure that quantifies the similarity between two curves by finding the minimum "leash length" required for two agents traversing each curve simultancously while maintaining their respective temporal order. As a well-adopted similarity measure computer vision and vehicle tracking (Borgefors, 1984), Fréchet distance may be reasonable choice for comparing 3D robot.
15

---

Top Drowar (sein tick))
Seevol
Jproft
close drawel
moove
knook-X
pick
place
knoos
place X
picx
knnool
dramer
Plaos Fruit
Rostook Drowor
Move DrsweT
place
place X Upright
place drawer
into middle
place
from drewer
close draaver
open
place X dramer dinser
close top pick x from drawer
open BOME
closa drawner
dinwerr
Figure 9: Semantic relevance measures how the semantic skills of rollout trajectories compare to the semantic skills of the most similar training trajectories, as measured by motion similarity. For the seen skill (close top drawer), the most similar training trajectories are largely of the same semantic skill. For the unseen skills, the most similar training trajectories are composed of semantically different tasks.
30
7.5
20
20
5.0
2.5
20
0.0
-25
(a) "close top drawer" (from Training Dataset)
(b) "place fruit"
(c) "move within drawer" (d) "pick from chair"
Figure 10: First-interaction height alignment compares the relative difference between the z-height of the first gripper interactions of query trajectories to the first gripper interactions of the most similar training trajectories, measured by motion similarity. The red line represents the baseline relative difference of the query trajectory compared with itself, which would be a difference of 0.0. The unseen skills in general see large variance the difference of first-interaction heights of the query trajectories compared to the most similar training trajectories.
end-effector waypoint trajectories since it is order-preserving and parameterization independent (Holladay Srinivasa, 2016).
Specifically, consider two trajectories and r' where each trajectory contains waypoints T {Po. P1, ..., Pm) and -' r'= (= and d(pt, pt) is a distance measure like Euclidean distance. Then, using the notation [1: denote removing the first element and returning the rest of the sequence r, the Fréchet distance FD is recursively defined as:
FD(T,7') = (ttp::/.))...mmf/////./r/l//l/ad
this work, each waypoint is the sensed end-effector center position and the distance measure is Euclidean distance. Note that the orientation and interaction (closing/opening action) are not taken into consideration.
C.2 ADDITIONAL SAMPLES OF TRAJECTORY SIMILARITIES
Figure 12 shows additional examples of evaluation trajectories and their most similar trajectories in the training dataset.
D ADDITIONAL VISUALIZATION
Fig. 13 visualizes the example rollouts of unseen skills. Fig. 14 demonstrates more example trajectories from image generation models and correpoondnng rollouts.
16

---

025
0.30
013
0.00
200
90
60
(a) "close top drawer" (from Training Dataset)
(b) "place fruit"
(c) "move within drawer" (d) "pick from chair"
We visualize the distribution of Fréchet distances of query trajectories to the most similar training trajectories, as measured by motion similarity. The red line represents the median of the average distance between evaluation trajectories and the most similar training trajectories. Query trajectories of unseen skills in general see larger Fréchet distances the most similar training trajectories, compared to query trajectories from training skills.
Figure 11:
Fig 16 shows the rollouts of example evaluations in realistic scenarios mentioned in Sec. 4.5. We showcase additional evaluations in Fig. 17. Notably, we find that RT-Trajectory is quite robust various simultancous visual distribution shifts including conditions, new objects, and new furniture textures. addition, these realistic "in the wild" evaluations were ran in controlled laboratory environments, so the evaluations often required generalization to new heights or furniture geometries (different shaped drawers, cabinets, or tables).
buildings,
backgrounds, new distractors, new lighting
E CASE STUDIES IN EMERGENT CAPABILITIES AND BEHAVIORS
E. PROMPT ENGINEERING
Fig. 18 illustrates two examples of prompt engineering mentioned in Sec. 4.5. For instance, if the user wants to prompt RT- RT-Trajectory to place an object at a high position, it is better to draw trajectory that first reaches higher peak, and then move downward the target.
E.2 RETRY BEHAVIOR
Compared to non-learning methods, RT-Trajectory is able to recover from execution failures. Fig. 19 illus- trates the retry behavior emerged when RT-Trajectory is opening the drawer given the trajectory sketch gener- ated by prompting I.I with Code Policies (CaP) mentioned in Sec. 3.4. After failure attempt open the drawer by its handle, the robot retried to grasp the edge of the drawer, and managed to pull the drawer.
E.3 HEIGHT-AWARE DISAMBIGUATION FOR RT-Trajectory (2.5D)
trajectories (without depth information) are visually ambiguous for distinguishing whether the robot should move its arm deeper higher. We find that height-aware color grading for RT RT-Trajectory (2.5D) can effectively help reduce such ambiguity, as shown Fig. 20.
PROPERTIES OF TRAJECTORIES GENERATED BY DIFFERENT APPROACHES
Fig. shows the comparison of trajectories generated by different methods.
17

---

13
18
57
0.8
18
6.
3.5
-02
19
10-
80 6%
Figure 12: Evaluation trajectories for new skills and their closest trajectories from the training set. Each row shows three frames of skill evaluation rollout, with the executed trajectory and similar training set trajectories overlaid, as well as depicting the trajectories in an orthographic projection in robot base frame looking at the robot from the front and the side. As can be seen, the policy is able to follow the desired trajectories closely and achieve the tasks. While in many cases, in particular in image space, some of the similar trajectories from the training set look very close the executed trajectory, the front and side view rows to 4 reveal that the policy at some crucial point has to and successfully does deviate from what it has during training. F.g., in a bottle, while nearest training trajectories are from close mi didle drawer, which doesn't 't move the gripper down far enough. Additionally, row is example where for swivel chair prompt trajectory there coincidentally are many very closely matching move X near training trajectories. However, the prompt here specifies not close the gripper the first contact point, which the policy is able to respect.
the prompt says go all the way down to pick up
18

---

Figure 13: Example rolllouts of unseen skills. The trajectory sketch overlaid on the initial image is visualized. From top to bottom: Place Fruit, Upright and Move, Fold Towel, Move within Drawer, Restock Drawer, Pick from Chair, Swivel Chair.
19

---

Figure 14: Example trajectories from image generation models. Each row shows the trajectory sketch
overlaid on the first frame and the rollout. The language instructions open miedle drawer, place orange into middle drawer, move Tup can near blue plastic bottle.
(a)
(b)
(c)
(d)
Figure 15: Example RT-Trajectory evaluations in realistic scenarios involving (a) novel articulated objects requiring new motions, (b) manipulation on new surfaces in new buildings in new heights, (c) interacting with pivot-hinge cabinet despite training only on sliding-hinge drawers, and (d) circuitous tabletop patterns extending beyond direct paths in the training dataset. Full rollouts are shown in Fig. 16 and the supplemental video at https: /ttp//ttttttttt trrattraa jectory github.. io/.
20

---

(a)
(b)
(c)
(d)
(e)
(f)
(g)
Figure 16: Qualitative examples of emergent capabilities of RT- RT-Trajectory in realistic scenarios beyond the training settings: (a) new articulated objects requiring novel motion strategies, (b) new circuitous motions requiring multiple turns, (c) new living room setting with a new height, object, and background, (d) new bathroom setting with precise picking from cup, (e) new bedroom setting with drawer at new height, (f) new kitchen setting with an unseen pan requiring placement onto new stove, and (g) new kitchen setting with new pivot hinge that requires new motion for opening and closing.
21

---

Figure 17: Visualizing additional interesting examples of RT-Trajectory's 's generalization performance in new scenarios. These include a novel kitchen room setting with unseen cup and unseen placemat, a new living room room setting with new manipulation objects with new furniture pieces in new heights, and bathroom setting with harsh lighting and different table height.
22

---

(a) The objective is place the apple the middle stage.
(b) The objective to place the apple onto the top stage.
Figure 18: Case studies in prompt engineering. Each row shows the trajectory sketch overlaid on the first frame and the corresponding rollout. As seen in the first two rows, suboptimal trajectory prompts result in failures. However, by keeping the initial scene conditions identical but simply improving the trajectory prompt, the policy able to succeed.
Figure 19: Example of retry behavior. The first image the trajectory sketch generated from the CaP overlaid on the initial observation. The remaining images show the rollout. The robot first attempts open the drawer by grasping its handle, but fails (2nd image). Then, it retries to open the drawer by grasping the edge instead.
23

---

Figure 20: Comparison between RT-Trajectory (2D) and RI-Trajectory (2.5D). Given the same 2D trajectory generated by the CaP, RT-Trajectory (2.5D) lifts the object while RT-Trajectory (2D) moves the object to deeper position due to ambiguity 2D trajectory.
Pick green jalapeno chip bag
Pick robar blueberry
Move 7up near
plassic
Pick orange can from top drawer and place
on counter
orange
Initial Image
Human drawing
Human demo video
Prompting LLMs
Image generation model
Comparison of trajectories generated from different methods
Figure 21: Comparison of trajectories generated by different methods. Each row shows the initial image (Ist column) followed by trajectory sketches overlaid on it.
24


Reviews:
[{"review_id": "qftr6CvxsW", "reviewer": "Reviewer_zAA2", "summary": "The paper proposes to abstract robot policies via rough sketches. The paper identify sketeches as a modality that is concise but yet expressive enough to represent a task, and allow for generalisation to novel tasks. The trajectory sketch in question in projected into the view of a calibrated camera. The paper explores when the trajectories are hand-drawn, derived from demonstration videos, prompted from LLMs, and generated from image generation models.\n\nThe paper is highly novel in using sketches as the abstraction for which tasks are conditioned on, and, by extensive experimental results, show the benefits of this representation for policy learning. The method appears sound, and the reviewer also appreciates the detailed evaluation on real world robot experiments and the discussion on emergent capabilities -- the retry behaviour was particularly interesting to observe. A question that might be interesting to ponder about is: does the smoothness or other property of the trajectory affect the quality of the task conditioning? Although abstracting the task as a sketch and conditioning on it is novel. The reviewer would like to point out that there have been previous attempts at leveraging human sketches to provide robot demonstrations in a paradigm known as \"diagrammatic teaching\" (https://arxiv.org/abs/2309.03835; https://openreview.net/forum?id=6cUysxHoL1), and should be viewed as related work.\n\nOverall, this paper is novel, the method is sound and convincing with a solid amount of real robot experiments. The usage of trajectory sketches as a way of specifying tasks has the potential to produce many new oportunities in robot learning.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "See above", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIuC5CCSl4", "reviewer": "Reviewer_Ryeg", "summary": "The paper proposes to use a rough trajectory of the end-effector in the image space as a way of task specification. This enables more flexible task specification during inference time, including a human drawing, with a human video demonstrating the task, or from LLM output. Experiments validate the effectiveness of the method and additionally demonstrate a small amount of motion generalization.", "questions": "Figure 8 is confusing to me. From the left visualization, the query trajectories are well captured by the training trajectories, suggesting no extrapolation and mostly interpolation. On the other hand, the right part is difficult to understand. Can the authors clarify or change the claim accordingly?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated, with a simple but effective modification to task specification. The paper makes a meaningful contribution to the literature and is well executed. \n2. The paper is well presented, with good writing and visuals.", "weaknesses": "1. Generating the trajectory itself might not be trivial. For example, in the case when a person draws the trajectory, the person needs to understand the trajectory of the end-effector (including any joint limits) and project it to a 2D space. This can become a bottleneck, especially for more difficult tasks (e.g. large-range motion, long-horizon tasks). More analysis of the robustness of the trajectory specification could strengthen the paper. \n\n2. As mentioned in the limitation section already, the method seems to be limited to a fixed-view camera with calibration, limiting the applicability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGSEdb2IwG", "reviewer": "Reviewer_YmAB", "summary": "This paper proposes an approach to task specification based on providing, at test time, an approximate trajectory to accomplish the task. More specifically, the trajectory is provided as a 2D image, representing the projection on the camera of the end-effector position, the interaction points (where the gripper should be opened and closed), and possibly the height of the end effector. I see the approach as an instantiation of the general problem of trajectory tracking under constraints. The main contribution is adapting this idea to make it compatible with sensorimotor models and imitation learning. The approach is evaluated over multiple experiments in the real world, where it outperforms baselines without test-time goal conditioning and end-goal conditioning only.", "questions": "The main actions to improve the paper are in my opinion:\n\n1. Better positioning of the work with respect to previous literature.\n2. Motivation behind task selection\n3. Quantitative experiments in the setup of Sec. 4.5", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are:\n1. The proposed approach, despite being a small extension of the well-known body of literature on trajectory tracking, is original. Using a trajectory to define the task at test time is the default for all control systems. The fact that the trajectory is not feasible is generally not a problem for feedback systems (See Borrelli et al., Predictive control for linear and hybrid systems). However, representing such a trajectory as an image and conditioning a model on it is an interesting idea.\n2. While I don't agree with some of the experimental setup (more on this below), I found the analysis on trajectory conditioning (Sec 4.3) and measuring generalization (Sec. 4.4) to be quite interesting and well executed.\n3. The paper writing is very good and easy to follow. The figures help in understanding the core concepts.", "weaknesses": "The main weaknesses I see are:\n1. The paper does not make a strong enough connection between the traditional idea of trajectory tracking and the proposed approach. For example, there is no section in the related work covering the history of this technique and its recent applications to robot learning. As mentioned above, I still see some innovations in the proposed approach (sensorimotor controllers are generally conditioned on a low-dimensional trajectory, not an image of it)—however, this lack of perspective results in overclaiming the system's novelty.\n2. The comparison against baselines in Sec 4.2 is not apples to apples: while the proposed approach gets an input a trajectory on __how__ to complete the task, the other methods only access a higher level instruction (e.g. language), or the final goal represented as an image. It is, therefore, not surprising that with more and cleaner information, the proposed approach outperforms the baselines. I find the experiments in Table I to be much more informative. They show that when the trajectory is very coarse, e.g. when drawn by a human, the proposed approach can track it better than the off-the-shelf IK. However, these results also show that the approach is, on average, worse than generating a trajectory with a VLM and tracking it with an IK, which somehow is against the main paper's contribution.\n3. Each subsection seems to evaluate on different \"tasks\", and it is unclear how the task selection is done. For example, why change the task from 4.2 to 4.3? And why even change it in the comparisons of Table 1? Note that my point is not that the task were handpicked, but rather that this dynamics makes drawing insights from experiments challenging.\n4. The paper has as its main claim generalization. Still, its main experiments on generalization over scenes and tasks only occupy a small part of the paper (Sec. 4.5). At the very least, this part should become quantitative since this is what would back up the claims on generalization.\n\nSome rather minor limitations:\n1. The ordering of Fig. 2 seems rather arbitrary. For example, in my subjective opinion, I see trajectory and code to be much closer than what is shown right now. Therefore, either more justification is needed, or the figure should be removed. \n2. The approach makes strong assumptions that contradict the claim on scalability. Specifically, the two main assumptions are: (1) the camera should be static at training and test time, as well as fully calibrated, and (2) the end effector should always be visible in the camera. This makes data collection and evaluation very challenging and limits the set of tasks that can be done.\n3. In Fig. 8, I don't get what \"unrelated semantic categories\" mean for a low-level skill. At the level of trajectory, why is generalization over semantics a question? Isn't it obvious due to the abstraction of the skill? Or did I misunderstand the meaning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)