RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 19:15:47

Abstract: Natural language and images are commonly used as goal representations in goal-conditioned imitation learning. However, language can be ambiguous and images can be over-specified. In this work, we study hand-drawn sketches as a modality for goal specification. Sketches can be easy to provide on the fly like language, but like images they can also help a downstream policy to be spatially-aware. By virtue of being minimal, sketches can further help disambiguate task-relevant from irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of trajectories paired with synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch performs comparably to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch handles sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please visit http://rt-sketch.github.io.

Main paper: 
 introduction: 
 Robots operating alongside humans in households, workplaces, or industrial environments have an immense potential for assistance and autonomy, but careful consideration is needed of what goal representations are easiest for humans to convey to robots, and for robots to interpret and act upon.Instruction-following robots attempt to address this problem using the intuitive interface of natural language commands as inputs to language-conditioned imitation learning policies (Brohan et al., 2023b; a; Karamcheti et al., 2023; Lynch & Sermanet, 2020; Lynch et al., 2023) . For instance, imagine asking a household robot to set the dinner table. A language description such as "put the utensils, the napkin, and the plate on the table" is under-specified or ambiguous. It is unclear how exactly the utensils should be positioned relative to the plate or the napkin, or whether their distances to each other matter or not. To achieve this higher level of precision, a user may need to give lengthier descriptions such as "put the fork 2cm to the right of the plate, and 5cm to the leftmost edge of the table.", or even online corrections ("no, you moved too far to the right, move back a bit!") (Cui et al., 2023; Lynch et al., 2023) . While language is an intuitive way to specify goals, its qualitative nature and ambiguities can make it both inconvenient for humans to provide without lengthy instructions or corrections, and for robot policies to interpret for downstream precise manipulation.On the other hand, using goal images to specify objectives and training goal-conditioned imitation learning policies either paired with or without language instructions has shown to be quite successful in recent years (Jiang et al., 2022; Jang et al., 2022) . In these settings, an image of the scene in its desired final state could fully specify the intended goal. However, this has its own shortcomings: access to a goal image is a strong prior assumption, and a pre-recorded goal image can be tied to a particular environment, making it difficult to reuse for generalization. Between natural language, which lacks granularity to unambiguously specify goals, and images, which overspecify goals in unnecessary detail, leading to the need for internet-scale data for generalization, we recognize that current frameworks lack a goal representation which adequately captures user intent in a convenient yet expressive manner. While natural language is highly flexible, it can also be highly ambiguous or require lengthy descriptions. This quickly becomes difficult in long-horizon tasks or those requiring spatial awareness. Meanwhile, goal images over-specify goals in unnecessary detail, leading to the need for internet-scale data for generalization.To this end address these challenges, we study hand-drawn sketches as a convenient yet expressive modality for goal specification in visual imitation learning. By virtue of being minimal, sketches are still easy for users to provide on the fly like language Yet unlike language, they (1) provide more, but allow for more spatial-aware task specification. Like goal images, sketches readily integrate with off-the-shelf policy architectures that take visual input, but provide an added level of goal abstraction that ignores unnecessary pixel-level details. Finally, the quality and selective inclusion/exclusion of details in a sketch can help a downstream policy distinguish task relevant from irrelevant details. without needing to faithfully preserve pixel-level details as in an image, and (2) help a downstream policy disambiguate task-relevant from -irrelevant objects based on their selective inclusion, exclusion, or level of detail. Furthermore, sketches readily integrate with off-the-shelf policy architectures that take visual representations as input.In this work, we present RT-Sketch, a goal-conditioned policy for manipulation that takes a userprovided hand-drawn sketch of the desired scene as input, and outputs actions. The novel architecture of RT-Sketch modifies the original RT-1 language-to-action Transformer architecture (Brohan et al., 2023b) to consume visual goals rather than language, allowing for flexible conditioning on sketches, images, or any other visually representable goals. To enable this, we concatenate a goal sketch and history of observations as input before tokenization, omitting language. We train RT-Sketch on a dataset of 80K trajectories paired with synthetically produced goal sketches, generated by an image-to-sketch stylization network trained from a few hundred image-sketch pairs. We evaluate RT-Sketch across six manipulation skills on real robots involving tabletop object rearrangements on a countertop with drawers, subject to a wide range of scene variations. These skills include moving objects near to one another, knocking a can sideways, placing a can upright, closing a drawer, and opening a drawer. Experimentally, we find that RT-Sketch performs on a similar level to image or language-conditioned agents in straightforward settings. When language instructions are ambiguous, or in the presence of visual distractors, we find that RT-Sketch achieves ∼ 2X more spatial precision and alignment scores, as assessed by human labelers, over language or goal imageconditioned policies (see Fig. 1 (right) ). Additionally, we show that RT-Sketch can handle different levels of input specificity, ranging from rough sketches to more scene-preserving, colored drawings (see Fig. 1 (left)). 
related work: 
 In this section, we discuss prior methods for goal-conditioned imitation learning which operate on traditional goal representations. We also highlight ongoing efforts towards image-sketch conversion, which open new possibilities for goal-conditioning modalities which are underexplored in robotics.Goal-Conditioned Imitation Learning Despite the similarity in name, our learning of manipulation policies conditioned on hand-drawn sketches of the desired scene is different from the notion of policy sketches (Andreas et al., 2017) , symbolic representations of task structure describing its subcomponents. Reinforcement learning (RL) is not easily applicable in our scenario, as it is nontrivial to define a reward objective which accurately quantifies alignment between a provided scene sketch and states visited by an agent during training. We instead focus on imitation learning (IL) techniques, particularly the goal-conditioned setting (Ding et al., 2019) .Goal-conditioned IL has proven useful in settings where a policy must be able to handle spatial or semantic variations for the same task (Argall et al., 2009) . These settings include rearrangement of multiple objects (Brohan et al., 2023b; a; Lynch et al., 2023; Manuelli et al., 2019) , kitting (Zakka et al., 2020) , folding of deformable objects into different configurations (Ganapathi et al., 2021) , and search for different target objects in clutter (Danielczuk et al., 2019) . However, these approaches tend to either rely on language (Brohan et al., 2023b; Lynch & Sermanet, 2020; Lynch et al., 2023; Karamcheti et al., 2023; Shao et al., 2020) , or goal images (Danielczuk et al., 2019) to specify variations. Follow-up works enable multimodal conditioning on either goal images and language (Jang et al., 2022) , in-prompt images (Jiang et al., 2022) , or image embeddings (Manuelli et al., 2019; Zakka et al., 2020; Ganapathi et al., 2021) . However, all of these representations are ultimately derived from raw images or language in some way, which overlooks the potential for more abstract goal representations that are easy to specify but preserve spatial awareness, such as sketches.In addition to their inflexibility in terms of goal representation, goal-conditioned IL tends to overfit to demonstration data and fails to handle even slight distribution shift in new scenarios (Ross et al., 2011) . For language-conditioning, distribution shift can encompass semantic or spatial ambiguity, novel instructions or phrasing, or unseen objects (Jang et al., 2022; Brohan et al., 2023b) . Goalimage conditioning is similarly susceptible to out-of-distribution visual shift, such as variations in lighting or object appearances, or unseen background textures (Burns et al., 2022; Belkhale et al., 2023) . We instead opt for sketches which are minimal enough to combat visual distractors, yet expressive enough to provide unambiguous goals. Prior work, including (Barber et al., 2010) and (Porfirio et al., 2023) , have shown the utility of sketches over pure language for navigation and limited manipulation settings. However, the sketches explored in these works are largely intended to guide low-level motion at the joint-level for manipulation, or provide explicit directional cues for navigation. Cui et al. (2022) considers sketches amongst other modalities as an input for goalconditioned manipulation, but does not explicitly train a policy conditioned on sketches. They thus came to the conclusion that the scene image is better than the sketch image at goal specification. Our result is different and complementary, in that policies trained to take sketches as input outperform a scene image conditioned policy, by 1.63x and 1.5x in terms of Likert ratings for perceived spatial and semantic alignment, subject to visual distractors.sketches have gained increasing popularity within the computer vision community for applications such as object detection (Chowdhury et al., 2023a; Chowdhury et al., 2023; Chowdhury et al., 2022) , visual question answering (Qiu et al., 2022; Qiu et al., 2023) , and scene understanding (Chowdhury et al., 2023b) , either in isolation or in addition to text and images. When considering how best to incorporate sketches in IL, an important design choice is whether to take sketches into account (1) at test time (i.e., converting a sketch to another goal modality compatible with a pre-trained policy), or (2) at training time (i.e., explicitly training an IL policy conditioned on sketches). For (1), one could first convert a given sketch to a goal image, and then roll out a vanilla goal-image conditioned policy. This could be based on existing frameworks for sketch-to-image conversion, such as ControlNet (Zhang & Agrawala, 2023) , GAN-style approaches (Koley et al., 2023) , or text-to-image synthesis, such as InstructPix2Pix (Brooks et al., 2023) or Stable Diffusion (Rombach et al., 2022) . While these models produce photorealistic results under optimal conditions, they do not jointly handle image generation and style transfer, making it unlikely for generated images to match the style of an agent observations. At the same time, these approaches are susceptible to producing hallucinated artifacts, introducing distribution shifts (Zhang & Agrawala, 2023) .Based on these challenges, we instead opt for (2), and consider image-to-sketch conversion techniques for hindsight relabeling of terminal images in pre-recorded demonstration trajectories. Recently, Vinker et al. (2022b; a) proposes networks for predicting Bezier curve-based sketches of input image objects or scenes. Sketch quality is supervised by a CLIP-based alignment metric. While these approaches generate sketches of high visual fidelity, test-time optimization takes on the order of minutes, which does not scale to the typical size of robot learning datasets (hundreds to thousands of demonstration trajectories). Meanwhile, conditional generative adversarial networks (cGANs) such as Pix2Pix (Isola et al., 2017) have proven useful for scalable image-to-image translation. Most related to our work is that of Li et al. (2019) , which trains a Pix2Pix model to produce sketches from given images on a large crowd-sourced dataset of 5K paired images and line drawings. We build on this work to fine-tune an image-to-sketch model on robot trajectory data, and show its utility for enabling downstream manipulation from sketches. 
sketch-conditioned imitation learning: 
 In this section, we will first introduce our problem of learning a sketch-conditioned policy. We will then discuss our approach to train an end-to-end sketch-to-action IL agent. First, in Section 3.1, we discuss our instantiation of an auxiliary image-to-sketch translation network which automatically generates sketches from a reference image. In Section 3.2, we discuss how we use such a model to automatically hindsight relabel an existing dataset of demonstrations with synthetically generated goal sketches, and train a sketch-conditioned policy on this dataset.Problem Statement Our goal is to learn a manipulation policy conditioned on a goal sketch of the desired scene state and a history of interactions. Formally, we denote such a policy by π sketch (a t |g, {o j } t j=1 ), where a t denotes an action at timestep t, g ∈ R W ×H×3 is a given goal sketch with width W and height H, and o t ∈ R W ×H×3 is an observation at time t. At inference time, the policy takes a given goal sketch along with a history of RGB image observations to infer an action to execute. In practice, we condition π sketch on a history of D previous observations rather than all observations from the initial state at t = 1. To train such a policy, we assume access to a dataset D sketch = {g n , {(o n t , a n t )} T (n) t=1 } N n=1 of N successful demonstrations, where T (n) refers to the length of the n th trajectory in timesteps. Each episode of the dataset consists of a given goal sketch and a corresponding demonstration trajectory, with image observations recorded at each timestep. Our goal is to thus learn the sketch-conditioned imitation policy π sketch (a t |g, {o j } t j=1 ) trained on this dataset D sketch .Training a sketch-conditioned policy requires a dataset of robot trajectories that are each paired with a sketch of the goal state achieved by the robot. Collecting such a dataset from scratch at scale, including the trajectories themselves and manually drawn sketches, can easily become impractical. Thus, we instead aim to learn an image-to-sketch translation network T (g|o) that takes an image observation o and outputs the corresponding goal sketch g. This network can be used to postprocess an existing dataset of demonstrations D = {{(o n t , a n t )} T (n) t=1 } N n=1 with image observations by appending a synthetically generated goal sketch to each demonstration. This produces a dataset for sketch-based IL:D sketch = {g n , {(o n t , a n t )} T (n) t=1 } N n=1 .RT-1 Dataset In this work, we rely on an existing dataset of visual demonstrations collected by prior work (Brohan et al., 2023b) . RT-1 is a prior language-to-action imitation learning agent trained on a large-scale dataset (80K trajectories) of VR-teleoperated demonstrations that include skills such as moving objects near one another, placing cans and bottles upright or sideways, opening and closing cabinets, and performing pick and place on countertops and drawers (Brohan et al., 2023b) .Here, we repurpose the RT-1 dataset and further adapt the RT-1 policy architecture to accommodate sketches, detailed in Section 3.2.We acknowledge that there are innumerable ways for a human to provide a sketch corresponding to a given image of a scene. In this work, we make the following assumptions about input sketches for a controlled experimental validation procedure. In particular, we first assume that a given sketch respects the task-relevant contours of an associated image, such that tabletop edges, drawer handles, and task-relevant objects are included and discernible in the sketch. We do not assume contours in the sketch to be edge-aligned or pixel-aligned with those in an image. We do assume that the input sketch consists of black outlines at the very least, with shading in color being optional. We further assume that sketches do not contain information not present in the associated image, such as hallucinated objects, scribbles, or textual annotations, but may omit task-irrelevant details that appear in the original image.To train an image-to-sketch translation network T , we collect a new dataset i) different representations of the same image o i , in order to account for the fact that there are multiple, valid ways of sketching the same scene.D T = {(o i , g 1 i , . . . , g L (i) i )} M i=1 consisting of M image observations o i each paired with a set of goal sketches g 1 i , . . . , g L (i) i . Those represent L (To collect D T , we take 500 randomly sampled terminal images from demonstration trajectories in the RT-1 dataset, and manually draw sketches with black lines on a white background capturing the tabletop, drawers, and relevant objects visible on the manipulation surface. While we personally annotate each robot observation with a single sketch only, we add this data to an existing, much larger non-robotic dataset (Li et al., 2019) . This dataset captures inter-sketch variation via multiple crowdsourced sketches per image. We do not include the robot arm in our manual sketches, as we find a minimal representation to be most natural. Empirically, we find that our policy can handle such sketches despite actual goal configurations likely having the arm in view. We collect these drawings using a custom digital stylus drawing interface in which a user draws an edge-aligned sketch over the original image (Appendix Fig. 15 ). The final recorded sketch includes the user's strokes in black on a white canvas with the original image dimensions.We implement the image-to-sketch translation network T with the Pix2Pix conditional generative adversarial network (cGAN) architecture, which is composed of a generator G T and a discriminator D T (Isola et al., 2017) . The generator G T takes an input image o, a random noise vector z, and outputs a goal sketch g. The discriminator D T is trained to discriminate amongst artificially generated sketches and ground truth goal sketches. We utilize the standard cGAN supervision loss to train both (Li et al., 2019; Isola et al., 2017) :L cGAN = min G T max D T E o,g [log D T (o, g)] + E o,g [log(1 -D T (o, G T (o, g))](1)We also add the L 1 loss to encourage the produced sketches to align with the ground truth sketches as in (Li et al., 2019) . To account for the fact that there may be multiple valid sketches for a given image, we only penalize the minimum L 1 loss incurred across all L (i) sketches provided for a given image as in Li et al. (2019) . This is to prevent wrongly penalizing T for producing a valid sketch that aligns well with one example but not another simply due to stylistic differences in the ground truth sketches. The final objective is then a λ-weighted combination of the average cGAN loss and the minimum alignment loss:EQUATIONIn practice, we supplement the 500 manually drawn sketches from D T by leveraging the existing larger-scale Contour Drawing Dataset (Li et al., 2019) . We refer to this dataset as D CD , which contains 1000 examples of internet-scraped images containing objects, people, animals from Adobe Stock, paired with L (i) = 5 crowd-sourced black and white outline drawings per image collected on Amazon Mechanical Turk. Visualizations of this dataset are provided in Appendix Fig. 4 . We first take a pre-trained image-to-sketch translation network T CD (Li et al., 2019) trained on D CD , with L (i) = 5 sketches per image. Then, we fine-tune T CD on D T , with only L (i) = 1 manually drawn sketch per robot observation, to obtain our final image-to-sketch network T . Visualizations of the sketches generated by T for different robot observations are available in Fig. 5 . With a means of translating image observations to black and white sketches via T (Section 3.1), we can automatically augment the existing RT-1 dataset with goal sketches. This results in a dataset, which we refer to as D sketch , which can be used for training our algorithm, RT-Sketch.The original RT-1 datasetD lang = {i n , {(o n t , a n t )} T (n) t=1} N n=1 consists of N episodes with a paired natural language instruction i and demonstration trajectory {(o n t , a n t )} T n t=1 . We can automatically hindsight-relabel such a dataset with goal images instead of language goals (Andrychowicz et al., 2017) . Let us denote the last step of a trajectory n as T (n) . Then the new dataset with image goals instead of language goals isD img = {o n T (n) , {(o n t , a n t )} T (n) t=1 } N n=1, where we treat the last observation of the trajectory o n T (n) as the goal g n . To produce a dataset for π sketch , we can simply replaceo n T (n) with ĝn = T (o n T (n) ) such that D sketch = {ĝ n , {(o n t , a n t )} T (n) t=1 } N n=1 .To encourage the policy to afford different levels of input sketch specificity, we in practice produce goals by ĝn = A(o n T (n) ), where A is a randomized augmentation function. A chooses between simply applying T , T with colorization during postprocessing (e.g., by superimposing a blurred version of the ground truth RGB image over the binary sketch), a classical Sobel operator (Sobel, 1968) for edge detection, or not applying any operators, which preserves the original ground truth goal image (Fig. 2 ). By co-training on all representations, we intend for RT-Sketch to handle a spectrum of specificity going from binary sketches; colorized sketches; edge detected images; and goal images (Appendix Fig. 5 ).In our setting, we consider goals provided as sketches rather than language instructions as was done in RT-1. This change in the input representation necessitates a change in the model architecture. The original RT-1 policy relies on a Transformer architecture backbone (Vaswani et al., 2017) . RT-1 first passes a history of D = 6 images through an EfficientNet-B3 model (Tan & Le, 2019) producing image embeddings, which are tokenized, and separately extracts textual embeddings and tokens via FiLM (Perez et al., 2018) and a Token Learner (Ryoo et al., 2021) . The tokens are then fed into a Transformer which outputs bucketized actions. The output action dimensionality is 7 for the end-effector (x, y, z, roll, pitch, yaw, gripper width), 3 for the mobile base, (x, y, yaw), and 1 for a flag that can select amongst base movement, arm movement, and episode termination. To retrain the RT-1 architecture but accommodate the change in input representation, we omit the FiLM language tokenization altogether. Instead, we concatenate a given goal image or sketch with the history of images as input to EfficientNet, and extract tokens from its output, leaving the rest of the policy architecture unchanged. We visualize the RT-Sketch training inputs and policy architecture in Fig. 2 . We refer to this architecture when trained only on images (i.e., an image goal-conditioned RT1 policy) as RT-Goal-Image and refer to it as RT-Sketch when it is trained on sketches as discussed in this section.Training RT-Sketch We can now train π sketch on D π sketch utilizing the same procedure as was used to train RT-1 (Brohan et al., 2023b) , with the above architectural modifications. We fit π sketch using the behavioral cloning objective function. This aims to minimize the negative log-likelihood of an action provided the history of observations and a given sketch goal (Torabi et al., 2018) :J(π sketch ) = N n=1 T (n) t=1 log π sketch (a n t |g n , {o j } t j=1 ) 
experiments: 
 We seek to understand the ability of RT-Sketch to perform goal-conditioned manipulation as compared to policies that operate from higher-level goal abstractions like language, or more overspecified modalities, like goal images. To that end, we test the following four hypotheses:H1: RT-Sketch is successful at goal-conditioned IL. While sketches are abstractions of real images, our hypothesis is that they are specific enough to provide manipulation goals to a policy. Therefore, we expect RT-Sketch to perform on a similar level to language goals (RT-1) or goal images (RT-Goal-Image) in straighforward manipulation settings.H2: RT-Sketch is able to handle varying levels of specificity. There are as many ways to sketch a scene as there are people. Because we have trained RT-Sketch on sketches of varying levels of specificity, we expect it to be robust against variations of the input sketch for the same scene.H3: Sketches enable better robustness to distractors than goal images. Sketches focus on taskrelevant details of a scene. Therefore, we expect RT-Sketch to provide robustness against distractors in the environment that are not included in the sketch compared to RT-Goal-Image that operates on detailed image goals.H4: Sketches are favorable when language is ambiguous. We expect RT-Sketch to provide a higher success rate compared to ambiguous language inputs when using RT-1.Policies We compare RT-Sketch to the original language-conditioned agent RT-1 (Brohan et al., 2023b) , and RT-Goal-Image, a policy identical in architecture to RT-Sketch, but taking a goal image as input rather than a sketch. All policies are trained on a multi-task dataset of ∼ 80K real-world trajectories manually collected via VR teleoperation using the setup from Brohan et al. (2023b) . These trajectories span a suite of common office and kitchen tasks such as picking and placing objects, reorienting cups and bottles upright or sideways, opening and closing drawers, and rearranging objects between drawers or a countertop.Evaluation protocol To ensure fair comparison, we control for the same initial and goal state of the environment across different policy rollouts via a catalog of well-defined evaluation scenarios that serve as references for human robot operators. For each scenario, we record an initial image (RGB observation) of the scene, the goal image (with objects manually rearranged as desired), a natural language task string describing the desired agent behavior to achieve the goal, and a set of hand-drawn sketches corresponding to the recorded goal image. At test time, a human operator retrieves a particular evaluation scenario from the catalog, aligns the physical robot and scene according to a reference image using a custom visualization utility, and places the relevant objects in their respective locations. Finally, the robot selects one of the goal representations (language, image, sketch, etc.) for the scenario as input to a policy. We record a video of the policy rollout for downstream evaluation (see Section 4.2). We perform all experiments using the Everyday Robot2 , which contains a mobile base, an overhead camera, and a 7-DoF manipulator arm with a parallel jaw gripper. All sketches for evaluation are collected with a custom manual drawing interface by a single human annotator on a tablet with a digital stylus.Performance Metrics Defining a standardized, automated evaluation protocol for goal alignment is non-trivial. Since binary task success is too coarse-grained and image-similarity metrics like frame-differencing or CLIP (Radford et al., 2021) tend to be brittle, we measure performance with Figure 3 : Goal Alignment Results: Average Likert scores for different policies rating perceived semantic alignment (Q1) and spatial alignment (Q2) to a provided goal. For straightforward benchmark manipulation tasks, RT-Sketch performs comparably and in some cases better than RT-1 and RT-Goal-Image in terms of both metrics, for 5 out of 6 skills (H1). RT-Sketch further exhibits the ability to handle sketches of different levels of detail (H2), while achieving better goal alignment than baselines when the visual scene is distracting (H3) or language would be ambiguous (H4). Error bars indicate standard error across labeler ratings.two more targeted metrics. First, we quantify policy precision as the distance (in pixels) between object centroids in achieved and ground truth goal states, using manual keypoint annotations. Although leveraging out-of-the box object detectors to detect object centroids is a possibility, we want to avoid conflating errors in object detection (imprecise bounding box, wrong object, etc.) from manipulation error of the policy itself. Second, we gather human-provided assessments of perceived goal alignment, following the commonly-used Likert (Likert, 1932) rating scheme from 1 (Strongly Disagree) to 7 (Strongly Agree), for:• (Q1) The robot achieves semantic alignment with the given goal during the rollout.• (Q2) The robot achieves spatial alignment with the given goal during the rollout.For Q1, we present labelers with the policy rollout video along with the given ground-truth language task description. We expect reasonably high ratings across all methods for straightforward manipulation scenarios (H1). Sketch-conditioned policies should yield higher scores than a languageconditioned policy when a task string is ambiguous (H4). Q2 is instead geared at measuring to what degree a policy can spatially arrange objects as desired. For instance, a policy can achieve semantic alignment for the instruction place can upright as long as the can ends up in the right orientation. For Q2, we visualize a policy rollout side-by-side with a given visual goal (ground truth image, sketch, etc.) to assess perceived spatial alignment. We posit that all policies should receive high ratings for straightforward scenarios (H1), with a slight edge for visual-conditioned policies which implicitly have stronger spatial priors encoded in goals. We further expect that as the visual complexity of a scene increases, sketches may be able to better attend to pertinent aspects of a goal and achieve better spatial alignment than image-conditioned agents (H3), even for different levels of sketch specificity (H4). We provide a visualization of the assessment interface for Q1 and Q2 in Appendix Fig. 16 . We note that we perform these human assessment surveys across 62 individuals (non-expert, unfamiliar with our system), where we assign between 8 and 12 people to evaluate each of the 6 different skills considered below.In this section, we present our findings related to the hypotheses of Section 4. Tables 4.2 and 4.2 measure the spatial precision achieved by policies in terms of pixelwise distance, while Fig. 3 shows the results of human-perceived semantic and spatial alignment, based on a 7-point Likert scale rating. We evaluate 6 skills from the RT-1 benchmark (Brohan et al., 2023b) : move X near Y, place X upright, knock X over, open the X drawer, close the X drawer, and pick X from Y. For each skill, we record 15 different catalog scenarios, varying both objects (16 unique in total) and their placements.In general, we find that RT-Sketch performs on a comparable level to RT-1 and RT-Goal-Image for both semantic (Q1) and spatial alignment (Q2), achieving ratings in the 'Agree' to 'Strongly Agree' range on average for nearly all skills (Fig. 3 (top)). A notable exception is upright, where RT-Sketch essentially fails to accomplish the goal semantically (Q1), albeit with some degree of spatial alignment (Q2). Both RT-Sketch and RT-Goal-Image tend to position cans or bottles appropriately and then terminate, without realizing the need for reorientation (Appendix Fig. 8 ). This behavior results in low centroid-distance to the goal (darker gray in Table 4 .2 (left)). RT-1, on the other hand, reorients cans and bottles successfully, but at the expense of higher error (Appendix Fig. 8 , light color in Table 4 .2 (left)). In our experiments, we also observe the occurrence of excessive retrying behavior, in which a policy attempts to align the current scene with a given goal with retrying actions such as grasping and placing. However, performing these low-level actions with a high degree of precision is challenging, and thus excessive retrying can actually disturb the scene leading to knocking objects off the table or undoing task progress. In Table 4 .2, we report the proportion of rollouts in which we observe this behavior across all policies. We note that RT-Goal-Image is most susceptible to this failure mode, as a result of over-attending to pixel-level details and trying in excess to match a given goal exactly. Meanwhile, RT-Sketch and RT-1 are far less vulnerable, since both sketches and language provide a higher level of goal abstraction.In this table, we further see that RT-Goal-Image has a tendency to over-attend to pixel-level details, which can result in excessive retrying behavior and failure to terminate when attempting to rearrange objects to exactly match a given goal image (darker gray in Section 4.2 (right), denoting more frequent failures).We next assess RT-Sketch's ability to handle input sketches of varied levels of detail (free-hand, edge-aligned line sketch, colorized line sketch, and a Sobel edge-detected image as an upper bound).Free-hand sketches are drawn with a reference image next to a blank canvas, while line sketches are drawn on a semi-transparent canvas overlaid on the image (see Appendix Fig. 15 ). We find such a UI to be convenient and practical, as an agent's current observations are typically available and provide helpful guides for sketching lines and edges. Across 5 trials each of the move near and open drawer skills, we see in Section 4.2 that all types of sketches produce reasonable levels of spatial precision.As expected, Sobel edges incur the least error, but even free-hand sketches, which do not necessarily preserve perspective projection, and line sketches, which are far sparser in detail, are not far behind. This is also reflected in the corresponding Likert ratings (Fig. 3 (left, bottom)). Free-hand sketches already garner moderate ratings (around 4) of perceived spatial and semantic alignment, but line sketches result in a marked performance improvement to nearly 7, on par with the upper bound of providing an edge-detected goal image. Adding color does not improve performance further, but leads to interesting qualitative differences in behavior (see Appendix Fig. 9 ).Free-Hand Line Sketch Color Sketch Sobel Edges Move Near 7.21 ± 2.76 3.49 ± 1.38 3.45 ± 1.03 3.36 ± 0.66 Drawer Open 3.75 ± 1.63 3.34 ± 1.08 2.48 ± 0.50 2.13 ± 0.25 Table 2 : RT-Sketch Spatial Precision across Sketch Types (RMSE (centroid-distance) in px. We report the spatial precision achieved by RT-Sketch subject to different input modalities. As expected, for less detailed and more rough sketches, RT-Sketch achieves lower precision (lighter shading), and for richer representations RT-Sketch is more precise (bolded, darker shading). Still, there is a relatively small difference in performance between line, color, and edge-detected representations, indicating RT-Sketch's ability to afford different levels of input specificity.) H3: Next, we compare the robustness of RT-Sketch and RT-Goal-Image to the presence of visual distractors. We re-use 15 move X near Y trials from the catalog, but introducing 5 -9 distractor objects into the initial visual scene after alignment. This testing procedure is adapted from RT-1 generalization experiments referred to as medium-high difficulty (Brohan et al., 2023b) . In Section 4.2 (left, bottom), we see that RT-Sketch exhibits far lower spatial errors on average, while producing higher semantic and spatial alignment scores over RT-Goal-Image( Fig. 3 (middle, bottom)). RT-Goal-Image is easily confused by the distribution shift introduced by distractor objects, and often cycles between picking up and putting down the wrong object. RT-Sketch, on the other hand, ignores task-irrelevant objects not captured in a sketch and completes the task in most cases (see Appendix Fig. 10 ).H4: Finally, we evaluate whether sketches as a representation are favorable when language goals alone are ambiguous. We collect 15 scenarios encompassing 3 types of ambiguity in language instructions: instance ambiguity (T1) (e.g., move apple near orange when multiple orange instances are present), somewhat out-of-distribution (OOD) language (T2) (e.g., move left apple near orange), and highly OOD language (T3) (e.g., complete the rainbow) (see Appendix Fig. 11 ). While the latter two qualifications should intuitively help resolve ambiguities, they were not explicitly made part of the original RT-1 training (Brohan et al., 2023b) , and hence only provide limited utility. In Section 4.2 (left, bottom), RT-Sketch achieves nearly half the error of RT-1, and a 2.39-fold and 2.79-fold score increase for semantic and spatial alignment, respectively (Fig. 3 (right, bottom)). For T1 and T2 scenarios, RT-1 often tries to pick up an instance of any object mentioned in the task string, but fails to make progress beyond that (Appendix Fig. 12 ). This further suggests the utility of sketches to express new, unseen goals with minimal overhead, when language could otherwise be opaque or difficult to express with only in-distribution vocabulary (Appendix Fig. 13 ).Limitations and Failure Modes Firstly, the image-to-sketch generation network used in this work is fine-tuned on a dataset of sketches provided by a single human annotator, and we have yet to stresstest the generalization capabilities of RT-Sketch at scale with sketches produced by different people. Secondly, we note that RT-Sketch shows some inherent biases towards performing certain skills it was trained on, and occasionally performs the wrong skill. For a detailed breakdown of RT-Sketch's limitations and failure modes, please see Appendix C). 
conclusion: 
 We propose RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. To enable such a policy, we first develop a scalable way to generate paired sketch-trajectory training data via an image-to-sketch translation network, and modify the existing RT-1 architecture to take visual information as an input. Empirically, we show that RT-Sketch not only performs on a comparable level to existing language or goal-image conditioning policies for a number of manipulation skills, but is amenable to different degrees of sketch fidelity, and more robust to visual distractors or ambiguities. Future work will focus on extending hand-drawn sketches to more structured representations, like schematics or diagrams for assembly tasks. While powerful, sketches are not without their own limitations -namely ambiguity due to omittted details or poor quality sketches. In the future, we are excited by avenues for multimodal goal specification that can leverage the benefits of language, sketches, and other modalities to jointly resolve ambiguity from any single modality alone. 
 
 Appendix: In this section, we provide further details on the visual goal representations RT-Sketch sees at train and test time (Appendix A), qualitative visualizations of experimental rollouts (Appendix B), limitations (Appendix C) of RT-Sketch, as well as the interfaces used for data annotation, evaluation, and human assessment (Appendix D). To best understand RT-Sketch's performance and see its emergent capabilities, please refer to our website. 3 .Since the main bottleneck to training a sketch-to-action policy like RT-Sketch is collecting a dataset of paired trajectories and goal sketches, we first train an image-to-sketch translation network T mapping image observations o i to sketch representations g i , discussed in Section 3. To train T , we first take a pre-trained network for sketch-to-image translation (Li et al., 2019) trained on the ContourDrawing dataset of paired images and edge-aligned sketches (Fig. 4 ). This dataset contains L (i) = 5 crowdsourced sketches per image for 1000 images. By pre-training on this dataset, we hope to embed a strong prior in T and accelerate learning on our much smaller dataset. Next, we finetune T on a dataset of 500 manually drawn line sketches for RT-1 robot images. We visualize a few examples of our manually sketched goals in Fig. 5 under 'Line Drawings'. (Li et al., 2019) . For each image, 5 separate annotators provide an edge-aligned sketch of the scene by outlining on top of the original image. As depicted, annotators are encouraged to preserve main contours of the scene, but background details or fine-grained geometric details are often omitted. Li et al. (2019) then train an image-to-sketch translation network T with a loss that encourages aligning with at least one of the given reference sketches (Eq. ( 2)).Notably, while we only train T to map an image to a black-and-white line sketch ĝi , we consider various augmentations A on top of generated goals to simulate sketches with varied colors, affine and perspective distortions, and levels of detail. To further interpret RT-Sketch's performance, we provide visualizations of the precision metrics and experimental rollouts. In Fig. 7 , we visualize the degree of alignment RT-Sketch achieves, as quantified by the pixelwise distance of object centroids in achieved vs. given goal images. In Fig. 8 , Fig. 9 , Fig. 10 , and Fig. 12 , we visualize each policy's behavior for H1, H2, H3 and H4, respectively. Fig. 11 visualizes the four tiers of difficulty in language ambiguity that we analyze for H4. To best understand RT-Sketch's performance, we kindly refer interested readers to our website containing a much more detailed overview and videos of all policies.Figure 7 : Spatial Precision Visualization: We visualize four trials of RT-Sketch on the Move Near skill, along with the measured spatial precision in terms of RMSE. To evaluate spatial precision, we have a human annotator annotate the frame that is visually most aligned, and then keypoints for the object that was moved in this frame and in the provided reference goal image. For each of the four trials, we visualize the rollout frames until alignment is achieved, along with the labeled object centroids and the offset in achieved vs. desired positions. The upper right example shows a failure of RT-Sketch in which the apple is moved instead of the chip bag, incurring a high RMSE. These visualizations are intended to better contextualize the numbers from Section 4.2.Figure 8 : H1 Rollout Visualization: We visualize the performance of RT-1, RT-Sketch, and RT-Goal-Image on two skills from the RT-1 benchmark (upright and knock). For each skill, we visualize the goal provided as input to each policy, along with the policy rollout. We see that for both skills, RT-1 obeys the semantic task at hand by successfully placing the can upright or sideways, as intended. Meanwhile, RT-Sketch and RT-Goal-Image struggle with orienting the can upright, but successfuly knock it sideways. Interestingly, both RT-Sketch and RT-Goal-Image are able to place the can in the desired location (disregarding can orientation) whereas RT-1 does not pay attention to where in the scene the can should be placed. This is indicated by the discrepancy in position of the can in the achieved versus goal images on the right. This trend best explains the anomalous performance of RT-Sketch and RT-Goal-Image in perceived Likert ratings for the upright task (Fig. 3 ), but validates their comparably higher spatial precision compared to RT-1 across all benchmark skills (Section 4.2).Figure 9 : H2 Rollout Visualization: For the open drawer skill, we visualize four separate rollouts of RT-Sketch operating from different input types. Free-hand sketches are drawn without outlining over the original image, such that they can contain marked perspective differences, partially obscured objects (drawer handle), and roughly drawn object outlines. Line sketches are drawn on top of the original image using the sketching interface we present in Appendix Fig. 15 . Color sketches merely add color infills to the previous modality, and Sobel Edges represent an upper bound in terms of unrealistic sketch detail. We see that RT-Sketch is able to successfully open the correct drawer for any sketch input except the free-hand sketch, without a noticeable performance gain or drop. For the free-hand sketch, RT-Sketch still recognizes the need for opening a drawer, but the differences in sketch perspective and scale can occasionally cause the policy to attend to the wrong drawer, as depicted.Figure 10 : H3 Rollout Visualization: We visualize qualitative rollouts for RT-Sketch and RT-Goal-Image for 3 separate trials of the move near skill subject to distractor objects. In Column 2, we highlight the relevant non-distractor objects that the policy must manipulate in order to achieve the given goal. In Trial 1, we see that RT-Sketch successfuly attends to the relevant objects and moves the blue chip bag near the coke can. Meanwhile, RT-Goal-Image is confused about which blue object to manipulate, and picks up the blue pepsi can instead of the blue chip bag (A). In Trial 2, RT-Sketch successfully moves an apple near the fruit on the left. A benefit of sketches is their ability to capture instance multimodality, as any of the fruits highlighted in Column 2 are valid options to move, whereas this does not hold for an overspecified goal image. RT-Goal-Image erroneously picks up the green chip bag (B) instead of a fruit. Finally, Trial 3 shows a failure for both policies. While RT-Sketch successfully infers that the green can must be moved near the red one, it accidentally knocks over the red can (C) in the process. Meanwhile, RT-Goal-Image prematurely drops the green can and instead tries to pick the green chip bag (D). 11 ): One source of ambiguity in language descriptions is mentioning an object for which there are multiple instances present. For example, we can easily illustrate three different desired placements of an orange in the drawer via a sketch, but an ambiguous instruction cannot easily specify which orange is relevant to pick and place. In all rollouts, RT-Sketch successfully places the correct orange in the drawer, while RT-1 either picks up the wrong object (A), fails to move to the place location (B), or knocks off one of the oranges (C). Although in this case, the correct orange to manipulate could easily be specified with a spatial relation like pick up the left/middle/right orange, we show below in Appendix Fig. 13 that this type of language is still out of the realm of RT-1's semantic familiarity.Figure 13 : H4 Rollout Visualization (T2-3 as visualized in Fig. 11 ): For T2, we consider language with spatial cues that intuitively should help the policy disambiguate in scenarios like the oranges in Fig. 12 . However, we find that RT-1 is not trained to handle such spatial references, and this kind of language causes a large distribution shift leading to unwanted behavior. Thus, for the top rollout of trying to move the chip bag to the left where there is an existing pile, RT-Sketch completes the skill without issues, but RT-1 attempts to open the drawer instead of even attempting to rearrange anything on the countertop (A). For T3, we consider language goals that are even more abstract in interpretation, without explicit objects mentioned or spatial cues. Here, sketches are advantageous in their ability to succinctly communicate goals (i.e. visual representation of a rainbow), whereas the corresponding language task string is far too underspecified and OOD for the policy to handle (B).While RT-Sketch is performant at several manipulation benchmark skills, capable of handling different levels of sketch detail, robust to visual distractors, and unaffected by ambiguous language, it is not without failures and limitations. In Fig. 14 , we visualize the failure modes of RT-Sketch. One failure mode we see with RT-Sketch is occasionally re-trying excessively, as a result of trying to align the scene as closely as possible. For instance, in the top row, Rollout Image 3, the scene is already well-aligned, but RT-Sketch keeps shifting the chip bag which causes some misalignment in terms of the chip bag orientation. Still, this kind of failure is most common with RT-Goal-Image (Section 4.2), and is not nearly as frequent for RT-Sketch. We posit that this could be due to the fact that sketches enable high-level spatial reasoning without over-attending to pixel-level details.One consequence of spatial reasoning at such a high level, though, is an occasional lack of precision. This is noticeable when RT-Sketch orients items incorrectly (second row) or positions them slightly off, possibly disturbing other items in the scene (third row). This may be due to the fact that sketches are inherently imperfect, which makes it difficult to reason with such high precision.Finally, we see that RT-Sketch occasionally manipulates the wrong object (rows 4 and 5). Interestingly, we see that a fairly frequent pattern of behavior is to manipulate the wrong object (orange in row 4) to the right target location (near green can in row 4). This may be due to the fact that the sketch-generating GAN has occasionally hallucinated artifacts or geometric details missing from the actual objects. Having been trained on some examples like these, RT-Sketch can mistakenly perceive the wrong object to be aligned with an object drawn in the sketch. However, the sketch still indicates the relative desired spatial positioning of objects in the scene, so in this case RT-Sketch still attempts to align the incorrect object with the proper place.Finally, the least frequent failure mode is manipulating the wrong object to the wrong target location (i.e. opening the wrong drawer handle). This is most frequent when the input is a free-hand sketch, and could be mmitigated by increasing sketch detail (Section 4.2).Figure 15 : Sketching UI: We design a custom sketching interface for manually collecting paired robot images and sketches with which to train T , and for sketching goals for evaluation. The interface visualizes the current robot observation, and provides the ability to draw on a digital screen with a stylus. The interface supports different colors and erasure. We note that intuitively, drawing on top of the image is not an unreasonable assumption to make, since current agent observations are far more readily available than a goal image, for instance. Additionally, the overlay is intended to make the sketching interface easy for the user to provide, without having to eyeball edges for the drawers or handles blindly.Figure 16 : Assessment UI: For all skills and methods, we ask labelers to assess semantic and spatial alignment of the recorded rollout relative to the ground truth semantic instruction and visual goal. We show the interface above, where labelers are randomly assigned to skills and methods (anonymized). The results of these surveys are reported in Fig. 3 .http://rt-sketch-anon.github.ioeverydayrobots.comhttp://rt-sketch-anon.github.io 


Decision: The reviewers were positive about this paper. There are only a couple of questions that the authors might comment on:

-- At what level of abstraction is the sketch no longer able to convey goal effectively? It would be interesting to probe this question.

-- How might we generate a more diverse set of sketches? This seems related to the first question.

Post review: good job.

Reviews: review: title: title: Review
paper_summary: paper_summary: Please see review below
main_review: main_review: This paper presents an approach for goal-conditioned imitation learning, where the goal is specified by a user sketch. To tackle this, the authors propose RT-sketch -- essentially a robot transformer model that additionally conditions on goal images. The technical contributions here mainly lie in the construction of the sketch dataset: this dataset contains "sketches" generated by edge detectors and generative networks, as well as real manually sketched goal images. These are then conditioned on to train robot policies with transformer models. A major focus of this work is to get diverse sketches in the dataset so that it can handle a variety of sketches. Various permutations, such as colorization of part os the sketch have been investigated.

The motivations here are justified -- sketches provide a fairly specific and precise interface to communicate with robots that would be challenging to achieve with language, especially when considering spatial attributes. The weaknesses are in the practicalities of the system, the sketches provided are either quite detailed and accurate even when the problem setting is quite simple. e.g. the chest of drawers sketched have handles on each of the individual drawers, even when most of these are irrelevant; the number of circular objects in the bowl matches the real number (the sketch has 8 objects in the bowl in fig 7.). It's reasonable to expect that the method breaks at some level of abstraction: I would be interested to see additional experiments to identify where this boundary of the level of abstraction is. 

Finally, although sketching desired goals is novel, there have been attempts to integrated sketches into specifications for robots that are note referenced here. These include sketching to instruct motion trajectories [1] and sketching to specify affordances [2]. These are relevant related works that should be mentioned.

[1] Zhi et al., Instructing Robots by Sketching: Learning from demonstration via probabilistic diagrammatic teaching, 2024.
[2] Masnadi et al., A Sketch-Based System for Human-Guided Constrained Object Manipulation, 2020.
questions: questions: Please see review above.
limitations: limitations: 3
review_summary: review_summary: Please see review above.
title: title: Well-presented and novel approach for goal-conditioning
paper_summary: paper_summary: The paper presents a goal-conditioned policy for manipulation using “sketches” as the conditioning in contrast to language or camera images, which the authors claim can be ambiguous or contain distractors respectively. Results are quantitatively good, and can drive future multimodal goal-specification research.
main_review: main_review: Strengths:

I find the contributions of this paper simple yet well-motivated. Sketches can help disambiguate task-relevant from distractor objects, and are easy to provide on-the-fly just like language. Such kind of exploration for goal-conditioned policy learning is impactful for the field of robotics. I would also categorize this paper to be one that tries finding ways of using non-robotics datasets (contour drawing dataset in this case) to help improve visuomotor learning performance, which is an interesting and potentially useful direction in robot learning.

Weaknesses:
1. The suite of tasks considered for evaluating the policy could be diversified. Many tasks considered in recent papers are object-centric and have low tolerances for error (e.g. Stanford’s “make coffee” task) which creates bottlenecks that drastically reduces task success rates. Can the authors consider experimenting on such high tolerance tasks, or add discussion in limitations as to which kinds of tasks would have to be excluded for now given the reported mean-squared errors?
2. Why did the authors not experiment with the state-of-the-art Diffusion Policy with sketch and real images as goals?
questions: questions: Please refer to weaknesses mentioned above.
limitations: limitations: 2
review_summary: review_summary: I believe the paper presents a novel approach for goal-conditioning that can drive future multimodal goal-specification research. However, some more experiments/discussion might be needed to convince readers of potential general efficacy of this approach in robot learning.
title: title: A creative task definition with a successful method.
paper_summary: paper_summary: This paper introduces the task of using sketches as a way to command a robot to do table top and cabinet related tasks. The compare this to language and photograph conditioned methodologies. They show that sketches, as input have a great advantage over other inputs in a variety of circumstances.
main_review: main_review: This paper was of high quality. It was very clearly written and organized. It was very creative and original for the authors to think of sketches as a modality to control robots.  This work is highly significant since it introduces a new input to robots and compares it to other inputs.

There are several strengths of the paper. The writing and organization of the paper are very clear. The authors do thorough evaluation of the presented model. The four tested hypotheses offer a thorough investigation of what might be useful or advantageous of using sketches as inputs to robots. The authors tie their work well to its motivation. The video summarized the paper very clearly.

There are no huge weaknesses of this paper that I could determine. The authors acknowledge that the tasks are relatively limited to the training tasks. It would be nice to assess the user experience with controlling a robot via sketches. Sketches can be difficult and time consuming compared to language instructions. I would like to see a user study comparing the experience of using different modalities, but I think that that is out of scope for this paper.
questions: questions: The participants in the evaluation were unpaid. What was the general motivation for these people to participate?

The table setting example is compelling. I agree it would be very challenging to describe these arrangements with language. Could you please have the robot perform a demonstration of this example? If it is infeasible to try this, could you please explain whether you think RT-Sketch would be capable of doing this?

MISC:
Typo Line 321: "RT-Sketchachieves"
limitations: limitations: 3
review_summary: review_summary: This paper proposes an interesting novel task. The paper is well written and the evaluation is thorough with interesting results.