Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches

Authors: Peihong Yu, Amisha Bhaskar, Anukriti Singh, Zahiruddin Mahammad, Pratap Tokekar
Primary area: applications to robotics, autonomy, planning
Keywords: robotics, learn from demonstration, reinforcement learning

Abstract:
Training robotic manipulation policies traditionally requires numerous demonstrations and/or environmental rollouts. While recent Imitation Learning (IL) and Reinforcement Learning (RL) methods have reduced the number of required demonstrations, they still rely on expert knowledge to collect high-quality data, limiting scalability and accessibility. We propose Sketch-to-Skill, a novel framework that leverages human-drawn 2D sketch trajectories to bootstrap and guide RL for robotic manipulation. Our approach extends beyond previous sketch-based methods, which were primarily focused on imitation learning or policy conditioning, limited to specific trained tasks. Sketch-to-Skill employs a Sketch-to-3D Trajectory Generator that translates 2D sketches into 3D trajectories, which are then used to autonomously collect initial demonstrations. We utilize these sketch-generated demonstrations in two ways: to pre-train an initial policy through behavior cloning and to refine this policy through RL with guided exploration. Experimental results demonstrate that Sketch-to-Skill achieves $\sim$96\% of the performance of the baseline model that leverages teleoperated demonstration data, while exceeding the performance of a pure reinforcement learning policy by $\sim$170\%, only from sketch inputs. This makes robotic manipulation learning more accessible and potentially broadens its applications across various domains.

Decision: reject
Reviewer scores: 5, 3, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730247332486

thread_type:
official_review

<thread_object>
creator: reviewer_cvp5

<content>
summary:
This paper presents a method for training robotic manipulation policies through the use of 2D human trajectory sketches. The pipeline depends on a generator that takes a pair of 2D human trajectory sketches, drawn on the same scene captured from two different view points, and generates 3D trajectories (sequences of 3D points). This generator must be pre-trained using a dataset of sketches and corresponding 3D trajectories. Next, the 3D trajectories are used to train an agent with imitation learning, and then the agent is finetuned via reinforcement learning, using a discriminator-based reward to ensure consistency between the agent behavior and the 3D trajectories that came from the sketches. The method is shown to perform well on 6 tasks in MetaWorld and on a button pressing task in the real world.

strengths:
The paper is well-written and easy to follow. The idea is interesting at a high-level and demonstrated on MetaWorld, a common benchmark, which could help others reproduce the work.

weaknesses:
**Practicality of Approach.** The approach depends on a trained sketch-to-3D trajectory generator, which itself must be trained on a sufficient set of data. It seems that the burden of collecting this dataset would outweigh the benefits of using this generator as an alternative to providing human demonstrations. Furthermore, teleoperating a handful of demonstrations seems like a more general approach. Demonstrations can be collected for any task that the human can demonstrate, while the performance of the proposed approach is constrained to the capabilities of the trained generator. For example, it seems that the current method can only be using for position-controlled tasks, and not tasks where the robot must control both position and orientation. Furthermore, the trained generator might not generalize well beyond the training data, which likely needs to be collected on the same set (or very similar set) of tasks, or at the very least in the same domain (e.g. same robot, same workspace). 

**Method Novelty.** The method itself seems limited in novelty -- the core component is the learned sketch-to-3D trajectory generator. Apart from that, it seems like an existing RL-finetuning algorithm is used (IBRL), with the only critical changes being (1) the choice of initial dataset and (2) the use of a discriminator. However, the discriminator itself seems to be unneeded, having little to no performance gain over not using it (Figure 8), while needed careful tuning for the discriminator reward weight (Figure 9). Even in the real robot experiments, it seems as though no discriminator is used.

**Experiment Issues.** The tasks presented in the paper appear to be limited in complexity. For example, the TD3 (RL baseline) can solve several of the tasks with just a sparse reward (Figure 5). I would suggest using some more challenging tasks for evaluation, perhaps from some other common benchmarks such as [robomimic](https://robomimic.github.io/), [RLBench](https://sites.google.com/view/rlbench), or [ManiSkill](https://www.maniskill.ai/home). There is also insufficient evaluation on some of the components of the method. In particular, how well does the sketch-to-3D generator generalize to new settings? Can it be applied to new tasks or new domains outside of the training data? How important is the architecture choice of the generator? Is the VAE crucial, or would other architectures work just as well? Is the choice of RL-finetuning method also critical, or would another method such as GAIL (as used in this work: https://arxiv.org/abs/1802.09564) work as well? Finally, in the real-world experiment, if BC already achieves 80%, what is the point of RL finetuning, if it achieves the same performance? Furthermore, training the sketch generator seemed to require 85 teleoperated trajectories itself (from Appendix). Training the 3D generator and then running RL seems much more painful than just training an agent directly on the teleoperated trajectories.

**Minor Issues.**

- line 71: "a more approach" - missing a word?

questions:
- Sec 4.1 - is there any quantitative evaluation as well, or is it all qualitative?
- What if the two human sketches are inconsistent? It might be tricky to ensure that they are consistent.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579977829

thread_type:
official_review

<thread_object>
creator: reviewer_ne7n

<content>
summary:
This work presents an interesting approach to extending IBRL by incorporating a trajectory generation method reminiscent of RT-Sketch. The core idea of generating 3D trajectories from pairs of 2D sketches, augmented by noise injection for data diversity, is novel.  The addition of a discriminator to guide the IBRL policy towards these generated trajectories further strengthens the link between user-provided sketches and learned robot behavior. However, the paper leaves some questions unanswered.

Firstly, the authors show a pair of sketches in Fig. 2, but it's not quantitatively clear how robust the trajectory generator is to variations in the spatial correlation between these sketches. Some of the analysis is present in appendix B, but adding information like the benefits of each augmentation strategy, sample variations in sketches, etc. would be beneficial. For example, how much spatial variation do you add?

In line 369, how many hand-drawn sketches do you collect?. Is it also three (i.e. number of expert tele-operated policies used)? The ablation study on Discriminator reward weighting in Fig. 9 is good, but the choice of reward weights seems arbitrary. A more logical progression (e.g., 0.1, 0.01, 0.001) would make the analysis clearer, or explain your rationale for choosing the current set of weights. While the appendix includes a no-discriminator condition (Fig 14), this crucial ablation should be in the main paper.

Looking ahead, the authors briefly mention "time parameterization of the trajectory" in future work (line 529).  This raises the question of how they envision solving the increased difficulty in achieving correlation between two sketches when time is factored in.

Finally, the paper would benefit from a discussion situating this method within the broader context of recently popular Vision-Language Agents (VLAs) for robotic manipulation, such as OpenVLA. This is particularly relevant given the increasing use of real-world robot-collected data in VLAs, whereas Sketch-to-Skill is trying to reduce tele-operated data collection. How do the authors see sketches complementing this data? Moreover, given the prevalence of LFD and imitation learning in VLA pre-training and fine-tuning, how do the authors position the advantages of RL in this setting? Including this in a related work or discussion section would be helpful.

Some minor points:
*   There is a typo on line 071: "we present a more approach."
*   For improved accessibility, the figures could be made more color-blind friendly, potentially by incorporating textures. For example, add textures to the bar plots in Fig 7.
*   In Fig. 3, why is there an asterisk after "Replay Buffer"?

strengths:
provided in summary

weaknesses:
provided in summary

questions:
provided in summary
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580784965

thread_type:
official_review

<thread_object>
creator: reviewer_beax

<content>
summary:
This work aims to use human-drawn sketches to learn RL policies for robotic manipulation tasks. They use a trajectory generator to generate demonstrations from sketches, then use IBRL to train a policy from the generated demonstrations. The paper additionally tests whether there are benefits in (1) generating more demonstrations per sketch and (2) performs guided exploration using a discriminator.

strengths:
Good results on a novel method, which is able to generally preserve performance while removing the need for high-quality teleoperated demonstration data. Each piece of the method is explained clearly, and the method is designed logically (e.g. exploit the demonstrated smoothness of the latent space to generate more demos per sketch with controlled noise). The results are presented clearly and effectively, and the Appendix provides useful information. Tests in the real-world show practical applicability of the method.

weaknesses:
Firstly, some additions to improve performance seem to have ambiguous results:

(a) Discriminator to guide exploration: In Fig. 6, the performance with and without the discriminator seem fairly close. The policy trained with discriminator only seems to perform better in ButtonPressTopdownWall, while the policy without discriminator seems to perform the same if not marginally better in the rest of the tasks.

(b) Additional demos per sketch: In Fig. 8, 10 demos per sketch + discriminator has similar average performance in ButtonPress (though lower lows), similar performance to others in CoffeePush, and has better average but lower max performance in BoxClose. It is mentioned that there is diminishing performance with more demos, but it does not seem clear that more demos per sketch is necessarily better? In fact, in Fig. 14, 10 demos + discriminator seems worse than others.

Secondly, it seems one benefit of sketching vs teleoperating is that sketched demonstrations can be made much faster and without expensive hardware. So, the usage of only 3 demos, and the absence of demo ablations (specifically on the number of sketches/teleoperated demonstrations provided) is somewhat odd. Perhaps MetaWorld tasks are too easy when using more demonstrations, but there exist harder standard LfD benchmarks which the authors did not test on.

questions:
Are there additional clarifications which would explain why the authors chose to keep the discriminator and 10 demos per sketch? Perhaps a different way to present the data (e.g. raw success rate on more eval episodes)?

Does the method scale to more sketches (and how does it compare to IBRL when providing more teleoperated data)?

Have you tried this method with other backbone RL algorithms (e.g. ensemble SAC, REDQ) to improve sample efficiency or final performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667632166

thread_type:
official_review

<thread_object>
creator: reviewer_k4pp

<content>
summary:
This paper explores the use of a sketch-based method as the means to provide robots with expert demonstrations that show how various manipulation tasks should be completed. These demonstrations can then be used to bootstrap the process of training control policies via behaviour cloning and reinforcement learning. The proposed system consequently consists of 1) a module that transforms a pair of 2d sketches (which correspond to two different viewpoints of a scene) into 3d trajectories, 2) a behaviour cloning step to generate an initial control policy, and 3) an RL module that refines this policy. Comparisons to teleoperation-based demonstrations and pure RL show how well this method works for six manipulation tasks that are relatively simple.

strengths:
I like the way the paper is written, and the clear research questions that are articulated / studied in Chapter 4. Within the context of the relatively simple manipulation tasks which are chosen, I think the paper is very thorough with its analysis, and the evaluation does indeed shine a positive light on sketch-based methods as the means to provide expert demonstrations. Nevertheless, as I explain below, I am afraid that this approach will not scale to manipulation tasks which are more complex.

weaknesses:
The methodology is overall sound, but nothing stands out as particularly novel. In addition, and more importantly, the manipulation tasks which are chosen for experiments are too basic. They do not showcase adaptive behaviours (e.g. what happens if an object that needs to be picked up is accidentally dropped, or if it otherwise does not behave/move as originally intended) or multi-stage, longer-horizon motions (e.g. opening a cardboard box before picking objects up from it, or folding a t-shirt). Teleoperation has been shown to be effective in such settings due to its real-time feedback and high-throughput high-DOF nature. I fear that the sketch-based approach presented here is simply too limited and will quickly become overly cumbersome as the task difficulty increases. For example, even providing a time-dependent orientation for the robot's end effector would likely be non-trivial through the pair of 2d sketches used as input for the system that is proposed in this paper. I would be happy to reconsider this standpoint if further experiments are presented for manipulation tasks that are considerably more complex.

questions:
As mentioned above, I have significant concerns over the scalability of the proposed sketch-based method for complex, real-world manipulation tasks. As robotics manipulation problems go, the tasks which are chosen in this work fall well within the toy-model domain, and as such, the findings do not apply to the real-world problems which we expect robots to be able to learn how to undertake through expert demonstrations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679096105

thread_type:
official_review

<thread_object>
creator: reviewer_vobb

<content>
summary:
SKETCH-TO-SKILL leverages 2D human-drawn sketches to bootstrap and guide RL for robotic manipulation, making the approach accessible and potentially impactful. However, based on the presented experiments, the practical potential remains unclear. The real-world demonstrations focus on relatively simple tasks, making it challenging to fully gauge the framework's effectiveness or scalability in more complex, nuanced environments. More diverse and demanding experiments, along with detailed explanations, would help reveal whether this approach can truly generalize and perform under varied real-world conditions.

strengths:
- The paper introduces an innovative approach by leveraging human-drawn 2D sketches to initialize and guide reinforcement learning in robotic manipulation.

- SKETCH-TO-SKILL provides an alternative in robot learning by enabling task training from simple sketches, which reduces the reliance on teleoperation data, specialized hardware, and advanced expertise. This approach potentially broadens access to robotic training methods.

weaknesses:
1. The examples shown focus on short, simple trajectories, so it’s difficult to gauge how well the framework would handle longer, more complex paths. This leaves open the question of how trajectory complexity and length impact the learning cost.

2. For human users, sketching complex trajectories might require a lot more effort than the straightforward examples presented. 

3. There’s also limited information on how the framework would deal with cases where paths overlap or are partially obscured. This could present practical challenges, both in terms of creating the sketches and the model’s ability to interpret them accurately.

4. The paper doesn’t clarify if there are specific guidelines for choosing sketch perspectives. It seems each task might need its own camera setup, which could complicate consistency and generalization in real applications.

5. The discriminator doesn’t appear to contribute significantly to the results, which makes it unclear how essential it is or if it’s been optimized effectively for generalization.

6. The experiments are carried out in simplified environments without much complexity or occlusion, so it’s hard to assess how well the model would perform in more realistic, cluttered settings.

7. The real-world tasks demonstrated are quite basic, which doesn’t fully showcase the framework’s potential across a broader range of applications. More varied real-world tests could provide deeper insights into its versatility.

8. There’s little information on the types of sketches and common sketching errors anticipated. Understanding how the model handles imperfections and what types of errors might impact learning outcomes would add clarity.

questions:
1. How does trajectory complexity affect the learning cost, both for the model and for users creating these sketches?

2. How feasible is it for users to sketch accurately for more complex, multi-step tasks, and is this approach realistic for those scenarios?

3. How does the model manage paths that overlap or are partially obscured? Are there any specific recommendations for users to handle these situations in sketches?

4. Are there guidelines for choosing the best perspective for sketching, and does each task require a different camera setup? If so, how might this impact the model’s ability to generalize?

5. Has the discriminator been optimized for generalization, and what tangible impact does it have on the model’s overall performance?

6. How sensitive is the model to changes in the scene, such as additional objects or occlusions?

7. Would testing on a wider variety of real-world tasks help to bring out a clearer picture of the model’s strengths and limitations?

8. What are the most common sketching errors expected in real-world use? How does the model handle these, and which types of errors are most likely to affect performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733253550082

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We extend our sincere thanks to all reviewers for their insightful feedback, which has significantly improved the quality and scope of our work. The new edits are highlighted in blue and red in the updated manuscript. We are particularly encouraged by the recognition of several key aspects of our paper:

1. **Innovative Approach:** Reviewers appreciated the novelty of using human-drawn sketches to streamline data collection, offering a scalable alternative to traditional methods in robot learning. Particularly Reviewer beAX believes our method produced “*Good results on a novel method…removing the need for high-quality teleoperated demonstration data. … method is designed logically*”. whereas reviewer vobB also mentions “*paper introduces an innovative approach…*”, Reviewer ne7n says “*The core idea of generating 3D trajectories…is novel*”
2. **Comprehensive Experiments:** Our thorough evaluation across both simulated and real-world tasks was noted, emphasizing the robustness and applicability of the framework. Reviewer k4pp thinks “*the paper is very thorough with its analysis, and the evaluation does indeed shine a positive light…*”, Reviewer cvP5 appreciated we *“demonstrated on MetaWorld…could help others reproduce the work”*. 
3. **Real-World Impact:** The potential of our approach to make robotic learning more accessible by reducing reliance on expensive teleoperation setups and enabling intuitive data collection through sketches was recognized. Reviewer beAX says *“Tests in the real-world show practical applicability of the method*” and Reviewer vobB mentions*“This approach potentially broadens access to robotic training methods.”*

---

### Key Updates During Rebuttal

### Handling Complex Tasks

To demonstrate our framework's scalability to more complex, long-horizon tasks (vobB, k4pp),  we included new experiments on benchmarks like **MetaWorld Assembly** and **RoboMimic Can Pick-and-Place** and **Real-world Bread-Pick-and-Place** (Figure 15, 17 and 18), which require multi-stage planning. These results demonstrate our framework’s ability to generalize and perform effectively on challenging tasks. 

### Importance of the Discriminator

Further experiments have highlighted the discriminator's crucial role (vobB, beAX, cvP5), particularly in hard tasks. We demonstrated its significant impact through additional experiments, particularly in hard tasks. For instance, on the Assembly task (Figure 18), success rates improved from **20% to 85%** with the discriminator, underscoring its critical role in guiding RL. 

### Robustness of the Sketch-to-3D Generator

In response to queries about the generator’s ability to generalize and handle sketch variations, our ablation studies validated the **VAE-based generator's** effectiveness. It consistently produces smooth, accurate trajectories even from imperfect sketches and adapts well to unseen tasks without retraining (Tables 2 and 3). (ne7n, cvP5)

### Real-World Application

Addressing concerns about real-world applicability (vobB, beAX), we expanded our experiments to include settings with randomized clutter, such as Bread Pick-and-Place and Toaster Button-Press (Figures 17 and 18), demonstrating robust performance in unstructured environments.  Furthermore, as highlighted in our discussions, while sketches do not replace the precise control offered by teleoperation for complex tasks, they provide an accessible and cost-effective means of guiding robotic learning in environments where high precision is not paramount. Our approach effectively leverages human-drawn sketches to initiate learning, broadening the applicability of sketch-based methods in practical robotics scenarios.

### Comparison to Teleoperation

Clarifying the advantages over teleoperation (k4pp, cvP5), we showed that sketches significantly reduce reliance on resource-intensive demonstration collection methods. Our approach achieves **96% of teleoperated performance** and a **170% improvement over pure RL**, offering a scalable and efficient alternative.

---

### Final Remarks

The reviewers have acknowledged the improvements made to our manuscript, with all but one increasing their scores or maintaining an already high score of 8 following our responses. One reviewer maintained their original score, expressing ongoing concerns about the application of sketch-based demonstrations for complex, real-world tasks. Our method builds upon existing research on sketch-based instructions for robots, extending its utility to reinforcement learning environments. While sketch-based methods are not intended to replace high-precision teleoperation for all tasks, they offer a valuable and accessible complement, especially in scenarios where teleoperation is impractical or unavailable. We hope this context and the positive changes most reviewers recognize will be considered in the final decision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an approach for combined imitation and reinforcement learning with the scaffolding of sketches. This is a borderline paper with arguments on both the positive and the negative side. On the positive side, the idea of generating 3d trajectories from 2d sketches is very interesting. Further the tricks to improve diversity of trajectories by tuning the noise is interesting as well. On the negative side, there are strong concerns on the applicability of this work for cutting edge robotics problems. After the rebuttal period, the reviewers, with a couple who are experts in robotics, engaged in significant discussion. The following comment summarizes the concerns quite succinctly and I think will benefit the authors as they improve this work for a future submission. 

"The main claim that has emerged, it would appear, is that the proposed method can be an easily-accessible complement to teleoperation. This claim, however, is very feeble. If one already has a robot system in place, would (for example) a VR handset device for teleop really be so inaccessible? And if the proposed sketch-based system can only be used for very simple tasks, then what good is it that it's accessible anyhow? If I saw a feasible path towards extending this system for the problems roboticist actually struggle with (e.g. get a robot to fold laundry), then I would be much more positive. But, despite the author's attempts to answer this concern, I remain convinced that the path forward for this work is exceedingly difficult, and it would likely be the case that teleop is still the main way to go in terms of providing robot demonstrations."