SketchAgent: Language-Driven Sequential Sketch Generation
Authors: name: Yael Vinker, name: Tamar Rott Shaham, name: Kristine Zheng, name: Alex Zhao, name: Judith E Fan, name: Antonio Torralba

Sketching serves as a versatile tool for externalizing ideas, enabling rapid
exploration and visual communication that spans various disciplines. While
artificial systems have driven substantial advances in content creation and
human-computer interaction, capturing the dynamic and abstract nature of human
sketching remains challenging. In this work, we introduce SketchAgent, a
language-driven, sequential sketch generation method that enables users to
create, modify, and refine sketches through dynamic, conversational
interactions. Our approach requires no training or fine-tuning. Instead, we
leverage the sequential nature and rich prior knowledge of off-the-shelf
multimodal large language models (LLMs). We present an intuitive sketching
language, introduced to the model through in-context examples, enabling it to
"draw" using string-based actions. These are processed into vector graphics and
then rendered to create a sketch on a pixel canvas, which can be accessed again
for further tasks. By drawing stroke by stroke, our agent captures the
evolving, dynamic qualities intrinsic to sketching. We demonstrate that
SketchAgent can generate sketches from diverse prompts, engage in
dialogue-driven drawing, and collaborate meaningfully with human users.