AirSketch: Generative Motion to Sketch
Authors: Hui Xian Grace Lim
Xuanming Cui
Yogesh S Rawat
Ser-Nam Lim
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative
Hand Motion
Sketch
Primary area: generative_models

Abstract: Illustration is a fundamental mode of human expression and communication. Certain types of motion that accompany speech can provide this illustrative mode of communication. While Augmented and Virtual Reality technologies (AR/VR) have introduced tools for producing drawings with hand motions (air drawing), they typically require costly hardware and additional digital markers, thereby limiting their accessibility and portability. Furthermore, air drawing demands considerable skill to achieve aesthetic results. To address these challenges, we introduce the concept of AirSketch, aimed at generating faithful and visually coherent sketches directly from hand motions, eliminating the need for complicated headsets or markers. We devise a simple augmentation-based self-supervised training procedure, enabling a controllable image diffusion model to learn to translate from highly noisy hand tracking images to clean, aesthetically pleasing sketches, while preserving the essential visual cues from the original tracking data. We present two air drawing datasets to study this problem. Our findings demonstrate that beyond producing photo-realistic images from precise spatial inputs, controllable image diffusion can effectively produce a refined, clear sketch from a noisy input. Our work serves as an initial step towards marker-less air drawing and reveals distinct applications of controllable diffusion models to AirSketch and AR/VR in general.

AirSketch: Generative Motion to Sketch
Hui Xian Grace Lim
Xuanming Cui xu9790220ucf..o edu
Yogesh S Rawat yogesh&erev. uuf.ucuuu
Ser-Nam Lim
hxgraceduef.co edu
sernamßucf edu
University Central Florida
Tord Tockking
Ganeond
Test Toviing
Geverna
Goorsten
Figure Given hand drawing video, we first use hand detection algorithm extract the tracking image (Hand Tracking). We then take this noisy tracking image and generate clean sketch (Generated), which faithfully and aesthetically resembles the intended sketch (Ground-truth).
Abstract
Illustration is fundamental mode of human expression and communication. Cer- tain types of motion that accompany speech can provide this illustrative mode of communication. While Augmented and Virtual Reality technologies (AR/VR) have introduced tools for producing drawings with hand motions (air drawing), they typ- ically require costly hardware and additional digital markers, thereby limiting their accessibility and portability. Furthermore, air drawing demands considerable skill to achieve aesthetic results. To address these challenges, AirSketch, aimed at generating faithful and visually coherent sketches directly from hand motions, eliminating the need for complicated headsets markers. We devise a simple augmentation based self-supervised training procedure, enabling controllable image diffusion model to learn translate from highly noisy hand tracking images to clean, aesthetically pleasing sketches, while preserving the from the original tracking data. We present air drawing
introduce the concept
essential visual datasets to study this problem. Our findings demonstrate that beyond producing photo-realistic images from precise spatial inputs, controllable image diffusion effectively produce a refined, clear sketch from a noisy input. Our work serves an initial step towards marker-less air drawing and reveals distinct applications of controllable diffusion models to AirSketch and AR/VR in general. Code and dataset are available under:
com/bxxgrcce//DddeeFsll.
Authors contributed equally.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Hand gestures are an essential element communication 43 air drawing) can depict visual aspects of an object. This form of expression is frequently used to visually supplement verbal communication and is used in various practical applications, including conceptual discussions, overcoming language barriers, and aiding in visual design.
In particular, iconic hand motions (i.e.
Popular AR/VR tools like Google's Tiltbrush (20) and HoloARt (5) create visuals via hand motions, but are inconvenient. These applications generally require head-mounted displays and digital hand markers, which are costly and hinder portability. Furthermore, their weight and temperature make them unsuitable for continuous use [60, [ 13). and so are inconvenient for spontaneous usage. Yet, these devices provide accurate positioning, stabilization, and varied brush controls and are crucial to producing high-quality drawings.
Can we generate sketches from hand motions without additional sensors markers? In order to enhance accessibility and convenience, we aim to generate sketches from hand motions videos captured using any standard camera embedded in devices like smartphones smart glasses.
One could clearly deploy hand tracking algorithms 2 74 to turn these hand motion videos into sketches However, creating air drawings with hand tracking algorithm alone presents several challenges. These include the user's drawing ability, physical fatigue, and inaccuracies in hand tracking. Noise in hand tracking can severely distort sketch, rendering it almost unrecognizable.
The objective. therefore, to generate clean sketches that faithfully represent the user's intent, from highly noisy and distorted hand motion input. This task requires the model to possess a deep understanding of shape and object priors, enabling to discern and correct deformed motion cues while filtering out undesirable noise. We refer to this task as Generative Motion to Sketch.
There are many approaches this task, with different architecture and data modalities. The input modality might include learned video representations, coordinate sequences from a hand tracking algorithm, or a rasterized image. Depending the modality, the task may also be reformulated as video-to-sketch, sequence-to-seqeenee introduces interesting opportunities for rich exploration of all these diverse approaches.
smage-to-image or a combination thereof. This diversity
We explore the use of controllable image Diffusion Models (DM) in generating sketches from motion. Existing work such as ControlNet (76) and T21Adapter 4 generate photo-realistic images given spatially-pres -preciss conditioning images. We explore a different use case by using controllable DMs to "reconstruct" clean sketches from severely distorted and noisy input images obtained with hand tracking algorithm. We propose simple, augmentation-ban based, self-supervised training procedure and construct two air drawings datasets for evaluation purposes.
Our experiments show that with our augmentation basedd training, controllable image DMs are able to recognize and interpret correct visual cues from noisy tracking images, some of which even appear to be nearly unrecognizable to the human eye, and generate sketches faithfully and aesthetically, while being robust to unseen objects. Moreover, we show that through simple augmentations, the model is capable of sketch-completion and text-instructed stroke styling. Finally, we conduct ablations to investigate 1) the effects of different augmentations, 2) the contribution of text prompts to sketch generation and 3) the effect of different levels of input 'chaos' on the quality of resulting generations.
In summary, in this paper:
1. We conduct pilot study of AirSketch, sketch generation from marker-less air drawing, and provide two air drawing datasets for evaluation.
2. We propose a controllable DM approach that generates faithful and aesthetic sketches from air-drawn tracking images with self-supervised, augmentation-baal training procedure, and conduct ablation studies to prove its effectiveness and robustness.
3. We explore different way of using spatially-conditioned DMs and reveal some of their interesting properties in the context of sketch generation. We hope our experiments shed new light on understanding the properties of controllable DMs and facilitate future research.

---

2 Related Works
2.1 Sketching in AR/VR
There are many methods for drawing in AR and VR. Applications such Google's Tilt Brush (20]. Mozilla's s A-Painter [45). Oculus' Quill [48), and HoloARt [5] display user strokes as lines or
tubes that can extend in all three dimensions. Many sketching applications such as these require
combination of VR/AR headsets and controllers to use. Since drawing freehand with six degrees of freedom makes it difficult to draw steady lines and surfaces, applications like AdaptiBrush (59) and
Drawing on Air 28 use trajectory-basee motion of ribbons to render strokes predictably and reliably.
Just-a-Line [21] and ARCore Drawing [26] are smartphone-based AR drawing applications, where users draw lines by moving smartphone in the air [21]. or drawing on smartphone screen [26].
2.2 Sketch Applications
Representing Sketches. A sketch can be represented in both vector and pixel space, and with varying levels of abstraction. A sketch can be represented as a rasterized image [40 75
50 37 32
sequence of coordinates [23] [6. [8]. a set of Bezier curves [67 [8]. or a combination g These
representation methods are suited for different tasks. For example, the rasterized image representation
is commonly used in Sketch-Basses Image Retrieval (SBIR) in order to compare sketches with images, while the coordinate sequence representation is often used for sketch generation. On the other hand, sketch can also depict the same object at varying abstraction levels, thereby imposing further challenges on downstream tasks, especially rketch-based retrieval in 2D 33
62 and
3D 66 41 14 31.
Sketch Generation. Most existing sketch generation methods adopt the vector representation 23 56 38 and view sketch generation as an auto-regressive problem on coordinate sequences. These sequences are typically represented by lists of tuple (őz, dy. p). where dz and 6y represent the offset distances of x and y from the previous point, and p a one-hot vector indicating the state of the pen. Sketch-RNN 23 uses a Variational Autoencoder (VAE) with a bi-directional RNN as the encoder and an auto-regressive RNN as the decoder. Sketch- Bert training regime. Instead of ruto-regresiive sketch generation, SketchGAN [39 takes in rasterized sketch image and uses a cascade GAN to iteratively complete the sketch. Sketch-Pix2Seg [12] adds vision encoder on top of Sketch-RNN and thus allow the model to take in an image input and reconstruct the sketch using coordinate sequences.
3X follows the BERT [5] model design and
Nonetheless, these methods generate sketches in the input sketch modality by taking in sketch and reconstructing the exact same sketch, or by predicting endings given incomplete sketches. In contrast, our work considers the task for generating sketches from hand motions. Specifically, we adopt the image representation for sketch generation as opposed to using coordinate sequences, This offers several advantages: it 1) maintains constant computation complexity with regards to sketch length, 2) drawing- invariant, which is especially favorable in our case as we consider extremely noisy sketches input, and 3) allows us to utilize large pretrained image generative models.
2.3 Image Diff'usion Models
Diffusion Probabilistic Model. First introduced by Sohl-Dickstein et al. [64] diffusion proba-
bilistic models have been widely applied in image generation [24 655 29 To reduce computational cost, Latent Diffusion Models [57] project input images to lower dimension latent space, where
diffusion steps are performed. Text-to-image diffusion models 47 55 54
61 51)
achieve text
control over image generation by fusing text embeddings obtained by pre-trained language models such as CLIP [52) and T5 [53] into diffusion UNet (58) via cross attention.
Controllable Image Generation. Beyond text, various recent works have focused on allowing more fine-graine controls over the diffusion process. This can be done via prompt engineering [73] 36 78) manipulating cross-attention
10 27 71), or training additional conditioning adapters [44 76
ControlNet (76) learns trainable copy of the frozen UNet encoder and connects it to the original UNet via zero convolution. T21Adapter trains lightweight adapter to predict residuals that are added to the UNet encoder at multiple scales.
3

---

Figure 2: Examples of air drawing videos. Left: synthetic hand drawing; Right: real hand drawing.
In particular, both ControlNet and T2IAdapter take spatially-precise conditional images such as canny edge, depth, and skeleton images extracted from the original image. Koley al. [33 considers human sketches as a condition and observes that human sketches have shapes that are deformed compared to canny edge or depth images, resulting in generations that lack photo-realism. Therefore, they avoid using the apatial-conditioning approach by training sketch adapter to transform the sketch into language tokens that replace the text embeddings as the input to the cross attention.
In our work, we show that the caatial-onditiooing does not have to be "hard" constraint, at least not in the domain of sketch generation. Through proper augmentations, we discover that we can indeed teach ControlNet to map from noisy input condition to clean output.
2.4 Hand tracking and gesture recognition
Hand tracking and gesture recognition are used for several purposes, including communication and interaction in an AR environment [2] [4 and VR [I], for image-based pose estimation in sign language recognition 7
and many others [35] [TI]. Many of these require depth-sensing hardware, such as work done by Oikonomidis et. al. using a Kinect sensor 49). or use deep learning for pose estimation
making difficult to integrate them into lightweight systems.
Hand pose estimation such as MediaPipe Hands (74] and OpenPose [9] take in RGB images or video and return the coordinates of 21 landmarks for each hand detected, and MediaPipe Hands is lightweight enough to integrate into even mobile devices.
3 Air-Drawing Dataset
In order to thoroughly evaluate our model, we need datasets with sketch-video pairs. Popular sketch datasets include Sketchy (33. TUBerlin (17), and Quick, Draw! 22 There are also hand motion datasets that associate hand motions with semantic meaning. such as Sign Language MNIST [66). How2Sign [16). and the American Sign Language Lexicon Video Dataset [46]. However, there are no datasets that associate sketches with air drawing hand motions, prompting us to create our own sketch-video pair datasets for evaluation purposes.
Synthetic Air-Drawing Dataset. We use samples from the Quick, Draw! dataset 22 as the intended ground-trus truth sketch; each stroke is represented as a sequence of timestamps and coordinates. 3D human arm asset animated in the Unity engine [30] using inverse kinematics and rotation constraints, see Figure 2 (left). While following stroke coordinates, the index finger is extended, and when the stroke ends, the hand is in a closed fist. The videos have an aspect ratio of 1920 by 1080 pixels and were recorded at 60 frames per second. We choose 50 common object categories from Quick, Draw! dataset, each with 100 sketches form our synthetic dataset with a total of 5000 sketch-videa pairs. This synthetic Air-Drawing dataset simulates the scenario where users have perfect drawing ability, and the errors are solely introduced by the tracking algorithm.
Real Air- Air-Drawing Dataset. A human user attempts to replicate sketches from the Quick, Draw! dataset by moving their index finger through the air. The videos were recorded with aspect ratio of 1280 by 720 pixels and at 30 frames per second. We take 10 samples per category used in the synthetic dataset, resulting in 500 video-sketch pairs.

---

Prompt:
fish
Training
Random
CLiP
ControlNet
Augmentations
Jitters
Decoder
Distortion
blok
Lx
Brase
Misplacement
x.
Inference Result
Inference
Downbook
mack
Trainable
Hand tracking Algorithm
Frozen
Figure 3: The overall pipeline for training and inference. During training, randomly apply augmentations to the original ground-truth sketch form distorted view. The distorted view is passed to ControlNet, which asked to generate original, undistorted sketch. During inference, a hand- hand-tracking algorithm used on hand motion video to create the input.
4 Methods
Our method trains a controllable DM to recover the clean sketch from a noisy one, which we discuss in Sections 4.1 and 4.2 As input training time, simulate noisy sketches produced from the hand tracking algorithm, discussed in Section [.3] Finally, we evaluate our model using two different datasets: dataset of 3D animated hand motion videos, and small dataset of real hand motion videos. The creation of these two datasets is discussed Section 3
direct application
4.1 Preliminary: Controllable DMs
Diffusion models involve forward and inverse diffusion process. Given input image æo, the forward process gradually adds noise to 2g to form a noisy input æt at time step t as:
Ft = v 00 (VT o4)c, (O,
where 0e = IT Ols, and az = 3r is determined by pre-defined noise scheduler 24
The reverse process trains denoising UNet eg(.) to predict the noise added to input 2p- In the context of controllable generation, in ControlNet (76) and T21Adapter [44, with of conditions including text prompt c and an additional condition c₁, the overall loss can be defined as:
C Ben.ttce,,,,,-N(((0.1. ||e E0 t, Ct, c)l
4.2 Training Controllable DMs for Sketch Recovery
We adopt ControlNet [76 as our primary controlling approach. Our training procedure is illustrated in Figure 3 Due to the lack of sketch-video pair datasets, we devise a self-supervised, augmentation- based training procedure. During training, for each sketch image, we randomly sample combinations of augmentations A(-) and apply to zo to get the distorted view A(zo). ). It is then used as the input to ControlNet's conditioning adapter. Hence, the loss function can be re-written as:
L Ego. B.o.t..c..eneeNV...)) ||e
co( Ct2 A(ar)))
Therefore, unlike regular controllable DMs where the conditioning adapter takes in edge-like maps and predicts spatial-conditioning signals be injected to the UNet, adapter learns both the spatial-conditionnng signals and a mapping from the distorted to the clean input: A(xo) + Zn-
4.3 Sketch Augmentations
We categorize the prevalent errors from air drawings into three types: 1) user-induced artifacts such as hand jitters and stroke distortions, 2) hand tracking errors such as inaccurate hand landmark predictions, unintended strokes, and 3) aesthetic shortcomings related to the user's drawing proficiency.

---

In order to closely replicate these artifacts, we carefully examine typical noise found in real tracking samples and divide them into 3 categories: local, structural, and false strokes. For each category. we observe several types of artifacts, and apply augmentations to introduce each artifact to an input sketch. Visual examples for these augmentations are shown in Figure
Original
Distortion
Spikes
Distort&& Misolacement
Stroke Resize
Transitional
Rondom
Local Augmentation
Structural Augmentation
False Strokes
Figure 4: Visual examples of different augmentations.
Local artifacts include jitters, stroke-wise distortion, and random spikes. Jitters are common and become especially obvious when drawing speed is low. Stroke-wise distortion happens when fail to properly draw the desired shape primitive (e.g. drawing a rectangle but ending up with unwanted curvy edges), potentially due lack of friction and visual landmarks. Random spikes can arise from an accidental jerk of a user's hand or an incorrect detection from the tracking algorithm.
Structural artifacts include sketch-level distortion, incorrect stroke size, and misplacement. Sketch- level distortion refers to an overall distorted aspect ratio; incorrect stroke size and misplacement occur when the user unintentionally draws stroke too small/large, and/or at the wrong position.
False strokes refer to unintentionally drawn strokes which commonly occur in three situations: entering or exiting the canvas, transitioning between strokes, and hesitating during drawing.
Unlike real images where shapes and scales strictly defined, sketches more versatile and thus do not have a clear boundary for correctness. For example, a slight change of scale during augmentation does falsify the sketch. Therefore, we carefully tune the augmentation parameters such that the resulting augmented sketches are, in general, obviously incorrect.
Experiments
Datasets and Implementation Details. In training, we subset of 100 categories from the Quick, Draw! dataset (22). Because large portion of Quick, Draw! sketches are not well-drawn, we first calculate the CLIP Image-Text similarity between each sketch and their corresponding category and select the top 5% from each category, resulting in 60K sketches. Note that the sketches used for training are mutually exclusive with the sketches used for generating synthetic and real tracking images used during evaluation. To test for generalizability, we select 10 categories with similar statistics from the rest and exclude them from training.
We primarily use Stable Diffusion XL (51) (SDXL) in our experiments, and adhere to the original ControlNet training and inference procedures. During both training and inference phases, we use text prompts in the format of "a black and white sketch of a <category>" to guide the model generation. We also finetune SDXL the Quick, Draw! dataset with Low-Rank Adaptation (25) (LoRA) in order to "equip" the model with the basic ability to generate sketches in the appropriate style.
Evaluation Metrics. We primarily focus on using faithfulness, or the similarity between the generated sketch and the ground-truth sketch, as our model performance. Due to the versatility of sketches, we adopt multiple metrics to ensure comprehensive measurements. On the pixel-level, we use SSIM (70) to measure detailed local structural similarity, and Chamfer Distance (2) (CD) for global comparison, as CD is less sensitive to local density mismatch. Taking perceptual perspective, we adopt LPIPS [77]. CLIP (52) Image-to-Image similarity (I2I), and CLIP Image-to- Text similarity (I2T) between sketches and their corresponding text prompts to measure "recognizability"
We benchmark our model on the similarity between the ground -ruth sketch and the hand tracking image. We then train ControlNet on sketches but without any augmentation as our second baseline.
Details in Appendix A.2

---

Table 1: Results the similarity between generated and ground-truth sketches from Quick, Draw! dataset. "Tracking" refers hand tracking images, and "Gen." refers to generated images. "w/ Aug." " refers to whether sketch augmentations have been applied. "CLIP 121/2TT refers to CLIP Image-to-Iman -oa-to--/ag//mmg--tooooo Text similarity.
Dataset
Backhone
Aug.
SSIM (f)
CD (
LPIPS ()) (2) CLIP 12T (T)
Seen Categories
Tracking
synth.
0.59
20.12
0.36
0.80
0.22
Gen. synth.
SDXL.
X
0.59
20.11
0.37
0.79
0.23
Gen. synth.
SD1.5
V
0.60
17.98
0.35
0.80
0.26
Gen. synth.
SDXL
0.64
17.39
0.33
0.85
0.28
Tracking
real
0.55
32.36
0.42
0.76
0.21
Gen. real
SDXL
X
0.55
31,99
0,41
0.79
0.21
Gen. real
SD1.5
0.59
27.59
0.38
0,80
0.27
Gen. real
SDXL
0.64
25.46
0.36
0.84
0.29
Unseen Categories
Tracking synth.
0.59
20.47
0.36
0,80
0.22
synth.
SDXL.
0.59
20.32
0.35
0.81
0.22
Gen. synth.
SD1.5
0.60
17.50
0.35
0.80
0,26
Gen. synth.
SDXI.
0.64
17.27
0.34
0.85
0.27
Tracking
real
0.54
33.92
0.42
0.76
0.21
Gen.
real
SDXL SD1.5 SDXL.
X
0.55
33.53
0.41
0.78
0.21
Gen. real
0.61
27.67
0.38
0.80
0.27
Gen.
real
24.26
0.38
0,85
0.28
Govedd-mah
Grendemahh
Trackin
Gaveradd
Granndmm
SLI
Figure 5: Generations TUBerlin datasct.
5.1 Results and Analysis
Faithfulness. In Figure D we clearly observe the ControlNet trained with augmentations successfully identifies the visual cues from the noisy tracking image, removes artifacts, and generates the clean sketches, while being aesthetic and semantically coherent. Unsurprisingly, ControlNet trained without augmentations fails to make any improvement from the tracking. In Figure 5
we show
additional results where the model trained on TUBerlin [7 dataset.
Table shows quantitative results for measuring the faithfulness of generated sketch images to ground-truth. For both synthetic and real datasets, observe noticeable performance gain. For example, with the real dataset and SDXL, SSIM increases by 10% in SSIM, LPIPS decreases by 6%, and CD decreases by 21%.
In Table 2 and Figure 6 we show comparison between ours and Sketch-Pix2Seq (P2S). Since P2S performs best when the model is trained only one category, randomly pick 10 categories

---

Table 2: Quantitative comparison be- tween Peetch-Pix2Seq (P2S.) and ours on a subset of 10 classes.
Figure 6: A comparison between generations from Sketch-Pix2Seg (top) and ours (bottom).
(1)
(D)
121 (?) (TT
Tracking
0.5
32.36
0.21
P2S.
0.58
30.19
0.82
0.26
Tradkon
Connersed
Gronddmeh
Larrand-trih
Figure 8: Generations on unseen categories from Quick, Draw! dataset.
train 10 P2S models, and compare with ours using the selected categories. For each P2S model, following the original work, we first train 60M steps for reconstructing the exact same input sketch, and subsequently train 40M steps for constructing the clean sketch given the noisy tracking image. In Table 2 our SSIM and CD are noticeably higher than P2S, suggesting better faithfulness. This validated Figure 6 where can observe while P2S is able generate semantically correct sketches, it fails to follow the input tracking faithfully. Moreover, as P2S requires to train separate model on each individual category and has no generalizability, hard to adapt to real-world usage.
In Figure 7 we show visualizations of the ControlNet hidden states across denoising steps for both the base- line model, trained with the original ControlNet recipe (top row), and our model with noise-augmented training (bottom row). The baseline Control- Net hidden states closely approximate the input edge structure early in the de- noising process but fail converge to- ward a clean sketch representation, re- sulting in largely static visualizations.
16
20
25
Genernaiio
Imper
Figure 7: A comparison of visualization of ControlNet hidden states throughout denoising process from baseline approach without augmentation (top) and with augmentations (bottom).
In contrast, the hidden states from our augmete training progressively reveal coherent and accurate sketch outline throughout the denoising process.
Generalizability. From Figure g are not seen during training, suggesting that the trained ControlNet learns a robust category-agnostt mapping from the noisy to clean sketch. The generalizability is also verified in Table [ where the resulting scores for unseen categories are close to scores obtained on the seen categories.
that model generalizes well categories which
5.2 Ablations
Role of Augmentations. In Table we provide similarity scores between ground-truth and gener- ated sketches when different combinations of augmentations are applied in training. We observe that local augmentations plays an important role detailed local similarity: with only local augmentation applied, SSIM increases more than other metrics, with 10% improvement to the baseline, yet increases by only 4% and 0% when only false strokes and structural augmentations are applied. Con- versely, false strokes and structural augmentation have larger effects the global similarity of the

---

Table 3: Model trained seen categories with different combinations of augmentations. All the experiments are conducted using Stable D and trained for steps.
Local
Structural
False Strokes
SSIM (1)
CD (U)
LPIPS (4)
CLIP 121 (1)
CLIP 12T (t)
Orig. vs Tracking
0.55
32.36
0.42
0.76
0.21
0.60
31.49
0.39
0.83
0.27
0.55
31.14
0,41
0.80
0.24
0.57
29.04
0.40
0.82
0.24
0.60
31.21
0.39
0.83
0.28
0.61
29.82
0.37
0.83
0.28
0.60
28.97
0.39
0.82
0.23
0.62
27.33
0.37
0.84
0.29
Tracking
wi Pronet
wio Pronpc
Ground Irthh
Tracking
wi Paoroge
Pronntt
Ground-muzi
Tracking
Ground
Figure 10:
Examples of incorrect generation on unseen categories due to the absence of text prompt (w/o Prompt), comparing to correct generations when prompt is present (w/ Prompt).
generated sketch: CD decreases by 9% and 3% when only false strokes and structural augmentations are applied, respectively.
In Figure we provide visual results when different combinations of sketch augmentations applied during training. We observe that local augmentations are indeed crucial to removing jitters and correcting deformed lines, while false stroke augmentations ensure that the model does not falsely follow the spatial-cor ppatial-conditions introduced by these false strokes. The structural augmentations are less significant, likely because structural artifacts are common as local artifacts and false strokes.
Effect of Text Prompts. In- put tracking image conditions are extremely noisy and may not even possess obvious visual cues about the nature of the intended sketch. It then important to investigate the effect of text guid- ance on the generated output and examine if our augmentation- based training significantly con- tributes to the generation, if it guided purely by text prompts.
Tracking
Studuaal Fabe Stoke
Lical
Folse Smooe Stractnn -Seactanl
Local
False Shokk LLoaa
Groand-
tralk
In Table we show the similarity scores between the ground-truth and generated sketches with or without prompt on seen or un- categories. When no aug- mentation is applied during train- ing, there is little to perfor- mance gain, even with a text prompt. When augmentations are present, in most cases, there is noticeable improvement across all metrics. Such results verify the necessity of our augmentation- based training procedure.
Figure 9: Qualitative results when different combinations of augmenta- tions are applied during training. Each column represents the generated sketches when the model is trained with combination of augmen- tations, c.g. Column are generated sketches when only structural augmentations are applicd during training.
We find that the model tends to rely more on text prompts when working with unseen categories. When prompts are not present, CLIP I2T and CD drop 24.7% 4.77% and 21.4% respectively on the unseen categories, versus decrease of 4.9% and 10.3% on the seen categories. In Figure I0 show

---

Table 4: Quantitative results real air drawing dataset with/without prompts, "w/ Aug." refers to with/without augmentations. "w/ P." refers to with/without prompts. "12I" refers to CLIP Image-Image similarity between trackingggenertted and ground-truth sketch, and "12T" refers CI.I CLIP Image- Text similarity between sketches texts associated with their class labels.
Figure 11: The impact of level of chaos of the track- ing on the faithfulness on CD. We divide CD between tracking and ground- truth sketch equally into four bins. Within each bin, we plot the mean of CD between ground-truth sketch, tracking, and generated image with/without prompt.
generated sketch, based
Seen Categories
w/ Aug. w/ P.
SSIM (1) CD CD(L) LPIPS (1)
121 (1) 12T (1)
Tookin
Gr vi hoepy so Proos
Seen Categories
Onig. Chamfer Distance
Tracking
0.55
32.36
0.42
0.76
0.21
201
X
0.55
0.79
0.21
X
X
0.56
32.09
0.40
0.78
0.21
0.63
26.70
0.36
0.81
0.26
Unseen Categories
Unseen Categorics
"Trakiing
0.54
33.92
0.42
0.76
0.21
X
0.56
33.66
0.40
0.78
0.22
0.63
24.26
0.38
0.84
0.28
0.61
30.47
0.39
0.80
0.23
(2577 48 (12) Orig. vs Tracking Chamfer Distance
(8704), R5.77
examples the model failing to generate the correct sketch when no prompt is present, under unseen categories. Consider the bottom left example of Figure T0] the ground-truth sketch (Ground-truth) represents a snail. With the corresponding text prompt, the model successfully generates a sketch of snail, following visual cues in the tracking image (Tracking) such the spiral shape. When prompt is given, the model falsely, yet reasonably, generates a fan (w/o Prompt) a category seen during training. In fact, in the absence of prompt and given input from an unseen category, we find that the failure cases tend to be generated sketches of the seen categories.
investigate how the amount of chaos in the imput conditioning tracking image, measured by the CD between the ground-trum and tracking images, affects generation. The improvements ground-truth and tracking images is large. On the other hand, the CD between ground-truth and generated sketches almost the same when the tracking image is already close the ground-truth sketch, indicating that the generation is being faithful to the ground-truth sketch. We also observe that when the level of chaos is high for the unseen categories (bottom right), the performance gain with text guidance is most obvious, suggesting that the model relies more on text guidance for correct generation. Conversely, the model depends less on the text prompts when the input chaos level is low or under seen categories. Such results are also in line with Figure and Table 4 as discussed above.
Effect of level of chaos conditional input. In Figure
more obvious when the CD between the
Conclusions
In this paper, we tackle the problem of marker-less air drawing by leveraging spatially-controlledd diffusion model. We devise a simple augmentation- based data-free training procedure to learn mapping from noisy to clean sketches. We collect two hand drawing datasets and verify that the trained model can effectively generate coherent and faithful sketch from an extremely noisy tracking image, and exhibits decent generalizability.
Limitations and Future Work.
We provide a stepping stone for the proposed task by creating a framework that establishes correspondence between hand motions and clean, pleasing, and coherent illustrations, and providing datasets to evaluate these tasks. However, this work does not explore the possibility of using hand gestures to create complex, full color images that image diffusers are known for. It also assumes that the desired output sketch simple and often cartoon-lite as a majority of the Quick, Draw! sketches are drawn with few lines and simple shapes.
Societal Impact. Generative models for media are generally prone to misuse which can lead the spread of inappropriate and harmful content. While this work is based in the sketch domain, there could still be adverse adoption for generating mocking content and bullying, especially by younger users.
10

---

References
[1] Alex Adkins. The Importance of Hand Motions for Communication and Interaction in Virtual Reality. PhD thesis, Clemson University, 2022.
[2] M. Akmal Butt and P. Maragos. Optimum design of chamfer distance transforms. IEEE Transactions Image Processing, 7(10):1477-1484, 1998. doi: 10.1109/83.718477.
[3] Alaa Al-Obodi, Ameerh Al-Hanine, Khalda Al-Harbi, Maryam Al-Dawas, and Amal Al- Shargabi. A saudi sign language recognition system based on convolutional neural networks. Building Services Engineering Research and Technology. 13, 11 2020. doi: 10.37624/JJER/113.
11 11.2020.3328-333
[4] Rahaf Alawwad, Ouiem Behir, and Mohamed Maher Ben Ismail. Arabic sign language recognition using faster r-cnn. International Journal of Advanced Computer Science and Applications, 12:692-700, 03 2021. doi: 10. 00.14466//AAASAA2220.00000
[5] Judith Amores Fernandez and Jaron Lanier. Holoart video showcase: Painting with holograms in mixed reality. pages 466-466, 05 2017. doi: 10. 0..14553322763333449777.
[6] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In 2022 /EEE/CVF Conference Computer Vision and Puttern Recognition (CVPR). IEEE, June 2022. doi: 10. 10. 11100ccc::2288822222222222 220.5226688.22201666 URL BRR.http://dd doi org/
10. 1001/ 10..119//CVPR520 CWW.02688PR52688.. 2022.002222222222 01767
[7] Qanita Bani Baker, Nour Alqudah, Tibra Alsmadi, and Rasha Awawdeh. Image-based arabic sign language recognition system using transfer deep learning models. Applied Computational Intelligence and Soft Computing, 2023:1-11, 12 2023. doi: (0.11555/023/519500,
[8] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. Vectorization and rasterization: Self-supervised learning for sketch and handwriting, 2021.
[9] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. CoRR, abs/1812.008005 2018. URL
http://arxiv.. org abs / 1812. 08008
[10] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free free layout control with cross-attention guidance, 2023.
[11] Weiya Chen, Chenchen Yu, Chenyu Tu, Zehua Lyu, Jing Tang, Shiqi Ou, Yan Fu, and Zhidong Xue. A survey on hand pose estimation with wearable sensors and computer-vision-ue methods. Sensors, 20:1074, 02 2020. doi: 10.3390/s20041074.
[12] Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq; a model to generate sketches of multiple categories, 2017. URL
hbs////b///17999 04121
[13] Takanori Chihara and Akihiko Seo. Evaluation of physical workload affected by mass and center of mass of head-mounted display. Applied ergonomics, 68:204-212, 04 2018. doi: ab.1016/..peerr.201111 2017.
[14] Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Democratising 2d sketch 3d shape retrieval through pivoting. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 23218-23229, 2023. doi: 10.1109ICCV511000.223...3
[15] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[16] Amanda Cardoso Duarte, Shruti Palaskar, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giró-i-Nieto. How2sign: A large-scale multimodal dataset for continuous american sign language. CoRR, abs/2008.00 081 143,, 2020. URL URI. http:ttppss... tetttss//arree //arxiv
org/ a00//00.. 08143

---

[17]
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH), 31(4):441-444100 2012.
[18]
Rinon Gal, Yael Vinker, Yuval Alaluf, Amit H. Bermano, Daniel Cohen- Or, Ariel Shamir, and Gal Chechik. Breathing life into sketches using text-to-video priors. 2023.
[19] Liuhao Ge, Hui Liang. Junsong Yuan, and Daniel Thalmann. Robust 3d hand pose estimation in single depth images: from single-view cnn multi-view cnns. 06 2016. doi: 10.1109/CVPR. 2016.391.
[20] Google. Tiltbrush, 2016. https://www.iitruss..oom
[21] Google Creative Lab and Uncorked Studios.
Just a
line, March 2018 8.
[22] Google Research. Introducing the kaggle "quick. draw!" doodle recognition challenge, Septem- ber 2018. htw.:/ooggg.rerr..gggooooo
[23] David Ha and Douglas Eck. A neural representation of sketch drawings, 2017.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
[25] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models, 2021.
[26] Jonas Jongejan. ARCore Drawing, August 2017. drawing.
[27] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native cealeton-guided diffusion model for human image generation, 2023.
[28] Daniel Keefe. Robert Zeleznik, and David Laidlaw. Drawing on air: Input techniques for controlled 3d line illustration. IEEE Transactions on Visualization and Computer Graphics, 13 (5):1067-108,, 2007. doi: 10.1 1011009/T...00...... 1060.
[29] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023.
[30] Knife
Entertainment.
Realistic 100e
FPS
hands,
2018.
[31] Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song. How to handle sketch-abstraction in sketch-based image retrieval?, 2024.
[32] Subhadeep Koley, Ayan Kumar Bhunia, Anceshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song. You'll 'Il never walk alone: sketch and text duet for fine-grained image retrieval, 2024.
[33] Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song. It's all about your sketch: Democratising sketch control diffusion models, 2024.
[34] Taehee Lee and Tobias Höllerer. Handy ar: Markerless inspection of augmented reality objects using fingertip tracking. pages 83-90, 11 2007. ISBN 1--42244-14522-9 doi: 10.1 Wttp////...00..437333333
[35] Rui Li, Zhenyu Liu, and Jianrong Tan. A survey on 3d hand pose estimation: Cameras, methods, and datasets. Pattern Recognition, 93, 04 2019. doi: pat.oo//. 2019.00..019.044.. 9.044.26.
[36] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation, 2023.
[37] Fengyin Lin, Mingkang Li, Da Li, Timothy Hospedales, Yi-Zhe Song, and Yonggang Qi. Zero-shot everything sketch- based image retrieval, and in explainable style, 2023.
12

---

[38] Hangyu Lin, Yanwei Fu, Yu-Gang Jiang, and Xiangyang Xue. Sketch-bert: Learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt, 2020.
[39] Fang Liu, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang. Sketchgan: Joint sketch completion and recognition with generative adversarial network. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5823-5832, 2019.
URL. https://api stw..pppo.emmatiiccccollcll.. org/ hece/Corpuus.D... 182164433
[40] Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao. Deep sketch hashing: Fast free-hand sketch-based image retrieval, 2017.
[41] Ling Luo, Yulia Gryaditskaya, Tao Xiang, and Yi-Zhe Song. Structure-aware 3d vr sketch to 3d shape retrieval, 2022.
[42] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: A hands-on survey. IEEE Transactions on Visualization and Computer Graphics, 22, 01 2016. doi: 10.1 1..100/..2222.5555555 253408.
[43] David McNeill. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press., Chicago, Illinois, USA, 1992.
[44] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 2023.
[45] Mozilla. A-painter, 2016.
[46] Carol Neidle, Ashwin Thangali, and Stan Sclaroff. Challenges in the development of the american sign language lexicon video dataset (asllvd) corpus. 05 2012.
[47] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
[48] Oculus and Smoothstep. Quill, 2016. https://quill.rtt.
[49] lason Oikonomidis, Nikolaos Kyriazis, and Antonis Argyros. Efficient model-based 3d tracking of hand articulations using kinect. volume 1, 01 2011. doi: 10.5244/C.25. 101.
[50] Kaiyue Pang. Ke Li, Yongxin Yang, Honggang Zhang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. Generalising fine-grained sketch-based image retrieval. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 677-686, 2019, doi: 10.1109//VPR.2219.0000
[51] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[53] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2023.
[54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation, 2021.
[55] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
[56] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer-basee representation for sketched structure, 2020.
13

---

[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[59] Enrique Rosales, Chrystiano Araújo, Jafet Rodriguez, Nicholas Vining. DongWook Yoon, and Alla Sheffer. Adaptibrush: adaptive general and predictable vr ribbon brush. ACM Trans. Graph., 40(6), 2021. doi: 10. 0..1445/3478513334885111 URL https://doi. orgo 1145/
3478513 3480511
[60] Michael Rupp. Is it getting hot in here? the effects of headset microclimate temperature perceived thermal discomfort, vr sickness, and skin temperature. Applied Ergonomics, 1 114, 2024.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022.
[62] Anceshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine-grained or not, 2023.
[63] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics, 35:1-12, 07 2016. doi: 10. 00.1145/297782222255555
[64] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256 2265, Lille, France, 07-09 Jul 2015. html
PMLR. URL UtL/
mlr (ttpss//88//sol1-iccsttttete
[65] Jamming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL [ttl/tttts///pperrrr//
Settfooummid-Saee Sdt.iiiiitttttiiH.t.
[66] tecperson. Sign language mnist: Drop-in replacement for mnist for hand gesture recognition htmmm//dttasess/tatamngeeee
tasks, 2016. https://ww.kagge/
mnist.
[67] Yael Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-awan object sketching, 2022.
[68] Chengde Wan, Thomas Probst, Lue Van Gool, and Angela Yao. Self-supervised 3d hand pose estimation through training by fitting. pages 10845-10854, 06 2019. doi: 10.1 109/CVPR.2019 01111.
[69] Fang Wang, Le Kang, and Yi Li. Sketch-based 3d shape retrieval using convolutional neural networks, 2015.
[70] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 1304):600-612, 2004. doi: 10. 109/TTP.2003.8 819861.
[71] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion, 2023.
[72] Yan Yan, Ke Chen, Yu Xie, Song Yiming, and Yonghong Liu. The Effects of Weight on Comfort of Virtual Reality Devices, pages 239-248. 01 2019. ISBN 978-3-319-94705-11 doi: 10.1007/978-3331994770 27.
14

---

[73] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation, 2022.
[74] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung. Chuo-Ling Chang, and Matthias Grundmann. Mediapipe hands: On-device real-time hand tracking. CoRR, abs/2006. 10214, 2020. URL https: /wwpvvaaavvv////////i//aa//aaaaa org/ wtggabs////geoooooo.gogggeee 2006 10214
[75] Jingyi Zhang, Fumin Shen, Li Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Luc Van Gool. Generative domain-migration hashing for sketch-to-image retrieval. In Proceeding of European Conference Computer Vision (ECCV), 2018.
[76] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models, 2023.
[77] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[78] Tianjun Zhang. Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with gpt-4, 2023.
15

---

A Appendix
A.1 Details for Sketch Augmentations
Implementation.
Below we provide implementation details for each augmentation:
Local Augmentation includes types of sub-augmentations: stroke distortion, random spikes, and jitters. For stroke distortion, we generate number of wave functions each with random frequency, shift, and amplitude. We aggregate them to form a random curve, whose value used distort the ground-truth sketch. Spikes are implemented in two modes sharp spikes and smooth spikes. Sharp spikes are determined by the width and height which are randomly sampled from normal distribution; smooth spikes are implemented with Bezier curve, and the control points are randomly sampled from uniform distribution within predefined range along the gradient of the ground-trum truth sketch. Jitters are simply small perturbations sampled from pre-defined normal distribution and added to random locations along the ground-trui -trtt sketch.
Structural Augmentation also include 3 types of sub-augmentations: sketch-wise distortion & relocation, stroke-wise misplacement, and stroke-wist resize. For sketch-wise distortion& relocation, we randomly shrink and change aspect ratio of the whole sketch and reposition the sketch in the canvas. For stroke-wise misplacement/resize, we randomly move/resize each stroke within a pre-defined range.
False Strokes includes two types: transitional and random false strokes. For transitional false strokes, we simply draw lines between the transition of each stroke; for random false strokes, we randomly draw number of extra lines the canvas.
In the above augmentations, misplacement and stroke resize under structural augmentation are mutuuall-excllsiivee applied. This is because it is highly probable that the combination of the two will completely destroy the visual cues of the resulting sketch image. For all other augmentations, we set 50% chance for each to be applied each sketch sample during training.
A.2 Training Details
Held-out categories. Because different object categories have different drawing difficulty, sketch and tracking samples belonging to different categories exhibit large variance in statistics (CLIP Image- Text similarity between ground-truth sketch and corresponding text, CLIP Image-Image similarity, Chamfer Distance, or SSIM between ground-truth and tracking image). To allow for fair comparison of the performance between seen and held-out categories, we first partition categories into 10 clusters using K-Means Clustering, and randomly sample one category from each cluster as the held-out categories. The held-out categories are:
[car, face, cow, snail, diamond, candle, angel, cat, grapes, sun)
Training configurations. All training is conducted diffusion model with LORA, we set LORA rank to 4, per device batch size to 16, learning rate to 55 gradient accumulation steps 4, and train for 6K steps the Quick, Draw! dataset. For our augmentation- based ControlNet training, we set per device batch size to 8, learning rate to -b gradient accumulation steps 4, and the proportion of empty prompts 25%.
Nvidia H100 GPUs. For finetuning the
A.3 Comparison on Egocentric Hand Tracking Algorithm
An example of hand landmarking with MediaPipe, OpenPose, and NRSM is shown in Figure 12 We can see that only MediaPipe's hand landmarker comparatively accurate and consistent, while OpenPose often fails to detect landmarks, and NSRM is less accurate. However, even though MediaPipe seems to accurately predict hand landmarks, the resulting tracking image, as shown in Figure and Figure 15 still contains large amount of noise.
16

---

OpenPose
Mediapipe
NSRM
Figure 12: A comparison hand landmarking done by MediaPipe, OpenPose, and NSRM respec- tively. MediaPipe in general provides most accurate hand landmarks, while OpenPose often struggling detect the hand, and NSRM not able to provide accurate landmarks.
?7
Figure 13: Auto-completion given a partially-drawn sketch in two modes: line segment missing (Column 1&5) and whole strokes missing (Column 3&7). Bordered images the generated results.
A.4 Sketch Completion & Text-conditioned styling.
also perform other sketch-based tasks such as sketch completion and text-conditioned styling. Unlike prior works 339 38 on sketch completion that require sophis- ticated architectures training procedures, could achieve sketch completion by simply adding random erasure to the set of augmentations during training. In Figure [3] we show two types of sketch completion: completion with small missing line segments (column 1 5),, and completion with whole strokes missing (column 3&7). In addition, as shown in Figure 14] we can also utilize the text-conditioning capacity of DMs prompt for different stroke styles, such as different color and thickness, therefore alleviating the need for fine-grained style control on the user's part.
With simple augmentations,
A.5 Visual Comparison between Controllable DMs
In Figure 15 we compare the generated sketches between ControlNet and T2IAdapter, when trained under our augmentation-bass procedure. We can observe that both ControlNet and T2IAdapter generate visually coherent sketches and generally follow the input tracking image. Nonetheless, by
Generated
Ground-truth
Tracking
Generated
Ground
Figure 14: Generation with text instructions for different line colors and brush thickness. Bordered images are the generated results. We use the prompt: "A sketch of <category>, <color> lines. <thickness> brush."
17

---

Hand Tracking
ControlNet
121Adapler
Ground-iruth
Hand Tracking
ControlNet
T21Adapler
Ground-truth
Figure 15: More inference results on ControlNet with our augmentation-basee based training procedure. We additionally show inference results using T21Adapter in method.
looking into details, ControlNet's generations are more faithful to the visual cues from the tracking images. For example, the butterfly (bottom right) generated by ControlNet resembles the shapes of the wings from tracking image closely, while the generation from T2IAdapter looks much less similar.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are true and accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See "Limitations and Future Work" in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There were no theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Paper details the model used, the data used in the training process, and the data used in inference. These are also all publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

reproducible, they Answer: [Yes]
Justification:
We release the full code and dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiuueee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper states that is follows ControlNet's provided training and inference methods, and states any time hyperparameter is different.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars not reported because it would be too computationally expensive; also, the improvement on quantitative results are obvious across multiple metrics, and are backed by qualitative results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix A.2.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///harrrppc.ccoo/p//////aa
Answer: [Yes]
Justification: Research conducted in the paper conform, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The work in this paper outputs sketch-like, oftentimes cartoon-like, images that might be misued for generating inappropriate memes, especially in the teenager category.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data and model used in this paper have already been vetted pose minimal risk for misuse by their creators.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing assets (i.e. Unity game engine and 3D assets used) were properly cited, see Section 3.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The creation and contents of the two new datasets are well documented in Section 3.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does involve crowdsourcing nor resesarch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing resesarch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.


Reviews:
[{"review_id": "MFQUVaNYPc", "reviewer": "Reviewer_V9Hw", "summary": "The authors investigate the problem of sketch generation from the finger's motion trajectory, which is interesting. To make it, the authors adopt a standard ControlNet pipeline, while highlighting the importance of data augmentation (adding noise to clean sketches to mimic the hand-tracking image). Hence the major contribution of this paper is the tweak that makes ControlNet applicable for generating clean sketches from the continuous messy finger trajectories.", "questions": "Please refer to the weaknesses.", "limitations": "The major limitation may be the limited practical usage as I listed in weaknesses.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors repurpose ControlNet to produce a clean sketch from a severely deformed and line-messy input, which is useful for finger motion to sketch generation and could be potentially valuable for other related HCI applications. \n- The results look promising, and the application of sketch-completion and text-instructed stroke styling are interesting.\n- The experimental results are extensive and validate the effectiveness of the proposed augmentation strategies.", "weaknesses": "- The major concern is about the novelty of the technical contribution. Personally, the proposed augmentation methods are more like practical tricks for better cleaning up messy sketches.\n- The methods of accepting raster-format input for sketch generation should be also compared by training these models with the same augmented sketches, such as [a][b][c].\n- The practical usage seems limited since: (1) the proposed method is only validated on simple sketches, so no way to justify whether any scalability issue when facing complex scenarios; (2) the user probably needs to edit (e.g., deleting lines) while drawing, how to deal with it?\n- The collected real air-drawing dataset is very small.\n\n[a] Chen, Yajing, et al. \"Sketch-pix2seq: a model to generate sketches of multiple categories.\"\n\n[b] Yang, Lan, et al. \"Sketchaa: Abstract representation for abstract sketches.\" \n\n[c] Zang, Sicong, Shikui Tu, and Lei Xu. \"Controllable stroke-based sketch synthesis from a self-organized latent space.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Concern about the novelty of the paper**\nThank you for your feedback. While augmentation is a well-known method, it has never been applied for sketch generation, and our finding that it can be used as an effective self-supervised pre-training task for airsketch is in our opinion a non-trivial contribution. Further, we would like to reiterate that the contribution of our work is not only about an implementation ‘trick’, but also \n1. a novel task of generative motion to sketch, with a dataset. There is no other work, including those cited by the reviewer, that achieves the task nor was the task previously proposed. We believe it will open the doors to many interesting applications by bringing generative modeling into the AR space. \n2. A concrete baseline method that has been thoroughly evaluated and analyzed, with performance that we believe will be useful for future research and applications. \n3. A novel way of viewing and exploiting controllable diffusion models.\nWe are happy to discuss this further.\t\n#### **Comparison to raster-format input for sketch generation**\nWe thank the reviewer for mentioning these works. As per requested, we run Sketch-Pix2Seq on a subset of 10 classes. For each class, we train a separate Sketch-Pix2Seq model by first training 50K steps without augmentation, followed by another 50K steps with augmentation. Results are summarized in Table [1] below and figure [4](left) in the rebuttal PDF. In particular, we find that while Sketch-Pix2Seq produces semantically coherent sketches, it often cannot identify and follow visual cues from the messy tracking image, as shown in Figure [1](left). It thus leads to a noticeable improvement in terms of CLIP I2I/I2T, but a marginal improvement over SSIM/CD.\n\n|                   | SSIM (↑) | CD (↓)       | LPIPS (↓)     | CLIP I2I (↑) | CLIP I2T (↑) |\n|-------------------|----------|--------------|---------------|--------------|--------------|\n| Tracking          | 0.5      | 32.36        | 0.42          | 0.76         | 0.21         |\n| Sketch-Pix2Seq    | 0.58±1.7$e^{-3}$ | 30.19±0.60 | 0.38±1.5$e^{-3}$  | 0.82±0.78$e^{-3}$ | 0.26±0.41$e^{-3}$ |\n| Ours              | **0.63**±0.21$e^{-3}$ | **25.45**±0.26 | **0.36**±1.1$e^{-3}$ | **0.84**±2.0$e^{-3}$ | **0.29**±0.86$e^{-3}$ |\n\nIn addition, we also want to point out that works like [a][c] could only fit a model on one or few classes at a time and struggle to generalize beyond – while both [a][c] aim at learning across multiple classes, their maximum number of categories is only 5, which is far less than what is needed in our scenario. Therefore we did not include them for benchmarking. \n\n#### **Practical usage**\n(1) *-- \"the proposed method is only validated on simple sketches, so no way to justify whether any scalability issue when facing complex scenarios\"*\n* We run additional experiments on the TUBerlin dataset. Results are summarized in Figure [1] in the rebuttal PDF, which shows our method is able to generate faithful and coherent sketches given more complex objects.\n* All previous works in sketch generation [a][c] also only consider simple sketches from Quick, Draw! dataset.\n\n(2) *-- \"the user probably needs to edit (e.g., deleting lines) while drawing, how to deal with it?\"*\n\nWhile sketch editing is indeed an interesting topic, we believe this may be out of scope of this work and could become an interesting future direction. \n\nAs our work is the first to study motion to sketch in a generative manner, we acknowledge (Line 41-45, Introduction) there is a large space of exploration and improvement. Meanwhile, to deploy such models into production-ready AR/VR systems, certain software engineering designs are still needed, which is yet not our primary concern in this work.\n\n#### **Size of the dataset**\nWe acknowledge that the real dataset is small for training tasks. However, these are difficult datasets to collect, especially for a first work in this task. We are still in the process of expanding the dataset to support future work. On the other hand, this underlines the necessity of our augmentation-based training, which is self-supervised and does not require labeled data. \n\n\n[a] Chen, Yajing, et al. \"Sketch-pix2seq: a model to generate sketches of multiple categories.\" \n\n[b] Yang, Lan, et al. \"Sketchaa: Abstract representation for abstract sketches.\" \n\n[c] Zang, Sicong, Shikui Tu, and Lei Xu. \"Controllable stroke-based sketch synthesis from a self-organized latent space.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n5mSlYV2Qn", "reviewer": "Reviewer_gzU1", "summary": "This paper addresses a new task: sketch generation from marker-less air drawing. The authors trained a spatially conditioned diffusion model to generate sketches from noisy hand tracking results and text prompts. During training, they devised an augmentation-based training procedure.", "questions": "1. Could the authors please provide more results that is not from the Quick, Draw! dataset?\n2. Could the authors please generation results of simple geometry drawing?", "limitations": "1. The input of this model is a complete drawing of the user, which means the user cannot see their half-way drawing. This is not favorable in many applications.\n2. This method only takes the hand tracking algorithm's output as input. It ignores the rich information from the video, which may help to achieve more precise control over drawing.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good paper writing. Easy to follow.\n2. This paper addressed a new task: marker-less sketch generation, which is useful in AR/VR applications.\n3. Experiments are performed on both real and synthetic datasets.\n4. The idea of \"soft\" spatial constraint is novel, which harnesses the capability of image generation model to de-noise the output of hand tracking algorithms.", "weaknesses": "1. The model may overfit on the sketches from the Quick, Draw! dataset, as the diffusion model is finetuned on this dataset.\n2. The ability to generate simple geometries without certain semantics is not explored. For example, can the model generate a simple curve following the user's drawing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Model overfitting on Quick, Draw! Dataset**\nIn Figure[3] in the rebuttal PDF we show inference results on unseen classes during both augmentation-based training and LoRA finetuning, where our method still maintains its performance.\n\nOn the other hand, we agree with the reviewer that due to the finetuning, the model is “overfit” to the dataset’s style. However, fine-tuning diffusion models to a specific style for consistent outputs is a common practice in the image generation community and is not generally regarded as a limitation.\n\n#### **Results from a different dataset**\nIn Figure [1] in the rebuttal PDF, we show results from additional experiments on the TUBerlin dataset, which contains more complicated sketches compared to Quick, Draw! Dataset. As can be seen in Figure [1], our method is able to generate faithful and coherent sketches given more complex objects.\n\n#### **Simple geometries**\nIn Figure[2] in the rebuttal PDF we show examples from simple geometries shapes such as triangles, circles and squares, and the model is able to draw them faithfully.\n\n#### **Can’t see halfway drawing** \n*-- \"The input of this model is a complete drawing of the user, which means the user cannot see their half-way drawing. This is not favorable in many applications.\"*\n\nFirst, we have shown that our model is capable of auto-completing sketches with partial input (Figure[10] Appendix), and therefore the input is not limited to complete drawing. From a research perspective, interactive sketching and editing in a generative manner is indeed interesting, which we will consider in our followup works. From a practical AR/VR standpoint, certain software engineering designs can also be incorporated to enable users seeing half-way drawing/performing interactive sketching. While we do not focus on the engineering side in this work, we will be happy to engage the reviewer about this during the discussion period.\n\n#### **Lose wealth of information from the video**\n*-- “This method only takes the hand tracking algorithm's output as input. It ignores the rich information from the video, which may help to achieve more precise control over drawing.”*\n\nWe fully agree with the reviewer’s observation, which in fact aligns with our initial approach: translating video content directly to sketch under an Encoder-Decoder framework. However we found it very hard to learn meaningful trajectory information from video encoders such as VideoMAE. Additionally, there is no large-scale, high quality dataset that includes the rich information found in videos and yet has highly detailed sketches as targets. Creating such a dataset is also time consuming. However, we do agree that by considering additional information such as video features or motion velocities, the generation quality can be improved, which we leave for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FegSlebaQ4", "reviewer": "Reviewer_Gs1n", "summary": "The paper presents a technique for generating raster handwritten sketches based on the tracking of the hand (from egocentric video), with the target application scenario of sketching in AR/VR. \n\nThe training is done mostly based on Quick, Draw! dataset of sketches combined with generated hand videos using Unity (5k samples). In addition, 500 real videos were collected by the authors for the purposes of training and evaluation.\n\nThe model is ControlNet-based (most experiments using Stable Diffusion XL).\n\nThe authors show qualitatively that this technique allows for generating sketches that are quite close to the ground truth, and can be extended to sketch completion. They also show the importance of providing a text prompt describing the object category that needs to be generated. The also show the importance of using the data augmentation when generating the synthetic dataset, to ensure that synthetic sketch data is close to the real data obtained by hand tracking algorithm.", "questions": "1. Could you please clarify / show whether SDXL that has only been tuned on Quick, Draw classes that are not used for evaluation, could perform equally well? \n\n2. How does the method perform on other sketches that are further away from the training data (ex not the cluster centers for clustering of the training data)?\n\n3. Could you please clarify the exact dataset used for evaluation? Was it only comprised of real videos, or of synthetic training samples as well? What was the train-test split?\n\nI am willing to increase my score if the authors could provide answers to Q 1/2 and planning to decrease it if there is insufficient evidence that the model can perform well beyond the training Quick, Draw classes.", "limitations": "The authors have generally addresses the limitations of their work (with the exception of the data release, see above).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality. The paper presents a first-of-its-kind approach and a first-of-its-kind dataset to train the model on.\n2. Clarity. The writing is mostly clear and easy to follow.\n3. Quality. The ablation study is well done and highlights the importance of the main decisions made by the authors.", "weaknesses": "The main weakness of the approach is the quality of the analysis, which brings under question the significance of the technique. \n\nIn particular, the authors use Stable Diffusion model LoRA-tuned on Quick, Draw dataset (lines 209-210) for all their experiments. While the evaluation is done on a held-out set of Quick, Draw classes (appendix, line 521), it seems that the tuning could allow leakage of the set of classes to the model generation - suggesting that instead of the ability to generalize to unseen classes, the model actually simply memorized  the class set. \n\nAnother similar issue is the technique for the selection of the held-out set of classes (appendix, lines 514-520) where the held-out classes were selected from the K-Means clustering of the classes in Quick, Draw - meaning that they are intentionally similar to the ones used during training.\n\nMore generally, the unanswered question of the applicability of this technique to anything beyond the given set of 50 common sketching classes is the main weakness of the paper - generalization to truly unseen sketch classes, ability of SDXL and CLIP to deal with more complex objects or multiple objects on the same image is unclear. \n\nThe second weakness is the limited reproducibility of the approach - the authors have not made even the synthetic dataset public (even though they claim to intend to), citing (appendix, line 659) \"As of the submission of the paper, the new datasets presented in this paper need to be cleaned to ensure personal information of the creator isn’t released unintentionally\" - which is unlikely to be an issue for synthetic data.\n\nThe third weakness is the set of evaluation metrics selected by the authors. While it does a good job of highlighting the importance of decisions made by the authors, for me as the reader, the most important missing piece of information about the quality of this approach that I am interested in is \"how often does the output of the model actually depict the object of the same class as the user intended to draw?\" - which could be obtained by running a high-enough quality classifier on the generated data or looking to the closest neighbour in CLIP embedding space with all of the class categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Weakness 1.1: Finetuning samples leaking**\n*-- “While the evaluation is done on a held-out set of Quick, Draw classes (appendix, line 521), it seems that the tuning could allow leakage of the set of classes to the model generation - suggesting that instead of the ability to generalize to unseen classes, the model actually simply memorized the class set.”*\n\nIn Figure[3] we run inference on less-common classes that are not seen during both augmentation-based training and LoRA finetuning, and we can clearly observe that the model maintains its performance over these unseen classes. \n\n#### **Weakness 1.2: Regarding K-Means clustering for partitioning seen/unseen classes**\n*-- “... the held-out classes were selected from the K-Means clustering of the classes in Quick, Draw - meaning that they are intentionally similar to the ones used during training.”*\n\nWe would like to clarify:\n* The K-Means clustering is based solely on evaluation data (our collected synthetic/real hand videos), not training data. \n* We perform the clustering over the order of chaos of classes – how different is the trajectory image from the ground-truth (SSIM/CD), and how similar is it to its corresponding text label (CLIP I2T). We are *not* using it to intentionally split semantically similar classes into seen/unseen classes (e.g. assigning “dog” to seen and “gray dog” to unseen). \n* We are *not* sampling unseen classes from one particular cluster that “resembles the training data”, but uniformly from all clusters, such that the averaged baseline statistics between seen/unseen are similar and can thus be compared.\n\nDifferent classes from the Quick, Draw! dataset have vastly different qualities. Without such a procedure it will be impossible to quantitatively draw comparisons two sets of classes. In addition, as we show in our ablations, order of chaos has a clear impact on model performance. By drawing unseen classes uniformly from cluster centers, we can perform such ablations (e.g. Figure [8] main paper) in different order of chaos. \n\nWe apologize for any ambiguous wording that causes the confusion, and will improve the expression in the final draft of the paper.\n\n#### **Weakness 1.3: Applying to anything out of the 50 classes/more complex objects?**\nWe show that our model is indeed capable of generalizing out of the seen classes and to generate more complex sketches. For results on classes out of the 50 classes please see Figure[3] in rebuttal PDF and response under Weakness 1.1. For results on more complex objects please see Figure [1] in rebuttal PDF for results on TUBerlin dataset and general response 1. \n\n#### **Weakness 2: Synthetic dataset has not been made public.**\nOur datasets will become available when the paper is accepted to a venue! It is a very common practice for researchers to wait to release code and datasets until their paper is accepted, to avoid having their work misappropriated by other researchers prior to publication; we hope the reviewer understands. We will absolutely release all code and datasets once the paper has been accepted into a venue.\n\n#### **Weakness 3: Evaluation metrics** \n*-- “The third weakness is the set of evaluation metrics selected by the authors…I am interested in is \"how often does the output of the model actually depict the object of the same class as the user intended to draw?\"*\n\nIn the paper we indeed report the CLIP image-to-text similarity (CLIP I2T) to measure how likely is the output depicting the intended class. However if the reviewer believes a different metric is needed, we are happy to provide further evaluation.\n\n#### **Q1: Could you please clarify / show whether SDXL that has only been tuned on Quick, Draw classes that are not used for evaluation, could perform equally well?**\nYes, please see Figure[3] in the rebuttal PDF and response under Weakness 1.1.\n\n#### **Q2: How does the method perform on other sketches that are further away from the training data (ex not the cluster centers for clustering of the training data)?**\nWe hope that the clarification about the K-Means selection and the evaluation on the unseen classes/datasets answers this question.\n\n#### **Q3: Could you please clarify the exact dataset used for evaluation? Was it only comprised of real videos, or of synthetic training samples as well? What was the train-test split?**\n\nWe used the collected synthetic and real hand motion datasets only for evaluation. In the main paper (e.g. Table[1]) we report the testing accuracy separately under synthetic and real dataset. The training samples were all augmented-clean sketch pairs from the Quick, Draw! dataset and do not overlap with the evaluation samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7lnzlKax4", "reviewer": "Reviewer_THqa", "summary": "The authors present an interesting application of multiple vision components merged together. They introduce AirSketch to generate sketches directly from hand gestures. They present a self-supervised training procedure and augmentations to with an image diffusion model to generate the realistic sketches. They also present two datasets building on existing datasets and Unity engine. They claim that controllable image diffusion can produce a clear sketch from a noisy input and present a new technique for drawing without markers.", "questions": "Is there anything specific in the model that can solve for jitters? Any particular component of the pipeline that authors think is helping with robustness? \nCan something other than Quick, Draw! be used for generating the synthetic dataset?\nFor ControlNet, the authors use a batchsize of 8. Is that enough to maintain statistics with layers like BatchNorm? \nHave the authors looked at more human-like augmentations to help the model better generalize to unseen objects and become more robust to the existing ones? Pls. see Atoms of Recognition, Extreme Image Transformations, etc.", "limitations": "The authors find that the model relies on text prompts more when applied to unseen categories. The hypothesis is based on the absolute numbers. The authors should also try and show something like a saliency map for both visual and textual elements of the model to better understand the parts that the model is focusing on. \nThe authors did not release the code with this submission but promise a future release. Please make sure that the code does accompany the paper for better reproducibility. \nThe authors also list that there is no need for statistical significance. However, can SSIM not be a good metric to see if the differences between the generated maps and ground truth is significant, when not relying on the human eye?\nSSIM and CD by themselves are weak metrics to measure similarity, and are prone to errors. The authors must address that in the text and also tell what might be good ways to mitigate that (i am not asking to devise a new metric, just let the reader clearly know). \nThe technology can also have a negative societal impact, perhaps not directly but as a derivative of this being applied to another area. The authors should try and address those areas.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is clearly written and covers most of the related works in sufficient depth. \nThe authors clearly motivate why the problem is important and what could be some clear applications of the idea from an AR/VR perspective (which has become a popular field of research lately).\nThe authors highlight the lack of hand drawing to sketch datasets and to that end present novel datasets.\nThe authors also cover the different cases of failures and show that the augmentations can make it robust.", "weaknesses": "The idea presented in the paper might be a combination of existing ones and can be easily materialized by using a combination of different models, but it is nevertheless an interesting area. \nVery limited description of the hand tracker. The authors should expand on it.\nDidn’t find enough motivation for why only Quick, Draw! was used as the starting point?\nIs there a reason for choosing Unity over other engines? Can stable diffusion be used to generate the hand gesture videos for the dataset? \nThe metrics used for evaluations can all be prone to errors. I am not sure if there is sufficient time to get IRB and run human experiments, but that could significantly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Limited description of the hand tracker**\nWe thank the reviewer for pointing this out! While we did discuss hand trackers in the appendix (A.3), we will add a more explicit description to the main body of the paper.\n\n#### **Why only Quick, Draw! was used as the starting point?**\n1. When drawing in the air without controllers or headsets, quick and convenient everyday sketches are likely to resemble the simplistic sketches found in the Quick, Draw! dataset, rather than the more elaborate sketches in the TUBerlin dataset.\n2. All prior works on sketch generation, such as [a,b,c], only consider the Quick, Draw! dataset. \n3. The Quick, Draw! dataset is the biggest sketch dataset, and is the only dataset with timestamp information for each point in the drawing. We needed this information to create the synthetic dataset, where the hand moves at a speed accurate to how the sketch was truly drawn.\n\n#### **Is there a reason for choosing Unity over other engines? ​​Can stable diffusion be used to generate the hand gesture videos for the dataset?**\nGenerating hand gesture videos with Stable Diffusion would be much harder than simply using a graphics engine, especially since it is prone to incorrectly generating hands (even despite recent improvements). Unity is a very mature graphics engine and does the job for us, but we are totally open to suggestions from the reviewer for future work.\n\n#### **IRB and human experiments**\nUnfortunately, we won't have enough time within the rebuttal period to get that done. Our institution has an extensive IRB protocol. \n\n#### **Is there anything specific in the model that can solve for jitters?**\nWe account for jitters (as a result of the hand tracking and human tremors) by introducing the jitter augmentation during training. The model then learns to clean up jitters when they are encountered in the input. \n\n#### **Any particular component of the pipeline that authors think is helping with robustness?**\nThe part of the pipeline that we think helps the most with the robustness are our suite of augmentations, which helps our model learn what aberrations in the sketches need to be removed. We made sure to carefully develop our augmentations to account for common errors in sketching that are encountered in our use-case, whether it be from human error, the hand tracking, or simply the nature of the task.\n \n#### ** other datasets than Quick, Draw! for generating the synthetic dataset?**\nUnfortunately since Quick, Draw! Is the only dataset providing timestamp associated with drawing coordinates, it is the only available dataset for creating synthetic drawing videos. \n\n#### **For ControlNet, the authors use a batch size of 8. Is that enough to maintain statistics with layers like BatchNorm?**\nControlNet’s standard training regime has a batch size of 4, so our batch size of 8 is likely not too small.\n\n#### **Have the authors looked at more human-like augmentations to help the model better generalize to unseen objects and become more robust to the existing ones?**\nWe thank the reviewer for the suggestions to include more “human-like” augmentations and bringing up interesting topics in Atoms of Recognition and Extreme Image Transformations[c,d]; we will consider them for improving robustness in our future work. Meanwhile, the way we conduct our augmentations are designed to simulate a comprehensive range of “human-like” errors found in the noisy sketches produced by our air drawing. \n\n#### **Statistical significance**\nWhile it is too computationally intensive to conduct training multiple times, we re-conduct our main result in Table [1] from the paper by performing inference 10 times on each sample with different seedings, and report the mean&std shown in the table below.\n|w/ Aug.|SSIM (↑)|CD (↓)|LPIPS (↓)|CLIP I2I (↑)|CLIP I2T (↑)|\n|-|-|-|-|-|-|\n||||**Seen Categories**|\n|✘| 0.55±0.03$e^{-3}$ | 31.92±0.12$e^{-3}$|0.41±4.0$e^{-3}$|0.77±0.78$e^{-3}$|0.21±0.41$e^{-3}$|\n|✓| **0.64**±0.14$e^{-3}$ |**25.13**±0.22|**0.36**±1.0$e^{-3}$|**0.84**±1.2$e^{-3}$ | **0.29**±0.93$e^{-3}$|\n||||**Unseen Categories**|\n|✘|0.55±0.08$e^{-3}$| 33.53±0.31$e^{-3}$|0.41±3.8$e^{-3}$|0.78±0.84$e^{-3}$|0.21±0.51$e^{-3}$|\n|✓|**0.63**±0.45$e^{-3}$|**24.26**±0.71|**0.38**±1.9$e^{-3}$|**0.85**±2.8$e^{-3}$|**0.28**±1.7$e^{-3}$|\n\nWe also conducted a Paired T-test for significance. However as the means have a clear margin and std is low, the p-values are all low in the scale of 1e-14. Since this is not a rigorous test, we are not including the results here.\n\n#### **The authors should also try and show something like a saliency map for both visual and textual elements of the model to better understand the parts that the model is focusing on.**\nFigure [4] (right) in the rebuttal PDF shows the visualization of ControlNet’s hidden state throughout the reverse diffusion steps. We can clearly observe that while the model trained with augmentation (bottom) is able to infer the clean sketch contour (t>=16), the one trained without augmentation (top) fails to do so. \n\n#### **Data & Code release**\nWe would like to assure the reviewer that we will absolutely release the code and datasets with the paper for better reproducibility, once the paper is accepted to a venue. \n\n#### **Negative societal impact**\nWe thank the reviewer for bringing this up. The only potential negative impact we could think of is the use of our model for generating malicious content. However since we focus on sketch generation, the impact is much less than a regular image generation model. However, we would love to hear what the reviewer thinks, so we can include it in the final version of our paper.\n\n[a] A Neural Representation of Sketch Drawings\n\n[b] Sketch-pix2seq: a model to generate sketches of multiple categories\n\n[c] Controllable stroke-based sketch synthesis from a self-organized latent space\n\n[d] Atoms of recognition in human and computer vision\n\n[e] Extreme Image Transformations Improve Latent Representations in Machines", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for spending time reading our paper and providing insightful feedback. We appreciate reviewers finding our proposed task and approach novel, interesting, and useful (V9Hw, gzU1, Gs1n), and the experiments being extensive and convincing (V9Hw, Gs1n, THqa). We address each of the reviews separately below but provide some common responses here. 

### Generalizability on unseen classes 
Figure[3] in the rebuttal PDF shows inference results on less-common classes not seen during both LoRA fine-tuning and augmentation-based training. We could clearly observe that the model maintains its performance over these unseen classes. 

### Experiments on TUBerlin dataset
We run additional experiments on the TUBerlin dataset following the same training procedure from the main paper, and results are shown in Figure[1] in the rebuttal PDF. Our method produces faithful and coherent results with much more complicated sketches compared to Quick, Draw!, demonstrating its scalability to more complex scenarios (Reviewer V9Hw, gzU1).

Meta-review: The main weakness of this paper is centralized in the technical contribution of this paper, while the author emphasizes the task of generative motion to sketch as the main contribution of this paper. Overall, the reviewers appreciate the newly created task in this paper. Given these concerns raised by reviewers, the author provided detailed and professional rebuttals.
I hope the author can revise the paper according to the suggestions from reviewers to make their paper stronger for the camera-ready.
Decision: Accept (poster)