Title: AirSketch: Generative Motion to Sketch
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generative

Hand Motion

Sketch
Abstract: Illustration is a fundamental mode of human expression and communication. Certain types of motion that accompany speech can provide this illustrative mode of communication. While Augmented and Virtual Reality technologies (AR/VR) have introduced tools for producing drawings with hand motions (air drawing), they typically require costly hardware and additional digital markers, thereby limiting their accessibility and portability. Furthermore, air drawing demands considerable skill to achieve aesthetic results. To address these challenges, we introduce the concept of AirSketch, aimed at generating faithful and visually coherent sketches directly from hand motions, eliminating the need for complicated headsets or markers. We devise a simple augmentation-based self-supervised training procedure, enabling a controllable image diffusion model to learn to translate from highly noisy hand tracking images to clean, aesthetically pleasing sketches, while preserving the essential visual cues from the original tracking data. We present two air drawing datasets to study this problem. Our findings demonstrate that beyond producing photo-realistic images from precise spatial inputs, controllable image diffusion can effectively produce a refined, clear sketch from a noisy input. Our work serves as an initial step towards marker-less air drawing and reveals distinct applications of controllable diffusion models to AirSketch and AR/VR in general.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an interesting application of multiple vision components merged together. They introduce AirSketch to generate sketches directly from hand gestures. They present a self-supervised training procedure and augmentations to with an image diffusion model to generate the realistic sketches. They also present two datasets building on existing datasets and Unity engine. They claim that controllable image diffusion can produce a clear sketch from a noisy input and present a new technique for drawing without markers.\n\nSTRENGTHS:\nThe paper is clearly written and covers most of the related works in sufficient depth. \nThe authors clearly motivate why the problem is important and what could be some clear applications of the idea from an AR/VR perspective (which has become a popular field of research lately).\nThe authors highlight the lack of hand drawing to sketch datasets and to that end present novel datasets.\nThe authors also cover the different cases of failures and show that the augmentations can make it robust.\n\nWEAKNESSES:\nThe idea presented in the paper might be a combination of existing ones and can be easily materialized by using a combination of different models, but it is nevertheless an interesting area. \nVery limited description of the hand tracker. The authors should expand on it.\nDidn’t find enough motivation for why only Quick, Draw! was used as the starting point?\nIs there a reason for choosing Unity over other engines? Can stable diffusion be used to generate the hand gesture videos for the dataset? \nThe metrics used for evaluations can all be prone to errors. I am not sure if there is sufficient time to get IRB and run human experiments, but that could significantly strengthen the paper.\n\nQUESTIONS:\nIs there anything specific in the model that can solve for jitters? Any particular component of the pipeline that authors think is helping with robustness? \nCan something other than Quick, Draw! be used for generating the synthetic dataset?\nFor ControlNet, the authors use a batchsize of 8. Is that enough to maintain statistics with layers like BatchNorm? \nHave the authors looked at more human-like augmentations to help the model better generalize to unseen objects and become more robust to the existing ones? Pls. see Atoms of Recognition, Extreme Image Transformations, etc.\n\nLIMITATIONS:\nThe authors find that the model relies on text prompts more when applied to unseen categories. The hypothesis is based on the absolute numbers. The authors should also try and show something like a saliency map for both visual and textual elements of the model to better understand the parts that the model is focusing on. \nThe authors did not release the code with this submission but promise a future release. Please make sure that the code does accompany the paper for better reproducibility. \nThe authors also list that there is no need for statistical significance. However, can SSIM not be a good metric to see if the differences between the generated maps and ground truth is significant, when not relying on the human eye?\nSSIM and CD by themselves are weak metrics to measure similarity, and are prone to errors. The authors must address that in the text and also tell what might be good ways to mitigate that (i am not asking to devise a new metric, just let the reader clearly know). \nThe technology can also have a negative societal impact, perhaps not directly but as a derivative of this being applied to another area. The authors should try and address those areas.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a technique for generating raster handwritten sketches based on the tracking of the hand (from egocentric video), with the target application scenario of sketching in AR/VR. \n\nThe training is done mostly based on Quick, Draw! dataset of sketches combined with generated hand videos using Unity (5k samples). In addition, 500 real videos were collected by the authors for the purposes of training and evaluation.\n\nThe model is ControlNet-based (most experiments using Stable Diffusion XL).\n\nThe authors show qualitatively that this technique allows for generating sketches that are quite close to the ground truth, and can be extended to sketch completion. They also show the importance of providing a text prompt describing the object category that needs to be generated. The also show the importance of using the data augmentation when generating the synthetic dataset, to ensure that synthetic sketch data is close to the real data obtained by hand tracking algorithm.\n\nSTRENGTHS:\n1. Originality. The paper presents a first-of-its-kind approach and a first-of-its-kind dataset to train the model on.\n2. Clarity. The writing is mostly clear and easy to follow.\n3. Quality. The ablation study is well done and highlights the importance of the main decisions made by the authors.\n\nWEAKNESSES:\nThe main weakness of the approach is the quality of the analysis, which brings under question the significance of the technique. \n\nIn particular, the authors use Stable Diffusion model LoRA-tuned on Quick, Draw dataset (lines 209-210) for all their experiments. While the evaluation is done on a held-out set of Quick, Draw classes (appendix, line 521), it seems that the tuning could allow leakage of the set of classes to the model generation - suggesting that instead of the ability to generalize to unseen classes, the model actually simply memorized  the class set. \n\nAnother similar issue is the technique for the selection of the held-out set of classes (appendix, lines 514-520) where the held-out classes were selected from the K-Means clustering of the classes in Quick, Draw - meaning that they are intentionally similar to the ones used during training.\n\nMore generally, the unanswered question of the applicability of this technique to anything beyond the given set of 50 common sketching classes is the main weakness of the paper - generalization to truly unseen sketch classes, ability of SDXL and CLIP to deal with more complex objects or multiple objects on the same image is unclear. \n\nThe second weakness is the limited reproducibility of the approach - the authors have not made even the synthetic dataset public (even though they claim to intend to), citing (appendix, line 659) \"As of the submission of the paper, the new datasets presented in this paper need to be cleaned to ensure personal information of the creator isn’t released unintentionally\" - which is unlikely to be an issue for synthetic data.\n\nThe third weakness is the set of evaluation metrics selected by the authors. While it does a good job of highlighting the importance of decisions made by the authors, for me as the reader, the most important missing piece of information about the quality of this approach that I am interested in is \"how often does the output of the model actually depict the object of the same class as the user intended to draw?\" - which could be obtained by running a high-enough quality classifier on the generated data or looking to the closest neighbour in CLIP embedding space with all of the class categories.\n\nQUESTIONS:\n1. Could you please clarify / show whether SDXL that has only been tuned on Quick, Draw classes that are not used for evaluation, could perform equally well? \n\n2. How does the method perform on other sketches that are further away from the training data (ex not the cluster centers for clustering of the training data)?\n\n3. Could you please clarify the exact dataset used for evaluation? Was it only comprised of real videos, or of synthetic training samples as well? What was the train-test split?\n\nI am willing to increase my score if the authors could provide answers to Q 1/2 and planning to decrease it if there is insufficient evidence that the model can perform well beyond the training Quick, Draw classes.\n\nLIMITATIONS:\nThe authors have generally addresses the limitations of their work (with the exception of the data release, see above).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper addresses a new task: sketch generation from marker-less air drawing. The authors trained a spatially conditioned diffusion model to generate sketches from noisy hand tracking results and text prompts. During training, they devised an augmentation-based training procedure.

STRENGTHS:
1. Good paper writing. Easy to follow.
2. This paper addressed a new task: marker-less sketch generation, which is useful in AR/VR applications.
3. Experiments are performed on both real and synthetic datasets.
4. The idea of "soft" spatial constraint is novel, which harnesses the capability of image generation model to de-noise the output of hand tracking algorithms.

WEAKNESSES:
1. The model may overfit on the sketches from the Quick, Draw! dataset, as the diffusion model is finetuned on this dataset.
2. The ability to generate simple geometries without certain semantics is not explored. For example, can the model generate a simple curve following the user's drawing?

QUESTIONS:
1. Could the authors please provide more results that is not from the Quick, Draw! dataset?
2. Could the authors please generation results of simple geometry drawing?

LIMITATIONS:
1. The input of this model is a complete drawing of the user, which means the user cannot see their half-way drawing. This is not favorable in many applications.
2. This method only takes the hand tracking algorithm's output as input. It ignores the rich information from the video, which may help to achieve more precise control over drawing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the problem of sketch generation from the finger's motion trajectory, which is interesting. To make it, the authors adopt a standard ControlNet pipeline, while highlighting the importance of data augmentation (adding noise to clean sketches to mimic the hand-tracking image). Hence the major contribution of this paper is the tweak that makes ControlNet applicable for generating clean sketches from the continuous messy finger trajectories.\n\nSTRENGTHS:\n- The authors repurpose ControlNet to produce a clean sketch from a severely deformed and line-messy input, which is useful for finger motion to sketch generation and could be potentially valuable for other related HCI applications. \n- The results look promising, and the application of sketch-completion and text-instructed stroke styling are interesting.\n- The experimental results are extensive and validate the effectiveness of the proposed augmentation strategies.\n\nWEAKNESSES:\n- The major concern is about the novelty of the technical contribution. Personally, the proposed augmentation methods are more like practical tricks for better cleaning up messy sketches.\n- The methods of accepting raster-format input for sketch generation should be also compared by training these models with the same augmented sketches, such as [a][b][c].\n- The practical usage seems limited since: (1) the proposed method is only validated on simple sketches, so no way to justify whether any scalability issue when facing complex scenarios; (2) the user probably needs to edit (e.g., deleting lines) while drawing, how to deal with it?\n- The collected real air-drawing dataset is very small.\n\n[a] Chen, Yajing, et al. \"Sketch-pix2seq: a model to generate sketches of multiple categories.\"\n\n[b] Yang, Lan, et al. \"Sketchaa: Abstract representation for abstract sketches.\" \n\n[c] Zang, Sicong, Shikui Tu, and Lei Xu. \"Controllable stroke-based sketch synthesis from a self-organized latent space.\"\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nThe major limitation may be the limited practical usage as I listed in weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}