CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
Authors: Kevin Frans
Lisa Soros
Olaf Witkowski
Venue: NeurIPS 2022 Accept (2022)
Keywords: image synthesis
clip
computer vision
language to text
creativity
art

Abstract: CLIPDraw is an algorithm that synthesizes novel drawings from natural language input. It does not require any additional training; rather, a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, which biases drawings towards simpler human-recognizable shapes. Results compare CLIPDraw with other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw.

CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
Kevin Frans
Massachusetts Institute of Technology, Cambridge, MA, USA Cross Labs, Cross Compass Ltd., Tokyo, Japan kvfrans@csail mit.edu
L. B. Soros
Cross Labs, Cross Compass Ltd., Tokyo, Japan
Olaf Witkowski
Cross Labs, Cross Compass Ltd., Tokyo, Japan
Earth-Life Science Institute, Tokyo Institute of Technology, Japan College of Arts and Sciences, University of Tokyo, Japan
Abstract
CLIPDraw is algorithm that synthesizes novel drawings from natural language input. It does not require any additional training; rather, a pre-trained CLIP language-image encoder is used metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, which biases drawings towards simpler buman-recognizable shapes. Results compare CLIPDraw with other synthesis- through-optimizatee methods, well highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably pro- ducing drawings diverse styles, and sealing from simple to complex visual representations as stroke count increases.
Introduction
When humans hear a description of a scene, it's easy to imagine what it may look like. Conversely, when we construct mental image, it's easy then describe that scene. At some level, humans have a deeply coupled representation for textual and visual structures key to understanding our world.
The recent introduction of CLIP (Rafford et al. 2021). dual language-image encoder, is large step towards unifying textual and visual information. In CLIP model, both text and images are mapped onto the same representational space, thus enabling the similarity between images and textual descriptions to be measured. When trained on large amounts of data, CLIP representations have been shown to solve robust range of image-based recognition tasks.
This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather pre-trained CLIP model is used as a metric for maximizing similarity between the given description and generated drawing. Rather than photorealistic images, CLIPDraw aims to synthesize simple drawings that nevertheless match the prompt. Thus, CLIPDraw optimizes a set vector strokes rather than pixel images, a constraint that biases drawings towards simple human-recognizable shapes.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

"A drawing of cat".
"Horse cating cupcake".. "A 3D rendering of temple".
"Family vacation to Walt Disney World".
"Self".
Figure I: Various drawings synthesized by CLIPDraw, along with the corresponding description prompts used. CLIPDraw synthesizes images from text by performing gradient descent over set of RGBA Bézier curves, with the goal of minimizing cosine distance between the CLIP encodings of generated images and description prompts. CLIPDraw does not require learning new model, and can generally synthesize images within a minute on a typical GPU.
The aim of this work is to present CLIPDraw testbed for exploring language-iman -magee relationships and synthesizing Al-assisted artwork, as well as to showcase various nuances of the method. Re- sults compare between CLIPDraw and other optimization-ban -based text-to-image methods, along with highlighting several interesting behaviors:
By adjusting descriptive adjectives, such as "watercolor" or "3D rendering", CLIPDraw produces drawings of vastly different styles.
CLIPDraw often matches the description prompt in creative ways, such as writing words from the prompt inside the image itself, interpreting ambiguous nouns in multiple ways.
Running CLIPDraw with low stroke count results in cartoonish drawings, while high stroke counts tend result in realistic renderings.
By giving CLIPDraw abstract prompts, such "happiness" or "self", we examine what visual concepts the CLIP model associates with them.
CLIPDraw behavior can be further controlled through the use of negative prompts, such as "a messy drawing", to encourage the opposite behavior.
Related Work
Text-to-Image Synthesis. This work greatly draws from the field of text-to-image synthesis, whose primary aim is to generate images that correctly match a given textual description. In recent years, focus has been on methods that aim learn a direct post-to-image mapping function, often in the form of conditional GAN (Goodfellow et al.] 2014a Mirza and Osindero 2014 Reed et al. [016)] Frolov et al. 202Tp. Commonly used datasets include Dxford-120 Flowers Nilssbck and Zisserman 2008), CUB-200 200 Birds (Wah et al., 2011). and COCO (Lin et al. 2014), all of which contain natural images and captions describing them. While GAN-based methods have enabled considerable progress towards photorealistic image synthesis, strong autoregressive models have achieved similar quality results (Oord al., 2017| Chen al. 2020), with the recent DALL-E model (Ramesh et al. (021) showcasing the benefit of scaling text-to-image synthesis networks to large capacity. In comparison to text-to-image generative models, which require large amounts of training, this work follows framework of synthesis through optimization, in which images are generated through evaluation-time optimization against a given metric.
Synthesis Through Optimization. Instead of directly learning an image generation network, an alternative method of image synthesis is optimize towards a matching image during evaluation time. This framework is often referred to as activation maximization (Erhan et al.) 2009 Nguyen al. 2016 [Mordvinsee al. 2013), where random image is optimized through backpropogation to increase certain neuron activations of pretrained network. Activation
Activation-maximizz maximization methods have produced highly realistic images, however a challenge to understand the meaning of a neuron activation, the objective minimize the distance between the produced image and given description phrase,
activation. CLIPDraw builds off set of methods where rather than maximizing
defined by powerful CLIP iaguage-image encoder (Fernando al. 2021: Murdock) 2021

---

CLIP Model
"A speceship flying sturry sky".
Text Encoding
Auymcaited Banch
Ciudient Loop
Voctor
Image Angment
CLIP
Cosine Distance
Reader
Model
Image Encoding
Loss
Figure 2: CLIPDraw iteratively synthesizes images through evaluation-time gradient descent. Starting from random set of Bézier curves, the position and colors of the curves are optimized so that the generated drawings best match the given description prompt. Before being passed into the CLIP encoder, drawings augmented into multiple perspective-shifted copies.
Iter.
Lier.
lier.
lier,
Iter. 100
Iter. 150
200
Iter.
Figure 3: A typical CLIPDraw run gradually forms curves into concrete shapes. Here, background of star-shaped structures develops into large spaceship. More pronounced stars eventually appear, in addition to Darth- Vader-aaeeekee figure on the spaceship.
Crowson et al. 2022 Galatolo et al. 2021 ). A key issue in synthesis through optimization is that the produced images often leave the space of natural images (Nguyen et al. 2015), or fool the system through adversarial means (Goodfellow et al. 2014b), thus body of work aims to discover 'natural image priors' to constrain produced images Nguyen et al, 2016 2017). While a typical solution is to constrain optimization to the generative space of a GAN, this setup be expensive evaluate, and only allows synthesis images producible by the GAN generator. Because CLIPDraw focuses synthesizing drawings rather than realistic pictures, CLIPDraw instead limits optimization set of vector curves. This constraint results in stroke-based images, which must capture larger features such shapes and outlines, rather than fine ine-grained textures.
Vector Graphics. This work builds largely from work by al. (2020) which introduces differentiable renderer for vector graphics. Image generation methods that operate over vector images have uraditionally required vector-based dataset, however recent work has shown how differentiable renderers can be used to bypass this limitation (Reddy et al. 2021 Shen and Chen 2021 Kotovenko ct al. 202T). CLIPDraw uses a differentiable renderer as a representation for generating drawings; namely set of RGBA Bézier curves are optimized rather than pixels.
draft of this work was previously released the public research community, and a number of follow-up papers have since been published. StyleCLIPDraw (Schaldenbrand et al. (20TT) introduces an explicit image-based style loss the CLIPDraw framework. CLIP-CLOPP (Mrowski et al., (022) extends CLIPDraw to use image patches rather than strokes, and CLIPasso (Vinker et al.] 2022) uses CLIIP to extract sketches from photos. TTan and Ha (202)) pursure optimization via evolutionary strategies rather than gradient descent. CL.IPDraw has already influenced the image synthesis community in its pre-release form, and we hope a formal publication will continue to encourage collaboration.
Follow Up Work to CLIPDraw.
3 Method
The objective of CLIPDraw is to synthesize a drawing that matches a given description prompt (Figure D.. Specifically, a pre-trained CLIP model is used as a judge. A CLIP model contains two networks an image encoder and textual encoder - which both map their respective inputs into a shared encoding space of 51 512-length vector. Similarity is measured via the cosine distance between two encodings. Thus, the goal of CLIPDraw is to produce an image which, when encoded via CLIP, matches the CLIP encoding of the prompt.
Drawings Cr.PDraw are represented by a set of differentiable RGBA Bézier curves, following the method by Li et al. (2020). Each curve is parametrized by 3-5 control points, along with thickness

---

Algorithm CLIPDraw
Input: Description Phrase desc; Iteration Count Z; Curve Count N; Augment Size D; Pre-trained CLIP model.
Begin:
Encode Description Phrase. EncPhr = CLIP(desc)
Initialize Curves. Curves..N- RandomCurvel)
i = 0 to / do
Render Curves to Pixels. Pixels DiffRendert Curves)
Augment the Image, AugBatch, Augment( Pixels)
Encode Image. Encimg CLiP(AugBatchy
Compute Loss. Loss = Sim (innre EncPhr, Encimg)
Backprop. Curves Minimize(Loss)
end for
and an RGBA color vector. Drawings initially begin with curves randomly distributed throughout the image. with white background. During optimization, the number of curves and control points is fixed, however the positions of the points along with the thickness and color vectors can be optimized via gradient descent.
The CLIPDraw algorithm (Algorithm | works by running evaluation-time gradient descent, as shown First, the description phrase is encoded via the CLIP model, and random set of N Bézier curves are initialized. During each iteration, the curves are rendered to a pixel image via the differentiable renderer, and the resulting image is then duplicated D times and augmented by random perspective shift and random crop-and-reste resize. The resulting batch of augmented images is passed into the CLIP image encoder, and the cosine distances to the description phrase summed to form the loss value. Because all operations are differentiable, gradient descent can be run through the entire loop, optimizing the parameters of the curves to decrease loss. This procedure is repeated I times, until convergence.
in Figure 2
The goal of the image augmentation is to force drawings to remain recognizable when viewed through various distortions. Without image augmentation, synthesi-ttrouuh-opiiiizzz methods often result in adversarial images that fulfill the numerical objective but are un- recognizable to humans. This work uses the torch. transforms. RandomPerspect and torch. transf orms. RandomResi Renddeeeezzddropp functions in sequence. Note that the specific details of the augmentation were not the focus of this work, furthermore, trials with different augmentations show that augmentation choice does not influence synthesis to a noticeable degree (sce Appendix).
Figure 3 showcases the gradual synthesis of typical CLIPDraw drawing. Note that while the optimization process is largely deterministic, there is randomness in the initial curves and image augmentations, thus multiple runs of CLIPDraw can result in different drawings.
4 Results
In the following sections, various interesting behaviors of CLIPDraw highlighted through variety of examples. With the exception Figure [5] example images are picked to best convey the behavior in consideration. Focus is placed qualitative observations, unusual behavior, or recurring trends in CLIPDraw image synthesis.
Compared to methods that learn direct generative model, aptimization-basee synthesis methods do not require prior training. Instead, images are generated through an evaluation- time optimization loop, aiming to maximize given objective. This work specifically focuses on synthesizing images that match the CLIP encoding of a description prompt. The following methods are compared:
CLIPDraw, in which drawings are produced by set of RGBA Bézier curves. The control points, thickness, and colors of the curves can all be adjusted.
Pixel Optimization, which instead optimizes 224x224x3 matrix of RGB pixels. Otherwise, all algorithmic aspects are the same as CLIPDraw, including image augmentation.

---

"A drawing of cat"
"A painting afa sunset"..
"Underwater""
"Sheep wearing top hat".
"A 3D rendering of temple".
"Watercolor of truck".
"Third Eye".
TIRE HIRD
Crip
0
Pixel Optimization
BigGan Optimization
VOGAN
Oplimization
CLiPDraw Augment)
methods, all which share the objective of matching a given CL.IP-encoded description phrase. All methods optimize for cosine similarity, as shown in T CLIPDraw can produce a diverse set of heman--ecognizabll drawings based on simple strokes and shapes. CLIPDraw tends to result in simple drawings, often featuring multiple instances of concepts in a variety styles. On the other hand, Pixel Optimization creates interesting textures but fails to compose colors and shapes. BigGan and VQGAN Optimization can synthesize high-resolution images, but are constrained to the set of natural images the generators trained on, thus they tend to produce images featuring single object, and cannot achieve techniques such as writing text. CLIPDraw without image augmentation produces images that score high during training, but fail to retain these scores when tested augmentations, and are nonsense when viewed by humans. Cosine similarities for each prompt/method pair are shown Table I
Figure 4: Images synthesized via various waahhesi-tthroug--pppiiiii
BigGAN Optimization, in which images produced using pre-trained BigGAN gen- erator. The weights of the generator are frozen; only the latent vectors are optimized. Samples generated as by Murdock (221))
VQGAN Optimization, in which images are created through sampling a VQGAN codebook
Esser et al. 2021). Samples are generated as by Crowson et al. (2022).
CLIPDraw (No Augment), which is identical to CLIPDraw, except image augmentation is applied to the synthesized drawings.
These methods (Figure [5]. In CLIPDraw, stroke count 256, and 8 duplicates are used during image augmentation. CLIPDraw tends to result in diverse set of hugan-beco doodles. Pixel Optimization creates interesting textures but fails to compose colors and shapes. BigGAN and VQGAN Optimization synthesize high-resolution images, but constrained to the set of images the generator can produce. Images from CLIPDraw without image augmentation score high numerically, but are nonsense to the human eye.
the same CLIP matching objective for 250 steps of gradient descent
4.1 What kinds of visual techniques does CLIPDraw use to satisfy the textual description?
CLIPDraw often results in drawings that match their description prompts in multiple, unexpected ways, shown in Figure 5 A prime example is the prompt for "a painting of starry night sky". The
5

---

Prompt
CraP
Pixel Opt.
BigGan Opt
VQGAN Opt
(No Aug)
A drawing of cat.
.376 + 0005
.385 = .009
.325 + .014
.377 + .004
.240 .015
paint. of sunset.
.390 .004
.215 .010
.379 + .003
.379 .003
.325 - .015
Underwater.
.413 + .006
.385 = .009
.327 + .013
.358 + .006
+ 247
Sheep wearing a top hat.
.434 .010
.434 .010
.321 .016
.411 .009
215
A 3D rend. of temple.
467 + .005
.855 .009
.311 = .028
.430 + .008
.240 H .009
Watercol. firetruck.
.507 .007
.385 .009
.38 - .014
.457 .007
.233 0007
Third Eye.
436 + .010
385 .009
323 + .015
.368 + .005
.211 + .013
Table 1: CLIP cosine similarities for images shown above. Scores are computed by taking the average cosine similarity between the description prompt and 64 augmentations of the generated images. Each method run for 250 steps of gradient descent, with 10 starting seeds for every prompt, and resulting means and standard deviations are presented. In CLIPDraw results, stroke count is 256,, and duplicates are used during image augmentation. Images take around 1-2 minutes to synthesize on typical Colab GPU.
YEtL
VIKE
HiRD
Pop predietiona:
leapings 35.324
Pus: 20.789
caciogy
4.714
competibac: 3.14
H मोहाीeld
"A painting of night dky".
"Yeli taking selfic".
"Thind eye".
"Fast Food"
Figure 5: CLIPDraw often matches the description prompt through creative techniques, such as forming letters inside the images, interpreting ambiguous words in multiple ways.
drawing's s background features sky with prominent moon and few scattered stars. The drawing is rendered in painterly style, however it also features an actual canvas and painter. Inside the canvas, black and blue swirls resemble Van Gogh's 's 1889 "The Starry Night".
Another interesting behavior of CLIPDraw is its tendency to write words the drawing itself. In "Yeti taking selfie", letters resembling "Yeti" can be seen in the top-right corner. In "Third Eye", again words resembling "third" and "eye" are seattered throughout the image. times, the drawings contain symbols that are not literally the description, but are still associated, such as the prompt "H fiy; tft" (bicycle in Japanese) resembling Google Maps screenshot with a Japanese-like character in the comner
Prompt ambiguity also presents intriguing results. In "Fast Food", McDonald's logo along with set of hamburgers is shown. However, also present are two joggers in footrace, providing another interpretation of "fast". Included Figure Sggure5aare the top words predicted by CLIP as closest the image, showing that CLIP recognizes both "jogging" and "hamburger" as related to the synthesized drawing of "fast food".
4.2 Can CLIPDraw reliably produce drawings in different styles?
A useful feature of CLIPDraw is its ability to adjust not just the content of its drawings, but also the styles, based on the description prompts given. Part of this flexibilityy is due the robustness of pre-trained GAN generator, CLIPDraw
curve-based images: in comparison methods that drawings are not limited the space of natural images. Thus, a variety of styles can be produced, and these styles easily explorable through text.
As shown in Figure o synthesized image of cat can look vastly different depending on the descriptor words included. When asked for "drawing of a cat", CLIPDraw synthesized cartoonish depiction cat, comprised mostly of an outline and simple face. "realistic photograph" features more detailed shading, while "cat as 3D rendered in Unreal Engine" showcases complex lighting along with a depth-based blurring. Further styles feature a bias towards certain colors, such as the reds and greens of Japanese woodblock prints, the multi-color blends of watercolors.

---

"A drawing cat"
realistic photograph of car".
"A cat 3D rendered in Unreal Engine".
"A Japanese woodblock print of ome cat".
"A watereolor painting of cal".
"A wireframe model cal".
Figure 6: By adjusting adjectives, CLIPDraw can produce drawings of diverse styles. Styles vary not only the texture of the images, showease different representations of the underlying content, such as cartoonish cat when prompted for "drawing", versus cat in perspective when prompted for a "3D wireframe model".
et
16 Strokes
32 Strokes
Strokes
128 Strokes
256 Strokes
512 Strokes
Figure 7: When stroke count is increased, CLIPDraw produces drawings of increasing realism. Low-stroke drawings "The Eiffel Tower" opt for a cartoonish representation, while high-stroke drawings capture 3D depth, background content, and shading.
An interesting result is that adjusting descriptive adjectives not only changes the textures of the draw- ings, akin to Style Transfer methods (Gatys et al. 2015). but also changes its structural representation of the underlying content. For example. prompting for "a drawing" produces a flat cartoonish cat, while prompts like "a 3D wireframe" produce a cat in perspective, with depth and shadows.
4.3 How does the stroke count affect what drawings CLIPDraw produces?
When the stroke count is low, CLIPDraw tends to produce cartoonish abstract representations of the given description prompt. As stroke counts increase, drawings become more detailed and incorporate additional features. Figure shows "The Eiffel Tower" with various stroke counts. In the 16-stroke example, the tower is drawn as only a few straight lines. Higher stroke count images begin more details the Eiffel Tower itself, along with additional features such background colors and complex lighting.
A common thread in synthesis-through-optiiinn undesirable results; it is also necessary to constrain optimization to a suitable space of images, such as the natural images generated by GAN, any image made of strokes in the case of CLIPDraw. Limiting stroke count furthers this constraint. When optimizing within the space of 16-stroke images, it is hard to achieve details textures, thus synthesized drawings will reveal the most basic forms that make up a visual concept. As a tool for Al-assisted art, the stroke-count parameter presents an easy way of adjusting between "simple" and "complex".
methods is that pure optimization leads to
4.4 What happens abstract words are given as a description prompt?
When given abstract prompt without literal interpretation, CLIPDraw must utilize cultural con- nections to come up with visual concepts that relate to the description. Often, this results in drawings that contain symbols relating to the given prompt, such as in Figure g with "Happiness" containing smiling faces and fireworks, "Translation' showcasing English and Japanese- characters, and "Enlightenment" featuring prominent monk-like figure.
At times, synthesized drawings demonstrate concepts through more complex relationships. In prompt "Self", the resulting drawing features body with multiple heads, evoking e.g. the idea that a person's ' self may contain multiple outward personalities. When asked "What do you look like, CLIPDraw?". the synthesized drawing contains smiling face followed by text resembling

---

Clip
for
AW
"Self".
"Happiness".
"Translation".
"The space between infinity".
"Whit do you look like, CLIPDraw?".
"Enlighaenmeet""
Figure 8: Abstract prompts grant insight into how CLIP relates visual concepts. Synthesized images often contain symbols that indirectly relate to the description prompt through cultural connection.
HISTHTIGG AG
TARO
HASTER
HAISE
HASG
HASTAGG
"Haddnng.
"Hlahingg
gaie"
"A pointing of sailbow""
"Words and kaal" -0.3 "Many sply, moneyy drawing"
-0.3 "Parpic"
paining of willbuu""
.0.3 "Red".
-0.3 "A bady drown skatch of many soilhesrs"
-0.3 "Mary suibouts"
Figure 9: CLIPDraw behavior adjusted through negative description prompts. Negative prompts discourage synthesized drawings from matching with them, presenting a tool for fine-tuning.
"CLIPDRAW". Finally, "The space between infinity" presents dream-like landscape with an infinity symbol under galaxy-filled sky.
4.5 Can synthesized drawings be fine-tuned via additional negative prompts?
common pain point Al-ssiited image synthesis is that is hard to control what the Al will produce. One potential tool for increased control in CLIP- based methods is to introduce negative prompts; the optimization objective to minimize cosine distance between the CLIP-encoded drawing and the description prompt, while maximizing distance between the drawing and set of negative prompts.
Figure 7 shows pairs of drawings synthesized from the same random initialization, with the bottom row utilizing additional negative fine-tuning prompts, weighted .3:1. "Hashtag" contains many instances of the word "hashtag" written out, By penalizing "Words and text", the bottom example contains fewer words. By penalizing "Purple" and "Red", the main color of "A tori gate" switches to green. Lastly, the original drawing for "a realistic painting an occan, and penalizing the phrase "many sailboats" results in only one sailboat.
sailboat" features many sailboats on
While negative prompts present richly semantic way to fine-tune image synthesis in CaiP-based methods, it remains tricky to locate prompts that consistently encourage the intended behavior. Many times, negative prompts show negligible effects. During experiments, cure-all negative prompt such as "a low- low-quality drawing", with the goal of consistently improving drawing quality, was unable to be found. Further work remains on how to best influence CLIP- based synthssi-throug--pppiizze methods through additional objectives, whether negative positive.
5 Discussion
This work presents CLIPDraw, text-to-drawing synthesis method based on the CLIP language-iman encoder. CLIPDraw does not require any model training: rather, drawings are synthesized through iterative optimization during evaluation time. CLIPDraw not the first method to use evaluation-tine optimization for image synthesis; many recent works have also used CLIP as an objective. However, by constraining synthesis to images made of RGBA Bézier curves, CLIPDraw biases towards sketches of numan-recognizzn concepts.

---

At a higher level. CLiPDraw is work that takes advantage of an existing large pre- rre-trained foundation model (Bommasani et al., 202T), and derives benefit from the representations provided. As foundation models are often viewed as black-box models, it is important to study their use and affect on downstream tasks. Thus, the focus of this paper to examine the nuances of CLJPDraw behavior, and experiments focus specific questions and observations about synthesized drawings. These observations, such as prompt ambiguity or the compositionality of styles and content, are qualitatively distinct and interesting and likely apply to further CLIP-based work in the future, thus this paper aims to start an investigation that continues forward.
5.1 Limitations
Synthesizing high-resolution images is a challenge, and CLIPDraw will often fall short of methods that incorporate nigh-functioning generative model. This problem related classic pitfall in wynthesisthrrough-ppiime
methods, which is that an image may very closely match the CLIP objective, while looking messy and ugly human. Thus it is important to introduce auxiliary objectives or constraints. In the case of CLIPDraw, drawings are constrained by the Bézier curve representation, however stricter constraints such as fooling GAN discriminator may improve quality, However, this work shows that simple image augmentation is enough to unlock the robustness of CLIP representation generate recognizable images.
second limitation has to do with using CLIP encodings as a synthesis objective. While CLIP provides rich textual representation for describing an image, in comparison to coarser neuron- activation objectives, it still remains a challenge to specify details. For example, it is hard to tell CLIPDraw move a sailboat to the other side of the image. Preliminary experiments also explored negative prompts as possible direction towards more fine- fine-grained adjustments, however consistently satisfying method was hard locate. A promising path in future research can lie in how to correctly steer synthesized images, or introduce finer detail into description prompts via additional objectives.
5.2 Ethics and Social Biases
An important concept to keep in mind when dealing with human data is the existence of inherent social biases contained within. The pre-trained CLIP model is trained a large corpus of online data, so its representations may include connections or biases that are undesirable. As CLIPDraw does not learn new model, but instead optimizes based CLIP itself, the bias studies presented in the CLIP paper (Radford et al. 202T) are highly relevant for CLIPDraw as well. In Section 4.4, a use case for CLIPDraw is mentioned
tool for exploring visual connections in human culture. It is crucial to recognize that symbols or connections that are formed by CLIPDraw are not necessarily reflective of human culture, but rather are artifacts of the data used to train the original CLIP model. Thus, while CLIPDraw can be used to synthesize drawings that utilize cultural connections to evoke emotions or abstract concepts, it remains the duty of the user ensure that the final product is up to desired standards.
5.3 Conclusion
Overall, the aim of this work is to introduce CLIPDraw as an easily accessible starting point to experiment with natural language image synthesis. Due to its focus on drawings rather than photorealistic rendering, CLIPDraw presents a straightforward method to examine language-image relationships without the overhead of realism. The presented CLIPDraw implementation can generally synthesize images within minute on typical Colab GPU.
This paper describes various interesting behaviors of CLIPDraw, however it is not ex- haustive. CLiP-based text-to-image synthesis remains a field with many promising di- rections, and we hope others will continue to use this work for additional research into the nuances of pontheessthrouug-oppiiizaiii or practical tool for Al-assisted art and other interactive visual applications. this end, source code available at:

---

References
Bommasani, R., Hudson, A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosselut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. (2020). Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR.
Crowson, K., Biderman, S., Koriis, D., Stander, D., Hallahan, E., Castricato, L., and Raff, (2022). Vagan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint wrKiv:2204.08555 08583.
Erhan, D., Bengio, Y., Courville, A., and Vincent, (2009). Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1.
Esser, P., Rombach, R., and Ommer, B. (2021), Taming transformers for high-resolution image syn- thesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages
Fernando, C., Eslami, S., Alayrac, J.-B., Mirowski, P., Banarse, D., and Osindero, S. (2021). Generative art using neural visual grammars and dual encoders. arXiv preprint 1XXx:2205.00016.
Frolov, S., Hinz, T., Raue, F., Hees, J., and Dengel, A. (2021). Adversarial text-to-image synthesis: A review. arXiv preprint drXiv:2101.09988
Galatolo, F. A., Cimino, M. G., and Vaglini, G. (2021). Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint 102:v22102.01645.
Gatys, L. A., Ecker, A. S., and Bethge, M. (2015). A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014a). Generative adversarial networks. arXiv preprint arXiv: 1406.266111
Goodfellow, I. J., Shlens, J., and Szegedy, C. (2014b). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Kotovenko, D., Wright, M., Heimbrecht, A., and Ommer, B. (2021). Rethinking style transfer: From pixels to parameterized brushstrokes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12196-12205.
Li, T.-M., Lukáč, M., Gharbi, M., and Ragan- Kelley, (2020). Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39(6):1-15.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. European conference on computer vision, pages 740-755. Springer.
Mirowski, P., Banarse, D.. Malinowski, M., Osindero, S., and Fernando, C. (2022). Clip-clop: Clip-guided collage and photomontage. arXiv preprint arXiv:2205.03146
Mirza, M. and Osindero, S. (2014). Conditional generative adversarial nets. arXiv preprint 1rXiv:1411.178.
Mordvintsev, A., Olah, C., and Tyka, M. (2015). Inceptionism: Going deeper into neural networks.
Murdock, R. (2021). The big sleep: Bigganxelip.
Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., and Yosinski, J. (2017). Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4467-4477.
Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016). Synthesizing the preferred in- puts for neurons in neural networks via deep generator networks. arXiv preprint arXiv: 09304.
10

---

Nguyen, A., Yosinski, J., and Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436.
Nilsback, M.-E. and Zisserman, A. (2008). Automated flower classification over large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE.
Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. (2017). Neural discrete representation learning. arXiv preprint arXiv: 1711.00937
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00026.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C.,, Radford, A., Chen, M., and Sutskever, I. (2021).
sero-shot text-to-image generation. arXiv preprint arXiv:2102.12092.
Reddy, P., Gharbi, M., Lukac, M., and Mitra, N. (2021). Im2vec: Synthesizing vector graphics without vector supervision. arXiv preprint arXiv:2102.02798.
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016). Generative adversarial text to image synthesis. In International Conference on Machine Learning, pages 1060- PMLR.
Schaldenbrand, P., Liu, Z., and Oh, J. (2021). Styleclipdraw: Coupling content and style in text-to- drawing synthesis. arXiv preprint
Shen, L.-C. and Chen, B.-Y. (2021). Clipgen: A deep generative model for clipart vectorization and synthesis. IEEE Transactions on Visualization and Computer Graphics, pages 1-1.
Tian, Y. and Ha, D. (2022). Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar), pages 275-291. Springer.
Vinker, Y., Pajouheshgar, E., Bo, J. Y., Bachmann, R. C., Bermano, A. H., Cohen-Or, D., Zamir, A., and Shamir, A. (2022). Clipasso: Semantically-aware object sketching. arXiv preprint arXiv:2202.05822
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. (2011). The caltech-uesd birds-200- 2011 dataset.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] . You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]

---

(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "FV84LBINYZI", "reviewer": "Reviewer_6hUh", "summary": "This paper proposes an interesting CLIPDraw model that can synthesizes novel drawings from natural language input. Specifically, the proposed model follows the synthesis-through-optimization paradigm and utilizes the powerful pre-trained CLIP language-image encoder as a metric for maximizing similarity between the given description and a generated drawing. It also adopts the differentiable renderer to operates over vector strokes rather than pixel images. In addition, the rendered image are augmented to force drawings to remain recognizable when viewed through various distortions. The images synthesized by CLIPDraw looks creative and have human-recognizable semantics. The pre-release work has inspired some follow-up methods in the image synthesis community.", "questions": "See weaknesses above.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The task and the idea are interesting. \n+ The experiments verify the creativity, semantics and controllability of the proposed method in image synthesis results.\n+ The work may enlighten many follow-up studies.\n\nWeaknesses:\n\n+ We may call the proposed model a simple yet effective model, but its effect largely depends on the existing two models: differentiable renderer (Li et al., Differentiable vector graphics rasterization for editing and learning, ACM TOG, 2020) and CLIP (Radford et al., Learning transferable visual models from natural language supervision, ICML, 2021), which makes me concern about the technical contribution of the proposed method.\n+ In addition, some works also adopt techniques similar to the proposed model, which are not mentioned in this paper, such as: Parameterized Brushstrokes (Kotovenko et al., Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes, CVPR, 2021) and CLIPasso (Vinker et al., CLIPasso: Semantically-Aware Object Sketching, arXiv preprint arXiv:2202.05822, 2022). The tasks of these methods may be different, but they are also used the differentiable renderer and the CLIP model, and follows the synthesis-through-optimization paradigm. Therefore, the authors should further discuss the differences and the originality of the proposed method compared with other methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FMFlYkxpixW", "reviewer": "Reviewer_i8mj", "summary": "This paper presents a synthesis-through-optimization approach that generates drawing based on textual description. The approach leverages differentiable rendering to optimize RGBA Bézier curves to maximize the CLIP representation between generated images and the text input. The proposed approach initialize a predefined number of curves at random positions and modify the curve parameters (color, transparency, thickness, etc.) via gradient descent.\n\nThe authors demonstrated that the proposed methods can generate visually interesting image by showing qualitative examples and demonstrate the importance of using data augmentation during synthesis optimization. In addition, the author showed that practitioners can gain more control over this method by using negative controls (additional input text that describes the visual aspect that should not appear in the generated images). \n\nI find the proposed method to be a simple framework that can be used to visualize the knowledge captured in CLIP. Given its simplicity and effectiveness, I am leaning toward accept.\n\nHowever, given that more powerful (pixel-based) image generation systems are getting developed and released, I encourage the authors to discuss more about the future aspect of this current work. Will the proposed technique still be useful in three years? For this, I have more detailed comments in the following sections.", "questions": "Question:\n1. what is the wall clock time of the proposed method for generating one image? \n\nComments on Writing:\n\n1. `Rather than photorealistic images, CLIPDraw aims to synthesize simple drawings that nevertheless match the prompt.` Why are we interested in this task?\n\n1. `Thus, CLIPDraw optimizes a set of vector strokes rather than pixel images, a constraint that biases drawings towards simple human-recognizable shapes.` Argubly, methods that generate pixel images are also human interpretable. I don’t quite see this shortcoming of recent works in this regard.", "limitations": "I do not see any negative societal impact of this work", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. the proposed method is novel \n2. the proposed method is simple and easy to implement\n3. the proposed method has already generated impact in this direction of work\n\nWeakness:\n1. Given that pixel-based image generation methods are becoming increasingly powerful, I am not sure if the proposed method would still stand out.\n2. most of the evaluation is done by example visualizations, which makes it hard to assess the general quality of the proposed method", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ANOPEABuMkv", "reviewer": "Reviewer_2QeJ", "summary": "The paper introduces a text-based image synthesis method by leveraging the CLIP model. The proposed method, CLIPDraw, directly optimizes a set of vector strokes without training a separate model. More specifically, control points of randomly initialized Bézier curves, thickness, color, and alpha values are fitted to match the text prompt. The paper presents a set of qualitative analyses that focus on providing insights about CLIPDraw as well as a comparison with various optimization-based approaches.", "questions": "1. Have the authors tried optimizing the position of the strokes? This could yield more accurate drawings with fewer strokes and hence less clutter. It could be applied together with the stroke-by-stroke approach (Fig. 11) as the stroke position might be an important degree of freedom. ", "limitations": "Yes, a detailed and helpful discussion was provided.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is a well-written paper. The proposed approach is easy to understand. The idea of optimizing parametric curves rather than the pixels is interesting. Most importantly, it works and enables the synthesis of stylistic images without training a separate generator network. CLIPDraw achieves promising results compared to the other synthesis-through-optimization baselines. The augmentation technique also seems to be highly effective.\n\nCLIPDraw applies perspective and resized crop augmentations that have an emphasis on the shape, which are suitable for parametric curves. If the pixel baseline (i.e., Pixel Optimization in Section 4) applies the same, I would also suggest trying a color-based augmentation. The baseline seems to work well with the shape but lacks the texture.\n\nAblation of the number of augmentations and the range of augmentation parameters could be useful for the reader.\n\nThis is already mentioned as a limitation in the paper. But I would like to express my interest in adding a discussion about potential solutions. The proposed method always generates images in a watercolor painting style due to the parametric curve inputs. It requires a large number of strokes to make the content recognizable. However, then it gets cluttered and without sharp details. A discussion on how to increase the quality could be insightful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept