Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Authors: Marco Cipriano
Moritz Feuerpfeil
Gerard de Melo
Venue: ICML 2025 poster (2025)
Keywords: Image Generation
Scalable Vector Graphics
VQ-VAE
Differentiable Rasterizer
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a text-guided SVG generative model that creates vector graphics from natural language descriptions, learning from raster images without direct SVG supervision

Abstract: Scalable Vector Graphics (SVG) is a popular format on the web and in the design industry. However, despite the great strides made in generative modeling, SVG has remained underexplored due to the discrete and complex nature of such data. We introduce GRIMOIRE, a text-guided SVG generative model that is comprised of two modules: A Visual Shape Quantizer (VSQ) learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an Auto-Regressive Transformer (ART) models the joint probability distribution over shape tokens, positions and textual descriptions, allowing us to generate vector graphics from natural language. Unlike existing models that require direct supervision from SVG data, GRIMOIRE learns shape image patches using only raster image supervision which opens up vector generative modeling to significantly more data. We demonstrate the effectiveness of our method by fitting GRIMOIRE for closed filled shapes on the MNIST and Emoji, and for outline strokes on icon and font data, surpassing previous image-supervised methods in generative quality and vector-supervised approach in flexibility.

VECTOR GRIMOIRE: Codebook-based Shape Generation under Raster Image Supervision
Marco Cipriano" 11
Moritz Feuerpfeil
Gerard de Melo
Abstract
A A
A
Im2Vee Ours
Scalable Vector Graphics (SVG) a popular for- mat on the web and the design industry. How- ever, despite the great strides made in genera- tive modeling, SVG has remained underexplored due to the discrete and complex nature of such data. We introduce GRIMOIRE, text-guided SVG generative model that is comprised two modules: Visual Shape Quantizer (VSQ) learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an Auto-Regressive Transformer (ART) models the joint probability distribution over shape tokens, positions, and textual descriptions, allowing us to generate vector graphics from natural language. Unlike existing models that require direct super- vision from SVG data, GRIMOIRE learns shape image patches using only raster image supervi- sion which opens up vector generative modeling to significantly more data. We demonstrate the effectiveness of our method by fitting GRIMOIRE for closed filled shapes on MNIST and Emoji. and for outline strokes on icon and font data, surpass- ing previous image-supervised methods in gener- ative quality and the pector-supervisee approach in flexibility.
K A
Figure 1. Generative results for fonts and icons from GRIMOIRE and Im2Vec. Since Im2Vec does accept any conditioning, sample after training Im2 only on icons of stars or the letter A, respectively. For GRIMOIRE we the models trained on the full dataset conditioned on the respective class.
Isola et al., 2017: Saharia et al., 2022; Nichol al., 2021), SVG generation remains a less explored task. Prior work aimed at training deep neural network for this goal has primarily adopted large language models (LLMs) to address the problem (Wu et al., 2023; Tang et al., 2024). In general, existing approaches share two key limitations: they neces- sitate SVG data for direct supervision, which inherently limits the available data and increases the burden of data pre-processing, and they are not easily extendable when it comes to visual attributes such as color or stroke properties. The extensive pre-processing is required due to the diverse nature of an SVG file that can express shapes as a series of different basic primitives such as circles, lines, and squares each having different properties that can overlap and occlude each other.
1. Introduction
In the domain of computer graphies, Scalable Vector Graph- ies (SVG) has emerged as a versatile format, enabling the representation of 2D graphics with precision and scalabil- ity. SVG is an XML-based vector graphics format that describes a series of parametrized shape primitives rather than limited-resolution raster of pixel values. While mod- generative models have made significant advancements in producing high-quality raster images (Ho al., 2020;
An ideal generative model for SVG should however benefit from visual guidance for supervision, which is not possible when merely training to reproduce tokenized SVG primi- tives, as there is no differentiable mapping to the generated raster imagery. In this paper, we present GRIMOIRE, a novel pipeline explicitly designed to generate SVG files with only raster image supervision. Our approach incorporates dif- ferentiable rasterizer, DifIVG (Li al., 2020), bridge the vector graphics primitives and the raster image domain. We adopt a VQ-VAE recipe (Van Den Oord et al., 2017), which pairs codebook-basee discrete auto-encoder with an nuto-regressive Transformer that models the image space implicitly by learning the distribution of codes that resem- ble them. We find this approach particularly promising for
Equual contribution 'Hasso Plattner Institute. Correspondence to: Marco Cipriano mmaaro.cipriano@ee @hpi.de>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
vector graphies generation, as it breaks the complexity of this task into two stages. In the first stage of our method, we decompose images into primitive shapes represented as patches. A rector-quantize auto-encoder learns to encode and map each patch into a discrete codebook, and decode these codes to an SVG approximation of the input patch, which trained under raster supervision. In the second stage, the series of raster patches containing primitives are encoded and the prior distribution of codes is learned by an auto-regressive Transformer model conditioned on a tex- tual description. At inference, a full series of codes can be generated from textual input, or other existing shape codes. Therefore, GRIMOIRE supports text-to-SVG generation and SVG auto-completion as possible downstream tasks out-of- the-box.
keNUWA (Tang et al., 2024) introduced Vector Quantized Strokes to compress SVG strokes into codebook with SVG supervision and fine-tune a pre-trained Encoder-Decoder LLM to predict these tokens given textual input.
Another line of work has sought to incorporate visual super- vision. These approaches generally rely on recent advances in differentiable rasterization, which enables backpropaga- tion of raster-based losses through different types of vecto- rial primitives such Bézier curves, circles, and squares. The most important development in this area is DiffVG (Li et al., 2020), which removed the need for approximations and introduced techniques to handle antialiasing. They fur- ther pioneered training a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN; Goodfellow et al., 2014) on MNIST (LeCun et al., 1998) and QuickDraw (Ha & Eck, 2017). These generative capabilities have subsequently been extended in lm2Vec (Reddy et al., 2021), which adopts a VAE including a recurrent neural network to generate vector graphics as sets of deformed and filled circular paths, which are differentiably composited and rasterized, allowing for back proppaaopaa of a multi-resolution MSE-based pyramid loss. However, all of these models lack versatile condition- ing (such as text) and focus either image vectorization, i.e., the task of creating the closest vector representation of a raster prior, or vector graphics interpolation.
SVG generative models by
The key contributions of this work are:
1. We frame the problem of image-supervised SVG gener- ation the prediction of series of individual shapes and their positions on shared canvas.
2. We train the first text-conditioned generative model that learns to draw vector graphies with only raster image supervision.
3. We compare our model with alternative frameworks showing superior performance in generative capabili- ties on diverse datasets.
A different type of SVG generation enabled by DiffVG is painterly rendering (Ganin et al., 2018; Nakano, 2019), where an algorithm iteratively fits a given set of vector primitives to match an image, guided by deep perceptual loss function. To achieve this goal, CLIPDraw (Frans et al., 2022) rasterized set of randomly initialized SVG paths and encoded these with a pre-trained CI.IP (Radford et al., 2021) image encoder, iteratively minimizing the cosine distance between such embeddings and the text description. Vector Fusion (Jain al., 2023) and SVGDreamer (Xing al., 2024) leveraged Score Distillation Sampling (SDS; Poole et al., 2022) to induce abstract semantic knowledge from an off-the-shelf Stable Diffusion model (Rombach et al., 2022). Finally, some other approaches based on neural implicit representations model vector graphics as continuous functions encoded in neural networks (Thamizharasan et al., 2024: Zhang et al., 2024; Polaezek et al., 2025).
4. We release the code of this work to the research com- munity'.
2. Related Work
2.1. SVG Generative Models
The field of vector graphics generation has witnessed in- creasing interest. While a number of works predate the era of Large Language Models (Carlier et al., 2020; Wang & Lian, 2021), the most recent approaches (Lopes et al., 2019; Aoki & Aizawa, 2022; Wu et al., 2023; Tang et al., 2024) have recast the problem as an NLP task, learning distribu- tion over tokenized SVG commands. Iconshop (Wu et al., 2023) introduced a method of tokenizing SVG paths that makes them suitable input for causal language modeling. To add conditioning, they employed a pre-trained language model to tokenize and embed textual deseriptions, which are concatenated with the SVG tokens to form sequences, on which the auto-regressive Transformer can learn joint probability. Chat2SVG introduces a hybrid framework that leverages LLMs and image diffusion models to generate and refine SVG (Wu et al., 2024). StarVector trains a mul- timodal LLM on SVG data (Rodriguez et al., 2024). Stro-
2.2. Vector Quantization
VQ-VAE (Van Den Oord et al., 2017) is a well-known improved architecture for training Variational Autoen- coders (Kingma & Welling, 2013; Rezende et al., 2014). Instead of focusing on representations with continuous fea- tures as in most prior work (Vincent et al., 2010; Denton et al., 2016; Hinton & Salakhutdinov, 2006; Chen et al., 2016), the encoder in a VQ-VAE emits discrete rather than
hhtpp::// wwwphhhww..tptooo./tcct cwwm//pptpoo//eecrrrrriiiooo
2

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
VSQ Module
SVG
discrete codebook look
SVG Predition Head
Differentiable Rasterization
Oupur Coordinates
Other Attribuues
Discrete Encoder
Projection Layer
color Width
<g
Input Raster Patch
Quantized Embedding ng
Predicted Raster Patch
monu
Data Creation for ART training
ART Module
"The icon clock"
Trained VSQ Encoder
Discrete Patch Positions
BERT
sos
Auto RittoRegressive Transformer
V1
8g
Uy
0g
U/i
EOS
Cansei
Figure 2. Overview of training pipeline of GRIMOIRE, First, the VSQ module encodes raster input patches into discrete codes and learns optionally, geometric constraint. After this, each image
reconstruct each patch as an SVG shape using visual supervision encoded into series of discrete codes using the trained VSQ encoder. Finally, the ART module learns the joint distribution of these codes and the corresponding text description. At inference, only text or partial codes are provided for conditioning the ART module.
continuous codes, Each code maps to the closest embedding codebook of limited size. The decoder learns to recon- struct the original input image from chosen codebook embedding. Both the encoder-decodee architecture and the codebook trained jointly. After training, the autoregres- sive distribution over the latent codes is learnt by second model, which then allows for generating new images via ancestral sampling. Latent discrete representations were already pioneered in previous work (Mnih & Gregor, 2014; Courville et al., 2011). Mentzer et al. (2023) simplified the design of the vector quantization in VQ- IQ-VAE with a scheme called finite scalar quantization (FSQ), where the encoded representation of image is projected the nearest po- sition a low-dimensional hypercube. In this case, additional codebook must be learned, but rather it is given implicitly, which simplifies the loss formulation. Our work builds in part on the VQ- VAE framework and includes the FSQ mechanism.
tized vector into shape parameters of cubic Bézier curves through the decoder Dvsq- Instead of learning the code- book (Van Den Oord et al., 2017), we adopt the more ef- ficient approach of defining our codebook V as a set of equidistant points in hypercube with g dimensions. Each dimension has unique values: L 12,,22.. 141- The size of the codebook [V] is hence defined by the prod- uet of values of all dimensions. We define q = 5 and L = [7,5, 5,5, 5,5, 5] for a target codebook size of 4,375 unique codes, following the recommendations the original au- thors (Mentzer et al., 2023).
Before being fed to the encoder Evsq, each image RC is divided into patches S ($1, $2..... 8n)) with 128x where C = 3 is the number of channels. A set of discrete anchor coordinates O with 6 N2 being the center coordinate of s; the original image is also saved. The original image I can then reconstructed using S and θ.
(0,
3. Method
We experiment four datasets (see Section A.I). For MNIST, the patches are obtained by tiling each image in 6 x 6 grid. For Fonts and FIGR-8, each patch depicts part of the target outline. We also include preliminary results on Emojis, where our training data is created using the Segment Anything (SAM) model (Kirillov et al., 2023), which provides series of masks for the entire image. In this extraction pipeline, each mask produces one patch. The
3.1. Stage 1 Visual Shape Quantizer
The first stage of our model employs Visual Shape Quantizer (VSQ),* a vector-quantized auto-encoder, whose encoder Evso maps an input image onto a discrete code- book V through vector-quante quatti-quaniintiion and decodes that quan-
3

---

Vector Grimoire: Codebook- Shape Generation under Raster Image Supervision
Finally, the predicted path of Bézier curves from $ points passes through the differentiable rasterizer obtain raster output & aifiv(DDvoo) (vso (s:))). In order to learn to reconstruct strokes and shapes, we train the VSQ module using the mean squared error:
Raw Images
Extraction Method
Extracted Patch and Positions
Leccuus = (s
(1)
Bounding Coordinates the Original Image
Dvsq can be extended to predict continuous values for any visual attribute supported by the differentiable ras- terizer. Hence, we also propose series of other fully- connected prediction heads that can optionally enabled: Pwithh - I predicts the stroke width of the overall shape, and c - R outputs the stroke color or the filling color for the output of $ points- All the modules followed by a sigmoid activation function.
S
Tille Indexes
While Lisons would suffice for training the VSQ, operating only on the visual domain could lead to degenerate strokes and undesirable local minima. To mitigate this, we propose novel geometric constraint feeamn which punishes the placement of control point at irregular distances measured between all combinations of points predicted by 4 pointss
Sam Box Coordinates, Layer Hierarchy
Let P = (P..P2....,Pooe Pv+1) be the set of all start and end points of a curve with p; = (pf ..)) and p,, E [0, 1J. Then Pt. is defined as the Euclidean distance between two points ps and Py. Pj is defined as the mean scaled inner distance for point Pj to all other points in P, while 6, is the average squared deviation from that mean for point pj:
Figure 3. Overview of the data generation process for GRIMOIRE. For FIGR-8, we extract the outlines of each icon and create small centered raster segments. We save the original anchor position of each segment for the second stage of our training pipeline. For 6 Grid
the MNIST digits, we simply create patches from and save the index of each tile. For Emoji, we generate an overall segmentation mask with SAM and create a raster patch from each mask. The mask bounding hox coordinates can be used in the second stage similarly to the experiments on FIGR-8. The layer order is created by sorting patches by area. Fonts comes in vector format and can be casily manipulated to extract strokes, similarly to FIGR-8. More information about the outline extraction provided in Section A.3.
v+1
Pi.j
1
u+1
Piái
DJ
6j
Pj
(2)
-i-3) j
fyoom is finally defined as the average of the deviations for all start and end points in P. Zeomm is then weighted with and added to the reconstruction loss.
different extraction approaches and the resulting patches and coordinates are depicted in Figure 3.
The VSQ encoder Evsq maps each patch 8; CC11881128 to codes on the hypercube EvsQ 1218811 + V as follows. Each centered raster patch s, is encoded with ResNet-18 (He et al., 2016) into latent variable 2; E 2 = Rdx with = 512. Eventually, each of the & codes is projected to y dimensions through a linear mapping layer and finally quantized, resulting in & codes (u1, U2 Ug) Ui)) with e; V.
C geom
53
Lvso Crecoms + xxxxxx Eganm
(3)
==1
Here, a is hyperparameter. The overall scheme of GRI- MOIRE including the first stage of training is depicted in Fig- ure 2.
3.2. Stage 2 - Auto- Regressive Transformer
The decoder DvsQ consists of projection layer, which transforms all the & predicted codes back into the latent space 2, and a lightweight neural network T points which predicts the control points of cubic Bézier curves that form single connected path.
After the VSQ is trained, each patch 8; can be mapped onto code of the codebook V using the encoder Evsq and the quantization method. However, the predicted patch s; captured by the VSQ does not describe a complete SVG, as the centering leads loss of information about their

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
global position 0g on the original canvas. Also, the sequence of tokens is still missing the text conditioning. This is addressed in the second stage of GRIMOIRE. The second Transformer (ART)
(22x(xx++1)), which predicts connected strokes, and a 1-D
CNN o Apoiss 2 ((2x(x33)) which outputs closed shape.
stage consists of an Auto-Regresse that learns for each image / the joint distribution over the text, positions, and stroke tokens. A textual description I of is tokenized into T = (Ti; T2,.... Tr) using a pre-trained BERT encoder (Devlin et al.. 2018) and embedded. I is visually encoded by transforming its patches 8; onto i V via the encoder Evsq, whereas each original patch position 0g E θ is mapped into the closest position in a 256 x 256 grid, resulting in 256² possible position tokens. Special tokens <SOS>, <BOS>, and <EOS> indicate the start of a full sequence, beginning of the patch token sequence, and end of sequence, respectively. Patch tokens alternate with corresponding position tokens. The final input sequence for given image to the ART module becomes:
We use Eeeom only for the experiments with D and set a = 0.4. We opt to train the ResNet encoder from scratch during this stage, since the target images belong to a very specific domain. The amount of trainable parameters is 15.36.M for the encoder and 0.8M for the decoder. The final inference pipeline discards the encoder and only requires the lightweight trained decoder Dysq. hence resulting in faster inference.
5. Results
In this section, we report the results for all our experiments and discuss our findings. First, in Section 5.1 and Sec- tion 5.2, examine the quality of the reconstructions and generations produced by GRIMOIRE in comparison to existing methods. Then, in Section 5.3 we highlight the flex- ibility of our approach, demonstrating how GRIMOIRE can be easily extended to incorporate additional SVG features and more complex targets. Furthermore, in Section 5.4 we compare GRIMOIRE with some popular
x (<sos>, M,,.., Tt, <BOS>, 01, u1,...
.. Gn Up <EOS>)
The total amount of representable token values then has a magnitude of | + 256 +3 = 69, 914 for [V| = 4,375. A learnable weight matrix We e Rdx69.914 embeds the position and visual tokens into vector of size d. The BERT text embeddings are projected into the same d-dimensional space using trainable linear mapping layer. The ART module consists of 12 and 16 standard Transformer decoder blocks (with causal multi-head attention using 8 attention heads) for fonts and icons, respectively. The final loss for the ART module is defined as:
method
5.1. Reconstructions
Closed Paths. We begin by presenting the reconstruction results of our VSQ module on the MNIST dataset. In our experiments, we model each patch shape using a total of 15 segments. Increasing the number of segments beyond this point did not yield any significant improvement in recon- struction quality. Given the simplicity of the target shapes, we adopted a single code per shape.
CCausll
Σ logp(æ; Izzi
(4)
We also conducted a comparative analysis of the reconstruc- tion capabilities of our VSQ module against 1m2Vee. To assess the generative quality of our samples, we employed the Fréchet Inception Distance (FID; Heusel et al., 2017) and CLIPScore (Radford et al., 2021), both of which are computed using the image features of a pre-trained CLIP encoder. Additionally, to validate our VSQ module, we con- sidered the reconstruction loss Lrecons as it directly reflects the maximum achievable performance of the network and provides more reliable metric.
During inference, the input to the ART module is repre- sented as x = (<SOS>, .i,..., Tt, <BOS>), where new to- kens are predicted auto-regressively until the <EO5> token is generated. Additionally, visual strokes can be incorpo- rated into the input sequence to condition the generation process.
4. Experimental Setting
When training on FIGR-8, we utilize a contour-finding al- gorithm (Lorensen & Cline, 1987) to extract outlines from raster images, which are then divided into several shorter segments. Additional details regarding this extraction pro- cess can be found in Section A.3. In contrast, the Fonts dataset is natively available in vector format, making it easier to manipulate, similar to icons, before undergoing rasterization.
As shown in Table 1. our VSQ module consistently achieves lower reconstruction errors than Im2Vec across all MNIST digits. In Table 2, we also evaluate a subset with only digits zero, chosen for its challenging topology, where our method again outperforms Im2 For MNIST, we fill 1m2Vec's s predicted shapes to match the raster ground truth, while for other scenarios, we report both filled and unfilled versions.
Our reconstructions achieve higher CLIPScores in all cases. The only exception is FID, where 1m2Vec occasionally per-
We propose two variants of $ Ppeints described in Sec- tion 3.1, a fully-connected neural network @painks 2
5

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Table Results for reconstructions of GRIMOIRE and Ve2 the test-set including all classes. The last row includes post-processing.
MNIST
Fonts
FIGR-8
Model
MSE J
FID J
CLIP
MSE
FID 11
CLIP 11
MSE 41
FID 41
CLIP 11
Im2 (filled)
0.140
1.33
25.02
0.140
2.04
26.82
0.330
16.10
26.17
Im2Vec
n/a
n/a
n/a
0.050
5.64
26.72
0.050
13.90
26.17
VSQ
0.090
7.09
25.24
0.014
4.45
28.61
0.004
1.42
31.09
VSQ + PI
n/a
n/a
n/a
0.011
0.29
28.96
0.002
0.05
32.03
Table Results for reconstructions of GRIMOIRE and Im2Vec on the test-set, using the class reported next the dataset name. The last row includes post-processing
MNIST (0)
Fonts (A)
Icons (Star)
Model
MSE l
FID J
CLIP 11
MSE 11
FID 41
CLIP 11
MSE 4
FID 11
CLIP ft
Im2Vee (filled)
0.218
2.20
24.61
0.087
1.64
26.27
0.120
2.40
30.90
Im2Vec
n/a
n/a
n/a
0.060
6.33
25.78
0.110
11.17
30.40
VSQ
0.130
11.2
26.68
0.020
4.50
29.13
0.002
1.26
31.64
VSQ +
n/a
n/a
n/a
0.012
0.61
29.46
0.001
0.07
32.94
forms better. We attribute this the lower resolution of the ground truth images, which affects FID stability. CLIPScore mitigates this by directly measuring similarity to the textual description.
is specifically trained on such subset. In Table also report the values after training on the full datasets. In this case, GRIMOIRE substantially outperforms Im2Vec, which is unable to cope with the complexity of the data.
Strokes. We analyze VSQ reconstruction errors on Fonts and FIGR-8 under varying segment counts, codes per shape, and input stroke lengths. For Fonts, using more than one seg- ment per shape consistently degrades reconstruction, likely because the dataset's strokes require fewer Bézier curves. Shorter stroke length thresholds improve reconstruction, with MSE decreasing as the threshold moves from 11% to 7% to 4% of the image size. While shorter strokes are easier to model, overly short settings may cause worse predictions.
Finally, as GRIMOIRE quickly learns to map basic strokes or shapes onto its finite codebook and due to the similarities hetween those primitive traits among various samples in the dataset, we find GRIMOIRE converge even before completing full epoch any dataset. Despite the recon- struction error being considerably higher, we also notice reasonable domain transfer capabilities between FIGR-8 im- ages and Fonts when training the VSQ module only on one dataset and keeping the maximum stroke length consistent. Qualitative examples of the re-usability of the VSQ module are reported in the Appendix.
The best reconstructions are achieved by using multiple codes per centered stroke. The two-codes configuration has average decrease in MSE of 18.2 18.28%, 41.46%, and 26.09% for the respective stroke lengths. However, the best- performing configuration with two codes per shape is just 11.36% better than the best single code representative, which we believe does not justify twice the number of required visual tokens for the second stage training. Throughout our experiments, the configurations with multiple segments con- sistently benefit from our geometric constraint. Ultimately, for our final experiments, we choose (v 2, 1) for Fontsand (v= = 4, <= = 2) for FIGR-8.
5.2. Generations
Text Conditioning. We compare GRIMOIRE with Im2Vec by generating glyphs and icons and handwritten digits, and report the results in Table Despite Im2Vec being tailored for single classes only, our general model shows superior performance in CLIPScore for all datasets. Im2Vec shows generally lower FID score in the experiments with filled shapes, which we attribute again to the lower resolution of the ground truth images (MNIST) and bias in the metric itself CLIP struggles to produces meaningful visual em- beddings for sparse images (Chowdhury al., 2022) for Fontsand FIGR-8. In contrast, in the generative results on
Regarding the comparison with Im2Vec, Table 2 shows that text-conditioned GRIMOIRE on a single glyph or icon has superior reconstruction performance even if Im2Vec
6

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
unfilled shapes, GRIMOIRE almost consistently outperforms Im2 Vec by a large margin for glyphs and icons.
Ve =
0
=
Ve =
3
Ve = 10 = 15
Map Pin
Note that we establish new baseline results for the complete datasets, as Im2Vee does not support text or class condition- ing.
Document Cloud
Looking at qualitative samples Figure I, one can see that contrary to the claim that surplus shapes collapse to a point (Reddy et al., 2021), there multiple redundant shapes present in the generations of Im2Vec. A single star might then be represented by ten overlapping almost identical paths.
Overall, GRIMOIRE produces much cleaner samples with
less redundancy, which makes them easier edit and more visually pleasing. The text conditioning also allows for more flexibility. Furthermore, the generations are diverse, as can be seen in Figure 4, where we showcase multiple generations for the same classes from FIGR-8. Additional generations on all datasets are provided in the Appendix.
Figure 5. Different completions with varying number of context segments ve (marked in red). GRIMOIRE can meaningfully com- plete irregular starting positions of the context strokes.
-
-
Figure 6. Inputs (top) and corresponding reconstructions (bottom) generated by a VSQ model trained to predict additional visual attributes of the input strokes, such as color and stroke width. Input from the test-set.
Phone
Heart
Light bulb
Arrow
User
Home
Settings
Figure 4. Examples of text-conditioned icon generation from GRI- MOIRE.
Vector Conditioning. We also evaluate GRIMOIRE on an- other task previously unavailable for image-supervised vec- tor graphic generative models, which is text-guided icon completion. Figure 5 shows the capability of our model to complete an unseen icon, based on set of given context strokes that start at random positions. GRIMOIRE can mean- ingfully complete various amounts of contexts, even when the strokes of the context stem from disconnected parts of the icon. We provide a quantitative analysis in Section A.13. The results in this section are all obtained with the default pipeline that post-processes the generation of our model. detailed analysis of our post-processing is provided in Section A.4 and Section A.5.
VSQ module on input patches with varying attributes. Fig- ure shows the results, where strokes randomly colored using an eight-color palette and variable widths. The VSQ accurately learns these features without expanding the code- book or modifying the network architecture.
A similar evaluation on closed shapes (Figure 7) confirms that VSQ jointly encodes shape and color within a single code. In contrast, other vector generative models often rely rigid tokenization schemes, making extension to new attributes more complex.
Finally, we report preliminary results of GRIMOIRE on more complex colorful targets based on segmentation-based extraction approach. We validate this by reconstructing images of Emoji, following the MNIST setup but with erucial difference: each closed shape represents entire layer of the canvas rather than a tile. This allows for SVGs structured as editable layers, similarly to real-world use cases. Figure 8 illustrates per-layer reconstructions, while Figure presents the final composited outputs. Further details on these experiments are reported Section A.2.
5.3. Flexibility
We demonstrate the flexibility of GRIMOIRE with qualitative results on new SVG attributes. A key advantage of our two- stage generative pipeline is that the ART module remains independent of visual attributes, allowing the VSQ vector prediction head to be extended to any attribute supported by the differentiable rasterizer. Specifically, we enable stroke width and color prediction ( wiith and & color), training the

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Table Results for generation using GRIMOIRE and 1m2Vec. GRIMOIRE is trained with all the classes of the dataset and conditioned the respective class using the text description. FID uses test-data as target.
MNIST (0)
MNIST (Full)
Fonts (A)
Fonts (Full)
FIGR- 8(Star)
FIGR-S(Full)
Model
FID IJ
CLIP 11
FID J1
CLIP IT
FID 11
CLIP 11
FID 11
CLIP 11
FID 11
CLIP 11
FID T1
CLIP 11
Ie2 (filled)
2.22
24.69
w/a
1.20
25.81
n/a
I/
31.72
n/a
Tm2Vee
n/a
25.21
n/a
n/a
5,36
25.39
n/a
n/a
11.59
31.88
n/a
n/a
GRIMOIRE (ours)
12.25
26.60
9.25
25.25
5.61
30.60
1.67
28.64
6.25
32.24
0.64
29.00
Target
VSQ
Target
vsQ
Input
VSQ
Figure Reconstruction of MNIST digits when VSQ module also predicts the filling color. Input from the test-set.
Input
VSQ
Input
VSQ
Figure 9. Reconstructions of emojis from our VSQ, all SVG layers are rendered together. Input from the test-set.
Making this analysis quantitative is not straightforward. The FID score with regard to the image distribution is reliable thousands of samples, but the computational cost of SDS-based models requires to hours for few samples (e.g.. SVGDreamer) or uses costly proprietary models (e.g., Chat2SVG). We have hence used the PSNR on 20 generated samples from all models. The results in Table 4 highlight how all models fall short on our dataset distribution.
Figure 8. Reconstructions of individual layers of emojis from the
our VSQ. Input from the test-set.
5.4. Differrncces with SDS-based Methods
In this section, we clarify how our model differs from popu- lar Score Distillation Sampling (SDS) architectures.
Model
PSNR (dB)
Lack of target. SDS methods do not involve training, and rely on pretrained backbones, which produce more artis- tic and visually results, but also unbound to any specific target data. SDS methods lack any control on the target domain. To highlight this aspect, Figure 10 reports an example of class images adopted in this work, and shows the different generations obtained with GRIMOIRE and pop- ular SDS methods such VectorFusion, CLIPDraw, and SVGdreamer. GRIMOTRE produces simple yet diverse gen- crations that are coherent with its reference dataset. In con- trast, in all cases, the generations from SDS based methods appear distant from the target distribution, often partially ignoring the "black and white" suffix in the prompt.
GRIMOIRE (ART)
45.19
CLIPDraw
28.68
VectorFusion
36.62
SVGDreamer
34.53
Table 4. Average PSNR of 20 generated samples for the class "User" with GRIMOIRE and existing SDS-based methods.
Speed. SDS methods are also iterative by design, which means that generating results is extremely slow in practice. One of the motivations for training generative pipeline like GRIMOIRE is that, at inference time, producing a new sample merely takes the time of forward pass. Indeed,
8

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
GRIMOIRE is orders of magnitude faster than popular SDS based methods. In Table 5, report the generation time (time for inference and file saving) for an image with Grimoire, VectorFusion, and CLIPdraw. These values were obtained across 20 generations on one NVIDIA H100.
Impact Statement
This work aims to advance the field of text-to-SVG gen- eration by introducing a new framework that learns using only raster image supervision. GRIMOIRE enables broader accessibility training data. While the method may benefit applications in education, design, and accessibility, we do not foresee any immediate ethical concerns or adverse soci- etal impacts stemming from this research. Potential indirect effects are similar as for other Generative Al techniques. We hope that models that emit vector graphies can make it eas- for graphic designers build upon, enabling improved human Al workflows.
Model
Generation Time (s)
GRIMOIRE (ART)
2.34
CLIPDraw
100.19
VectorFusion
379.74 74
SVGDreamer
250.22
Table 5. Average generation times of icons given only the text prompt measured on five samples using one NVIDIA H100 GPU.
SDS-based methods extremely slow due to their iterative opti- mization strategy and result impractical for real-life applications.
References
Aoki, H. and Aizawa, K. Svg vector font generation for chinese characters with transformer. In 2022 IEEE In- ternational Conference Image Processing (ICIP), pp. 646-650. IEEE, 2022.
Datasets Samples
Vector
GRIMOIRE
CLIPDraw
SVGDreamer Fusion
Carlier, A., Danelljan, M., Alahi, A., and Timofte, R. Deepsvg: A hierarchical generative network for vector graphics animation. Advances in Neural Information Prucessing Systems, 33: 33:16351-16366 2020.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Figure 10. Text-conditioned generations from GRIMOIRE and based methods for "Heart" class. first row shows samples from FIGR-8. For CLIPDraw, VectorFusion and SVG- Dreamer, we used the prompt: "The icon heart, black and white.
Chowdhury, P. N., Sain, A., Bhunia, A. K., Xiang, T., Grya- ditskaya, Y., and Song, Y.-Z. Fs-coco: Towards under- standing of freehand sketches of common objects in con- text. In European Conference on Computer Vision, pp. 253-270. Springer, 2022.
6. Conclusion
Clouâtre, L. and Demers, M. Figr: Few-shot image gen- eration with reptile. arXiv preprint arXiv:1901 09119, 2019.
This work introduces GRIMOIRE, a novel framework for generating and completing complex SVGs using only raster image supervision. GRIMOIRE enhances the output quality over existing raster-supervised SVG models, while enabling flexible, lext-conditioned generation. We validate it on filled strategy
Courville, A., Bergstra, J., and Bengio, slab restricted boltzmann machine. In Proceedings of the fourteenth international conference artificial intelli- gence and statisties, pp. 233-2411 JMLR JMLR Workshop and
spike and
shapes using a tile-patching or segmentation-bant and on strokes with fonts and icons datasets. Our results show superior performance compared to existing models, even when adapted to specific image classes. We show that GRIMOIrE be seamlessly extended to support SVG attributes when included in the training data.
Conference Proceedings. 2011.
Denton, E., Gross, S., and Fergus, R. Semi-supervised learning with context context-conditional generative adversarial networks. arXiv preprint arXiv: 1oae::/1111.600 1.06430, 2016.
Acknowledgments
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 1810.0000 2018.
This work was conducted in part with funding from the HPI-MIT Designing for Sustainability Program.

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
emoji dataset. emoji-dataset. h co co/datasets vaa ha haw haalaaammmmmaaaaaa 2022.
LeCun, Y., Bottou, L.,, Bengio, Y., and Haffner, Gradient- based learning applied document recognition. Proceed- ings of the IEEE, ((11))2277-2322. 1998.
Frans, K., Soros, L., and Witkowski, O. Clipdraw: Explor- ing text-to-drawing synthesis through ingga encoders. Advances in Neural Information Processing Systems, 35:5207-5211 2022.
Li, T.-M., Lukáč, M., Gharbi, M., and Ragan- Kelley, J. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39 (6): 2020.
Ganin, Y., Kulkarni, T., Babuschkin, I., Eslami, S. A., and Vinyals, O. Synthesizing programs for images using rein- forced adversarial learning. International Conference on Machine Learning, pp. 1666-1675. PMLR, 2018.
Lopes, R. G., Ha, D., Eck, D., and Shlens, learned rep- resentation for scalable vector graphics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7930-7939, 2019.
Goodfellow, I., Pouget-Abadie. J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Lorensen, W. and Cline, H. E. Marching cubes: high resolution 3d surface construction algorithm. In Pro- ceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '87, pp. 163-169, New York, NY, USA, 1987. Association for Computing Machinery. ISBN 0897912276. doi: 10. 10.1 14.134/7740 URL https: //tp::///oooo./...//// 0115/37401.333
Ha, D. and Eck, D. A neural representation of sketch draw- 2017.
ings. arXiv preprint
He, K., Zhang, X., Ren, S., and Sun, Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 17:1:011110010 2017.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXivv2309. 15505, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Mnih, A. and Gregor, Neural variational inference and learning belief networks. In International Conference Machine Learning, pp. 1791-1799. PMLR, 2014.
Hinton, G. F. and Salakhutdinov, R. R. Reducing the di- mensionality of data with neural networks. science, 313 (5786):504-55 507, 2006.
Nakano, R. Neural painters: A learned differentiable constraint for generating brushstroke paintings. arXiv preprint 190.vv:9940004410 2019.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXhv:2112.10744 2021.
Isola, P., Zhu, J. Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vi- sion and pattern recognition, pp. 1125-1134, 2017.
Polaczek, S., Alaluf, Y., Richardson, E., Vinker, Y., and Cohen-Or, D. Neuralsvg: An implicit represen- tation for text-to-vector generation. arXiv preprint arXiv:2501.03992, 2025.
Jain, A., Xie, A., and Abbeel, Vectorfusion: Text-to-svg by abstracting paxel-based diffusion models. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1911-1920, 2023.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.111 2013.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 3748-8763 PMLR, 2021.
10

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Reddy, P., Gharbi, M., Lukac, M., and Mitra, N. J. Im2vec: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 7342-7351, 2021.
Wu, R., Su, W., and Liao, J. Chat2svg: Vector graphics gen- eration with large language models and image diffusion models. arXiv preprint arXiv:2411.16602. 2024.
Xing. X., Zhou, H., Wang. C., Zhang, J., Xu, D., and Yu, Q. Svgdreamer: Text guided svg generation with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. June 2024.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Zhang, P., Zhao, N., and Liao, Text-to-vector generation with neural path representation. ACM Transactions on Graphics (TOG), 43(4):1-13, 2024.
Rodriguez, A., Puri, A., Agarwal, S., Laradji, I. H., Rodriguez, P., Rajeswar, S., Vazquez, D., Pal, C., and Pedersoli, M. Starvector: Generating scalable vee- tor graphics code from images and text, 2024. URL https: /aaaiiivv.../... org/ ab//bb///// 2212. 11556.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolutinn image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022.
Saharia, C., Chan, W. Saxena, Si, L., Whang. J., Denton, E. L.. Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479-36494, 2022.
Tang, Z., Wu, C., Zhang, Z., Ni, M., Yin, S., Liu, Y., Yang. Z., Wang, L., Liu, Z., Li, J.. al. Strokenuwa: Tokeniz- ing strokes for vector graphic synthesis. arXiv preprint 1rXiv:2401.17093 2024.
Thamizharasan, V., Liu, D., Fisher, M., Zhao, N., Kaloger- akis, E., and Lukac, M. Nivel: Neural implicit vector layers for text-to-vector generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4589 4597, 2024.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., and Bottou, L. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 11(12), 2010.
Wang, Y. and Lian, 7. Deepvecfont: Synthesizing high- quality vector fonts via dual-modality learning. ACM Transactions on Graphics, 40(6), December 2021. doi: 10.1
Wu, R., Su, W., Ma, K., and Liao, J. Iconshop: Text-guided vector icon synthesis with autoregressive transformers. ACM Transactions on Graphics (TOG), 42(6):1-14, 2023. 2023.

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
A. Appendix
The appendix is organized as follows. In Section A.I, we provide information about the data used in our work; in Section A.2, we report additional details on the segmentation-batt based extraction method; in Section A.3 and Section A.4, we describe the pre-processing and post-processing applied in our experiments; in Section A.5, we show results with different post- processing methods; in Section A.6, we provide comparison with vector-supervised methods; in Section A.7, Section A.8, and Section A.9, we study sensitivity with regard to patch size, stroke length, and codebook size, respectively; in Section A.10, we report results of Im2 on other classes; in Section A. we show additional qualitative results on the effects of our geometric loss; in Section A. 12, we share some implementation details; in Section A. we show more results on GRIMOIRE conditioned strokes; in Section A. A.14, we explore domain transfer capabilities; in Section A.15, we analyze the codebook usage of our model; finally in Section A.17 and Section 18, we report more qualitative results and comparison on the reconstruction and generative of GRIMOIRE. We also report a glossary of all the notation in Section A.
A.1. Dataset
MNIST. We conduct our initial experiments on the MNIST dataset (LeCun et al., 1998). We upscale each digit to 128 x 128 pixels and generate the textual description using the prompt "x in black color", where is the class of each digit. We adopt the original train and test splits.
Fonts. For our experiments on fonts, we use a subset of the SVG-Fonts dataset (Lopes et al., 2019). We remove fonts where capital and lowercase glyphs are identical, and consider only 0-9, a-z, and A-Z glyphs, which leads to 32,961 unique fonts for a corpus of 22 samples. The font features- - such as type of character or style are extracted from the .TFF file metadata. The final textual description for a sample glyph g in font style s is built using the prompt: "[capital] g in s font", where "capital is included only for the glyphs A-Z. We use 80%, 10%, and 10% for training, testing, and validation respectively.
FIGR-8. We validate method on more complex data and further subset of FIGR-8 (Clouâtre & Demers, 2019). where we select the 75 majority classes (excluding "arrow"") and any class that contains those, e.g., the selection of "house" further entails the inclusion of "dog house". This procedure yields 427K samples, of which we select 90% for training, 5% for validation, and 5% for testing. We use the class names as textual descriptions without further processing besides minor spelling correction. Since the black strokes of FIGR-8 mark the background rather than the actual icon, we invert the full dataset before applying our additional pre-processing described in Section A.3.
Emoji. For our preliminary experiments with segmentation-guide patch extraction, we use subset of standard emoji images (emoji dataset, 2022). Specifically, we focus on images that primarily depict faces, selecting 107 for training and 20 for the test.
A.2. Segmenttaion-Guuided Patch Extraction
In this section, provide more information and example results on emoji generation. The model setup is similar to the experiments presented for the MNIST dataset with one fundamental difference: Each predicted closed shape targets one layer of the entire image canvas instead of a tile. This setting enables the prediction of final SVG that resembles real-world use cases where vector data is a set of editable layers, ultimately composited altogether. Our training data is created using the Segment Anything (SAM) model from Meta, which provides a series of masks for the entire image. In our extraction pipeline, each mask produces one layer. We quantize the original image into 4,096 possible colors and create a raster layer for each mask by using the median color in the original image for its respective mask. A qualitative example of the results from the extraction pipeline is shown in Figure 11. The image also depicts three-dimensional visualization of the final extracted layers sorted by their area.
Each layer is center-cropped based on the bounding boxes of the SAM mask. White padding of 10 pixels is added on all sides similarly to what was done for the MNIST. However, in this scenario, padding does not create any artifact and merely becomes an additional scaling factor, since the reconstructed shapes fit the whole image size. During VSQ training, the ground truth cropping bounding boxes are used to scale and shift back the points predicted by the VSQ into the original position. These shifting values and the hierarchy of the layers become the new target of the ART module. We introduced minor additional changes to cope with the increasing complexity of the data, especially the color imbalance due to the small number of samples: The VSQ module outputs RGB colors per shape, but the raster and ground truth layers are converted into the CIE-LAB color space before computing the loss. The color channels (AB) of each layer are weighted inversely to
12

---

Vector Grimoire: Codebook-basee Shape Generation under Raster Image Supervision
Figure 10. Original Image
Figure 10. SAM masks
Figure 10. Original Image
Figure 10. SAM masks
Figure 11. Layer extraction with SAM.
the frequency of the target color in the dataset. No weights are needed for the luminance channel. Figure 12 reports some examples of VSQ reconstructions by layer.
Notably, this reconstruction was achieved after training on only 110 emojis, and the results come from the set. Common shapes (such circles) and colors (such yellow) quickly learned, whereas complicated shapes remain challenging (e.g., shapes of the hair). Overall, this is already a large improvement to other raster-supervised SVG generative models. Im2Vec does not learn the colors. As stated in the original Im2Vec paper and found in the repository, it uses colors hard-coded to reflect the target image (e.g., one yellow and three black shapes when the target is simple emoji).
A.3. Pre- Pre-Processing
This section provides additional information regarding the pre- pre-processing and extraction techniques on the employed datasets.
Shapes. No pre- pre-processing is conducted for the MNIST dataset. Images are simply tiled using a x 6 grid and the central position of each tile the original image saved. For the Emoji dataset, each image is first processed with the SAM model generate masks covering every pixel. We filter masks with very small areas multiple connected components. Finally, convert the remaining masks into layer patches, assigning the foreground the median pixel value from the original image.
Strokes. For the FIGR-8 dataset, the pixels outlining the icons are isolated using a contour finding algorithm (Lorensen & Cline, 1987) and the coordinates are then used to convert them into vector paths. This simple procedure available in our code repository allows us to efficiently apply a standard propprcessiing pipeline defined in Carlier et al. (2020) and already adopted by other studies (Wu et al., 2023; Tang et al., 2024). The process involves normalizing all strokes and breaking them into shorter units if their length exceeds a certain maximum percentage of the image size. Finally, each resulting path
13

---

Vector Grimoire: Codebook-basee based Shape Generation under Raster Image Supervision
Figure 11. layers.
Figure 11. VSQ Reconstructions.
Figure 12. Inputs (left) input layers for the VSQ, (right) reconstructions of the model.
fragment is scaled, translated to the center of new canvas by placing the center of its bounding box onto the center of s, and rasterized to become part of the training data. Since strokes in S are all translated around the image center, the original center position of the bounding box in 7 is recorded for each and saved. These coordinates are discretized in range of 256 x 256 values. This approach is also used for Fonts, but since the data comes in vector format, there is no need for contour finding.
A.4. Post- Pest-Processing
Our approach introduces small discrepancies with the ground truth data during tokenization. The VSQ introduces small inaccuracies in the reconstruction of the stroke, and the discretization of the global center positions may slightly displace said strokes. The latter serve as the training data for the auto-regressive Transformer and therefore represent an upper limit to the final generation quality. Similarly for MNIST, the use of white padding on each patch to facilitate faster convergence results in small background gaps when rendering all shapes together. These small errors compound for the full final image and may become fairly visible in the reconstructions.
5 55
55
5
5555555
5
5
Figure 13. Different SVG post-processing methods visualized. From left to right: raw generation, results of applying PC and PI, results of applying PC and PI by only considering nearest neighbors of consecutive strokes.
While we opted not to modify the global reconstructions of MNIST generation, for FIGR-8 and Fonts, we make use of SVG post-processing similar to prior work (Tang et al., 2024), which introduced Path Clipping (PC) and Path Interpolations (PI). PC, the beginning
stroke is set the position of the end of the previous stroke. PI, a new stroke added that connects them instead. As we operate on visual supervision, the ordering of the start and end point of a stroke is not consistent. Hence, we adapt these two methods to consider the start and end point, but rather consider the nearest neighbors of consecutive strokes. We also add maximum distance parameter to the post- prosesrocessing in order to avoid intentionally disconnected strokes to get connected. See Figure 13, Figure 14 for qualitative depiction of this process and Section A.5 for a quantitative comparison.
A.5. Results with Different Post-Processe
In GRIMOTRE, the resulting full vector graphic generation is characterized by fragmented segments. This is because the output strokes of the VSQ decoder are each locally centered onto a separate canvas, and the auto-regressive Transformer, which is responsible for the absolute position of each shape, returns only the center coordinates of the predicted shape without controlling the state of connection between different strokes. To cope with this, in Section A.4, we introduced several post post-processing algorithms. In this section, we report additional information about the performance of each of
14

---

Vector Grimoire:
Codebook-based Shape Generation under Raster Image Supervision
them for the VSQ module (reconstruction) and the overall GRIMOIRE (generation). Table 6 shows that the PC technique consistently outperforms the alternatives across both datasets in terms of both FID and CLIPSSore.
Model
Fonts
FIGR-8
MSE
FID
CLIP
MSE
FID
CLIP
VSQ
0.0144
4.45
28.61
0.0045
1.29
31.17
VSQ (+PC)
0.0135
0.23
29.24
0.0023
0.10
31.97
VSQ (+PI)
0.0106
0.29
28.96
0.0028
0.07
32.0
GRIMOIRE
n/a
4.44
28.45
n/a
4.20
26.96
GRIMOIRE (+PC)
n/a
1.67
28.64
n/a
3.58
27.45
GRIMOIRE (+PI)
n/a
1.86
28.43
n/a
4.57
26.73
Table Reconstruction capabilities of our VSQ module and generative performance of GRIMOIRE with different post-processing techniques after training on Fonts and FIGR-8.
A.6. Comparison with Vector-supervised Methods
We extended our analysis to two vector-supervised methods - DeepSVG and IconShop training them on the same FIGR-8 data used for GRIMOIRE. Unlike GRIMOIRE, DeepSVG supports conditioning only on class identifiers; therefore, we assigned a unique identifier to each class in FIGR-8.
We also finetuned Llama 3.2 on FIGR-8 with minimal data pre-processing We believe this to be an insightful analysis that shows how tailored tokenization pipelines and extensive data reepprocessing are necessary for other vector-supervise models to perform effectively. We wish to highlight how raster and vector data provide very different supervising signals.
Llama. We fine-tuned Llama (instruction tuning) for three days on eight H100 GPUs. Minimal pre-processing includes rounding up the path coordinates to integer values. Upon inspection, this did not affect the quality of the image. We use the original chat template and included special tokens delimit the SVG code. The performance at inference appears very poor. The model predicts the most recurrent patterns in the dataset, resulting mainly in circular artifacts. The SVG syntax is, however, correct most of the time, allowing rendering.
DeepSVG. We train DeepSVG using the official training script. The model converges within few hours, but the results are also not good, yielding the lowest CLIPScore and FID among all models.
IconShop. We also re-trained the original IconShop model on the subset of FIGR-8 used in Grimoire. In this case, the performance of the model is comparable to Grimoire, resulting in slightly better CLIPscore and FID.
All results are reported in Table 7.
Model
CLIPScore
FID
Conditioning
Supervision
De
22. 10
58.03
Class
Vector
Llama 3.2
25.45
38,93
Prompt
Vector
Grimoire
29.00
0.64
Prompt Prompt
Raster
IconShop
31.18
0.40
Vector
Table Comparison of GRIMOIRE with vector-supervised methods. Llama was trained with almost minimal pre- pre-processing.
A.7. Analysis of Patch and Grid Sizes
In this section, we report results after training the VSQ module on MNIST using varying grid sizes (default: 5) and patch sizes (default: x28x 128). The patch size had a limited effect on performance, while increasing the number of tiles (i.e., smaller patches) improved results, likely due to the simpler topology. In all configurations, the reconstruction error remained below that of Im2..e. Table reports MSE on the test set.
15

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Patch Size
Tiles = 3
Tiles 5
Tiles = 8
32
0.093
0.092
0.078
64
0.092
0.090
0.071
128
0.090
0.094
0.078
Table 8. Effect of patch and grid size on performance.
A.8. Analysis of Stroke Length
To assess the impact of stroke properties on VSQ performance, we conducted two additional experiments:
Stroke length variations: We created patches with smaller or larger strokes. Results show that shorter strokes yield lower reconstruction errors, similarly to the grid size variations.
Multiple stroke predictions per patch: We extended the prediction head the VSQ output two strokes per patch instead of one (as in the paper). The results show that more than segment per shape consistently degrades the reconstruction quality. This suggests that the complexity of strokes in our dataset does not require multiple Bézier curves per patch.
Results for both experiments are reported in Table 9.
Stroke Length
Segments
Stroke Width
MSE
3.0
0.4
0.0049
5.0
0.66
0.011
8.0
1.06
0.023
3.0
2
0.4
0.0052
5.0
2
0.66
0.017
8.0
2
1.06
0.023
Table 9. Impact of stroke length, number of segments per patch, and width.
A.9. Analysis of Codebook Size
To better understand the influence the codebook size VV on the reconstruction quality, we trained the VSQ module on the FIGR8 dataset using all the codebook sizes originally proposed in the Finite Scalar Quantization paper-namee namely 240, 1,000, 4,375 (used in work), 15,360, and 64,000.
The results, presented in Table 10, reveal two key insights. First, increasing the codebook size leads substantial reduction in reconstruction error up to V 4.375, indicating that richer set of quantization centers significantly improves representation capacity in this range. However, beyond this threshold, further en enlarging the codebook yields only marginal gains. This suggests that the added complexity and computational cost associated with very large codebooks may not be justified by the modest improvements in performance.
These findings support choice of [ 4,375 balanced setting, offering strong performance with efficient resource usage.
MSE
240
0.0205
1000
0.0175
4375
0.0145
15360
0.0130
64000
0.0128
Table 10. MSE across varying codebook sizes.
16

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
A.10. Im2 on Other Classes
We conducted a more in-depth analysis of the generative capabilities in 1m2 after training on single subsets of FIGR-8, and compare the results with GRIMOIRE. We trained Vec on the top-10 10 classes of FIGR-8: Camera (8,818 samples), Home (7,837), User (7,480), Book (7,163), Clock (6,823), Flower (6,698), Star (6,681), Calendar (misspelt as caledar in the dataset, 6,230), and Document (6,221). Table 11 compares the FID and CLIPScore with GRIMOIRE. Note that we train our model only once on the full FIGR-8 dataset and validate the generative performance using lext-conditioning on the target class, whereas Im2 is unable to handle training on such diverse data. Despite 1m2 appearing to obtain higher scores on several classes such as User or Document, qualitative inspection reveals how the majority of the generated samples come the form of meaningless filled blobs or rectangles. The traditional metrics employed in this particular generative field, based on the pre-trained CLIP model, react very strongly to such shapes in contrast to more defined stroke images. Qualitative samples are given in Table 19. We further observe a low variance in the generations when 1m2 learns the representations of certain classes, such as star icons.
Model
camera
home
user
book
clock
cloud
flower
calendar
document
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
ImeV (Aille)
9.21
27.86
3,48
26.85
2.12
28.92
7.18
27.26
6.12
26.38
17.43
24.38
6.61
25.42
4.5
27.26
12.19
28.65
Im2Vec
9.05
27.18
9.19
25.95
27.01
8.63
25.84
5.09
25.69
25.58
24.38
23.34
26.22
16.62
26.71
GRIMOIRE
6.74
29.81
7.16
27.16
5.45
26.81
6.65
27.1
7.22
26.32
6.78
24.96
10.27
22.00
5.57
26.23
4.08
27.96
GRIMOTRE (+PC)
5.77
30.22
7.6
27.41
4.38
27.18
5.8
27.24
6.79
26.45
6.05
25.51
9.37
22.46
5.09
26.41
3.81
28.21
GRIMOTRE (+PI)
7.5
29.46
7.44
27.01
5.95
26.85
6.79
27.08
7.63
26.12
7.09
24.73
9.97
22.04
5.87
25.98
4.21
27.89
Table 11. Quality of generations for GRIMOIRE and Im2Vec for the top-10 classes in FIGR-8.
A.11. Qualitative Results of the Geometric Loss
The adoption of our geometric constraint improves the overall reconstruction error, which attribute the network being encouraged to elongate the stroke as much possible. The results in Figure 15 show the effects on the control points of the reconstructed strokes from the VSQ. With the geometric constraint, the incentive to stretch the stroke works against the MSE objective, which results in an overall longer stroke and therefore in greater connectedness in a full reconstruction and an overall lower reconstruction error. We also present an example with an excessively high geometric constraint weight (a = 5) demonstrating that beyond a certain threshold, the positive effect diminishes, resulting in degenerated strokes.
A.12. Implementation Details
We use Adam optimization and train the VSQ module for epoch for Fonts and FIGR-8 and five epochs for MNIST. We use a learning rate of = x 10-5, while the auto-regressive Transformer is trained for ~30 epochs with A = 6 x 10-4, The Transformer has a context length of 512. Before proceeding to the second stage, we filter out icons represented by fewer than ten or more than 512 VSQ tokens, which affects 12.16% of samples. We use p-sampling for our generations with GRIMOIRE. Training the VSQ module on six NVIDIA H100 takes approximately 48, 15, and 12 hours for MNIST, FIGR-8, and Fonts, respectively; the ART module takes considerably fewer resources, requiring around 8 hours depending on the configuration. Regarding Im2, we replace the Ranger scheduler with AdamW (Loshchilov & Hutter, 2017) and enable the weighting factor for the Kullback-Leibler (KL) divergence in the loss function to 0.1, as it was disabled by default in the code repository. preventing any sampling. We train Lm2Vec with six paths for 105 epochs with a learning rate of 10-4 with early stopping if the validation loss does not decrease after seven epochs. Regarding the generative metrics, we utilized CLIP with a ViT-16 backend for FID and CLIPScore.
A.13. Generative Scores with Completion
To evaluate if GRIMOIRE generalizes and learns to meaningfully complete previously unseen objects, we compare the CLIPScore and FID of completions with varying lengths of context. The context and text prompts are extracted from 1,000 samples of the test set of the FIGR-8 dataset. The results are shown in Table 12.
While GRIMOTRE can meaningfully complete unseen objects, the quality of these completions is generally lower than the generations under text-only conditioning. This is expected, as prompts in the test set are also encountered during training (the class names). The CLIPScore generally drops to its lowest point with the least amount of context and then recovers when more context is given to the model, which coincides with our qualitative observations that with only a few context strokes, GRIMOIRE occasionally ignores them completely or completes them in an illogical way, reducing the visual appearance.
17

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
Model
Fonts
FIGR-8
FID
CLIP
FID
CLIP
GRIMOIRE (w/o context)
1.67
28.64
3,58
27.45
GRIMOIRE (+ stroke context)
2.78
27.25
4.65
25.31
GRIMOIRE (+ stroke context)
3.16
27.25
5,46
25.54
GRIMOIRE (+ 12 stroke context)
2.95
27.57
6.04
25.85
GRIMOIRE (+ 24 stroke context)
2.25
28.12
6.05
26.39
Table 12. Generation quality of GRIMOIRE with different lengths of provided context on Fonts and FIGR-8. Post-processing is conducted for all setups. GRIMOIRE textual input for all generations.
A.14. Domain Transfer Capabilities for Reconstruction
To validate how the strokes learned during the first training stage adapt to different domains, our VSQ module to reconstruct Fonts after training on FIGR-8, 8, and vice versa. Figure 16 provides a qualitative example for each setting. Despite the loss value for each image being around one order of magnitude higher than the in-domain test-set (MSE* 0.05), the VSQ module uses reasonable codes to reconstruct the shapes and picks curves in the correct directions. Straight lines end up being the easiest to decode in both cases.
A.15. Codebook Usage for Strokes
As described in Section 3.1, for FSQ, we fixed the number of dimensions of the hypercube to and set the individual number
of values for each dimension as [7, 5, 5, 5, 5] for total codebook size of B 4.375. In this section, we wish to share some interesting findings about the learnt codebook. For this, shall use the VSQ trained on FIGR-8 with Reode 1, maximum stroke length of 3.0, and the geometric constraint with 0.2.
Usog
After training the VSQ FIGR-8, we tokenize the full dataset. The resulting VQ tokens stem from 60.09% of the codebook, while 39.91% % of the available codes remained unused. The ten most used strokes make up 41.24% of the dataset, while the top 24 and 102 strokes make up roughly 50% and 75%, respectively. These findings indicate that for these particular VSQ settings, one could experiment with smaller codebook sizes.
To balance the stroke distribution, one could use a different subset of FIGR-8. -. Currently, the classes "menu", "credit card", "laptop", and "monitor" contribute the most to the stroke imbalance, with 26%, 24.3%, 24.05%, and 23.8% of their respective strokes being the most frequent horizontal one in Table 13.
>
18.76%
12.26%
2.56%
1.73%
1.16%
1.12%
0.99%
0.94%
0.92%
0.80%
Table 13. Top ten most used strokes of the VSQ module trained on icons and their relative occurrences in our subset of FIGR-8.
A.16. Average Strokes in Codebook
In Section A.15, show the ten most used strokes of our trained VSQ, but after inspecting the full codebook, notice how neighboring codes often express very similar strokes. Therefore, to visualize the codebook more effectively, we plot mean and minimum reductions of the full codebook in Figure 17. Additionally, we tokenize the full FIGR-8 dataset and plot the same reductions in Figure 18 to show the composition of the dataset.
A.17. Qualitative Results Reconstruction
In Table 14 and Table 15, we provide several qualitative examples of vector reconstructions using Im2Vec and our VSQ module on the Fonts and FIGR-8 - datasets, respectively. We fill the shapes the images when using Im2Vec, since the model creates SVGs as series of filled circles and would not able to learn from strokes with small width. Im2 does not converge when trained on the full datasets, whereas returns some approximate reconstruction of the input when only a
18

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
single class is adopted. In contrast, the VSQ module generalizes over the full dataset.
A.18. Qualitative Results Generation
In this section, we provide qualitative examples of our reconstruction and generative pipeline, and compared those with Im2Vec. Table 16 reports few examples of icons generated with GRIMOIRE using only lext-conditioning on classes. In Table 17 report some generations for MNIST. In Table 18, report generative results for Fonts. Thanks to the conditioning, we can generate upper-case and lower-case glyphs in bold, italic, light styles, and more. As can be seen in the table, GRIMOIRE also learns to properly mix those styles only based on text. Finally, in Table 19, we report some generative results on icons and Fonts for Vm2 on a single class dataset. The results show how the pipeline typically fails to produce meaningful or sufficiently diverse samples.
A.19. Glossary of notation
Table 20 reports a description of all the important notations used in this work.
19

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
capital regular font
Capital regular font
capital in bolditalic font
4 in normal font
regular font
italic font
C
1142
Unfixed
Pred.
C
LR42
I
Fixing
P1
CI
O I
LR42
Fixing
PC
Figure 14. Some examples of text-conditioned glyph generation from GRIMOIRE. The first row shows the unfixed model predictions, the
second and third rows depict the final outputs with two different post-processing techniques.
c
e
Figure 14. Ground Truth
Figure 14.
Figure 14,
0.1
Figure 14. 5
Figure 15. Samples from the test set when training the VSQ module with and without our geometric constraint. Each stroke consists of two cubic Bézier segments. Embedded within each stroke, the red dots mark start and end points, while the green and blue dot pairs are the control points of each segment.
Figure 15. Icons Fonts.
Figure 15. Fonts icons.
Figure 16. Qualitative zero-shot reconstructions from the test-set of FIGR-8 and Fonts after training the VSQ module solely on the respective other dataset.
20

---

Vector Grimoire: Codebook-basee Shape Generation under Raster Image Supervision
Figure 16. codebook mean strokes
Figure 16. all codebook strokes
Figure 17. Different reductions of all 4,375 strokes from the VSQ codebook. The model seems to have learned expressive codebook decoder mapping as the figure on the left shows smooth and evenly distributed stroke profile and the figure on the right displays strokes in almost every direction.
Figure 17. FIGR-S 8 mean strokes exclud- ing top ten strokes
Figure 17, FIGR-8 mean strokes
Figure 17. all FIGR-8 strokes
Figure 18. Different reductions of all strokes from the tokenized FIGR-8 - dataset. The visualization on left shows the dominance of the two most occurring strokes, middle shows distribution strokes skewed. missing 39.9 1% strokes are also visible the right figure, where certain diagonal strokes that are available in the codebook never used.
21

---

Vector Grimoirer Codebook Codebook-basey based Shape Generation under Raster Image Supervision
Im2Vec
A
AAAAC
GT
A
USS
)
G1
( O |
Im2Vec
Z
1
GT
O 1
Z
OSA
Table 14, Esaapile of various coonstructionn of our VSQ module after training Fonts compared to reconstructions of Im2Vec reiined on the letter "A" (first row) and 1m2 trained on the full Fonts dataset (third row).
22

---

Vector Grimoire: Codebook-basee Shape Generation under Raster Image Supervision
GT
Im2Vec
GT
VSQ
GT
Im2Vee
GT
OSA
Table 15, Examples of various reconstructions of our VSQ module after training icons compared to reconstructions of Im2Vec trained on one class (first row) and Im2 trained on the full dataset (third row).
23

---

Vector Grimoire: Codebook-basee Shape Generation under Raster Image Supervision
L
Clock
Luggage
Shopping Bag
Camera
Like
Home
Mail
Share
Target
Arrow
Microphone
Gift
Clock
Eye
Cube
Sun
Bell
Bell
Smile
Clip
Smile
Share
Bin
Book
Home
Table 16. Examples of various samples generated with GRIMOIRE after training on icons, using only text conditioning.
24

---

Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision
002344 4
S 7778
Table 17. Examples of samples generated with GRIMOIRE for each digit of the MNIST dataset.
25

---

Vector Grimoire: bodebook-based Shape Generation under Raster Image Supervision
Star
User
Document
Camera
Book
Table 18, Examples of filled samples generated with 1m2Vec after training the model on specific classes the dataset, For most classes, Im2Vec could not capture the diversity of the data and failed to meaningfully converge.
26

---

Vector Grimoire: Codebook-baser Shape Generation under Raster Image Supervision
ATITIN
Capital
YRGIXE
Capital
Y r r
n
9
d
Minuseule
S
h
h
0
2
Minuseule
Regular
Regular
Italic
Bold
Bold-italic
Light
Table 19. Examples of various samples generated with GRIMOIRE after training on Fonts, using only text conditioning.
27

---

Vector Grimoire: Codebook-baser Shape Generation under Raster Image Supervision
Table 20. Glossary of all notations.
Network encoder
D Network decoder
T Image from the dataset
V Codebook
Codes from the codebook
Set of values per dimension of our codebook
7 Single dimensional value
q Number of dimensions of the codebook
Series of patches
Single patch
C Color channels
Number of patches
+ Set of discrete coordinates
0 Single coordinate pair
Latent space
d Dimension of latent
Latent embedding
$ Predicted patch
Number of segments
P Set of points Point pair
P Euclidian distance
Neural network
E
Number of codes
T
Text description
T Tokenized description
Text tokens
t Number of text tokens
28


Reviews:
[{"review_id": "sWDNdkcARb", "reviewer": "Reviewer_ojwR", "summary": "This paper presents GRIMOIRE, a novel text-guided generative model for scalable vector graphics (SVG). The model consists of two main components: a Visual Shape Quantizer (VSQ), which learns to reconstruct raster images as vector shapes through a discrete codebook; and an Auto-Regressive Transformer (ART), which models the joint distribution over shape tokens, positions, and textual descriptions to generate SVGs from natural language prompts. Unlike prior approaches requiring direct supervision from SVG data, GRIMOIRE is trained only with raster image supervision, enabling it to scale to larger datasets. The authors evaluate their method on tasks such as closed-shape reconstruction (MNIST, Emoji) and stroke-based generation (icons, fonts), demonstrating improved flexibility over SVG-supervised methods and competitive generative quality against image-supervised baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript and providing insightful feedback.\n\n**We conducted ablation experiments to address your concerns.**\nWe have explored **different patch and grid sizes** on the MNIST dataset, analyzed the **impact of stroke length and width** on FIGR8, and investigated the **effects of varying codebook sizes**. Additionally, we have included **comparisons with three vector-supervised methods**.\n\n---\n## Ablation: Patch and Grid Sizes (MNIST)\nWe trained the VSQ module on MNIST with different grid sizes (paper value: 5) and patch sizes (paper value: 128x128).\n\nKey findings:\n- **The patch size variations had minimal impact on model performance.**\n- **Grid size variations led to improvements for larger number of patches per image**, likely due to the simpler topology of smaller patches.  \n- **In all cases, the reconstruction error remains lower than Im2Vec.**  \n\nThe values in the table below reports the MSE on the test-set. \n\n| Patch Size | Tiles = 3 | Tiles = 5 | Tiles = 8 |\n|------------|-----------|-----------|-----------|\n| 32         | 0.093    | 0.092    | 0.078    |\n| 64         | 0.092    | 0.09      | **0.071** |\n| 128        | 0.09      | 0.094     | 0.078     |  \n---\n## Ablation: Stroke Length (FIGR8)\nTo assess the impact of stroke properties on VSQ performance, we conducted two ablations:\n\n1. **Stroke length variations:** We created patches with smaller or larger strokes. Results show that **shorter strokes yield lower reconstruction errors**, similarly to the grid size variations. \n2. **Multiple stroke predictions per patch:** We extended the prediction head of the VSQ to output two strokes per patch instead of one (as in the paper). Results show that more than one segment per shape consistently degrades the reconstruction quality. This suggests that the complexity of strokes in our dataset does not require multiple Bézier curves per patch. \n| Stroke Length | Segments | Stroke Width | MSE    |\n|----------------|----------|--------------|--------|\n| 3.0            | 1        | 0.4          | 0.0049 |\n| 5.0            | 1        | 0.66         | **0.011** |\n| 8.0            | 1        | 1.06         | 0.023  |\n| 3.0            | 2        | 0.4          | 0.0052 |\n| 5.0            | 2        | 0.66         | **0.017** |\n| 8.0            | 2        | 1.06         | 0.023  |\n---\n## Ablation: Codebook Size\nTo understand the impact of codebook size $|V|$, we trained the VSQ on FIGR8 using all the sizes proposed in the original Finite Scalar Quantization paper (240, 1000, 4375 [our paper], 15360, and 64000). \n\nThe results reported below highlight two key observations:\n- **The reconstruction error decreases significantly up to $|V| = 4375$ but shows only marginal improvements beyond this point.**\n- **Using excessively large codebooks does not justify the increased computational cost.** \n\n| V  | MSE |\n|-------|---------------------|\n| 240   | 0.0205             |\n| 1000  | 0.0175             |\n| 4375  | **0.0145**     |\n| 15360 | 0.0130             |\n| 64000 | 0.0128             |  \n---\n## Comparison with Vector-Supervised Models\n**We have added a comparison with three vector-supervised models**, including a publicly available LLM: **LLama 3.2, DeepSVG, and IconShop**. We trained the models on the SVG version of FIGR8. A detailed analysis is in our response to reviewer `sRe3`.  \n\n---\n## Missing References\nWe appreciate your feedback regarding missing references.\n**We have addressed this in our response to reviewer `f1Fu` and outlined how we will incorporate these references in the final manuscript.**  \n\n---\n## Other Questions\n> It would be helpful if the authors explicitly stated the goals of each experiment in the supplementary section.\n\nWe agree. **We will add a brief introductory sentence before each subsection of the Appendix.**\n\n> Broken image reference at line 322.\n\nThank you for catching this. We will remove the reference, as the corresponding figure is no longer part of the main manuscript.\n\n> Why use a raster-domain image encoder instead of a vector-domain encoder?\n\nA core novelty of Grimoire is that **the entire framework operates in the raster domain**.\nUsing a vector-domain encoder would contradict this fundamental approach, making the pipeline **no longer vector-free**.\n\n> How does the VSQ module differ from the VQ-based architecture in Im2Vec?\n\nIm2Vec utilizes an **RNN-based architecture**, which does not belong to the family of **VQ models using discrete embedding codebooks**.  \n\nAnother key methodological difference is that Im2Vec attempts **end-to-end SVG generation**, encoding an entire image and predicting the individual components with an RNN. In contrast, our VSQ first learns vector representations from image patches, then the ART model learns to arrange these patches in the correct sequence.\n\nThis modular approach significantly **enhances the performance of raster-supervised generative models** compared to Im2Vec.", "claims_and_evidence": "The proposed architecture builds incrementally upon established techniques in vector image representation and generation. It integrates standard components in a coherent manner. While the claims are plausible and generally supported by qualitative and quantitative evidence, the overall novelty is modest. The improvements are incremental, and while the results are reasonable, they do not clearly establish substantial advancement over existing methods.", "methods_and_evaluation_criteria": "The benchmark datasets used in this work are appropriate, as they align well with those used in prior literature, allowing for meaningful comparisons. However, the methodology would benefit from clearer descriptions of which components are used in each experiment. Since the paper combines several previously established techniques, it is important to explicitly state which variants or components are evaluated in each experiment.", "theoretical_claims": "There is no theoretical claim in this work.", "experimental_designs_or_analyses": "## Vector Quantization for SVG Representation (Tables 1 and 2):\n An ablation study would greatly strengthen the experimental section. For example, variations in encoder-decoder configurations (e.g., patch size, grid size, codebook size, or SVG command set) could demonstrate the robustness and contribution of the individual components. The current comparisons, such as with Im2Vec, leave some ambiguity about whether the improvements stem from the proposed method or the dataset/model choices.\n\n\n## SVG Generation (Table 3):\n The evaluation should include a comparison with IconShop to give a more comprehensive view of generative quality relative to recent state-of-the-art models.", "relation_to_broader_scientific_literature": "This work contributes to the growing field of vector image representation and generation by bridging the raster and vector domains. The approach is particularly promising for multimodal applications, including vision-language models and code generation involving SVG. Given the relevance to downstream applications and the increasing interest in multimodal generation, this research has strong potential impact.", "essential_references_not_discussed": "Several key works are missing that are essential for contextualizing this paper’s contribution:\nGoogle-Fonts (ICCV 2019): A foundational model for deep SVG generation combining image auto-encoders with SVG decoders.\n\n\nDeepSVG (NeurIPS 2020): Introduced transformer-based autoencoders for vector graphics; its SVG tokenization remains widely used.\n\n\nLIVE (CVPR 2022): Demonstrates SVG translation from raster images without score distillation loss.\n\n\nDeepVecFont / DeepVecFont v2 (SIGGRAPH Asia 2021, CVPR 2023): Addressed SVG command modeling and differentiable rasterization.\nThese works are highly relevant to both the architecture and training methodology proposed in GRIMOIRE and should be discussed in the paper. Notably, vector image representation has a history that precedes the large language model (LLM) era, contrary to the implication in the related work section.\n\nNote that representation for SVG is not starting from large language model era as mentioned in related works", "other_strengths_and_weaknesses": "## Strengths:\nProposes a new framework for SVG generation that avoids the need for direct SVG supervision, opening the door to larger training datasets.\nAddresses a relatively underexplored area in generative modeling, with practical applications in design, UI generation, and code synthesis.\n\n\n## Weaknesses:\nLacks sufficient citation and discussion of related foundational works.\nAblation studies are missing, making it hard to isolate the contributions of each component.\nExperimental gains over prior work are modest.", "other_comments_or_suggestions": "## LLM Baselines:\n s a non-essential suggestion (outside rebuttal), it would be informative to include SVG generation results from public large language models (e.g., OpenAI’s GPT). Although their SVG generation quality is currently limited, showing this comparison would highlight the advantage of GRIMOIRE.\n\n\n## Typos:\n There is a broken image reference at line 322:\n “Qualitative results in ?? confirm this behaviour on the MNIST dataset.”", "questions_for_authors": "Why use a raster-domain image encoder (e.g., ResNet-18) instead of a vector-domain encoder such as that used in DeepSVG? This choice impacts the learned latent representation, and a comparison would help clarify its effects.\n\n\nHow does the VSQ module differ from the VQ-based architecture in Im2Vec? The performance seems similar, so it's unclear how much the modification contributes to overall results.\n\n\nCan the authors include a comparison with IconShop for SVG generation? Given that IconShop is a recent and strong baseline, its inclusion would help contextualize the performance of GRIMOIRE.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Us8oAQU97c", "reviewer": "Reviewer_f1Fu", "summary": "The authors propose a SVG generative model GRIMOIRE which can be conditioned on a text prompt or a partially completed SVG. The primary innovation in the paper is training a VQ-VAE which tokenizes patches of rasterized SVGs into discrete tokens which crucially can be reconstructed into SVG primitives (primarily bezier curves). This tokenizer can be trained end-to-end on purely raster images by a clever use of a differentiable renderer (DiffVG) in the VQ-VAE decoder which enables backpropagting through the L2 pixel space reconstruction loss into and through the now differentiable continuous SVG parameters. Once such a tokenizer has been trained the authors use it to tokenize MNIST, Fonts and FIGR-8 dataset and train a standard autoregressive transformer using a next token prediction loss to create an auto-regressive generative model for SVGs. The primary baseline for comparison is Im2Vec.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and thoughtful feedback on our manuscript. \n\nGiven the time constraints of this rebuttal, we have focused on addressing the major concerns as follows.  \n\n---\n## **Code Length of the SVG**  \n\nFor Im2Vec, the number of paths and control points per path is fixed at eight and ten, respectively.  \n**Grimoire dynamically adjusts the number of strokes based on the target complexity.** \n\nAfter inspecting over 15,000 generated SVGs across all FIGR-8 classes, we found that **the average number of paths is 95**.  \n\n**We have uploaded samples and reconstructions for the “user” class for both models to our anonymous repository**. For the reconstructions we included the original ground truth for reference.  \n\nFor single-target reconstructions, Im2Vec tends to overlap strokes around the outline, whereas for more complex targets, the strokes collapse. We encourage the reviewer to directly inspect the SVG code length on the anonymous GitHub Repository: [Link](https://github.com/anon-papercode/15874/tree/main/Showcase).  \n\nWe found this analysis insightful and plan to incorporate it into the results section in the final version of the manuscript.  \n\n---\n## **Additional Baselines**  \n\n**Thank you for suggesting finetuning a standard LLM with minimal preprocessing.** This was an insightful suggestion, especially given that the tokenization pipelines of other vector-supervised models (e.g., IconShop) are a considerable limitation.  \n\n**We have fine-tuned LLaMA 3.2 on the same FIGR-8 subset used for Grimoire with minor preprocessing** and also added comparisons with other vector-supervised models using their respective tokenization pipelines (DeepSVG, IconShop). \n\nThis analysis has been included in our response to reviewer `sRe3`.  \n\n---\n## **Missing References**  \n\nWe appreciate all reviewers for highlighting important missing references, such as LIVE. \nWe plan to expand the related work section to incorporate all suggested papers. Specifically, we will:  \n\n1. **Before L87:** Introduce vector-supervised methods that predate the LLM era, citing **DeepSVG, Google-Fonts, and DeepVecFont**.  \n2. **Immediately after:** Among the LLM-based approaches, explicitly mention **StarVector** and **Chat2SVG**.  \n3. **At the end of the section:** Where we discuss SDS-based methods, include **SVGDreamer** and dedicate a small paragraph to neural implicit representations, citing **NiVel, Text-to-Vector Generation with Neural Path Representation, and NeuralSVG**.  \n\nWhile these works differ significantly in methodology, they address similar problems to Grimoire and will be appropriately cited. \n\nWe hope this revision sufficiently addresses concerns regarding missing references.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "- The benchmark datasets seem reasonable.\n- However in my opinion there are a number of missing baselines. In particular for the reconstruction experiments in Tables 1 and 2 that validate the VQ-VAE I would suggest also including LIVE (https://arxiv.org/abs/2206.04655) a seminal paper in the area of SVG reconstruction of raster images. StarVector: https://arxiv.org/abs/2312.11556 would also be an appropriate baseline for the generation experiments.\n- Additionally, as a simple baseline that would be very insightful, fine-tuning a standard LLM e.g. Llama on the Fonts and FIGR-8 datasets, which IIUC have the underlying SVG code available, using minimal preprocessing of the SVG code and using the standard Llama tokenizer directly on the SVG code. This will validate or invalidate the need for such a complex tokenization method and the claims in the paper about the need for more rasterized training data and complex pre-processing required in order to train a SVG generation model directly on the SVG code.\n- Beside all quantitative results I would have also liked to have seen the average SVG codelength of all methods (GRIMOIRE and Im2Vec) for each dataset. When available, the average SVG codelength in the ground truth dataset would also be helpful. It is currently very difficult to understand how fair the comparison between the methods is, without some way to know how much compression each method is doing. I have a suspicion given the number of bezier curves used to represent each, relatively simple, image patch that GRIMOIRE may be relatively uncompressed and verbose.\n- No measure of significance or error bars are provided on any of the quantitative results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Checked.", "relation_to_broader_scientific_literature": "A very nicely written related work section is provided.", "essential_references_not_discussed": "LIVE is definitely an important missing reference: https://arxiv.org/abs/2206.04655. There are also a series of follow-up papers building on top of LIVE.\n\nLikewise StarVector: https://arxiv.org/abs/2312.11556", "other_strengths_and_weaknesses": "Strengths:\n\n- I appreciate that the authors will make their code open source.\n- The use of DiffVG to enable learning a discrete codebook for SVG primitives, and the fact that this enables training the tokenizer on purely rasterized images is very clever and a very nice contribution of the paper.\n\nWeaknesses:\n- The only structural primitive available to the VQ-VAE decoder are Bezier curves. SVG itself supports other primitives e.g. <circle>, <line>, etc. which may be more appropriate, interpretable and compressed representations of certain image patches. But having to make a discrete choice between different primitives would make the decoder non-differentiable meaning that other than for stylization (stroke width, color) all structure in the SVG must be represented as relatively uncompressed, uninterpretable paths.\n- The need for specialized path extraction methods on a per dataset basis as shown in Figure 3 is a major weakness of the method. Surely a more general patch extraction method would have been possible to employ without a significant degradation in quality? It would have also enabled training a single joint codebook and generative model for all datasets enabling potential transfer learning and greatly enhancing the generality of the method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98kdfsNqYR", "reviewer": "Reviewer_dhph", "summary": "This paper introduces a text-guided SVG generation model, i.e., GRIMOIRE, using only raster image supervision. The SVG generation task is formulated as the prediction of a series of individual shapes and positions. The experiments demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\n\nI appreciate the authors' clarifications. Most of my concerns have been addressed by the rebuttal. I would lean to keep my score by involving the additional evaluations and discussions in the revised version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for taking the time to review our manuscript and providing valuable feedback.\n\n\n---\n## **Discussion on SDS-based Methods**  \n\nIn the final version of the manuscript, we plan to more clearly highlight the differences between SDS approaches and Grimoire at the end of section 2.1 as follows:\n\n> “These methods do not involve training for the vector generation process and instead rely on models trained on raster images for other tasks, making them difficult to extend to new data.”\n\nRegarding this point, we have also:\n1. incorporated two more recent SDS-based methods, **SVGDreamer** and **Chat2SVG**, and will include additional qualitative results in the final version of the manuscript. \n2. Added a quantitative comparison of these methods. This analysis has been included in our response to Reviewer `sRe3`.\n\n\nWe used **PSNR** to evaluate some generated samples from all the SDS-based models, highlighting how these methods fall short when compared to a specific dataset. We chose not to use **FID**, as it requires a large number of samples to be statistically significant, which was not feasible within the constraints of SDS methods: slow generations e.g. SVGDreamer, expensive inference e.g. Chat2SVG based on Claude APIs.\n\n---\n## **Other Questions**  \n\n> Are there any quantitative evaluations for text-conditioned SVG generation?\n\nWe have used CLIPScore and FID for text-conditioned generation across the paper. As mentioned above, we have now also added PSNR for the comparison with SDS-based methods.\n\n> The first paragraph states two aspects of the result, but there are four subsections here.\n\nThank you for your suggestion on improving the writing in Chapter 5. We plan to revise the beginning of the chapter to more clearly outline its subsections.", "claims_and_evidence": "The discussion of the existing SDS-based methods is not very clear. Sec. 2.1 only describes several related works without discussing the difference between the existing works and the proposed methods. Sec. 5.4 only shows visual comparisons with the existing works. It would be better to involve the analysis in the paper, rather than supp.", "methods_and_evaluation_criteria": "The proposed method is a reasonable and effective solution to SVG generation.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The current evaluation is not thorough. Only Im2vec is used for comparison. To make the result more convincing, additional existing SVG generation works should be compared and discussed.", "relation_to_broader_scientific_literature": "The proposed method enhances the generation quality over existing raster-supervised SVG models and enables flexible text-conditioned SVG generation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The overall pipeline is a reasonable solution to perform SVG generation. The experiments demonstrate the effectiveness of the proposed method.", "other_comments_or_suggestions": "It would be better to reorganize the structure of section 5. The first paragraph states two aspects of the result, but there are four subsections here.", "questions_for_authors": "Are there any quantitative evaluations for text-conditioned SVG generation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSt6YAIW99", "reviewer": "Reviewer_sRe3", "summary": "This work focuses on SVG generation. Its model consists of two modules: a visual shape quantizer learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an auto-regressive Transformer model jointly learns the distribution over shape tokens, positions and textual descriptions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's time and thoughtful feedback on our manuscript. \nGiven the time constraints of this rebuttal, we have focused on addressing the major concerns as follows.\n\n\n---\n## **Vector-based baselines**\n\nWe have extended our analysis to two vector-supervised methods – **DeepSVG and IconShop** – training them on the same FIGR-8 data used for Grimoire. Unlike Grimoire, DeepSVG supports conditioning only on class identifiers, therefore we assigned a unique identifier to each class in FIGR-8.\n\nUpon suggestion of reviewer `f1Fu`, **we have also finetuned Llama 3.2** on FIGR-8 with minimal data preprocessing. We believe this to be an insightful analysis that **shows how tailored tokenization pipelines and extensive data preprocessing are necessary for other vector-supervised models to perform effectively.**\n\nDespite raster and vector data providing very different supervising signals, we believe that this analysis ultimately helps better position our method.\n\n**Llama**. We fine-tuned Llama (instruction tuning) for three days on eight H100 GPUs. Minimal preprocessing includes rounding up the path coordinates to integer values. Upon inspection, this did not affect the quality of the image. We use the original chat template and included special tokens to delimit the SVG code. **The performance at inference appears very poor**. The model predicts the most recurrent patterns in the dataset, resulting mainly in circular artifacts. **The SVG syntax is, however, correct** most of the time, allowing rendering.\n\n**DeepSVG**. We train DeepSVG using the official training script. The model converges within a few hours, but **the results are also not good**, yielding the lowest CLIPScore and FID among all models.\n\n**IconShop**. We also re-trained the original IconShop model on the subset of FIGR-8 used in Grimoire. In this case, **the performance of the model is comparable to Grimoire**, resulting in slightly better CLIPscore and FID.\n\nAll results are reported in the table below.\n\n| Model     | CLIPScore | FID   | Conditioning | Supervision |\n|-----------|-----------|-------|--------------|-------------|\n| DeepSVG   | 22.10     | 58.03 | Class        | Vector      |\n| Llama 3.2 | 25.45     | 38.93 | Prompt       | Vector      |\n| Grimoire  | 29.00     | 0.64  | Prompt       | Raster      |  \n| IconShop  | 31.18     | 0.40  | Prompt       | Vector      |\n\n---\n## **Additional SDS-based methods**\n\n\nWe have added two more recent SDS-based methods (**SVGDreamer and Chat2SVG**) to our qualitative analysis for the final version of our manuscript. We have also included a **new quantitative analysis of all these models.**\n\nThe point of the results in section 4.5 is not comparing general generative capabilities but highlighting that besides the aesthetically pleasing results, **this family of models falls short in representing a specific target domain** and provides no way to be extended to new data. \n\nMaking this analysis quantitative is not straightforward. FID score between image distribution  is reliable on thousands of samples, but the computational cost of SDS-based models requires up to hours for a few samples (e.g. SVGDreamer) or uses costly proprietary models (e.g. Chat2SVG). \n\nWe have hence used the PSNR of 20 generated samples from all models. \n**The results highlight how all models fall short on our dataset distribution.**\n\n| Class         | Model         | Average PSNR (dB) |\n|---------------|---------------|-------------------|\n| **User**      | CLIPdraw      | 28.68             |\n|               | **Grimoire**  | **45.19**         |\n|               | VectorFusion  | 36.62             |\n|               | Chat2SVG      | 37.62             |\n|               | SVGDreamer    | 34.53             |\n| **Heart**     | CLIPdraw      | 28.54             |\n|               | **Grimoire**  | **45.66**         |\n|               | VectorFusion  | 38.54             |\n|               | Chat2SVG      | 37.88             |\n|               | SVGDreamer    | 34.44             |\n\n---\n## **Other questions and observations**\n> Does the extension to support stroke width and color prediction mentioned in Sec 5.3 require re-training of the proposed two modules? \n\nIt does require retraining the VSQ. This is also the case for vector-supervised methods. \n\nHowever, a significant difference is that our VSQ prediction heads are very easy to develop or enable and require no further changes, whereas vector-supervised methods like IconShop require redesigning their tokenizer and reprocessing all the data.\n\nFinally, if the VSQ must be retrained with some additional modules (e.g., stroke color and width), the rest of the network can be reloaded, speeding up convergence.\n\n> \"the contribution simply lies on avoiding using ground-truth paths.\"\n\nThe main novelty of our work is providing a text-to-SVG framework which learns from data where the vector paths **do not exist**. This is different from simply avoiding existing vector datasets.", "claims_and_evidence": "There is no comparison with IconShop on FIGR-8. This work does not use ground-truth paths as the supervision while Iconshop uses ground-truth paths. If there is no such direct comparison, it is impossible to identify the contribution of the proposed work: a) it achieves better performance than the one uses ground-truth path. If this is the case, the contribution is very significant, i.e., changing the traditional paradigm of SVG generation; or b) it does not achieve better performance than the one uses ground-truth path. If that is the case, the contribution simply lies on avoiding using ground-truth paths.", "methods_and_evaluation_criteria": "There is no qualitative comparison between baselines and the proposed method. The quantitative comparison makes sense.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "In the comparison to the SDS-based methods, the chosen baselines are not the most advanced ones [1][2][3] and the results of SDS-based methods look much worse than what are reported in the original paper. I suggest the authors make more explanation for it.\n\n[1] https://arxiv.org/pdf/2312.16476\n\n[2] https://arxiv.org/pdf/2405.10317\n\n[3] https://arxiv.org/pdf/2411.16602 (optional as it is too close to the submission deadline)", "relation_to_broader_scientific_literature": "It introduces a new paradigm of training SVG generation model which does not rely on SVG data (which requires lots of data for training) and is not SDS-based (which can be very slow). The idea of splitting the image to patches in VSQ stage is smart.", "essential_references_not_discussed": "The advanced SDS-based methods are not discussed [1][2][3] \n\n[1] https://arxiv.org/pdf/2312.16476\n\n[2] https://arxiv.org/pdf/2405.10317\n\n[3] https://arxiv.org/pdf/2411.16602 (optional as it is too close to the submission deadline)", "other_strengths_and_weaknesses": "no additional comments.", "other_comments_or_suggestions": "no additional comments", "questions_for_authors": "Does the extension to support stroke width and color prediction mentioned in Sec 5.3 requires re-training of the proposed two modules?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 2 “Weak reject” and 2 ”Weak accept.” It is a borderline paper that needs further discussion with the SAC to make the final decision. Currently, the AC is slightly leaning towards rejecting this paper based on the analyses of its strengths and weaknesses listed below.

Strengths:
1. A new text-guided SVG generation model was proposed using only raster image supervision.
2. The idea of splitting the image to patches in VSQ stage is interesting.

Weaknesses:
1. The technical contribution and the performance improvements over existing methods are incremental.
2. Many important related works (e.g., Google-Fonts, DeepSVG, LIVE, DeepVecFont, etc.) are missing, which should be cited, discussed and compared. 
3. Insufficent experiments have been conducted. 
4. Although the proposed method can be trained using only raster image supervision, it cannot outperform some approaches trained on ground-truth paths. Therefore, it fails to handle the practical real-world scenario where the model is trained on vector graphs and outputs high-quality SVGs.
Decision: Accept (poster)