Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision

## Abstract

Abstract Scalable Vector Graphics (SVG) is a popular format on the web and in the design industry. However, despite the great strides made in generative modeling, SVG has remained underexplored due to the discrete and complex nature of such data.
We introduce Grimoire , a text-guided SVG generative model that is comprised of two modules:
A Visual Shape Quantizer (VSQ) learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an Auto-Regressive Transformer (ART) models the joint probability distribution over shape tokens, positions, and textual descriptions, allowing us to generate vector graphics from natural language.
Unlike existing models that require direct supervision from SVG data, Grimoire learns shape image patches using only raster image supervision which opens up vector generative modeling to significantly more data.
We demonstrate the effectiveness of our method by fitting Grimoire for closed filled shapes on MNIST and for outline strokes on icon and font data, surpassing previous image-supervised methods in generative quality and the vector-supervised approach in flexibility.

## 1 Introduction

In the domain of computer graphics, Scalable Vector Graphics (SVG) has emerged as a versatile format, enabling the representation of 2D graphics with precision and scalability. SVG is an XML-based vector graphics format that describes a series of parametrized shape primitives rather than a limited-resolution raster of pixel values.
While modern generative models have made significant advancements in producing high-quality raster images (Ho et al., ; Isola et al., ; Saharia et al., ; Nichol et al., ), SVG generation remains a less explored task. Existing works that have aimed to train a deep neural network for this goal primarily adopted language models to address the problem (Wu et al., ; Tang et al., ).
In general, existing approaches share two key limitations: they necessitate SVG data for direct supervision which inherently limits the available data and increases the burden of data pre-processing, and they are not easily extendable when it comes to visual attributes such as color or stroke properties. The extensive pre-processing is required due to the diverse nature of an SVG file that can express shapes as a series of different basic primitives such as circles, lines, and squares – each having different properties – that can overlap and occlude each other.

An ideal generative model for SVG should however benefit from *visual guidance for supervision*, which is not possible when merely training to reproduce tokenized SVG primitives, as there is no differentiable mapping to the generated raster imagery.

In this paper, we present Grimoire (Shape Generation with raster image supervision), a novel pipeline explicitly designed to generate SVG files with only raster image supervision. Our approach incorporates a differentiable rasterizer, DiffVG (Li et al., ), to bridge the vector graphics primitives and the raster image domain. We adopt a VQ-VAE recipe (Van Den Oord et al., ), which pairs a codebook-based discrete auto-encoder with an auto-regressive Transformer that models the image space implicitly by learning the distribution of codes that resemble them.
We find this approach particularly promising for vector graphics generation, as it breaks the complexity of this task into two stages.
In the first stage of our method, we decompose images into primitive shapes represented as patches. A vector-quantized auto-encoder learns to encode and map each patch into a discrete codebook, and decode these codes to an SVG approximation of the input patch, which is trained under raster supervision. In the second stage, the series of raster patches containing primitives are encoded and the prior distribution of codes is learned by an auto-regressive Transformer model conditioned on a textual description. At inference, a full series of codes can be generated from textual input, or other existing shape codes. Therefore, Grimoire supports text-to-SVG generation and SVG auto-completion as possible downstream tasks out-of-the-box.

The key contributions of this work are:

- 1.
We frame the problem of image-supervised SVG generation as the prediction of a series of individual shapes and their positions on a shared canvas.
- 2.
We train the first text-conditioned generative model that learns to draw vector graphics with only raster image supervision.
- 3.
We compare our model with alternative frameworks showing superior performance in generative capabilities on diverse datasets.
- 4.
Upon acceptance, release the code of this work to the research community.

Figure: Figure 1: Generative results for fonts and icons from Grimoire and Im2Vec. Since Im2Vec does not accept any conditioning, we sample after training Im2Vec only on icons of stars or the letter A, respectively. For Grimoire we use the models trained on the full dataset conditioned on the respective class.
Refer to caption: x1.png

Figure: Figure 2: Overview of Grimoire. On the left, the training process of our VSQ module is depicted, where raster input patches are encoded into discrete codes and reconstructed as SVG shapes using visual supervision. In the top right, each image is encoded into a series of discrete codes using the trained VSQ encoder and its textual description. The bottom right illustrates how the ART module learns the joint distribution of these codes and the corresponding text.
Refer to caption: x17.png

## 2 Related Work

### 2.1 SVG Generative Models

The field of vector graphics generation has witnessed increasing interest.
Following the extraordinary success of Large Language Models (LLM), the most recent approaches (Lopes et al., ; Aoki and Aizawa, ; Wu et al., ; Tang et al., ) have recast the problem as an NLP task, learning a distribution over tokenized SVG commands. Iconshop (Wu et al., ) introduced a method of tokenizing SVG paths that makes them suitable input for causal language modeling. To add conditioning, they employed a pre-trained language model to tokenize and embed textual descriptions, which are concatenated with the SVG tokens to form sequences that the auto-regressive Transformer can learn a joint probability on.

StrokeNUWA (Tang et al., ) introduced Vector Quantized Strokes to compress SVG strokes into a codebook with SVG supervision and fine-tune a pre-trained Encoder–Decoder LLM to predict these tokens given textual input. However, both of these approaches suffer from a number of limitations. First, they require a corpus of SVG data for training, which hinges upon large pre-processing pipelines to remove redundancies, convert non-representable primitives, and standardize the representations.

Secondly, there is no supervision of the visual rendering, which makes the models prone to data quality errors, e.g., excessive occlusion of shapes.
Finally, these models lack any straightforward extensibility towards the inclusion of new visual features such as colours, stroke widths, or fillings and alpha values.

Hence, another line of work has sought to incorporate visual supervision. These approaches generally rely on recent advances in differentiable rasterization, which
enables backpropagation of raster-based losses through different types of vectorial primitives such as Bézier curves, circles, and squares. The most important development in this area is DiffVG (Li et al., ), which removed the need for approximations and introduced techniques to handle antialiasing.

They further pioneered image-supervised SVG generative models by training a Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN) (Goodfellow et al., ) on MNIST (LeCun et al., ) and QuickDraw (Ha and Eck, ). These generative capabilities have subsequently been extended in Im2Vec (Reddy et al., ), which adopts a VAE including a recurrent neural network to generate vector graphics as sets of deformed and filled circular paths, which are differentiably composited and rasterized, allowing for back-propagation of a multi-resolution MSE-based pyramid loss. However, all of these models lack versatile conditioning (such as text) and focus on either image vectorization, i.e., the task of creating the closest vector representation of a raster prior, or vector graphics interpolation. We show in Section 5 that these approaches fail to capture the diversity and complexity of datasets such as FIGR-8, and generate repetitive samples.

A different type of SVG generation enabled by DiffVG is painterly rendering (Ganin et al., ; Nakano, ), where an algorithm iteratively fits a given set of vector primitives to match an image, guided by a deep perceptual loss function. To achieve this goal, CLIPDraw (Frans et al., ) rasterized a set of randomly initialized SVG paths and encoded these with a pre-trained CLIP (Radford et al., ) image encoder, iteratively minimizing the cosine distance between such embeddings and the text description. A similar approach was adopted by CLIPasso (Vinker et al., ) to translate images into strokes. Vector Fusion (Jain et al., ) leveraged Score Distillation Sampling (SDS) (Poole et al., ) to induce abstract semantic knowledge from an off-the-shelf Stable Diffusion model (Rombach et al., ). A very similar approach, based on SDS, has also been applied to fonts (Iluz et al., ). However, all painterly rendering methods come as iterative algorithms making them very computationally expensive and impractical in real world use cases. They frequently create many unnecessary and redundant shapes to minimize the perceptual loss.

### 2.2 Vector Quantization

VQ-VAE (Van Den Oord et al., ) is a well-known improved architecture for training Variational Autoencoders (Kingma and Welling, ; Rezende et al., ). Instead of focusing on representations with continuous features as in most prior work (Vincent et al., ; Denton et al., ; Hinton and Salakhutdinov, ; Chen et al., ), the encoder in a VQ-VAE emits discrete rather than continuous codes. Each code maps to the closest embedding in a codebook of limited size. The decoder learns to reconstruct the original input image from the chosen codebook embedding. Both the encoder–decoder architecture and the codebook are trained jointly. After training, the autoregressive distribution over the latent codes is learnt by a second model, which then allows for generating new images via ancestral sampling. Latent discrete representations were already pioneered in previous work (Mnih and Gregor, ; Courville et al., ), but none of the above methods close the performance gap of VAEs with continuous latent variables, where one can use the Gaussian reparametrization trick, which benefits from much lower variance in the gradients. Mentzer et al. () simplified the design of the vector quantization in VQ-VAE with a scheme called finite scalar quantization (FSQ), where the encoded representation of an image is projected to the nearest position on a low-dimensional hypercube. In this case, no additional codebook must be learned, but rather it is given implicitly, which simplifies the loss formulation.
Our work builds in part on the VQ-VAE framework and includes the FSQ mechanism.

## 3 Method

Figure: Figure 3: Overview of the data generation process for Grimoire. For the MNIST digits, we simply create patches from a $6\times 6$ Grid. For FIGR-8, we extract the outlines of each icon and create small centered raster segments. We save the original anchor position of each segment for the second stage of our training pipeline. More information about the outline extraction is provided in Section 7.2. Fonts comes in vector format and can be easily manipulated to extract strokes, similarly to FIGR-8.
Refer to caption: x18.png

### 3.1 Stage 1 – Visual Shape Quantizer

The first stage of our model employs a Visual Shape Quantizer (VSQ), a vector-quantized auto-encoder, whose encoder $E_{\mathrm{VSQ}}$ maps an input image $I$ onto a discrete codebook $\mathbb{V}$ through vector-quantization and decodes that quantized vector into shape parameters of cubic Bézier curves through the decoder $D_{\mathrm{VSQ}}$. Instead of learning the codebook (Van Den Oord et al., ), we adopt the more efficient approach of defining our codebook $V$ as a set of equidistant points in a hypercube with $q$ dimensions. Each dimension has $l$ unique values: $L=[l_{1},l_{2},\ldots,l_{q}]$. The size of the codebook $|\mathbb{V}|$ is hence defined by the product of values of all $q$ dimensions.
We define $q=5$ and $L=[7,5,5,5,5]$ for a target codebook size of 4,375 unique codes, following the recommendations of the original authors (Mentzer et al., ).

Before being fed to the encoder $E_{\mathrm{VSQ}}$, each image $I\in\mathbb{R}^{C\times H\times W}$ is divided into patches
$\mathbf{S}=(s_{1},s_{2},\ldots,s_{n})$, with $s_{i}\in\mathbb{R}^{C\times 128\times 128}$, where $C=3$ is the number of channels. A set of discrete anchor coordinates $\mathbf{\Theta}=(\theta_{1},\theta_{2},\ldots,\theta_{n})$ with $\theta_{i}\in\mathbb{N}^{2}$ being the center coordinate of $s_{i}$ in the original image $I$ is also saved. The original image $I$ can then be reconstructed using $S$ and $\Theta$.

We experiment on three datasets (see Section 4). For MNIST, the patches are obtained by tiling each image into a $6\times 6$ grid. For Fonts and FIGR-8, each patch depicts part of the target outline as shown in Figure 3.
We utilize a contour-finding algorithm (Lorensen and Cline, ) to extract outlines from raster images, which are then divided into several shorter segments. Additional details regarding this extraction process can be found in Section 7.2. In contrast, the Fonts dataset is natively available in vector format, making it easier to manipulate, similar to icons, before undergoing rasterization.

The VSQ encoder $E_{\mathrm{VSQ}}$ maps each patch $s_{i}\in\mathbb{R}^{C\times 128\times 128}$ to $\xi$ codes on the hypercube $E_{\mathrm{VSQ}}:\mathbb{R}^{C\times 128\times 128}\mapsto V$ as follows.
Each centered raster patch $s_{i}$ is encoded with a ResNet-18 (He et al., ) into a latent variable $z_{i}\in\mathcal{Z}\subset\mathbb{R}^{d\times\xi}$ with $d=512$. Successively, each of the $\xi$ codes is projected to $q$ dimensions through a linear mapping layer and finally quantized, resulting in $\hat{z}_{i}\in\mathbb{N}^{q}$. The final code value $v_{i}\in\mathbb{V}$ is then computed as the weighted sum of all $q$ dimensions of $\hat{z}_{i}$:

$$ $v_{i}=\sum_{j=1}^{q}\hat{z}_{ij}\cdot b_{j},$ $$

where the basis $b_{j}$ is derived as $b_{j}=\prod_{k=1}^{j-1}l_{k}$, with $b_{1}=1$.
This transformation ensures that each unique combination of quantized values $\hat{z}_{i}$ is mapped to a unique code $v_{i}$ in the codebook $\mathbb{V}$.

This approach avoids auxiliary losses on the codebook while maintaining competitive expressiveness.

The decoder $D_{\mathrm{VSQ}}$ consists of a projection layer, which transforms all the $\xi$ predicted codes back into the latent space $\mathcal{Z}$, and a lightweight neural network $\Phi_{\textrm{points}}$, which predicts the control points of $\nu$ cubic Bézier curves that form a single connected path. We propose two variants of $\Phi_{\textrm{points}}$, a fully-connected neural network $\Phi^{\textrm{stroke}}_{\textrm{points}}:\mathcal{Z}\mapsto\mathbb{R}^{(2
\times(\nu\times 3+1))}$, which predicts connected strokes, and a 1-D CNN $\Phi^{\textrm{shape}}_{\textrm{points}}:\mathcal{Z}\mapsto\mathbb{R}^{(2\times
(\nu\times 3))}$, which outputs a closed shape.

Finally, the predicted path of $\nu$ Bézier curves from $\Phi_{\textrm{points}}$ passes through the differentiable rasterizer to obtain a raster output $\hat{s}_{i}=\textrm{DiffVG}(D_{\mathrm{VSQ}}(E_{\mathrm{VSQ}}(s_{i})))$.
In order to learn to reconstruct strokes and shapes, we train the VSQ module using the mean squared error:

$$ $\mathcal{L_{\text{recons}}}=(s-\hat{s})^{2}.$ $$

$D_{\mathrm{VSQ}}$ can be extended to predict continuous values for any visual attribute supported by the differentiable rasterizer. Hence, we also propose series of other fully-connected prediction heads that can optionally be enabled: $\Phi_{\textrm{width}}:\mathcal{Z}\mapsto\mathbb{R}$ predicts the stroke width of the overall shape, and $\Phi_{\textrm{color}}:\mathcal{Z}\mapsto\mathbb{R^{C}}$ outputs the stroke color or the filling color for the output of $\Phi^{\textrm{stroke}}_{\textrm{points}}$ and $\Phi^{\textrm{shape}}_{\textrm{points}}$, respectively. All the modules are followed by a sigmoid activation function.

While $\mathcal{L_{\text{recons}}}$ would suffice for training the VSQ, operating only on the visual domain could lead to degenerate strokes and undesirable local minima. To mitigate this, we propose a novel geometric constraint $\mathcal{L_{\text{geom}}}$, which punishes control point placement of irregular distances measured between all combinations of points predicted by $\Phi^{\textrm{stroke}}_{\textrm{points}}$.

Let $P=(p_{1},p_{2},...,p_{\nu+1})$ be the set of all start and end points of a stroke with $p_{i}=(p^{x}_{i},p^{y}_{i})$ and $p^{x}_{i},p^{y}_{i}\in[0,1]$. Then $\rho_{i,j}$ is defined as the Euclidean distance between two points $p_{i}$ and $p_{j}$, $\overline{\rho}_{j}$ is defined as the mean scaled inner distance for point $p_{j}$ to all other points in $P$, and $\delta_{j}$ as the average squared deviation from that mean for point $p_{j}$:

$$ $\overline{\rho}_{j}=\frac{1}{\nu}\sum_{\begin{subarray}{c}i=1\\ i\neq j\end{subarray}}^{\nu+1}\frac{\rho_{i,j}}{|i-j|}\qquad\delta_{j}=\frac{1 }{\nu}\sum_{\begin{subarray}{c}i=1\\ i\neq j\end{subarray}}^{\nu+1}\left(\frac{\rho_{i,j}}{|i-j|}-\overline{\rho}_{ j}\right)^{2}$ $$

$\mathcal{L_{\text{geom}}}$ is finally defined as the average of the deviations for all start and end points in $P$. $\mathcal{L_{\text{geom}}}$ is then weighted with $\alpha$ and added to the reconstruction loss.

$$ $\mathcal{L_{\text{geom}}}=\frac{1}{\nu+1}\sum_{j=1}^{\nu+1}\delta_{j}\quad \quad\mathcal{L_{\text{geom}}}=\mathcal{L_{\text{geom}}}+\alpha\times\mathcal{ L_{\text{geom}}}$ $$

We use this component only for the experiments with $\Phi^{\textrm{stroke}}_{\textrm{points}}$ and set $\alpha=0.4$.
We opt to train the ResNet encoder from scratch during this stage, since the target images belong to a very specific domain. The amount of trainable parameters is $15.36M$ for the encoder and $0.8M$ for the decoder. We stress the importance of the skewed balance between the two parameter counts, as the encoding of images is only required for training the model and encoding the training data for the auto-regressive Transformer in the next step. The final inference pipeline discards the encoder and only requires the trained decoder $D_{\mathrm{VSQ}}$, hence resulting in more lightweight inference. The overall scheme of Grimoire including the first stage of training is depicted in Figure 2.

### 3.2 Stage 2 – Auto-Regressive Transformer

After the VSQ is trained, each patch $s_{i}$ can be mapped onto an index code $v_{i}$ of the codebook $V$ using the encoder $E_{\mathrm{VSQ}}$ and the quantization method. However, the predicted patch $\hat{s_{i}}$ captured by the VSQ does not describe a complete SVG, as the centering leads to a loss of information about their global position $\theta_{i}$ on the original canvas. Also, the sequence of tokens is still missing the text conditioning. This is addressed in the second stage of Grimoire. The second stage consists of an Auto-Regressive Transformer (ART) that learns for each image $I$ the joint distribution over the text, positions, and stroke tokens.
A textual description $T$ of $I$ is tokenized into $\mathcal{T}=(\tau_{1},\tau_{2},\ldots,\tau_{t})$ using a pre-trained BERT encoder (Devlin et al., ) and embedded. $I$ is visually encoded by transforming its patches $s_{i}$ onto $v_{i}\in V$ via the encoder $E_{\mathrm{VSQ}}$, whereas each original patch position $\theta_{i}\in\Theta$ is mapped into the closest position in a $256\times 256$ grid resulting in $256^{2}$ possible position tokens. Special tokens <SOS>, <BOS>, and <EOS> indicate the start of a full sequence, beginning of the patch token sequence, and end of sequence, respectively. Each patch token is alternated with its position token. The final input sequence for a given image to the ART module becomes:

$$ $x=(\texttt{<SOS>},\tau_{1},\ldots,\tau_{t},\texttt{<BOS>},\theta_{1},v_{1}, \dots\theta_{n},v_{n},\texttt{<EOS>})$ $$

The total amount of representable token values then has a dimensionality of $|V|+256^{2}+3=69,914$ for $|V|=\text{4,375}$. A learnable weight matrix $W\in\mathbb{R}^{d\times\text{69,914}}$ embeds the position and visual tokens into a vector of size $d$. The BERT text embeddings are projected into the same $d$-dimensional space using a trainable linear mapping layer. The ART module consists of $12$ and $16$ standard Transformer decoder blocks with causal multi-head attention with $8$ attention heads for fonts and icons, respectively. The final loss for the ART module is defined as:

$$ $\mathcal{L}_{\text{Causal}}=-\sum_{i=1}^{N}\log p(x_{i}\mid x_{<i};\theta)$ $$

During inference, the input to the ART module is represented as $x=(\texttt{<SOS>},\tau_{1},\ldots,\tau_{t},\texttt{<BOS>})$, where new tokens are predicted auto-regressively until the <EOS> token is generated. Additionally, visual strokes can be incorporated into the input sequence to condition the generation process.

## 4 Data

MNIST. We conduct our initial experiments on the MNIST dataset (LeCun et al., ). We upscale each digit to $128\times 128$ pixels and generate the texual description using the prompt “x in black color", where x is the class of each digit. We adopt the original train and test split.

Fonts. For our experiments on fonts, we use a subset of the SVG-Fonts dataset (Lopes et al., ). We remove fonts where capital and lowercase glyphs are identical, and consider only 0–9, a–z, and A–Z glyphs, which leads to 32,961 unique fonts for a corpus of $\sim$2M samples. The font features – such as type of character or style – are extracted from the .TTF file metadata.
The final textual description for a sample glyph $g$ in font style $s$ is built using the prompt: “[capital] $g$ in $s$ font”, where “capital ” is included only for the glyphs A-Z. We use 80%, 10%, and 10% for training, testing, and validation respectively.

FIGR-8. We validate our method on more complex data and further use a subset of FIGR-8 (Clouâtre and Demers, ), where we select the 75 majority classes (excluding “arrow”) and any class that contains those, e.g., the selection of “house” further entails the inclusion of “dog house”. This procedure yields 427K samples, of which we select 90% for training, 5% for validation, and 5% for testing. We use the class names as textual descriptions without further processing besides minor spelling correction. Since the black strokes of FIGR-8 mark the background rather than the actual icon, we invert the full dataset before applying our additional pre-processing described in Section 7.2.

## 5 Results

This section presents our findings in two primary categories. First, we examine the quality of the reconstructions and generations produced by Grimoire in comparison to existing methods. Second, we highlight the flexibility of our approach, demonstrating how Grimoire can be easily extended to incorporate additional SVG features.

### 5.1 Reconstructions

Closed Paths. We begin by presenting the reconstruction results of our VSQ module on the MNIST dataset. In our experiments, we model each patch shape using a total of $15$ segments. Increasing the number of segments beyond this point did not yield any significant improvement in reconstruction quality. Given the simplicity of the target shapes, we adopted a single code per shape.

We also conducted a comparative analysis of the reconstruction capabilities of our VSQ module against Im2Vec. To assess the generative quality of our samples, we employed the Fréchet Inception Distance (FID) (Heusel et al., ) and CLIPScore (Radford et al., ), both of which are computed using the image features of a pre-trained CLIP encoder. Additionally, to validate our VSQ module, we considered the reconstruction loss $\mathcal{L_{\text{recons}}}$, as it directly reflects the maximum achievable performance of the network and provides a more reliable metric.

**Table 1: Results for reconstructions of Grimoire and Im2Vec on the full datasets. The last row includes post-processing.**
|  | MNIST | Fonts | FIGR-8 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ |
| Im2Vec (filled) | 0.140 | 1.33 | 25.02 | 0.140 | 2.04 | 26.82 | 0.330 | 16.10 | 26.17 |
| Im2Vec | n/a | n/a | n/a | 0.050 | 5.64 | 26.72 | 0.050 | 13.90 | 26.17 |
| VSQ | 0.090 | 7.09 | 25.24 | 0.014 | 4.45 | 28.61 | 0.004 | 1.42 | 31.09 |
| VSQ + PI | n/a | n/a | n/a | 0.011 | 0.29 | 28.96 | 0.002 | 0.05 | 32.03 |

As shown in Table 1, our VSQ module consistently achieves a lower reconstruction error compared to Im2Vec across all MNIST digits. In Table 2, we also report the reconstruction error for a subset of the dataset, selecting the digit zero due to its particularly challenging topology. Again, our method exhibits superior performance with lower reconstruction errors. For MNIST, we fill the predicted shapes from Im2Vec, since the raster ground truth images are only in a filled format. However, we present both filled and unfilled versions for all other scenarios.

The CLIPScore of our reconstructions is higher in both cases. Notably, FID is the only metric where Im2Vec occasionally shows superior results. We attribute this to the lower resolution of the ground truth images, which introduces instability in the FID metric. The CLIPScore, however, mitigates this issue by comparing the similarity with the textual description.

Strokes. For Fonts and FIGR-8, we conduct a deeper investigation to validate the reconstruction errors of VSQ under different configurations, varying the amount of segments and codes per shape, and the maximum length of the input strokes. Our findings show that for Fonts, more than one segment per shape consistently degrades the reconstruction quality, possibly because the complexity of the strokes in our datasets does not require many Beziér curves to reconstruct an input patch. We also find that shorter thresholds on the stroke length help the reconstruction quality, as the MSE decreases when moving from $11\%$ to $7\%$ and eventually to $4\%$ of the maximum stroke length with respect to the image size. Intuitively, shorter strokes are easier to model, but could also lead to very scattered predictions for overly short settings.

The best reconstructions are achieved by using multiple codes per centered stroke. The two-codes configuration has an average decrease in MSE of 18.28%, 41.46%, and 26.09% for the respective stroke lengths. However, the best-performing configuration with two codes per shape is just 11.36% better than the best single code representative, which we believe does not justify twice the number of required visual tokens for the second stage training.
Throughout our experiments, the configurations with multiple segments do consistently benefit from our geometric constraint.
Ultimately, for our final experiments we choose $(\nu=2$, $\xi=1)$ for Fonts, and $(\nu=4$, $\xi=2)$ for FIGR-8.

Regarding the comparison with Im2Vec, Table 2 shows that the text-conditioned Grimoire on a single glyph or icon has superior reconstruction performance even if Im2Vec is specifically trained on that subset of data. In Table 1, we also report the values after training on the full datasets. In this case, Grimoire substantially outperforms Im2Vec, which is unable to cope with the complexity of the data.

Finally, as Grimoire quickly learns to map basic strokes or shapes onto its finite codebook and due to the similarities between those primitive traits among various samples in the dataset, we find Grimoire to converge even before completing a full epoch on any dataset. Despite the reconstruction error being considerably higher, we also notice reasonable domain transfer capabilities between FIGR-8 images and Fonts when training the VSQ module only on one dataset and keeping the maximum stroke length consistent. Qualitative examples of the re-usability of the VSQ module are reported in the Appendix.

**Table 2: Results for reconstructions of Grimoire and Im2Vec on subsets. The last row includes post-processing.**
|  | MNIST (0) | Fonts (A) | Icons (Star) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | MSE $\downdownarrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ |
| Im2Vec (filled) | 0.218 | 2.20 | 24.61 | 0.087 | 1.64 | 26.27 | 0.120 | 2.40 | 30.90 |
| Im2Vec | n/a | n/a | n/a | 0.060 | 6.33 | 25.78 | 0.110 | 11.17 | 30.40 |
| VSQ | 0.130 | 11.2 | 26.68 | 0.020 | 4.50 | 29.13 | 0.002 | 1.26 | 31.64 |
| VSQ + PI | n/a | n/a | n/a | 0.012 | 0.61 | 29.46 | 0.001 | 0.07 | 32.94 |

### 5.2 Generations

Text Conditioning. We compare Grimoire with Im2Vec by generating glyphs and icons and handwritten digits, and report the results in Table 3. Despite Im2Vec being tailored for single classes only, our general model shows superior performance in CLIPScore for all datasets. Im2Vec shows a generally lower FID score in the experiments with filled shapes, which we attribute again to the lower resolution of the ground truth images (MNIST) and a bias in the metric itself as CLIP struggles to produces meaningful visual embeddings for sparse images (Chowdhury et al., ) as for Fonts, FIGR-8. In contrast, in the generative results on unfilled shapes, Grimoire almost consistently outperforms Im2Vec by a large margin for glyphs and icons.

Note that we establish new baseline results for the complete datasets, as Im2Vec does not support text or class conditioning.

**Table 3: Results for generations of Grimoire and Im2Vec. Grimoire is trained on the full dataset and conditioned to the respective classes using the text description.**
|  | MNIST (0) | MNIST (Full) | Fonts (A) | Fonts (Full) | FIGR-8(Star) | FIGR-8(Full) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | FID $\downdownarrows$ | CLIP $\upuparrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ | FID $\downdownarrows$ | CLIP $\upuparrows$ |
| Im2Vec (filled) | 2.22 | 24.69 | n/a | n/a | 1.20 | 25.81 | n/a | n/a | 2.97 | 31.72 | n/a | n/a |
| Im2Vec | n/a | 25.21 | n/a | n/a | 5.36 | 25.39 | n/a | n/a | 11.59 | 31.88 | n/a | n/a |
| Grimoire (ours) | 12.25 | 26.60 | 9.25 | 25.25 | 5.61 | 30.60 | 1.67 | 28.64 | 6.25 | 32.24 | 3.58 | 27.45 |

Looking at qualitative samples in Figure 4 and Figure 1, one can see that contrary to the claim that surplus shapes collapse to a point (Reddy et al., ), there are multiple redundant shapes present in the generations of Im2Vec. A single star might then be represented by ten overlapping almost identical paths. The qualitative results in Figure 5 confirm this behaviour on the MNIST dataset. We also show that setting Im2Vec to predict only one single SVG path leads the model to compress the shape area and use its filling as a stroke width.

Overall, Grimoire produces much cleaner samples with less redundancy, which makes them easier to edit and visually more pleasing. The text conditioning also allows for more flexibility.
The generations are also diverse, as can be seen in Figure 4 where we showcase multiple generations for the same classes from FIGR-8. Additional generations on all datasets are provided in the Appendix.

Figure: Figure 4: Examples of text-conditioned icon generation from Grimoire.
Refer to caption: x19.png

Figure: Figure 5: Generative results for the MNIST dataset from Grimoire and Im2Vecwith the number of predicted paths fixed to one and ten respectively. Since Im2Vec does not accept any conditioning, we sample after training Im2Vec only on the digit Zero. For Grimoire, we use the models trained on the full dataset conditioned on the respective class.
Refer to caption: x40.png

Vector Conditioning.
We also evaluate Grimoire on another task previously unavailable for image-supervised vector graphic generative models, which is text-guided icon completion. Figure 6 shows the capability of our model to complete an unseen icon, based on a set of given context strokes that start at random positions. Grimoire can meaningfully complete various amounts of contexts, even when the strokes of the context stem from disconnected parts of the icon. We provide a quantitative analysis in Section 7.8. The results in this section are all obtained with the default pipeline that post-processes the generation of our model. A detailed analysis of our post-processing is provided in Section 7.3 and Section 7.4.

Figure: Figure 6: Different completions with varying number of context segments $\nu_{\text{context}}$ (marked in red). Grimoire can meaningfully complete irregular starting positions of the context strokes.
Refer to caption: x49.png

### 5.3 Flexibility

Finally, we demonstrate the flexibility of Grimoire through additional qualitative results on new SVG attributes.
One of the advantages of splitting the generative pipeline into two parts is that the ART module can be fully decoupled from the visual attributes of the SVG primitives. Instead, the vector prediction head of the VSQ can be extended to include any visual attribute supported by the differentiable rasterizer.
Specifically, we activate the prediction heads $\Phi_{\textrm{width}}$ and $\Phi_{\textrm{color}}$ —outlined in Section 3.1— to enable learning of stroke width and color, respectively. We train the VSQ module on input patches while varying the values of those attributes and present the qualitative outcomes in Figure 7, where each stroke is randomly colored using an eight-color palette and a variable stroke width. The VSQ module accurately learns these features without requiring altering the size of the codebook or modifying any other network configurations.

A similar analysis is conducted with closed shapes, and the results are reported in Figure 8, showing that the VSQ module jointly maps both shape and color to a single code. This highlights the minimal requirements of Grimoire in supporting additional SVG features. In contrast, other state-of-the-art vector-based generative models often rely on complex tokenization pipelines, making the extension to new SVG attributes more cumbersome and less flexible.

Figure: Figure 7: Inputs (top) and corresponding reconstructions (bottom) generated by a VSQ model trained to predict not only the shape but also the visual attributes of the input strokes, such as color and stroke width.
Refer to caption: extracted/5910441/figures/visual_attributes/visual_attributes_GT.png

Figure: Figure 8: Reconstruction of MNIST digits when the VSQ module also predicts the filling color. The left side shows the tiling of the original raster images, the right side reports the reconstructions from the VSQ module. No post-processing is applied.
Refer to caption: extracted/5910441/figures/appendix/generations/grim/mnist/orig_3.png

## 6 Conclusion

This work presents Grimoire, a novel framework for generating and completing complex SVGs, trained solely on raster images. Grimoire improves existing raster-supervised SVG generative networks in output quality, while offering significantly greater flexibility through text-conditioned generation. We validate Grimoire on filled shapes using a simple tile-patching strategy to create the input data, and on strokes using fonts and icons datasets. Our results demonstrate the superior performance of Grimoire compared to existing models, even when adapted to specific image classes. Additionally, we show that Grimoire can be seamlessly extended to support new SVG attributes when included in the training data.

Future work could explore incorporating additional vector primitives, expanding visual features, or employing a hierarchical approach to patch extraction.

## 7 Appendix

### 7.1 Glossary of Notation

Due to the number of notation used in this work, in Table 4 we have reported a recap of the most important with a brief description of their meaning.

**Table 4: Glossary of relevant notations in this work.**
| Notation | Meaning |
| --- | --- |
| $E$ | Network encoder |
| $D$ | Network decoder |
| $I$ | Image from the dataset |
| $V$ | Codebook |
| $v$ | Codes from the codebook |
| $L$ | Set of values per dimension of our codebook |
| $l$ | Single dimensional value |
| $q$ | Number of dimensions of the codebook |
| $\mathbf{S}$ | Series of patches |
| $s$ | Single patch |
| $C$ | Color channels |
| $n$ | Number of patches |
| $\mathbf{\Theta}$ | Set of discrete coordinates |
| $\theta$ | Single coordinate pair |
| $\mathcal{Z}$ | Latent space |
| $\hat{z}$ | Projected embedding |
| $d$ | Dimension of latent space |
| $z$ | Latent embedding |
| $\hat{s}$ | Reconstructed patch |
| $\nu$ | Number of segments |
| $P$ | Set of points |
| $p$ | Point pair |
| $\rho$ | Euclidian distance between two points |
| $\Phi$ | Generic Neural network |
| $\xi$ | Number of codebook codes |
| $T$ | Text description |
| $\mathcal{T}$ | Tokenized description |
| $\tau$ | Text token |
| $t$ | Number of text tokens |

### 7.2 Pre-Processing

This section provides additional information regarding the pre-processing and extraction techniques on the employed datasets.

Shapes. No pre-processing is conducted for the MNIST dataset. Images are simply tiled using a $6\times 6$ grid and the central position of each tile in the original image is saved.

Strokes. For the FIGR-8 dataset, the pixels outlining the icons are isolated using a contour finding algorithm [Lorensen and Cline, ] and the coordinates are then used to convert them into vector paths. This simple procedure available in our code repository allows us to efficiently apply a standard pre-processing pipeline defined in Carlier et al. [] and already adopted by other studies [Wu et al., , Tang et al., ]. The process involves normalizing all strokes and breaking them into shorter units if their length exceeds a certain maximum percentage of the image size. Finally, each resulting path fragment is scaled, translated to the center of a new canvas $s$ by placing the center of its bounding box onto the center of $s$, and rasterized to become part of the training data. Since strokes in $S$ are all translated around the image center, the original center position $\theta$ of the bounding box in $I$ is recorded for each $s$ and saved. These coordinates are discretized in a range of $256\times 256$ values.
This approach is also used for Fonts, but since the data comes in vector format, there is no need for contour finding.

### 7.3 Post-Processing

Our approach introduces small discrepancies with the ground truth data during tokenization. The VSQ introduces small inaccuracies in the reconstruction of the stroke, and the discretization of the global center positions may sligthly displace said strokes. The latter serve as the training data for the auto-regressive Transformer and therefore represent an upper limit to the final generation quality.
Similarly for MNIST, the use of white padding on each patch to facilitate faster convergence results in small background gaps when rendering all shapes together, as shown in Figure 5.
These small errors compound for the full final image and may become fairly visible in the reconstructions.

Figure: Figure 9: Different SVG post-processing methods visualized. From left to right: raw generation, results of applying PC and PI, results of applying PC and PI by only considering nearest neighbors of consecutive strokes.
Refer to caption: extracted/5910441/figures/svg_fixing.png

While we opted not to modify the global reconstructions of MNIST generation, for FIGR-8 and Fonts, we make use of SVG post-processing similar to prior work [Tang et al., ], which introduced Path Clipping (PC) and Path Interpolations (PI). In PC, the beginning of a stroke is set to the position of the end of the previous stroke. In PI, a new stroke is added that connects them instead. As we operate on visual supervision, the ordering of the start and end point of a stroke is not consistent. Hence, we adapt these two methods to not consider the start and end point, but rather consider the nearest neighbors of consecutive strokes. We also add a maximum distance parameter to the post-processing in order to avoid intentionally disconnected strokes to get connected. See Figure 9, Figure 10 for a qualitative depiction of this process and Section 7.4 for a quantitative comparison.

Figure: Figure 10: Some examples of text-conditioned glyph generation from Grimoire. The first row shows the unfixed model predictions, the second and third rows depict the final outputs with two different post-processing techniques.
Refer to caption: x64.png

### 7.4 Results with different Post-processing

In Grimoire, the resulting full vector graphic generation is characterized by fragmented segments. This is because the output strokes of the VSQ decoder are each locally centered onto a separate canvas, and the auto-regressive Transformer, which is responsible for the absolute position of each shape, returns only the center coordinates of the predicted shape without controlling the state of connection between different strokes. To cope with this, in Section 7.3, we introduced several post-processing algorithms. In this section, we report additional information about the performance of each of them for the VSQ module (reconstruction) and the overall Grimoire (generation). Table 5 shows that the PC technique consistently outperforms the alternatives across both datasets in terms of both FID and CLIPScore.

**Table 5: Reconstruction capabilities of our VSQ module and generative performance of Grimoire with different post-processing techniques after training on Fonts and FIGR-8.**
| Model | Fonts | FIGR-8 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | MSE | FID | CLIP | MSE | FID | CLIP |
| VSQ | 0.0144 | 4.45 | 28.61 | 0.0045 | 1.29 | 31.17 |
| VSQ (+PC) | 0.0135 | 0.23 | 29.24 | 0.0023 | 0.10 | 31.97 |
| VSQ (+PI) | 0.0106 | 0.29 | 28.96 | 0.0028 | 0.07 | 32.0 |
| Grimoire | n/a | 4.44 | 28.45 | n/a | 4.20 | 26.96 |
| Grimoire (+PC) | n/a | 1.67 | 28.64 | n/a | 3.58 | 27.45 |
| Grimoire (+PI) | n/a | 1.86 | 28.43 | n/a | 4.57 | 26.73 |

### 7.5 Im2Vec on Other Classes

We conducted a more in-depth analysis of the generative capabilities in Im2Vec after training on single subsets of FIGR-8, and compare the results with Grimoire. We trained Im2Vec on the top-10 classes of FIGR-8: Camera (8,818 samples), Home (7,837), User (7,480), Book (7,163), Clock (6,823), Flower (6,698), Star (6,681), Calendar (misspelt as *caledar* in the dataset, 6,230), and Document (6,221). Table 6 compares the FID and CLIPScore with Grimoire. Note that we train our model only once on the full FIGR-8 dataset and validate the generative performance using text-conditioning on the target class, whereas
Im2Vec is unable to handle training on such diverse data.
Despite Im2Vec appearing to obtain higher scores on several classes such as User or Document, a qualitative inspection reveals how the majority of the generated samples come in the form of meaningless filled blobs or rectangles. The traditional metrics employed in this particular generative field, based on the pre-trained CLIP model, react very strongly to such shapes in contrast to more defined stroke images. We refer reviewers to the qualitative samples in Figure 20. We further observe a low variance in the generations when Im2Vec learns the representations of certain classes, such as star icons.

**Table 6: Generative results for Grimoire and Im2Vec for the top-10 classes in FIGR-8.**
| Model | camera | home | user | book | clock | cloud | flower | calendar | document |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP | FID | CLIP |  |
| Im2Vec (filled) | 9.21 | 27.86 | 3.48 | 26.85 | 2.12 | 28.92 | 7.18 | 27.26 | 6.12 | 26.38 | 17.43 | 24.38 | 6.61 | 25.42 | 4.5 | 27.26 | 12.19 | 28.65 |
| Im2Vec | 9.05 | 27.18 | 9.19 | 25.95 | 6.33 | 27.01 | 8.63 | 25.84 | 5.09 | 25.69 | 25.58 | 24.38 | 6.8 | 23.34 | 6.61 | 26.22 | 16.62 | 26.71 |
| Grimoire | 6.74 | 29.81 | 7.16 | 27.16 | 5.45 | 26.81 | 6.65 | 27.1 | 7.22 | 26.32 | 6.78 | 24.96 | 10.27 | 22.00 | 5.57 | 26.23 | 4.08 | 27.96 |
| Grimoire (+PC) | 5.77 | 30.22 | 7.6 | 27.41 | 4.38 | 27.18 | 5.8 | 27.24 | 6.79 | 26.45 | 6.05 | 25.51 | 9.37 | 22.46 | 5.09 | 26.41 | 3.81 | 28.21 |
| Grimoire (+PI) | 7.5 | 29.46 | 7.44 | 27.01 | 5.95 | 26.85 | 6.79 | 27.08 | 7.63 | 26.12 | 7.09 | 24.73 | 9.97 | 22.04 | 5.87 | 25.98 | 4.21 | 27.89 |

### 7.6 Qualitative Results of the Geometric Loss

The adoption of our geometric constraint improves the overall reconstruction error, which we attribute to the network being encouraged to elongate the stroke as much as possible.
The results in Figure 11 show the effects on the control points of the reconstructed strokes from the VSQ. With the geometric constraint, the incentive to stretch the stroke works against the MSE objective, which results in an overall longer stroke and therefore in greater connectedness in a full reconstruction and an overall lower reconstruction error. We also present an example with an excessively high geometric constraint weight ($\alpha=5$) demonstrating that beyond a certain threshold, the positive effect diminishes, resulting in degenerated strokes.

Figure: Ground Truth
Refer to caption: extracted/5910441/figures/appendix/geometric_loss/gt.png

### 7.7 Implementation Details

We use AdamW optimization and train the VSQ module for 1 epoch for Fonts and FIGR-8 and five epochs for MNIST. We use a learning rate of $\lambda=2\times 10^{-5}$, while the auto-regressive Transformer is trained for $\sim$30 epochs with $\lambda=6\times 10^{-4}$. The Transformer has a context length of 512.
Before proceeding to the second stage, we filter out icons represented by fewer than ten or more than 512 VSQ tokens, which affects 12.16% of samples. We use p-sampling for our generations with Grimoire. Training the VSQ module on six NVIDIA H100 takes approximately 48, 15, and 12 hours for MNIST, FIGR-8, and Fonts, respectively; the ART module takes considerably fewer resources, requiring around 8 hours depending on the configuration.
Regarding Im2Vec, we replace the Ranger scheduler with AdamW [Loshchilov and Hutter, ] and enable the weighting factor for the Kullback–Leibler (KL) divergence in the loss function to 0.1, as it was disabled by default in the code repository, preventing any sampling. We train Im2Vec with six paths for 105 epochs with a learning rate of $\lambda=2\times 10^{-4}$ with early stopping if the validation loss does not decrease after seven epochs.
Regarding the generative metrics, we utilized CLIP with a ViT-16 backend for FID and CLIPScore.

### 7.8 Generative Scores with Completion

To evaluate if Grimoire generalizes and learns to meaningfully complete previously unseen objects, we compare the CLIPScore and FID of completions with varying lengths of context. The context and text prompts are extracted from 1,000 samples of the test set of the FIGR-8 dataset. The results are shown in Table 7.

While Grimoire can meaningfully complete unseen objects, the quality of these completions is generally lower than the generations under text-only conditioning. This is expected, as prompts in the test set are also encountered during training (the class names). The CLIPScore generally drops to its lowest point with the least amount of context and then recovers when more context is given to the model, which coincides with our qualitative observations that with only a few context strokes, Grimoire occasionally ignores them completely or completes them in an illogical way, reducing the visual appearance.

**Table 7: Generation quality of Grimoire with different lengths of provided context on Fonts and FIGR-8. Post-processing is conducted for all setups. Grimoire uses textual input for all generations.**
| Model | Fonts | FIGR-8 |  |  |
| --- | --- | --- | --- | --- |
|  | FID | CLIP | FID | CLIP |
| Grimoire (w/o context) | 1.67 | 28.64 | 3.58 | 27.45 |
| Grimoire (+ 3 stroke context) | 2.78 | 27.25 | 4.65 | 25.31 |
| Grimoire (+ 6 stroke context) | 3.16 | 27.25 | 5.46 | 25.54 |
| Grimoire (+ 12 stroke context) | 2.95 | 27.57 | 6.04 | 25.85 |
| Grimoire (+ 24 stroke context) | 2.25 | 28.12 | 6.05 | 26.39 |

### 7.9 Domain Transfer Capabilities for Reconstruction

To validate how the strokes learned during the first training stage adapt to different domains, we use our VSQ module to reconstruct Fonts after training on FIGR-8, and vice versa. Figure 12 provides a qualitative example for each setting. Despite the loss value for each image being around one order of magnitude higher than the in-domain test-set (MSE$\approx 0.05$), the VSQ module uses reasonable codes to reconstruct the shapes and picks curves in the correct directions. Straight lines end up being the easiest to decode in both cases.

Figure: Figure 12: Qualitative zero-shot reconstructions from the test-set of FIGR-8 and Fonts after training the VSQ module solely on the respective other dataset.
Refer to caption: extracted/5910441/figures/appendix/zeroshot/icons_on_fonts.png

### 7.10 Codebook Usage for Strokes

As described in Section 3.1, for FSQ, we fixed the number of dimensions of the hypercube to 5 and set the individual number of values for each dimension as $L=[7,5,5,5,5]$ for a total codebook size of $|B|=$ 4,375. In this section, we want to share some interesting findings about the learnt codebook. For this, we shall use the VSQ trained on FIGR-8 with $n_{\rm code}=1$, $n_{\rm seg}=2$, a maximum stroke length of $3.0$, and the geometric constraint with $\alpha=0.2$.

After training the VSQ on FIGR-8, we tokenize the full dataset. The resulting VQ tokens stem from 60.09% of the codebook, while 39.91% of the available codes remained unused. The ten most used strokes make up 41.24% of the dataset, while the top 24 and 102 strokes make up roughly 50% and 75%, respectively. These findings indicate that for these particular VSQ settings, one could experiment with smaller codebook sizes.

To balance out the stroke distribution, one could use a different subset of FIGR-8. Currently, the classes “menu”, “credit card”, “laptop”, and “monitor” are contributing the most to the stroke imbalance, with 26%, 24.3%, 24.05%, and 23.8% of their respective strokes being the most frequent horizontal one in Table 8.

**Table 8: Top ten most used strokes of the VSQ module trained on icons and their relative occurrences in our subset of FIGR-8.**
|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 18.76% | 12.26% | 2.56% | 1.73% | 1.16% | 1.12% | 0.99% | 0.94% | 0.92% | 0.80% |

### 7.11 Average Strokes in Codebook

In Section 7.10, we show the ten most used strokes of our trained VSQ, but after inspecting the full codebook we notice how neighbouring codes often express very similar strokes. Therefore, to visualize the codebook more effectively, we plot mean and minimum reductions of the full codebook in Figure 13. Additionally, we tokenize the full FIGR-8 dataset and plot the same reductions in Figure 14 to show the composition of the dataset.

Figure: Figure 13: Different reductions of all 4,375 strokes from the VSQ codebook. The model seems to have learned an expressive codebook-decoder mapping as the figure on the left shows a smooth and evenly distributed stroke profile and the figure on the right displays strokes in almost every direction.
Refer to caption: extracted/5910441/figures/appendix/codebook/reductions/codebook_mean_strokes.png

Figure: Figure 14: Different reductions of all strokes from the tokenized FIGR-8 dataset. The visualization on left shows the dominance of the two most occurring strokes, the middle shows that the distribution of strokes is skewed. The missing 39.91% of strokes are also visible in the right figure, where certain diagonal strokes that are available in the codebook are never used.
Refer to caption: extracted/5910441/figures/appendix/codebook/reductions/figr8_mean_strokes.png

### 7.12 Qualitative Results – Reconstruction

In Figure 15 and Figure 16, we provide several qualitative examples of vector reconstructions using Im2Vec and our VSQ module on the Fonts and FIGR-8 datasets, respectively. We fill the shapes of the images when using Im2Vec, since the model creates SVGs as series of filled circles and would not be able to learn from strokes with a small width. Im2Vec does not converge when trained on the full datasets, whereas it returns some approximate reconstruction of the input when only a single class is adopted. In contrast, the VSQ module generalizes over the full dataset.

### 7.13 Qualitative Results – Generation

In this section, we provide qualitative examples of our reconstruction and generative pipeline, and compared those with Im2Vec.
Figure 17 reports a few examples of icons generated with Grimoire using only text-conditioning on classes. In Figure 18 we report some generations for MNIST. In Figure 19, we report generative results for Fonts. Thanks to the conditioning, we can generate upper-case and lower-case glyphs in bold, italic, light styles, and more. As can be seen in the table, Grimoire also learns to properly mix those styles only based on text.
Finally, in Figure 20, we report some generative results on icons and Fonts for Im2Vec on a single class dataset. The results show how the pipeline typically fails to produce meaningful or sufficiently diverse samples.

Figure: Figure 15: Examples of various reconstructions of our VSQ module after training on Fonts compared to reconstructions of Im2Vec trained on the letter "A" (first row) and Im2Vec trained on the full Fonts dataset (third row).
Refer to caption: extracted/5910441/figures/appendix/recon/fonts/A/gt_filled_0.png

Figure: Figure 16: Examples of various reconstructions of our VSQ module after training on icons compared to reconstructions of Im2Vec trained on one class (first row) and Im2Vec trained on the full dataset (third row).
Refer to caption: extracted/5910441/figures/appendix/recon/icons/star/gt_filled_1.png

Figure: Figure 17: Examples of various samples generated with Grimoire after training on icons, using only text conditioning.
Refer to caption: x152.png

Figure: Figure 18: Examples of a samples generated with Grimoire for each digit of the MNIST dataset.
Refer to caption: x177.png

Figure: Figure 19: Examples of filled samples generated with Im2Vec after training the model on specific classes of the dataset. For most classes, Im2Vec could not capture the diversity of the data and failed to meaningfully converge.
Refer to caption: x187.png

Figure: Figure 20: Examples of various samples generated with Grimoire after training on Fonts, using only text conditioning.
Refer to caption: x217.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: As mentioned in the weakness section, this paper really needs to be positioned more properly against (more recent) works on raster-to-vector, token-based 2D/3D generation, and the abundance of vector graphics/CAD datasets. I currently do not buy into the utility of this patch-based mini stroke/shape representation, that is far from a true SVG representation. Clarifications over these points will help sway me - after all, there is nothing that is fundamentally incorrect about the approach. I might also be wrong with some of my claims - am a bit out of touch with this topic, as mentioned, so please do point out anything that I missed.
soundness: 3
strengths: - The overall pipeline design looks okay to me. It make sense to use a codebook to represent a set of instruction token, and an autoregressive transformer is a good way to interact with such a set of tokens. The design choice for encoding the images and the text prompts also are reasonable.
- Results look reasonable on the set of data shown, with considerable discussions. 
- Code is available, and appears to be quite reproducible.
summary: This paper proposed a patch-based approach for learning a discrete codebook for a set of mini strokes/shapes, and then using an autoregressive transformer that operates on this representation to perform "SVG" reconstruction and generation.
weaknesses: I am slightly out of touch with this class of problems recently, so I some of the stuff I mention below might not be that accurate or up to date. Would be happy to update that following clarifications. With that being said, I didn't get much out of this paper - both the problem studied and the technique used are far more well studied than what is depicted in this paper. I am aware of many approaches that are similar in terms of core ideas (autoregressive generation of 2D/3D content represented as a set of discrete instructions/parameters), many datasets that are far more complex than what is used in this paper, as well as multiple works that seem to provide better solutions to this problem (and not compared):

- Despite the claims in 2.1 about this work being different from other SVG generative models (and other CAD/sketch/3D content generation pipelines that follow similar idea of representing content with a set of tokenized instructions, none of which cited), this work is not doing anything that is that much different from these works. Use of vector quantization (e.g. SkexGen: CAD Construction Sequence Generation with Disentangled Codebooks) or grid based (quantized) features (e.g. ShapeFormer: Transformer-based Shape Completion via Sparse Representation) are both standard practices (just naming the first paper that came on top of my mind). In some sense, the paper is even a regression from some of these works, where the so called "SVG" representation used in this paper is not really SVG, but a patch wise curve/closed shape representation that lacks the structure of actual vector drawings.
- Despite the claim about lack of vector data, datasets like SketchGraph and DeepCAD offer more than enough data in vector/discrete formats, and datasets like Quick, Draw! (as cited briefly in this paper as well) already has a vector representation. Moreover, there are so many off the self approachs for vectorizing drawings (e.g. Deep Sketch Vectorization via Implicit Surface Extraction). With the vast abundance of such data, it just doesn't make sense to use the pseudo "SVG" representation in the paper, which has few of the benefits of actual vector drawings.
- Following the previous two points, the baselines used in the paper is both insufficient and inappropriate: of course the more wholistic representation used by Im2Vec  (which was also pretty ancient by the standard of current vision/graphics) won't have as much representation power as collection of patch-based strokes. However, it at least has a single, wholistic representation of the entire SVG, and thus is not only more flexible, but also allows various applications e.g. interpolation, editing, that is hard to achieve with the proposed method. Therefore, just showing that Im2Vec fails on some cases don't really make me buy into the usefulness of the proposed method: if the goal is to show "generative capabilities", then compare against many more recent SVG/sketch/CAD generation works; if the goal is to show that this works enables better raster-to-vector conversion and thus removes the need of curated dataset (of which there are many), then there's also many much more recent works to compare against; if the goal is to show that this patch-wise representation is a good representation, then more analysis should be done on how this is a good representation (and I don't think so, both from an intuitive level and by examining the various visual artifacts present in the qualitative examples) that is useful in practice (i.e. having at least some unique capacities of vector graphics, as opposed to just an image), then more analysis is needed (editing, interpolation, user studies, etc.).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * Are the results shown in Tables 1,2,3 and Figure 4,5,6,7,8, on the train set or the test set?
soundness: 3
strengths: 1) The idea of using raster image supervision is novel

2) Effectiveness of the proposed method is shown with extensive experiments and ablation
studies

3) Usefulness is shown on multiple use cases (a) text-conditioned generation, (b) text-
conditioned icon completion (This task was not possible with previous methods. The
proposed framework made it possible), and (c) ability to reconstruct vector images with
visual attributes such as color and stroke width.

Overall, the paper proposes a novel idea to use raster image supervision to train a generative
model for vector graphics images. It uses vector quantized autoencoder to learn a codebook for
primitive strokes+shapes, and then trains an auto-regressive transformer model to learn a joint distribution on
textual description and input images, as represented by codebook trained in the earlier stage. The
paper also puts a good effort in making extensive experiments, although comparisons could be made stronger (see next section for comments),  to demonstrate the usefulness of the proposed method.
summary: The paper addresses the problem of vector graphics image generation and proposes a novel approach to generate vector graphics images conditioned on text by using raster image supervision. The vector graphics image format is SVG, which is built using cubic bezier curves.

Existing gaps that motivate this paper are two fold: (a) Lack of abundant SVG data for training generative models for vector graphics generation, and (b) the difficulty of incorporating visual attributes such as color and strokes.

To address these challenges, the paper introduces a method to train a generative vector graphics model using just the raster images, eliminating the need for ground truth SVG training data. The proposed methods is a two-stage approach. The first stage decomposes an input raster image into primitive patches and learns a discrete codebook that maps each patch to a code. In the second stage, an auto-
regressive text-conditioned transformer is used to learn joint distribution of textual description and codes. The loss functions used are: (1) Reconstruction loss to reconstruct primitive strokes and shapes, (2)  Geometry loss for better placement of Bezier curve points,
(3) Causal loss for auto-regressive transformer resulting from causal multi-head attention in the decoder. As probably understood, this is a strongly supervised method. To summarize, the input to the method during training is an RGB image and a textual description of that image, while during inference, the method only takes a textual of an image we want the learned model to generate. At the output, we have a generated vector graphics image in a SVG format. The representation of SVG is Cubic Bezier curves, with control points and a starting point. The underlying modeling tools are vector-quantized auto-encoder for image encoding with ResNet as image encoder, DiffVG as differentiable rasterizer, and BERT to encode text, and an auto-regressive Transformer to generate strokes and curves in an auto-regressive fashion. The datasets used for experiments are: MNIST, Fonts, FIGR-8. In terms of evaluations, the paper looks at the following metrics: MSE, FID score and CLIP score.For comparisons, the method compares against one prior work from zcvpr 2021, Im2Vec.
weaknesses: 1) The paper could have been written in a better way to make it easier to understand.
Please see the following section for improvement suggestions

**Writing improvement suggestions:**

- The equation on line 265 (right side) does not make sense. I guess there is a spelling
mistake. Please number the equations so that it is easier to refer to.

- It would be helpful to have one more diagram explaining the stage-1 in more detail.
Figure-2 shows the steps at a very abstract level and it does not include all the steps e.g.
input and output of the model is not shown, the Geometry Loss mentioned in line 265 is
missing in the figure.

- The “Related Work” section in its current form is too elaborate and can be cut down by
writing it succinctly

- The “Method” section includes proposed method as well as the experimental details
together making it harder to read and understand. The experimental details such use of
contour finding algorithm (on line 215), variants used for $\Phi_{points}$ (on line 234-
236), the number of trainable parameters (on line 269), etc., can be factored out and
written in a separate section named “Experimental Setting”


2) Only one baseline (Im2Vec) was used for experiments. How about comparing to DeepSVG (from NeurIPS 2020) and ICON SHOP (from SIGGRAPH Asia 2023)? This would have made the evaluations thorough and helped get a better understanding of where the proposed approach stands, and what other gaps remain.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - The description of auto-encoder part (Line 219 to Line 232) is a bit unclear for me, with several math equations omitted according to my understanding. First, the latent variable $z_i$ is of dim $d \times \xi$, then "each of the $\xi$ codes is projected to q dimensions", does this mean the results are $d \times q$ and there are $\xi$ codes for each patch? What is the shape of the codes $v_i$? How does the "transformation ensures that each unique combination of quantized values is mapped to a unique code"?
- How to make sure that strokes from different patches are connected to each other?
soundness: 3
strengths: - The authors propose an image-supervised method to generate vector graphics, by fitting this task into a common used generation framework: vector-quantization auto-encoder with auto-regressive distribution learning. To assist auto-decoder learning, the authors also included novel components such as geometry losses.
- The authors have conducted detailed experiments to validate their method, both in terms of reconstruction and generation quality. The results also clearly showed that the proposed method can excel in the selected datasets compared with other baselines.
- The authors released the code and detailed instructions for great reproducibility.
summary: This paper presents a method called GRIMOIRE for generating scalable vector graphics. The authors first train a visual shape auto-encoder that maps 2D images into a discrete codebook. Specifically, each images is divided into patches and then encoded and quantized into a set of discrete tokens, and then the decoder transforms the latent codes back into the latent space, followed by a neural network to predict the control points of the Bezier curves. By using the differentiable rasterization, the Bezier curves can be rendered back into 2D images for end-to-end training. Then, the second stage involves learning an auto-regressive transformer to model the joint distribution over text, positions of stroke tokens. The authors did the experiments on three different datasets, including MNIST, FONTS, and FIGR-8. Results demonstrated that the proposed auto-encoder can reconstruct the images well and the generator can generate good results conditioned on text prompts.
weaknesses: - This paper focus on a relatively smaller problem: generating vector graphics on small datasets. The experiments are mostly done on simple datasets and I haven't seen more complex results. Compared with SDS-based methods, the proposed method are limited by the use of datasets and cannot produce more diverse results.
- Although the authors successfully applied the VQ-framework for generation, I think in terms of contribution, it's more like utilizing existing experience for some new tasks, so the technical contribution is not that significant for a higher score.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. For closed paths, the current evaluations are only conducted on the MINST dataset. I was wondering about more comparisons on EMOJIS and ICONS datasets, similar to the experiments in Im2Vec.

2. How to determine the number of patches that should be used for closed-filled shapes?

3. How to determine the weight in the geometric loss?
soundness: 3
strengths: The whole pipeline considers the vector graphics generation into two stages. The first VSQ stage can be trained under raster supervision, solving the issue that existing works often require direct supervision from SVG data. The second ART stage makes it possible to apply the proposed method to support several applications, including text-to-SVG generation and SVG auto-completion. The experimental results demonstrate the effectiveness of the proposed method.
summary: This paper focuses on the Scalable Vector Graphics (SVG) generation task. A text-guided SVG generative model, i.e., GRIMOIRE, is proposed to learn to draw vector graphics with only raster image supervision. The proposed model, which is formulated as the prediction of a series of individual shapes and positions, consists of two modules, one is a visual shape quantizer (VSQ), and the other is an auto-regressive transformer (ART). The effectiveness of the proposed method is demonstrated on both closed-filled shapes data and outline strokes data.
weaknesses: The main weaknesses are listed as follows.

1. The technical contribution of this paper is not very clear. Although the proposed method can be trained with only raster image supervision, the core modules, including DiffVG and VQ-VAE, come from existing works.

2. The proposed pipeline adopts different strategies for the MNIST digits and FIGR-8. I was wondering about the results using the same segmentation strategy, e.g., creating patches for both closed-filled shapes and outline strokes.

3. The current evaluation is not thorough. Either SVG reconstruction or text-to-SVG generation has been studied for several years. But this paper only compares the proposed method with one existing work, i.e., Im2vec, making the results less convincing.

4. For the ablation study, I cannot find any quantitative comparisons of the training losses for the proposed pipeline.

5. Although this paper states that some of the existing works are computationally expensive and impractical in real-world use cases, there are no detailed computational cost or running time analyses.

6. Minor issues. For the loss equation shown in L264-L266, the geometric loss should be added to the reconstruction loss, rather than itself.

Meta-review: The paper adopts the VQVAE framework for SVG generation using only image supervision by integrating a differentiable rasterizer. It received mixed/borderline ratings from the reviewers.

On the good side, the contribution of the work is recognized by most reviewers and the area chair. Applying the VQVAE framework to the field of SVG generation is an interesting and innovative approach. The use of visual guidance for supervision is convenient and eliminates the need for additional annotations, making the framework practical. The open-sourced code is a significant contribution to the community, enabling reproducibility and further exploration.

However, the writing has been a crucial concern for both Reviewer ymmo and Reviewer Zk2S. While some of Reviewer ymmo’s concerns were addressed, the overall clarity of the paper remains insufficient, especially the issues raised by Reviewer Zk2S. A specific issue lies in the inconsistent explanation of the mapping between patches and codes. 

For example: On L213 of the revised paper: “The VSQ encoder maps each patch to $\xi$ codes on the hypercube,” suggesting that a patch corresponds to $\xi$ codes. On L262: “Each patch can be mapped onto an index code  of the codebook,” implying that a patch corresponds to a single code. This inconsistency is confusing, and the rebuttal failed to clarify it. Moreover, the revised appendix exacerbates the confusion: In Section A.3, it states, “In practice, for our experiments, we keep $\xi=1$ and disable the product of all codes that create the final embedding for the decoder.” If a patch corresponds to only one code, why mention $\xi$ in the first place? Does this imply some special design? How could a patch produce more than one code? The appendix also introduces the concept of “product of all codes,” mentioned in both L922 and Figure 15, which never appears in the main paper. This unexplained term further muddles the presentation.

**The lack of clarity in these fundamental aspects makes the paper difficult to understand and unsuitable for acceptance until the writing is substantially improved for better comprehension.**

Lastly, some other issue: Reviewer Zk2S noted that the produced shapes are overly simple, and the paper lacks results on more complex and diverse datasets. Limited technical novelty as raised by Reviewer 6k4C and Reviewer sQs2. While the rebuttal addressed Reviewer sQs2’s concerns and provided more complex results, Reviewer 6k4C remains unconvinced. Reviewer 6k4C also found the quality of the results less satisfactory, and the rebuttal does not convince Reviewer 6k4C on this issue.

Decision: Reject