A Multi-Implicit Neural Representation for Fonts
Year: 2022

Abstract:
Fonts are ubiquitous across documents and come in a variety of styles. They are either represented in a native vector format or rasterized to produce fixed resolution images. In the first case, the non-standard representation prevents benefiting from latest network architectures for neural representations; while, in the latter case, the rasterized representation, when encoded via networks, results in loss of data fidelity, as font-specific discontinuities like edges and corners are difficult to represent using neural networks. Based on the observation that complex fonts can be represented by a superposition of a set of simpler occupancy functions, we introduce multi-implicits to represent fonts as a permutation-invariant set of learned implicit functions, without losing features (e.g., edges and corners). However, while multi-implicits locally preserve font features, obtaining supervision in the form of ground truth multi-channel signals is a problem in itself. Instead, we propose how to train such a representation with only local supervision, while the proposed neural architecture directly finds globally consistent multi-implicits for font families. We extensively evaluate the proposed representation for various tasks including reconstruction, interpolation, and synthesis to demonstrate clear advantages with existing alternatives. Additionally, the representation naturally enables glyph completion, wherein a single characteristic font is used to synthesize a whole font family in the target style.

heading: 1 Introduction
text: Fonts constitute the vast majority of documents. They come in a variety of styles spanning a range of topologies representing a mixture of smooth curves and sharp features. Although fonts vary significantly across families, they remain stylistic coherent across the different alphabets/symbols inside any chosen font family.
Fonts are most commonly stored in a vector form (e.g., a collection of spline curves) that is compact, efficient, and can be resampled at arbitrary resolutions without loss of features. This specialized representation, however, prevents the adaptation of many deep learning setups optimized for regular structures (e.g., image grids). In order to avoid this problem, custom deep learning architectures have been developed for directly producing vector output but they typically require access to ground truth vector data for training. This is problematic: first, collecting sufficient volume of vector data for training is non trivial; and second, vector representations are not canonical (i.e., same fonts can be represented by different sequence of vectors), which in turn requires hand-coded network parameters to account for varying number of vector instructions.
An alternate approach is to rasterize vectorized fonts and simply treat them as images. While this readily allows using image-based deep learning methods, the approach inherits problems of discretized
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
representations leading to aliasing artifacts and loss of sharp features. Further, the resultant images are optimized for particular resolutions and cannot be resampled without introducing additional artifacts.
Recently, implicit representation has emerged as an attractive representation for deep learning as, once trained, they can resampled at different resolutions without introducing artifacts. Unfortunately, implicit representations (e.g., signed distance fields) for fonts are often too complex to be represented accurately by neural networks. As a results, although deep implicits work for simple fonts, they can fail to retain characteristic features (i.e., edges and corners) for complex fonts.
Drawing inspiration from multi-channel SDFs [4], we observe that complex fonts can be expressed as a composition of multiple simple regions. For example, a local corner can be represented as a suitable composition of two half-planes, each of which can easily be individually encoded as deep implicit functions. We build on this idea by hypothesizing that complex fonts can also be encoded as suitable composition of global implicit functions. We call such a representation to be a multi-implicit neural representation, as each (global) implicit function is neurally encoded.
Thus, multi-implicits provide a simple representation that is amenable for processing by neural networks and the output fidelity remains comparable, even under resampling, to vector representations without losing edge or corner features. A remaining challenge is how to supervise such a network as there is no dataset with reference multi-implicits that be directly used. In this paper, we present a network structure and training procedure that allow multi-implicts to be trained using only local supervision. We describe how to extract necessary local supervision from vector input
and to adaptively obtain training information for the multi-implicits.
We extensively evaluated the proposed multi-implicits representation for various tasks including reconstruction, interpolation, and synthesis to demonstrate clear advantages over several existing state-of-the-art alternatives. Additionally, the representation naturally enables glyph completion, wherein a single characteristic font glyph is used to synthesize a whole font family in consistent style.

heading: 2 Related Work
text: Raster-based representation. One of the most intuitive representation of shapes is raster, i.e., representing a 2D shape by pixels or a 3D shape by voxels, which provides the grid-format data that is perfectly compatible with (regular) CNN models. Hence it has acted as a catalyst for many deep learning based methods for semantic editing of raster-based shapes. A general idea is to learn an encoding-decoding model and then manipulate the shape in the latent space. Sharing a similar spirit, many generative models [21, 5, 15] engaged in the raster-based generation and attribute transfer. [1, 25] narrowed down the scope to fonts specifically, which focus more on the shape instead of texture. While the raster-based representation has shown strong semantic editability as incorporated
with deep models, it is still limited by its intrinsic resolution. Since image super-resolution is an ill-posed problem, details cannot be fully recovered after upscaling.
Deep learning based vector graphs. Vector graphics (e.g., SVG) have been widely adopted as a scalable representation. Typically, it is constructed by sequences of Bézier curves, which are difficult to be modeled by traditional CNN models since the diverse sequence length across different shapes, as well as diverse types and numbers of Bézier curves for the similar or even the same shape. Therefore, RNN-based models (e.g., LSTM) are commonly adopted in recent works [17, 2] to learn the dynamic sequence of curves. Although the sequential modeling could achieve semantic editing and/or interpolation between shapes, it is still lagging behind CNN-based models (rasterbased methods) in terms of reconstruction accuracy because it has to model much longer temporal dependency. Also, there is no specific attention given to features like edges and corners.
Transformation from raster to vector. Instead of directly modeling raster or vector, an idea of achieving both scalability and editability is to model the transfer from raster to vector [20] (we will not consider vectorization methods that purely transfer images to vectors). Such an approach inherits the advantages of raster-based methods and supports easy editability. In addition, they output scalable vectors directly. However, this method would be still limited by the raster resolution, i.e., the output vector graph cannot capture enough details since the corresponding raster input may have already lost those finer details. Another challenge for such methods is the high complexity of the shape, e.g., interpolation between shapes with different topologies leads to undesirable intermediate shapes.
Deep implicit representation. Deep implicit functions [10, 3, 8, 22, 18, 23] have achieved great success in shape representation. They take advantage of deep learning techniques to fit an implicit function, which provides a continuous representation breaking the grid limitation of a raster domain. In addition, deep implicit functions model shapes spatially instead of modeling sequentially as aforementioned in deep learning based vector graphs. Therefore, deep implicit functions could preserve the editability like raster-based representation and potentially achieves scalable representation like vectors. Unfortunately, existing works seldom explore the scalability and fidelity of shapes in their representations, especially during editing, interpolation, and upscaling. For instance, sharp corners always suffer from editing and scaling. This is particularly problematic in the domain of fonts, and our work addresses this limitation via the proposed multi-implicit representations.

heading: 3 Methods
text: We represent fonts as the composition of multiple global implicit functions. An implicit representation has two key advantages: it allows for rendering at arbitrary resolution; it can be locally supervised in characteristic areas such as sharp edges and corners. Unlike a single implicit function, our multi-implicit representation can faithfully reconstruct sharp features with low reconstruction error (Figure 2).
Implicit Model
Image Rendering Multi-SDF Differentiable Rasterization
Multi-Curve Representation
Importance Sampling
+ z
L 2 regularization Corner Template Loss Global Shape Loss Gradient Loss + We train a generative neural network, as shown in the inset, that models fonts using this representation. Instead of directly learning the inside-outside status of the font image [3], we predict a set of distance fields of 2D shapes. Their composition is then fed into a differentiable rasterizer to produce the final image. We will show in Section 4 how this generative model enables a diverse set of font reconstruction and editing operations. Section 3.1 details differentiable rasterization of 2D distance fields, and corresponding corner preservation is discussed in Section 3.2. Finally, training losses and details are given in Section 3.3.

heading: 3.1 Differentiable Rasterization of Distance Field
text: As aforementioned, we use the signed distance field (SDF) to model 2D shapes. The common supervisions for SDF are SDF labels from the ground truth curves or distance transform on the silhouette [14]. Figure 3 compares the results between training with SDF and training with raster,
where training only on the ground truth SDF does not always result in a good raster image in terms of boundary smoothness. In contrast, training with the supervision of raster yields smoother boundary. Therefore, we will draw supervision on rasterized SDF to achieve better shape.
Sharing the spirit from vector graphics rasterization works [7, 11, 16, 19, 24], we simplify general vector graphics rendering by analytically approximating the point to curve distance as,
I(x, y) = K (min i∈F di(x, y)) g (x, y) , (1)
K(d) = ⎧⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎩ 1 if d > γ, k ( d γ ) = 1 2 + 1 4 (( d γ ) 3 − 3 ( d γ )) if − γ ≤ d ≤ γ, 0 if d < −γ, (2)
where di(x, y) indicates an SDF of distance from pixel (x, y) center to the closest point on the i-th curve corresponding the font F , and K denotes a function that approximates the opacity based on the distance value. There are many works for estimating di(x, y) from scene parameters but most of them are constrained by the choice of the parameterization. In this paper, we model each di(x, y) using an implicit neural network. The function g models the spatially-varying texture of the shape. For solid fonts, we set g = 1. In the supplemental we show examples of textured fonts where we make use of spatially varying g. In the function K, γ is the anti-alias range, and the kernel k is a radially symmetric continuous filter that satisfies the constraint k(1) = 0 and k(−1) = 1. In our work, we approximate k(⋅) using a parabolic kernel similar to [19, 13]. Note that the rasterization function I(x, y) has non-zero gradients only if (x, y) falls inside of the anti-aliasing range. We use a progressively decreasing anti-aliasing range strategy for better convergence and fidelity (see Sec. 3.3).

heading: 3.2 Multi-Curve Representation for Sharp Corners
text: We will lose details like sharp corners when upscaling bitmaps or sign distance functions. Resampling an implicit model that encodes the pixel values or signed distance values of a shape similarly suffers from blurry corners. An brute force solution is to train the implicit model with extremely high-resolution images, but this would drastically increase the burden of training, and still limited by the training resolution. Rather than
directly modeling corners, we represent corners as the intersection of multiple curves (e.g., lines or parabolas), drawing inspiration from traditional representations [9]. Note that even though these individual sub-curves may be smoothed after encoded by a deep model, the sharpness of their intersection will be preserved. With this insight, we construct sharp corners from multiple smooth curves
predicted by the implicit model. More specifically, the implicit model is designed to predict multiple SDFs (and rasterization of distance fields), each of which carries smooth curves/shapes decoupled from corners and edges as illustrated in Figure 4a.
Assume a shape F is represented by a set of curves C = {C1, C2,⋯, Cn}, where Cn is a binary map indicating whether a pixel is inside (i.e., 1) or outside (i.e., 0) the n-th curve, like the example in Figure 4a. In our scenario, Cn = K(dn), where dn is the n-th SDF channel estimated by the implicit model. Then, a function over all curves F (C) will fuse those curves to reconstruct the shape, preserving sharp corners. As illustrated in Figure 4a, two curves can sufficiently represent a corner, either convex or concave by adopting maximum or minimum as the function F . To represent a shape with arbitrary corners, however, it requires three curves at least. For example, F (C) = min(max(C1, C2), C3) can model all corners in a shape. Therefore, we set n = 3, i.e., C = {C1, C2, C3}. Since deep models are sensitive to the permutation of guidance signals during training, we use the median function as F , which achieves sharp corners and permutational invariance to the order of these curves. Thus, a corner-related loss on F (C) could be robust to the ordering ambiguity.
Based on F (C) (median function on three curves), a typical corner O is presented in Figure 4b, where intersection of two curves divides its local space into four quadrants, i.e., from Q1 to Q4. There are always two opposite quadrants that one is inside area (i.e., Q1) where corresponding values from F (C) are 1, and the other is outside area (i.e., Q4) where corresponding values from F (C) are 0. The rest two opposite quadrants (i.e., Q2 and Q3) are equal on F (C) but different on C. For example, the values on {C1, C2, C3} corresponding to the Q2 area is (1, 0, 0), so F (C) on Q2 is 0. Then, F (C) on Q3 should be 0 as well. However, the values on {C1, C2, C3} corresponding to the Q3 region must be different from (1, 0, 0), i.e., could be (0, 1, 0) or (0, 0, 1). If F (C) on Q2 and Q3 is 1, O is a concave corner. Otherwise, O is a convex corner. Such distribution of C along the four quadrants around a corner O is referred to as corner template.
The shapes are encoded as the multi-curve representation through an implicit model. The implicit model takes sample (x, y, l) (i.e., 2D spatial location (x, y) and glyph label l), as well as embedding z that indicates the font style, and outputs three channels of SDF {d1, d2, d3}. Then, the rasterization approach discussed in Differentiable Rasterization of Distance Field converts each di to Ci (i = 1, 2, 3). Finally, the median function F (C) renders the final shape. We optimize the global shape using the final render output and (locally) supervise each corner to be locally represented as an intersection of two curves. One could use the output of [4] to train a network. However, the edge coloring approach presented in [4] has no canonical form, which prevents a neural network from learning a continuous latent space between shapes. Please note that since we focus on corner supervision to ensure sharp corners at higher resolution resampling, it is unnecessary to constraint the model by multi-channel supervision globally.
In the optimization of rendered global shape, the median operation F (C) would route the gradients to the correspondingly active value only, i.e., only update a single channel at each location of C. However, at least two of the three channels of C at a certain location need to be updated to approaching the ground truth because of the nature of the median operator. Therefore, in the training stage, we use an approximation F̂ (C) defined as the average of the median and the closest value to the median, thus two channels will be updated.

heading: 3.3 Training Details
text: We use three losses on the shape of glyph: (i) a global shape loss that captures glyph shapes globally, (ii) corner template loss that supervises intersection of curves locally to make the shape robust against resampling and editing; and (iii) Eikonal loss to adhere to true SDFs.
- Area Samples - Edge Samples - Corner Samples For global shape training, since gradients are non-zero only at the antialiasing range, i.e., edges of the shape, we sample the edges of the rasterized glyph to train the implicit model. We shape 3 × 3 neighborhoods around the anti-alias pixel, where we have a sample of one value outside the shape (i.e., 0), one value inside the shape (i.e., 1), and everything in between. Meanwhile, we sample from homogeneous areas inside and outside of the glyph, such that the model does not fit a degenerate solution. The edge/corner-aware sampling is referred to as
importance sampling (see inset). Such non-standard sampling further motivates the use of implicit models, and importance sampling would significantly reduce the computational complexity for training on higher resolution shapes as compared to the traditional training on grid images.
We measure global shape loss between the final rendering from F (C) and rasterized glyph I . We use a differential approximation F̂ (C) at training time. All the edges, area, and corner samples from the raster image are used to train the implicit model via mean square error as,
Lglobal = E (F̂ (C) − I) 2 . (3)
For local corner template loss, we first perform corner detection. A corner is defined as a local where two curves intersect at an angle less than a threshold (the threshold is 3rad or 171◦ in our experiments). For each corner, we generate the corner template as discussed in section 3.2. The template size is 7× 7 corresponding to the image size of 128× 128. The size of the corner template is scaled based on the size of the image, but the size of the sampling neighborhood for the global shape training remains the same. We densely sample the edges and corners, and sparsely sample the homogeneous areas.
To represent glyph corners as the intersection of two curves, we supervise the corner samples by the corresponding corner templates. Since the render function F (C) is invariant to the order of the SDF/raster channels, the corner template loss inherits the permutation invariance to the channel order and, in order to avoid unnecessarily constraining the network, we only supervise Q2 and Q3 of the template using the loss,
Llocal = EO∈corner samples n
∑ i=2 min j∈{2,⋯,n} (COi − Tj(O)) 2 , (4)
where O indicates a corner sample from ground truth, and n is the number of channels (i.e., n = 3 in our setting). The correspondingly predicted curves of the corner O are denoted by COi , and the corresponding corner template is T (O) that has n channels indexed by j. The gradient loss aims to constraint the output of the implicit network to resemble a distance field this is so that the implicit re-sampling is more well behaved and resembles a closed continuous shape. A special case of Eikonal partial differential equations [6] posits that the solution to d(x, y; θ) must satisfy E∣∣∇d(x, y; θ)∣∣ = 1, where d(x, y; θ) denotes a simplified implicit model parameterized by θ. Since satisfying this constraint is not completely necessary for our desired solution, we loosen it to be greater than or equal to 1 as Eq. 5, which intuitively encourages the function to be monotonic.
Lgrad = { E∣1 − ∥∇d(x, y; θ)∥2∣ if ∥∇d(x, y; θ)∥2 < 1, 0 if ∥∇d(x, y; θ)∥2 ≥ 1, (5)
where ∥ ⋅ ∥2 represents the `2-norm, and ∣ ⋅ ∣ calculates the absolute values. Finally, the total loss is L = Lglobal + αLlocal + βLgrad + γ∥z∥2, (6)
where α, β, and γ are weights to balance these terms during the training.
Training Warm-up. Since the gradients mainly fall into the anti-aliasing range, network initialization would significantly affect the convergence. To this perspective, we set the initial anti-aliasing range to be the whole image range and slowly shrink it to k ⋅w−1 during the training, where w is image width, and k = 4 in our experiments. Such warm-up helps the model converge more consistently, and the estimated SDF is more well behaved. Comparison of SDF with and without warm-up is conducted in the supplementary.

heading: 4 Experiments
text: We evaluate our method against the tasks of reconstruction, interpolation, and generation. In reconstruction and interpolation, we compare our method to ImageVAE [12], DeepSVG [2], and Im2Vec [20], while we compare to DeepSVG and Attr2Font [25] in the generation task. The metrics for evaluating the rendered glyphs are mean squared error (MSE) and soft IoU (s-IoU), which is defined as
s-IoU(I1, I2) = ∥I1I2∥1/∥(I1 + I2)∣0,1∣∥1, (7)
where I1 and I2 are the images to compare, ∥ ⋅ ∥1 denotes the `1-norm, and ∣0, 1∣ clips the values to the interval of [0,1]. To evaluate the fidelity of glyph rendering in larger scales, glyphs are rendered at the resolution of 128, 256, 512, and 1024 from each method without changing the training resolution (64 × 64).
Reconstruction and Interpolation. We compute MSE and s-IoU over the training dataset at different resolutions to quantify how different algorithms capture the input training dataset. For a fair comparison, we train all the algorithms on the same dataset used by Im2Vec [20], which consists of 12,505 images. Table 1 displays MSE and s-IoU metrics on the training set. For ImageVAE, we perform bilinear interpolation to obtain higher resolution outputs. Our method outperforms the others on s-IoU, indicating better reconstruction of the glyph shapes. ImageVAE gets higher scores on MSE because its training objective aligns with the MSE metric. However, ImageVAE would show blurry shapes in interpolation and editing as demonstrated in Figure 6a, where we achieve more continuous interpolation/latent space. Even in reconstruction, as visualized in Figure. 5, ImageVAE cannot preserve sharp boundary and corners as compared to the other methods. Since DeepSVG and Im2Vec directly output vectors, they can always render shapes with clear boundaries, but they are limited in capturing the global shapes as compared to our method. Another advantage of our method over DeepSVG and Im2Vec is that we learn a smoother latent space, achieving better performance on interpolation as demonstrated in Table 2. Even interpolating between complex shapes, as shown in Figure 6b, our method performs better than the state-of-the-art Im2Vec. In Table 2 we present the MSE and s-IOU calculated between a random interpolated glyph and its nearest neighbour in the training dataset. This helps quantify similarity between training versus generation distribution.
We have also experimented with Fourier features, Sine activation and Sawtooth activation function with a feed forward network, due to the high frequency nature of these functions the latent space learned by such a network is not continuous. Since learning a continuous latent space is essential for applications like interpolation and font family generation from a complete or partial glyph, we resorted to using a LeakyReLU based activation function. However if the users main requirement is only faithful reconstruction of the input dataset, single channel fitting with fourier features trained
using our differential rasterization function should yield similar results as multi channel fitting with LeakyReLU activation.
Generation. To generate new fonts and the corresponding glyphs, first the auto-decoder implicit model is trained with latent vector z and glyph label (i.e., one-hot encoding) concatenated to spatial locations as the input. We train on 1,000 font families, i.e., 52,000 images, and test on 100 font families. In the inference stage, given an unseen glyph, we first find the optimal latent vector (i.e., font style) that makes the rendered glyph closest to the given glyph. More specifically, fixing the glyph label based on the given glyph, its font latent vector ẑ can be obtained by minimizing the distance between the raster of the given glyph and the predicted glyph using gradient descent. With the optimal ẑ, all the other glyphs with the same font style can be generated by iterating the glyph label. Figure 7 compares the font completion results between the baselines and ours, where a glyph “A” is given with unseen font style. Our results outperform the others in terms of global shape and sharpness of boundaries and corners. In general, raster-based methods (e.g., Attr2Font) tend to generate better shape but get blurry at corners. By contrast, vector-based methods (e.g., DeepSVG) eliminate the blurry effect while difficult to achieve good global shapes. It is a dilemma of generating better global shapes or better local corners in recent works. Our method is achieving both good shapes and corners. Table 3 provides statistical results that further demonstrates the superior generation capacity of our method.
We conduct a more challenging task, i.e., glyph completion, to explore the potential generation capacity of our method. As illustrated in Figure 8, given a partial glyph, it can still recover the whole glyph, as well as other glyphs with the same font style. In addition, sharp corners are still preserved.

heading: 5 Conclusion
text: We have presented multi-implicits – a new vector representation that is easy to process with neural networks and maintains 2D shape fidelity under arbitrary resampling. The representation is learned in a locally supervised manner allowing high precision recovery of corners and curves. The proposed multi-implicits representation is extensively evaluated in various font applications, including highresolution reconstruction, font style interpolation, font family completion, and glyph completion, and demonstrates clear advantages over prior image based and curve based approaches.
Broader Impact The proposed representation has the potential to be applied to other 2D vector objects, such as icons and animations, which can empower artists’ creativity and productivity. If trained on a handwriting dataset such a method could possibly be used for emulating a person’s handwriting for forgery.

Review summaries:
This paper proposes multi-implicit neural representation for fonts given the observation that it is hard to represent complex fonts with single neural representation. The author introduces a local supervision method that can train multi-implicit functions without knowing ground truth channels. Experimental results in various settings, e.g. reconstruction, interpolation, and generation, show that it outperformed the previous methods such as rasterized representation VAE and a couple of neural vectorized representations.

This paper aims to learn a representation for rasterized fonts that allow for re-scaling, reconstruction, interpolation, and generation by preserving local features such as sharp corners or smooth curves. The proposed method composes multiple implicit representations (i.e., signed distance fields) to model complex details observed in neutral and stylistic fonts. The core idea is to represent corners as an intersection of multiple curves, enabling the construction of sharp corners in arbitrary resolutions. This assumption is further supported by a local supervision objective for the corners. In the experiments, the authors present that their model is able to preserve sharp corners. The proposed model outperforms baselines both qualitatively and quantitatively.

This paper deals with the problem of glyph generation in fontset design. Whereas previous works used raster-based or vector-based representations (and combinations of the two) to learn generative models for fonts, this paper proposes the use of what they call “multi-implicit representations”. In the proposed method, implicit functions (in the style of “NeRF”) are learned to represent subregions of a glyph by combining multiple simple shapes (eg two half-planes to create a corner). These subregion-implicit models are learned with local supervision, and a multi-Signed Distance Field model learns to compose them into the global glyph.  

The paper proposes a method for representing vector graphic fonts with multi-implicit functions. The main goal is to allow the reconstruction of fonts at any resolution without loss of discontinuities such as corners and edges. Instead of representing a font shape as one SDF and then optimising the training process against this SDF, the paper proposes to represent a shape using multi-SDFs, and then differentiably rendering a shape against this multi-SDF representation and optimizing the input shape against this rendering. This allows the method to bypass the need for collecting a vast amount of vector graphics data for training.  The paper shows that this kind of multi-SDF representation helps in preserving discontinuities at corners and edges.