A Multi-Implicit Neural Representation for Fonts
Authors: Pradyumna Reddy
Zhifei Zhang
Zhaowen Wang
Matthew Fisher
Hailin Jin
Niloy Mitra
Venue: NeurIPS 2021 Poster (2021)
Keywords: Font Generation
Generative Neural Networks
Implicit Neural Networks

Abstract: Fonts are ubiquitous across documents and come in a variety of styles.  They are either represented in a native vector format or rasterized to produce fixed resolution images. In the first case, the non-standard representation prevents benefiting from latest network architectures for neural representations; while, in the latter case, the rasterized representation, when encoded via networks, results in loss of data fidelity, as font-specific discontinuities like edges and corners are difficult to represent using neural networks. Based on the observation that complex fonts can be represented by a superposition of a set of simpler occupancy functions, we introduce multi-implicits to represent fonts as a permutation-invariant set of learned implict functions, without losing features (e.g., edges and corners). However, while multi-implicits locally preserve font features, obtaining supervision in the form of ground truth multi-channel signals is a problem in itself. Instead, we propose how to train such a representation with only local  supervision, while the proposed neural architecture directly finds globally consistent multi-implicits for font families. We extensively evaluate the proposed representation for various tasks including reconstruction, interpolation, and synthesis to demonstrate clear advantages with existing alternatives. Additionally, the representation naturally enables glyph completion, wherein a single characteristic font is used to synthesize a whole font family in the target style. 


A Multi-Implicit Neural Representation for Fonts
Pradyumna Reddy
Zhifei Zhang 2
Zhaowen Wang"
Matthew Fisher
Hailin Jin
Niloy J. Mitra 1.2
University College London
2 Adobe Research
Abstract
Fonts are ubiquitous across documents and come in a variety of styles. They are either represented in native vector format or rasterized to produce fixed resolution images. In the first case, the non-standard representation prevents benefiting from latest network architectures for neural representations; while, in the latter case, the rasterized representation, when encoded via networks, results in loss data fidelity, as font-specific discontinuities like edges and corners are difficult to represent using neural networks. Based on the observation that complex fonts can be represented by superposition of set of simpler occupancy functions, introduce inltiiiimpiicits to represent fonts as cormuuaationinvarrin set learned implicit functions, without losing features (c.g.. edges and corners). However, while multi-implicits locally preserve font features, obtaining supervision in the form of ground truth multi-channel signals is problem in itself. Instead, propose how to train such a representation with only local supervision, while the proposed neural architecture directly finds globally consistent multi-implieets for font families. We extensively evaluate the proposed representation for various tasks including reconstruction, interpolation, and synthesis demonstrate clear advantages with existing alternatives. Additionally, the representation naturally enables glyph completion, wherein single characteristic font is used synthesize a whole font family in the target style.
1 Introduction
Fonts constitute the vast majority of documents. They come in variety of styles spanning range of topologies representing a mixture of smooth curves and sharp features. Although fonts vary significantly across families, they remain stylistic coherent across the different alphabets/symbols inside any chosen font family.
Fonts are most commonly stored in vector form (e.g., a collection of spline curves) that is compact, efficient, and can be resampled at arbitrary resolutions without loss of features. This specialized representation, however, prevents the adaptation of many deep learning setups optimized for regular structures (e.g., image grids). order avoid this problem, custom deep learning architectures have been developed for directly producing vector output but they typically require access ground truth vector data for training. This is problematic: first, collecting sufficient volume of vector data for training non trivial; and second, vector representations represented by different sequence of vectors). which in turn requires hand-coded network parameters to account for varying number of vector instructions.
not canonical (i.e., same fonts can be
alternate approach rasterize vectorized fonts and simply treat them as images. While this readily allows using image-based deep learning methods, the approach inherits problems of discretized
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Inpux
Reconstruction
IAE
Figure 1: Multi-implicit neural representation for high fidelity font reconstruction and generation. Note that while ours perform similar to ImageVAE at lower/training resolution, the advantage of ours become clear when we test at higher resolution (e.g, how corners continue to be preserved).
representations leading to aliasing artifacts and loss of sharp features. Further, the resultant images are optimized for particular resolutions and cannot be resampled without introducing additional artifacts.
Recently, implicit representation has emerged as an attractive representation for deep learning as, once trained, they can resampled at different resolutions without introducing artifacts. Unfortunately. implicit representations (e.g., signed distance fields) for fonts are often too complex to be represented accurately by neural networks. As a results, although deep implicits work for simple fonts, they can fail retain characteristic features (i.e., edges and corners) for complex fonts.
Drawing inspiration from multi-channel SDFs [4] we observe that complex fonts can be expressed composition of multiple simple regions. For example, local corner can be represented suitable composition of two half-planes, each of which can easily be individually encoded as deep implicit functions. We build this idea by hypothesizing that complex fonts can also be encoded as suitable composition of global implicit functions. We call such representation to be multi-implicit neural representation, each (global) implicit function is neurally encoded.
Thus, multi-implicits provide a simple representation that is amenable for processing by neural networks and the output fidelity remains comparable, even un- der resampling, vector representations without los- ing edge or corner features. A remaining challenge how supervise such network there is dataset with reference multi-implicits that be directly used. In this paper, we present network structure and train- ing procedure that allow multi-implicts to be trained using only local supervision. We describe how to extract necessary local supervision from vector input and adaptively obtain training information for the multi-implicits.
Origiaa Shaps
Ressoupling
ShF Rosseppinng
ReserplagOen)
Figure 2: Corner preserving capability of dif- ferent sampling methods.
We extensively evaluated the proposed multi-implicits representation for various tasks including reconstruction, interpolation, and synthesis demonstrate clear advantages over several existing state-of-the-ard alternatives. Additionally, the representation naturally enables glyph completion, wherein a single characteristic font glyph is used to synthesize whole font family in consistent style.
2 Related Work
Raster-based representation. One of the most intuitive representation of shapes raster, i.e.. representing a 2D shape by pixels or 3D shape by voxels, which provides the grid-format data that perfectly compatible with (regular) CNN models. Hence has acted catalyst for many deep learning based methods for semantic editing of raster-basee based shapes. A general idea is to learn an encoding-decoding model and then manipulate the shape in the latent space. Sharing a similar spirit, engaged in the raster-based generation and attribute transfer. 25] narrowed down the scope to fonts specifically, which focus more on the shape instead of texture. While the raster-based representation has shown strong semantic editability as incorporated
many generative models 21 5 15

---

with deep models, it is still limited by its intrinsic resolution. Since image super-resolution is an ill-posed problem, details cannot be fully recovered after upscaling.
Deep learning based vector graphs. Vector graphics (e.g., SVG) have been widely adopted as scalable representation. Typically, it is constructed by sequences of Bézier eurves, which difficult to be modeled by traditional CNN models since the diverse sequence length across different shapes, well as diverse types and numbers of Bézier curves for the similar or even the same shape. Therefore, RNN-based models (e.g., LSTM) are commonly adopted in recent works [7 2 to learn the dynamic sequence of curves. Although the sequential modeling could achieve semantic editing and/or interpolation between shapes, it is still lagging behind CNN-based models (raster- based methods) in terms of reconstruction accuracy because has model much longer temporal dependency. Also, there is no specific attention given to features like edges and corners.
Transformation from raster to vector. Instead of directly modeling raster vector, idea of achieving both scalability and editability is to model the transfer from raster to vector [20] (we will not consider vectorization methods that purely transfer images vectors). Such an approach inherits the advantages of raster-based methods and supports easy editability. In addition, they output scalable vectors directly. However, this method would be still limited by the raster resolution, i.e., the output vector graph cannot capture enough details since the corresponding raster input may have already lost those finer details. Another challenge for such methods is the high complexity of the shape, e.g.. interpolation between shapes with different topologies leads to undesirable intermediate shapes.
Deep implicit representation. Deep implicit functions 110
18 have achieved great success in shape representation. They take advantage of deep learning techniques to fit an implicit function, which provides continuous representation breaking the grid limitation of a raster domain. In addition, deep implicit functions model shapes spatially instead of modeling sequentially as afore- mentioned in deep learning based vector graphs. Therefore. deep implicit functions could preserve the editability like raster-based representation and potentially achieves scalable representation like vectors. Unfortunately, existing works seldom explore the scalability and fidelity of shapes in their representations, especially during editing, interpolation, and upscaling. For instance, sharp corners always suffer from editing and scaling. This particularly problematic in the domain of fonts, and our work addresses this limitation via the proposed multi- multi-implicit representations.
3 Methods
We represent fonts as the composition of multiple global implicit functions. An implicit representation has two key advantages: it allows for rendering at arbitrary resolution; it can be locally supervised in characteristic areas such as sharp edges and corners. Unlike single implicit function, our multi-implics representation can faithfully reconstruct sharp features with low reconstruction error (Figure 2)
We train generative neu- ral network, as shown in the inset, that models fonts us- ing this representation. In- stead of directly learning the inside-outside status of the font image [3), we pre- dict a set of distance fields of 2D shapes. Their com- position is then fed into differentiable rasterizer to produce the final image. We will show in Section 4 how this generative model enables a diverse set of font reconstruction and editing opera- tions. Section details differentiable rasterization of 2D distance fields, and corresponding corner preservation is discussed Section 3.2 Finally, training losses and details are given in Section 3.3
Multi-Curve Representation
Importance Sampling
Implicit Model
Multi-SDF
Differentiable Rusterization
Image Rendering
Lg regularization
Corner
Global Shape Loss
Gradient Loss
Template Loss
3.1 Differentiable Rasterization of Distance Field
As aforementioned,, we use the signed distance field (SDF) to model 2D shapes. The common supervisions for SDF are SDF labels from the ground truth curves or distance transform on the silhouette [1]. Figure compares the results between training with SDF and training with raster,
3

---

Ground Truth
Odl SDF Loe
SDP Raaer Insgg Low
Rake Image Low(Our)
Figure 3:
Reconstruction with difffrent training signals. Training using SDF (2nd column) does not ensure good reconstruction in raster domain, while training with both SDF and rasterized image (3rd column) and only rasterized SDF (ours) yields smoother boundary.
where training only the ground truth SDF does not always result in good raster image in terms of boundary smoothness. In contrast, training with the supervision of raster yields smoother boundary. Therefore, will draw supervision rasterized SDF to achieve better shape.
Sharing the spirit from vector graphics rasterization works I vector graphics rendering by analytically approximating the point to curve distance as,
24], we simplify general
1(x,0) v) K min (z, v) (z. ifF
(1)
1
if d
K(d)= = k (<4)) 4 1/2 +
( - -3(4)
- 7 s d s ?
(2)
0
if d -7i
where dz( y) indicates an SDF of distance from pixel (z, y) center the closest point on the i-th curve corresponding the font F, and K denotes a function that approximates the opacity based on the distance value. There are many works for estimating di (x, y) from scene parameters but most of them are constrained by the choice of the parameterization. In this paper, we model each da y) using an implicit neural network. The function g models the opatially-varying texture of the shape. For solid fonts, we set 1. In the supplemental we show examples of textured fonts where we make use of spatially varying g. In radially symmetric continuous filter that satisfies the constraint &(1) O and k(-1) 1. In our work, we approximate k(-) using a parabolic kernel similar to [3] Note that the rasterization function [(2, y) has non-zero gradients only if (I., y) falls inside of the anti-aliasing range. We use progressively decreasing anti-aliasing range strategy for better convergence and fidelity (see Sec. 3.3
function K, y is the anti-alias range, and the kernel is
3.2 Multi-Curve Representation for Sharp Corners
We will lose details like sharp corners when upscaling bitmaps or sign dis- tance functions. Resampling plicit model that encodes the pixel values signed distance values of a shape similarly suffers from blurry corners. An brute force solution is to train the implicit model with ex- tremely high-resolution images, but this would drastically increase the bur- den of training, and still limited by the training resolution. Rather than directly modeling corners, we represent corners as intersection multiple curves (e.g., lines or parabolas), drawing inspiration from traditional representations (9] Note that even though these individual sub-curves may be smoothed after encoded by deep model, the sharpness of their inter- section will be preserved. With this insight, construct sharp corners from multiple smooth curves
im-
Insad: Oussli
F max(C, Co)
F F-min(G, Cj)
(a) Possible corner preserving shapes repre- sented with n=2.
(b) Coreer tem- plate.
Figure 4: Intersection of eurves encode concave and convex corners.

---

predicted by the implicit model. More specifically, the implicit model is designed to predict multiple SDFs (and rasterization of distance fields), each of which carries smooth curves/shapes decoupled from corners and edges as illustrated in Figure 4a
Assume shape F is represented by a set of curves C = C2,,C2." C,. C..), where C, is binary map indicating whether pixel is inside (i.e., 1) or outside (i.e.. 0) the n-th curve, like the example in Figure 4a In our scenario, C,, [dd,,, where d, the n-th SDF channel estimated by the implicit model. Then, a function over all curves F(C) will fuse those curves to reconstruct the shape, preserving sharp corners. As illustrated in Figure 4a two curves can sufficiently represent corner, either convex or concave by adopting maximum or minimum as the function F. To represent a shape with arbitrary corners, however, it requires three curves at least. For example, F(C) min(max(C C2), C3) can model all corners in a shape. Therefore, we set = 3, i.c., C (C C2. C3). Since deep models are sensitive to the permutation of guidance signals during training, we use the median function as F, which achieves sharp corners and permutational invariance to the order of these curves. Thus, a corner-related loss on F(C) could be robust to the ordering ambiguity.
Based on F(C) (median function on three curves), a typical corner O is presented in Figure 4b) where intersection of two curves divides its local space into four quadrants, i.e., from Q1 to Qa. There are always two opposite quadrants that one is inside area (i.e., Q1) ) where corresponding values from F(C) are 1, and the other is outside area (i.c., Q4) where corresponding values from F(C) are O. The rest two opposite quadrants (i.e.. Q2 and Q3) are equal on F(C) but different on C. For example, the values on {C1. C22 C2}) corresponding to the Q2 area is (1, 0, 0), so F(C) on Q2 is O. Then, F(C) on Q3 should be 0 as well. However, the values on {C1, C2 C2) corresponding to the Q3 region must be different from (1, O, 0), i.e., could be (0, 1. 0) or (0, 0, 1). If F(C) on Q2 and Q3 is 1, O is concave corner. Otherwise, O is a convex corner. Such distribution of C along the four quadrants around a corner O is referred to as corner template.
The shapes are encoded as the multi-curve representation through an implicit model. The implicit model takes sample (z. y. 1) (i.e., 2D spatial location (z. y) and glyph label 1), as well as embedding that indicates the font style, and outputs three channels of SDF (d, d2, dz). Then, the rasterization approach discussed in Differentiable Rasterization of Distance Field converts each d: to C, (i 1, 3). Finally, the median function F(C) renders the final shape. We optimize the global shape using the final render output and (locally) supervise each corner to be locally represented as an intersection of two curves. One could use the output of 4 to train a network. However, the edge coloring approach presented in 44] has no canonical form, which prevents neural network from learning continuous latent space between shapes. Please note that since we focus on corner supervision to ensure sharp corners at higher resolution resampling, it is unnecessary to constraint the model by multi-channel supervision globally.
In the optimization of rendered global shape, the median operation F(C) would route the gradients to the correspondingly active value only, i.c., only update a single channel at each location of C. However, at least two of the three channels of C at a certain location need to be updated to approaching the ground truth because of the nature of the median operator. Therefore, in the training stage, we use an approximation F(C) defined as the average of the median and the closest value to the median, thus two channels will be updated.
3.3 Training Details
We use three losses on the shape of glyph: (i) global shape loss that captures glyph shapes globally, (ii) corner template loss that supervises intersection of curves locally to make the shape robust against resampling and editing; and (ii) Eikonal loss to adhere to true SDFs.
For global shape training, since gradients are non-zero only at the anti- aliasing range, i.e., edges of the shape, we sample the edges of the rasterized glyph to train the implicit model. We shape 3 neigh- borhoods around the anti-alias pixel, where we have sample of one value outside the shape (i.e., 0), one value inside the shape (i.e., 1), and everything in between. Meanwhile, we sample from homogeneous areas inside and outside of the glyph, such that the model does not fit a degenerate solution. The edge/corner- aware sampling is referred to
5

---

importance sampling (see inset). Such non-standard sampling further motivates the use of implicit models, and importance sampling would significantly reduce the computational complexity for training on higher resolution shapes as compared to the traditional training on grid images.
We measure global shape loss between the final rendering from F(C) and rasterized glyph T. We use differential approximation FCC) at training time. All the edges, area, and corner samples from the raster image are used to train the implicit model via mean square error as,
Litobal = E [ -))
For local corner template loss, we first perform corner detection. A corner is defined as a local where two curves intersect at angle less than threshold (the threshold is 3rad or 171" in our experiments). For each corner, we generate the corner template as discussed in section 3.2 The template size is 7 7 corresponding
the image size of 128 x 128. The size of the corner template is scaled based on the size of the image, but the size of the sampling neighborhood for the global shape training remains the same. We densely sample the edges and corners, and sparsely sample the homogeneous
To represent glyph corners as the intersection of two curves, we supervise the corner samples by the corresponding corner templates. Since the render function F(C) is invariant to the order of the SDF/raster channels, the corner template loss inherits the permutation invariance to the channel order and, in order to avoid unnecessarily constraining the network, only supervise Q2 and Q3 of the template using the loss,
(c, -T,10))) je(
Locct Oecomer samples
min
(4)
where indicates a corner sample from ground truth, and n is the number of channels (i.e., n = in our setting). The correspondingly predicted curves of the corner O are denoted by cP, and the corresponding corner template is T(O) that has n channels indexed by j.
The gradient loss aims to constraint the output of the implicit network to resemble distance field this is so that the implicit sampling is more well behaved and resembles a closed continuous shape. special case of Eikonal partial differential equations [6] posits that the solution to d(z, y/; 0) must satisfy E//V,( y; l)ll 1, where d(z; ys 0) denotes simplified implicit model parameterized by A. Since satisfying this constraint is not completely necessary for our desired solution, we loosen it to be greater than equal to as Eq. which intuitively encourages the function be monotonic.
E [[Vd(z, y 222l
if Vd(a, a y: 0)|l2 < 1, if Vd[a, y: 0)ll 2 1
C groa
where || ll2 represents the fy-norm, and calculates the absolute values. Finally, the total loss is Lygoboo + aooocaal (L 7||=112-
where a, 8, and weights to balance these terms during the training.
Training Warm-up. Since the gradients mainly fall into the anti-aliasing range, network initializa- tion would significantly affect the convergence. To this perspective, set the initial anti- anti-aliasing range be the whole image range and slowly shrink it k-w during the training, where w is image width, and & = 4 in our experiments. Such warm-up helps the model converge more consistently, and the estimated SDF is more well behaved. Comparison of SDF with and without warm-up is conducted in the supplementary.
4 Experiments
We evaluate our method against the tasks of reconstruction, interpolation, and generation. In reconstruction and interpolation, we compare our method to ImageVAE [12], DeepSVG [2] and Im2Vec (20), while we compare to DeepSVG and Aur2Font (25) in the generation task. The metries for evaluating the rendered glyphs are mean squared emror (MSE) and soft loU (s-IoU), which is defined as
s-LoU(41, 12) = ||4422l//1144 + 12)00.1111

---

Table 1: Comparison with baselines on reconstructing training samples different resolutions. In
training. we use resolution of 64 x 64. In testing, to achieve target resolution, we bilinearly upsample ImageVAI VAE output from 64, rasterize vector from DeepSVG and Im2Vec, and directly query ours.
MSE
s-loU f
Methods
128
256
512
1024
128
256
512
1024
ImageVAr VAI
.0072
.0120
.0160
.0186
.8522
.8466
.8482
.8494
Ve
.1022
.1081
.1108
.1121
.3073
.3124
.3162
.3164
Im2 Vec
.0435
.0518
.0557
.0571
.7279
.7293
.7294
.7294
Ours
.0118
.0170
.0201
.0218
.8750
.8978
.9035
.9049
where I and 1 are the images compare. || ||1 denotes the l -norm, and Jo, 1| clips the values to the interval of [0, ]. To evaluate the fidelity of glyph rendering in larger scales, glyphs are rendered at the resolution of 128, 256, 512, and 1024 from each method without changing the training resolution (64 x 64).
Input
ImageVAls
DeepSVG
Input
ImageVitl
DeepSVGG
8
8
8
8
1
Figure 5: Reconstruction examples (baseline vs. ours) with zoom-in box highlighting corners. We have vectorized the zero-level-set of the SDF output in piece-wise linear way. Please use digital-7oom to take closer look at the difference in reconstruction quality.
Reconstruction and Interpolation. We compute MSE and s-IoU over the training dataset at different resolutions to quantify how different algorithms capture the input training dataset. For fair comparison, we train all the algorithms on the same dataset used by Im2Vec [20], which consists of 12.505 images. Table [ displays MSE and s-IoU metrics on the training set. For Image VAE, we perform bilinear interpolation to obtain higher resolution outputs. Our method outperforms the others on s-IOU, indicating better reconstruction of the glyph shapes. Image VAE gets higher scores on MSE because its training objective aligns with the MSE metric. However, Image VAE would show blurry shapes interpolation and editing as demonstrated in Figure 6a| where we achieve more continuous interpolation/latenn space. Even in reconstruction, visualized in Figure. [5] Image VAE cannot preserve sharp boundary and corners as compared the other methods. Since DeepSVG and Im2Vec directly output vectors, they can always render shapes with clear boundaries, but they are limited capturing the global shapes as compared to our method. Another advantage of our method over DeepS VG and Im2Vec is that we learn a smoother latent space, achieving better performance on interpolation as demonstrated in Table 2 Even interpolating between complex shapes, as shown in Figure 6b] our method performs better than the state-of-the- Im2Vec. In Table 2 we present the MSE and S-OOU calculated between random interpolated glyph and its nearest neighbour in the training dataset. This helps quantify similarity between training versus generation distribution.
We have also experimented with Fourier features, Sine activation and Sawtooth activation function with feed forward network, due to the high frequency nature of these functions the latent space learned by such a network is not continuous. Since learning a continuous latent space is essential for applications like interpolation and font family generation from complete or partial glyph, we resorted to using LeakyRel. JU based activation function. However if the users main requirement only faithful reconstruction of the input dataset, single channel fitting with fourier features trained

---

using our differential rasterization function should yield similar results multi channel fitting with LeakyReLU activation.
Table 2: Comparison with baselines on interpolation at different resolutions.
MSE
s-IoU
Methods
128
256
512
1024
128
256
512
1024
VAI
.0181
.0183
.0185
.0185
.7715
.7721
.7731
.7734
De
.0544
.0556
.0569
.0575
.6337
.6347
.6365
.6372
Ve2
.0434
.0445
.0433
.0473
.7213
.7218
.7232
.7238
Ours
.0279
.0297
.0316
.0343
.8881
.1844
.8222
.8234
Sourcee
Target
Soused
Target
A
4
Interpolation
Imopplltiion
4
1n2Vee
4
4
4
4
4 4
Comparison of all methods.
(b) A challenging example where baselines fail.
Figure 6: Comparison of interpolation between two random font styles. We color different curves in the vector output to highlight details.
Generation. To generate new fonts and the corresponding glyphs, first the auto-decoder implicit model is trained with latent vector z and glyph label (i.e., one-hot encoding) concatenated to spatial locations as the input. We train on 1,000 font families, i.e., 52,000 images, and test on 100 font families. In the inference stage, given unseen glyph, we first find the optimal latent vector (i.e., font style) that makes the rendered glyph closest to the given glyph. More specifically, fixing the glyph label based the given glyph, its font latent vector å can be obtained by minimizing the distance between raster of the given glyph and the predicted glyph using gradient descent. With the optimal 2, all the other glyphs with the same font style can be generated by iterating the glyph label. Figure compares the font completion results between the baselines and ours, where glyph "A" is given with unseen font style. Our results outperform the others in terms of global shape and sharpness of boundaries and corners. In general, raster-based methods (c.g.. Attr2Font) tend to generate better shape but get blurry corners. By contrast, vector-based methods (e.g., DeepSVG) eliminate the blurry effect while difficult to achieve good global shapes. It is dilemma of generating better global shapes better local corners in recent works. Our method is achieving both good shapes and corners. Table provides statistical results that further demonstrates the superior generation capacity of our method.
We conduct a more challenging task, i.e., glyph completion, to explore the potential generation capacity of our method. As illustrated in Figure 8 given partial glyph, it can still recover the whole glyph, as well as other glyphs with the same font style. In addition, sharp corners are still preserved.
Table 3: Comparison with baselines on font generation task at different resolutions.
MSE J
s-loU f
Methods
128
256
512
1024
128
256
512
1024
De
.2577
.2788
.2854
.2911
.3584
.3613
.3651
.3672
Attr2Font
.2044
.2211
.2481
.2563
.6204
.6451
.6523
.6560
Ours
.0946
.1027
.1065
.1083
.8429
.8422
.8469
.8471
8

---

Input
E APORSTUVWER
DeepSVG
A O J Q h 1 mn D R
A
A
A A E C I J K MM OP C R S T VWXYZ a c e e f i
Attr2Font
j k
I m mn n y z
opqr r S t u V W X
A AA I J K M OPQRSTU V W X y Z a b e f g h
Ours
i
j k I m n
opqr S t u
W X y Z
Figure 7: Font completion examples (baseline ours) with zoom-in boxes highlighting the corners.
In)
SKEDEFGHIJKLINN uVwxyz Z
Figure 8: Glyph completion example. Given a partial glyph unseen in the training set, our method can complete the given glyph and other glyphs with the same font style. The zoom-in boxes highlight the corners. The mask region in the input is ignored during optimizing the latent vector E.
5 Conclusion
We have presented multi-implice new vector representation that is easy to process with neural networks and maintains 2D shape fidelity under arbitrary resampling. The representation is learned in locally supervised manner allowing high precision recovery of corners and curves. The proposed multi-implicits representation is extensively evaluated in various font applications, including high- resolution reconstruction, font style interpolation, font family completion, and glyph completion, and demonstrates clear advantages over prior image based and curve based approaches.
Broader Impact The proposed representation has the potential to be applied to other 2D vector objects, such as icons and animations, which can empower artists' creativity and productivity. If trained on handwriting dataset such a method could possibly be used for emulating person's 's handwriting for forgery.

---

References
[1] S. Azadi, M. Fisher, V. G. Kim, Z. Wang, B. Shechtman, and T. Darrell. Multi-content gan for few-shot font style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7564-7573, 2018.
A. Carlier, M. Danelljan, Alahi, and Timofte. Deepsvg: A hierarchical generative network for vector graphies animation. arXiv preprint arXiv:2007.11300 20000007.111., 2020.
[3] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5939 5948, 2019.
[4] V. Chlumský, Sloup, and Simeček. Improved corners with multi-channel signed distance fields. In Computer Graphics Forum, volume 37, pages 273-287, Wiley Online Library, 2018.
[5] Y. Choi, M. Choi, M. Kim, J.W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks multi-domain image-to-image translation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018.
[6] Crandall and P.-L. Lions. Viscosity solutions of hamilton-jacobe equations. Transactions of the American mathematical sociery, 277(1):1 42, 1983.
A. Fabris and A. R. Forrest. Antialiasing of curves by discrete pre-filtering In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 317-326, 1997.
[8] K. Genova, F. Cole, A. Sud, Sarna, and T. Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857-4866. 2020.
C. Green. Improved alpha-teste magnification for vector textures and special effects. In ACM SIGGRAPH 2007 courses, pages 1-18,27 2007.
[10] T. Groueix, M. Fisher. V. G. Kim, B. C. Russell, and M. Aubry. A papier-måché approach learning 3d surface generation. Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2 216-224, 2018.
[11] S. Gupta R. F. Sproull. Filtering edges for gray-scale displays. ACM SIGGRAPH Computer Graphics. 15(3):1-5, 1981.
[12] D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6113 2013.
[13] T.-M. Li. M. Lukáč, G. Michaël, and J. Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning, ACM Trans, Graph. (Proc. SIGGRAPH Asia), 193:11993: 2020.
[14] C.-H. Lin, C. Wang. and S. Lucey. Sdf-sm: Learning signed distance 3d object reconstruction from static images. In Advonces in Neural Information Processing Systems (NeurIPS), 2020.
[15] M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo, and S. Wen. Stgan: A unified selective transfer network for arbitrary image attribute editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3673-3682, 2019.
[16] C. Loop and J. Blian. Resolution independent curve rendering using programmable graphics hardware. In ACM SIGGRAPH 2005 Papers, pages 1000-1009, 2005.
[17] R. G. Lopes, D. Ha, D. Eck, and J. Shlens. A learned representation for scalable vector graphics. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 7930-7939, 2019.
[18] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 421. Springer, 2020.
[19] D. Nehab and H. Hoppe. Random-access rendering of general vector graphics. ACM Transactions on Graphics (TOG), 27(5):1-10. 2008
[20] P. Reddy, M. Gharbi, M. Lukac, and N. J. Mitra. Im2vec: Synthesizing vector graphics without vector supervision. arXiv preprini urXiv:2102.02798 2021.
[21] A. Sinha, A. Unmesh, Q Huang, and K. Ramani. Surfnet: Generating 3d shape surfaces using deep residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 6040 00496 2017.
10

---

[22] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[23] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and
S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. 2021.
[24] K. Turkowski. Anti-aliasing through the use of coordinate transformations. ACM Trunsactions on Graphics (TOG), 133):215-234, 1982.
[25] 7. L.. Yizhi Wang*, Yue Gao*. Attribute2font: Creating fonts you want from attributes. ACM Trans. Graph., 2020.


Reviews:
[{"review_id": "iMFhv8xW0uW", "reviewer": "Reviewer_Fqtj", "summary": "The paper proposes a method for representing vector graphic fonts with multi-implicit functions.\nThe main goal is to allow the reconstruction of fonts at any resolution without loss of discontinuities such as corners and edges.\nInstead of representing a font shape as one SDF and then optimising the training process against this SDF, the paper proposes to represent a shape using multi-SDFs, and then differentiably rendering a shape against this multi-SDF representation and optimizing the input shape against this rendering. This allows the method to bypass the need for collecting a vast amount of vector graphics data for training. \nThe paper shows that this kind of multi-SDF representation helps in preserving discontinuities at corners and edges.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe problem being addressed in the paper is an important one. The proposed approach of multi-SDF representation of fonts is interesting and helpful for the problem (as evidenced by the experiments in the paper).\n\nQuality: \nThe paper is a good read. It is well articulated and the goals are clearly stated. The paper compares exhaustively to prior works (both image-based and curve-based approaches) on different tasks such as high-resolution reconstruction, font style interpolation, font family completion, and glyph completion. Important ablation on the need for multi-SDF's to recover/preserve sharp corners when rendering fonts in high-resolution has been demonstrated using two resampling techniques -- implicit resampling (which densely resamples the implicit filed) and bilateral resampling (which upsamples the SDF prediction from the implicit model to the target resolution). The experiments show the effectiveness of the proposed approach in modeling vector graphic fonts, both in absolute terms and relatively.\n\nClarity: \nExcept for a few typos, the paper is well-written.\nL36 - \"resultant\" to \"resulting\"\nL62 - \"evaluated\" to \"evaluate\"\nL97 - \"as mentioned above\" (instead of \"as aforementioned\")\nL124 - same comment as above\n\nSignificance:\nI think the problem being addressed is getting attention from the community for the last year. The paper follows this trend and makes a modest contribution to this line of research (which is okay).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have made a comment about the positive societal aspect of the proposed approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hrs"}, "extra_text": {}}, {"review_id": "URw0br6MVPV", "reviewer": "Reviewer_Ex1G", "summary": "This paper deals with the problem of glyph generation in fontset design. Whereas previous works used raster-based or vector-based representations (and combinations of the two) to learn generative models for fonts, this paper proposes the use of what they call “multi-implicit representations”. In the proposed method, implicit functions (in the style of “NeRF”) are learned to represent subregions of a glyph by combining multiple simple shapes (eg two half-planes to create a corner). These subregion-implicit models are learned with local supervision, and a multi-Signed Distance Field model learns to compose them into the global glyph. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Major Comments\nThe paper presents impressive results, including great handling of heavily stylized fonts (Fig 6).\n\nThe method requires differential rendering, as well as sampling from the individual regions. Can the authors comment on the computational complexity of training this model? Does the individual-glyph training complicate the computational expense further? Is sampling slow, since it has to sample the subregions and compose them? Does sampling scale well with resolution?\n\nI am unsure on the utility of this method for designers. The output of the model is a combination of SDFs which need to be rendered in a custom renderer (within the model), and not a set of commands that can be manipulated in traditional font design software. This is in contrast with the papers that use vector graphics directly, which output a standard format. If this is true I believe it’s important to discuss the method’s limitations, including appropriate citations of methods that don’t have this limitation.\n\nNeRF is trained for a single scene. Is that also true for each glyph/font? It seems like the paper uses a z latent space that was trained with the method in [17]. If that’s correct, it might be worth citing it on line 280.\n\n# Minor Comments\nHow does it compare to “Learning implicit glyph shape representation?” I realize this is a new paper, but I’d be interested in reading a discussion on the differences of the two approaches\nhttps://arxiv.org/abs/2106.08573v1\n\nI am not as familiar with SDFs and implicit models as I am with font-generation models, so I defer the review of technical aspects of the implicit models to other reviewers\n\nOverall, the method is really interesting and the results seem really good. If my major comments are resolved I would consider increasing the score by 1.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, although I discuss some missing limitations above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ETh08rP0mlg", "reviewer": "Reviewer_iAuP", "summary": "This paper proposes multi-implicit neural representation for fonts given the observation that it is hard to represent complex fonts with single neural representation. The author introduces a local supervision method that can train multi-implicit functions without knowing ground truth channels. Experimental results in various settings, e.g. reconstruction, interpolation, and generation, show that it outperformed the previous methods such as rasterized representation VAE and a couple of neural vectorized representations.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. My main concern is that there is no clear experimental evidence that ‘multi-implicit’ functions are really necessary. Based on recent successful applications of neural implicit representations, e,g, NeRF and SIREN for 2d images, 3d shapes, etc, I do not see the reason why single implicit representation is not capable of generating complex fonts, and the author did not present the comparison against those. They mentioned the reason was SIREN and fourier feature papers do not learn a continuous latent space in the supplementary (L11-13). I do not think this is a valid reason, and in my opinion, it is very straightforward to just apply fourier features and sine activation function in the proposed method.\n\n2. In my opinion, the proposed training techniques, such as local supervision, are specifically designed for this particular task (edge and corner aware sampling). Although it might not be a good reason to underestimate the work, but I think it is a limitation in terms of broader impact.\n\n3. I encourage the authors to show ‘multi-implicit’ channels qualitative results in the main text. I could find a couple of examples in the supplementary, but it would also be great if the authors could explain more details about the results.\n\n4. Multi-curve representation seems to be an interesting approach, but as I mentioned earlier, I do not think there are experimental results that support the effectiveness of the proposed method.\n\n[minor comments]\n1. L276-277: Did you train like GAN? random sampling ‘z’ during training? I think more explanation is required.\n2. L213: edge/corner-award sampling -> edge/corner-aware sampling?\n3. L186: what would be the format of glyph label ‘l’? it was mentioned in L276, but better to explain when it shows up for the first time.\n4. How does interpolation work in experiment sections? linear interpolation between each ‘z’? then how did you get corresponding ‘z’ given the image.\n\n[Update]\nPost-Rebuttal: change the score 3 -> 4", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Author addressed the limitations in the supplementary, my comments and suggestions are included in the main review.\n\nI do not see any negative societal impact from this submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BCvprWza9lW", "reviewer": "Reviewer_DnKK", "summary": "This paper aims to learn a representation for rasterized fonts that allow for re-scaling, reconstruction, interpolation, and generation by preserving local features such as sharp corners or smooth curves. The proposed method composes multiple implicit representations (i.e., signed distance fields) to model complex details observed in neutral and stylistic fonts. The core idea is to represent corners as an intersection of multiple curves, enabling the construction of sharp corners in arbitrary resolutions. This assumption is further supported by a local supervision objective for the corners. In the experiments, the authors present that their model is able to preserve sharp corners. The proposed model outperforms baselines both qualitatively and quantitatively.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThe idea of representing fonts as a multi-channel signed distance field (SDF) is proposed in [4]. This paper incorporates it into learned SDFs by using the recent techniques in neural implicit surfaces. While the core idea is not novel, this combination enables learning canonical representations across several font families and hence allows for interesting applications such as super-resolution, interpolation, and exemplar-based font generation. \n\n**Quality**\nThe paper motivates the problem and the proposed method well. The paper focuses on learning a representation that can preserve the sharp corners in the downstream tasks. At the first glance, it does not sound like a challenging problem. The authors are able to illustrate the problem in the existing methods. The proposed approach for this problem is technically sound and its formulation seems correct. Experiments provide enough evidence showing the proposed method’s good performance compared to the selected baselines. The font completion results in Fig. 7 and the textured font results provided in the supplementary look impressive.\n\nIn my opinion, the main limitation of the submission is the baselines. The authors focus on font models and ignore other relevant baselines which leave some of the claims unsupported. Several SDF methods such as DeepSDF [1] and SIREN [2] could be used in the baselines to provide evidence for why multi-channel SDF is necessary. SDFs are very powerful and shown to be very effective in modeling complex 3D shapes. However, the authors claim otherwise in #41, “… implicit representations (e.g., signed distance fields) for fonts are often too complex to be represented accurately …”. I think the DeepSDF method could be a perfect baseline for this task. I am also wondering why the actual multi-channel SDF [4] is not considered in the reconstruction experiments. This would provide a piece of evidence for using a learning-based SDF.\nIn the ablations, it would be useful to see the contribution of design choices namely the corner loss, the sampling strategy, and using three channels instead of one (i.e., standard SDF with the corner loss and sampling).\n\nSome claims are requiring further discussion or evidence. \n- Related work Ln. 90-92: The proposed approach can also be categorized as a transformation from a raster to a continuous domain. How does it alleviate the problem of lack of finer details in the rasterized images?\n- Related work Ln. 82-85: is there evidence for the inferior performance of sequential models compared to CNN-based models?\n- Ln. 134 requires citation: “There are many works for estimating d(x; y) from scene parameters but most of them are constrained by the choice of the parameterization.” \n\n**Clarity**\nThe overall organization of the paper is good. It is well-written. The illustration provided in Fig. 4 is very helpful for understanding the core idea. \n\n**Significance**\nIt is an interesting problem, and the proposed method is very effective. It outperforms the baselines. However, in my opinion, the main problem is the selection of the baselines. It is not able to convince me if the proposed approach is really necessary. I am wondering whether a standard SDF could achieve similar results or not. \n\n[1] Park et al., DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation\n\n[2] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions\n\n-----------------------------------------------\n**Updated:**\nI thank the authors for their rebuttal and additional results supporting their submission. The clarifications and the new experiments show the advantage of the proposed multi-channel setup and the training objective in the raster domain. The local supervision seems to be useful in learning a canonical representation space, which is particularly helpful in modeling a large and diverse set of fonts. I raised my score by 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The broader impact statement highlights the positive impacts. There may exist potential misuse of the proposed method. For example, learning and synthesizing a person’s handwriting for forgery. Would it be possible?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-review of "A Multi-Implicit Neural Representation for Fonts"

The paper proposes representing vector fonts with multi-implicit functions. They were able to reconstruct vector fonts (at arbitrarily high resolution) while keeping specific font-discontinuities such as edges and corners, and also showcased their method to synthesize an entire font given one character only.

Reviewers were initially mixed. While reviewers generally agree that the proposed method works (and based on the results, seems to work really well), the concerns raised (esp by iAuP) is whether multi-implicit functions are indeed needed. DnKK raised a similar concern about whether standard SDF could attain similar results or not. The authors have responded to these concerns, and even produced some newer results in their response (They have also written a detailed general response with baseline comparisons). Reviewers were generally satisfied with their response, new results, and improvements made, and have increased their scores, and even persuaded the reviewer iAuP to improve their initial assessment.

In reviewer discussions, most reviewers are in favour of accepting the work. Reviewer iAuP strongly suggested that the authors clarify many issues they discussed in a detailed thread in the camera ready version. For instance, as they noted: *Given the author's additional experiments, it turns out that 'single implicts' w/ fourier features can reconstruct fonts as well as 'multi implicits', better in low resolution worse in high resolution, but roughly about the same.* So I would like to see the text clarified to the best of the authors' abilities (as they have pledged in the discussion thread) the text, and make sure the claims are accurate and clear, as to not mislead readers.

Given the generally high quality of the work, and fantastic results, which will be useful to the NeurIPS community and ML practitioners, I recommend accepting this work as a poster.

Decision: Accept (Poster)