A Multi-Implicit Neural Representation for Fonts

1 Introduction . Fonts constitute the vast majority of documents . They come in a variety of styles spanning a range of topologies representing a mixture of smooth curves and sharp features . Although fonts vary significantly across families , they remain stylistic coherent across the different alphabets/symbols inside any chosen font family . Fonts are most commonly stored in a vector form ( e.g. , a collection of spline curves ) that is compact , efficient , and can be resampled at arbitrary resolutions without loss of features . This specialized representation , however , prevents the adaptation of many deep learning setups optimized for regular structures ( e.g. , image grids ) . In order to avoid this problem , custom deep learning architectures have been developed for directly producing vector output but they typically require access to ground truth vector data for training . This is problematic : first , collecting sufficient volume of vector data for training is non trivial ; and second , vector representations are not canonical ( i.e. , same fonts can be represented by different sequence of vectors ) , which in turn requires hand-coded network parameters to account for varying number of vector instructions . An alternate approach is to rasterize vectorized fonts and simply treat them as images . While this readily allows using image-based deep learning methods , the approach inherits problems of discretized 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . representations leading to aliasing artifacts and loss of sharp features . Further , the resultant images are optimized for particular resolutions and can not be resampled without introducing additional artifacts . Recently , implicit representation has emerged as an attractive representation for deep learning as , once trained , they can resampled at different resolutions without introducing artifacts . Unfortunately , implicit representations ( e.g. , signed distance fields ) for fonts are often too complex to be represented accurately by neural networks . As a results , although deep implicits work for simple fonts , they can fail to retain characteristic features ( i.e. , edges and corners ) for complex fonts . Drawing inspiration from multi-channel SDFs [ 4 ] , we observe that complex fonts can be expressed as a composition of multiple simple regions . For example , a local corner can be represented as a suitable composition of two half-planes , each of which can easily be individually encoded as deep implicit functions . We build on this idea by hypothesizing that complex fonts can also be encoded as suitable composition of global implicit functions . We call such a representation to be a multi-implicit neural representation , as each ( global ) implicit function is neurally encoded . Thus , multi-implicits provide a simple representation that is amenable for processing by neural networks and the output fidelity remains comparable , even under resampling , to vector representations without losing edge or corner features . A remaining challenge is how to supervise such a network as there is no dataset with reference multi-implicits that be directly used . In this paper , we present a network structure and training procedure that allow multi-implicts to be trained using only local supervision . We describe how to extract necessary local supervision from vector input and to adaptively obtain training information for the multi-implicits . We extensively evaluated the proposed multi-implicits representation for various tasks including reconstruction , interpolation , and synthesis to demonstrate clear advantages over several existing state-of-the-art alternatives . Additionally , the representation naturally enables glyph completion , wherein a single characteristic font glyph is used to synthesize a whole font family in consistent style . 2 Related Work . Raster-based representation . One of the most intuitive representation of shapes is raster , i.e. , representing a 2D shape by pixels or a 3D shape by voxels , which provides the grid-format data that is perfectly compatible with ( regular ) CNN models . Hence it has acted as a catalyst for many deep learning based methods for semantic editing of raster-based shapes . A general idea is to learn an encoding-decoding model and then manipulate the shape in the latent space . Sharing a similar spirit , many generative models [ 21 , 5 , 15 ] engaged in the raster-based generation and attribute transfer . [ 1 , 25 ] narrowed down the scope to fonts specifically , which focus more on the shape instead of texture . While the raster-based representation has shown strong semantic editability as incorporated with deep models , it is still limited by its intrinsic resolution . Since image super-resolution is an ill-posed problem , details can not be fully recovered after upscaling . Deep learning based vector graphs . Vector graphics ( e.g. , SVG ) have been widely adopted as a scalable representation . Typically , it is constructed by sequences of Bézier curves , which are difficult to be modeled by traditional CNN models since the diverse sequence length across different shapes , as well as diverse types and numbers of Bézier curves for the similar or even the same shape . Therefore , RNN-based models ( e.g. , LSTM ) are commonly adopted in recent works [ 17 , 2 ] to learn the dynamic sequence of curves . Although the sequential modeling could achieve semantic editing and/or interpolation between shapes , it is still lagging behind CNN-based models ( rasterbased methods ) in terms of reconstruction accuracy because it has to model much longer temporal dependency . Also , there is no specific attention given to features like edges and corners . Transformation from raster to vector . Instead of directly modeling raster or vector , an idea of achieving both scalability and editability is to model the transfer from raster to vector [ 20 ] ( we will not consider vectorization methods that purely transfer images to vectors ) . Such an approach inherits the advantages of raster-based methods and supports easy editability . In addition , they output scalable vectors directly . However , this method would be still limited by the raster resolution , i.e. , the output vector graph can not capture enough details since the corresponding raster input may have already lost those finer details . Another challenge for such methods is the high complexity of the shape , e.g. , interpolation between shapes with different topologies leads to undesirable intermediate shapes . Deep implicit representation . Deep implicit functions [ 10 , 3 , 8 , 22 , 18 , 23 ] have achieved great success in shape representation . They take advantage of deep learning techniques to fit an implicit function , which provides a continuous representation breaking the grid limitation of a raster domain . In addition , deep implicit functions model shapes spatially instead of modeling sequentially as aforementioned in deep learning based vector graphs . Therefore , deep implicit functions could preserve the editability like raster-based representation and potentially achieves scalable representation like vectors . Unfortunately , existing works seldom explore the scalability and fidelity of shapes in their representations , especially during editing , interpolation , and upscaling . For instance , sharp corners always suffer from editing and scaling . This is particularly problematic in the domain of fonts , and our work addresses this limitation via the proposed multi-implicit representations . 3 Methods . We represent fonts as the composition of multiple global implicit functions . An implicit representation has two key advantages : it allows for rendering at arbitrary resolution ; it can be locally supervised in characteristic areas such as sharp edges and corners . Unlike a single implicit function , our multi-implicit representation can faithfully reconstruct sharp features with low reconstruction error ( Figure 2 ) . Implicit Model Image Rendering Multi-SDF Differentiable Rasterization Multi-Curve Representation Importance Sampling + z L 2 regularization Corner Template Loss Global Shape Loss Gradient Loss + We train a generative neural network , as shown in the inset , that models fonts using this representation . Instead of directly learning the inside-outside status of the font image [ 3 ] , we predict a set of distance fields of 2D shapes . Their composition is then fed into a differentiable rasterizer to produce the final image . We will show in Section 4 how this generative model enables a diverse set of font reconstruction and editing operations . Section 3.1 details differentiable rasterization of 2D distance fields , and corresponding corner preservation is discussed in Section 3.2 . Finally , training losses and details are given in Section 3.3 . 3.1 Differentiable Rasterization of Distance Field . As aforementioned , we use the signed distance field ( SDF ) to model 2D shapes . The common supervisions for SDF are SDF labels from the ground truth curves or distance transform on the silhouette [ 14 ] . Figure 3 compares the results between training with SDF and training with raster , where training only on the ground truth SDF does not always result in a good raster image in terms of boundary smoothness . In contrast , training with the supervision of raster yields smoother boundary . Therefore , we will draw supervision on rasterized SDF to achieve better shape . Sharing the spirit from vector graphics rasterization works [ 7 , 11 , 16 , 19 , 24 ] , we simplify general vector graphics rendering by analytically approximating the point to curve distance as , I ( x , y ) = K ( min i∈F di ( x , y ) ) g ( x , y ) , ( 1 ) K ( d ) = ⎧⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎩ 1 if d > γ , k ( d γ ) = 1 2 + 1 4 ( ( d γ ) 3 − 3 ( d γ ) ) if − γ ≤ d ≤ γ , 0 if d < −γ , ( 2 ) where di ( x , y ) indicates an SDF of distance from pixel ( x , y ) center to the closest point on the i-th curve corresponding the font F , and K denotes a function that approximates the opacity based on the distance value . There are many works for estimating di ( x , y ) from scene parameters but most of them are constrained by the choice of the parameterization . In this paper , we model each di ( x , y ) using an implicit neural network . The function g models the spatially-varying texture of the shape . For solid fonts , we set g = 1 . In the supplemental we show examples of textured fonts where we make use of spatially varying g. In the function K , γ is the anti-alias range , and the kernel k is a radially symmetric continuous filter that satisfies the constraint k ( 1 ) = 0 and k ( −1 ) = 1 . In our work , we approximate k ( ⋅ ) using a parabolic kernel similar to [ 19 , 13 ] . Note that the rasterization function I ( x , y ) has non-zero gradients only if ( x , y ) falls inside of the anti-aliasing range . We use a progressively decreasing anti-aliasing range strategy for better convergence and fidelity ( see Sec . 3.3 ) . 3.2 Multi-Curve Representation for Sharp Corners . We will lose details like sharp corners when upscaling bitmaps or sign distance functions . Resampling an implicit model that encodes the pixel values or signed distance values of a shape similarly suffers from blurry corners . An brute force solution is to train the implicit model with extremely high-resolution images , but this would drastically increase the burden of training , and still limited by the training resolution . Rather than directly modeling corners , we represent corners as the intersection of multiple curves ( e.g. , lines or parabolas ) , drawing inspiration from traditional representations [ 9 ] . Note that even though these individual sub-curves may be smoothed after encoded by a deep model , the sharpness of their intersection will be preserved . With this insight , we construct sharp corners from multiple smooth curves predicted by the implicit model . More specifically , the implicit model is designed to predict multiple SDFs ( and rasterization of distance fields ) , each of which carries smooth curves/shapes decoupled from corners and edges as illustrated in Figure 4a . Assume a shape F is represented by a set of curves C = { C1 , C2 , ⋯ , Cn } , where Cn is a binary map indicating whether a pixel is inside ( i.e. , 1 ) or outside ( i.e. , 0 ) the n-th curve , like the example in Figure 4a . In our scenario , Cn = K ( dn ) , where dn is the n-th SDF channel estimated by the implicit model . Then , a function over all curves F ( C ) will fuse those curves to reconstruct the shape , preserving sharp corners . As illustrated in Figure 4a , two curves can sufficiently represent a corner , either convex or concave by adopting maximum or minimum as the function F . To represent a shape with arbitrary corners , however , it requires three curves at least . For example , F ( C ) = min ( max ( C1 , C2 ) , C3 ) can model all corners in a shape . Therefore , we set n = 3 , i.e. , C = { C1 , C2 , C3 } . Since deep models are sensitive to the permutation of guidance signals during training , we use the median function as F , which achieves sharp corners and permutational invariance to the order of these curves . Thus , a corner-related loss on F ( C ) could be robust to the ordering ambiguity . Based on F ( C ) ( median function on three curves ) , a typical corner O is presented in Figure 4b , where intersection of two curves divides its local space into four quadrants , i.e. , from Q1 to Q4 . There are always two opposite quadrants that one is inside area ( i.e. , Q1 ) where corresponding values from F ( C ) are 1 , and the other is outside area ( i.e. , Q4 ) where corresponding values from F ( C ) are 0 . The rest two opposite quadrants ( i.e. , Q2 and Q3 ) are equal on F ( C ) but different on C. For example , the values on { C1 , C2 , C3 } corresponding to the Q2 area is ( 1 , 0 , 0 ) , so F ( C ) on Q2 is 0 . Then , F ( C ) on Q3 should be 0 as well . However , the values on { C1 , C2 , C3 } corresponding to the Q3 region must be different from ( 1 , 0 , 0 ) , i.e. , could be ( 0 , 1 , 0 ) or ( 0 , 0 , 1 ) . If F ( C ) on Q2 and Q3 is 1 , O is a concave corner . Otherwise , O is a convex corner . Such distribution of C along the four quadrants around a corner O is referred to as corner template . The shapes are encoded as the multi-curve representation through an implicit model . The implicit model takes sample ( x , y , l ) ( i.e. , 2D spatial location ( x , y ) and glyph label l ) , as well as embedding z that indicates the font style , and outputs three channels of SDF { d1 , d2 , d3 } . Then , the rasterization approach discussed in Differentiable Rasterization of Distance Field converts each di to Ci ( i = 1 , 2 , 3 ) . Finally , the median function F ( C ) renders the final shape . We optimize the global shape using the final render output and ( locally ) supervise each corner to be locally represented as an intersection of two curves . One could use the output of [ 4 ] to train a network . However , the edge coloring approach presented in [ 4 ] has no canonical form , which prevents a neural network from learning a continuous latent space between shapes . Please note that since we focus on corner supervision to ensure sharp corners at higher resolution resampling , it is unnecessary to constraint the model by multi-channel supervision globally . In the optimization of rendered global shape , the median operation F ( C ) would route the gradients to the correspondingly active value only , i.e. , only update a single channel at each location of C. However , at least two of the three channels of C at a certain location need to be updated to approaching the ground truth because of the nature of the median operator . Therefore , in the training stage , we use an approximation F̂ ( C ) defined as the average of the median and the closest value to the median , thus two channels will be updated .

Summary: This paper proposes multi-implicit neural representation for fonts given the observation that it is hard to represent complex fonts with single neural representation. The author introduces a local supervision method that can train multi-implicit functions without knowing ground truth channels. Experimental results in various settings, e.g. reconstruction, interpolation, and generation, show that it outperformed the previous methods such as rasterized representation VAE and a couple of neural vectorized representations.