A Multi-Implicit Neural Representation for Fonts

1 Introduction . Fonts constitute the vast majority of documents . They come in a variety of styles spanning a range of topologies representing a mixture of smooth curves and sharp features . Although fonts vary significantly across families , they remain stylistic coherent across the different alphabets/symbols inside any chosen font family . Fonts are most commonly stored in a vector form ( e.g. , a collection of spline curves ) that is compact , efficient , and can be resampled at arbitrary resolutions without loss of features . This specialized representation , however , prevents the adaptation of many deep learning setups optimized for regular structures ( e.g. , image grids ) . In order to avoid this problem , custom deep learning architectures have been developed for directly producing vector output but they typically require access to ground truth vector data for training . This is problematic : first , collecting sufficient volume of vector data for training is non trivial ; and second , vector representations are not canonical ( i.e. , same fonts can be represented by different sequence of vectors ) , which in turn requires hand-coded network parameters to account for varying number of vector instructions . An alternate approach is to rasterize vectorized fonts and simply treat them as images . While this readily allows using image-based deep learning methods , the approach inherits problems of discretized 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . representations leading to aliasing artifacts and loss of sharp features . Further , the resultant images are optimized for particular resolutions and can not be resampled without introducing additional artifacts . Recently , implicit representation has emerged as an attractive representation for deep learning as , once trained , they can resampled at different resolutions without introducing artifacts . Unfortunately , implicit representations ( e.g. , signed distance fields ) for fonts are often too complex to be represented accurately by neural networks . As a results , although deep implicits work for simple fonts , they can fail to retain characteristic features ( i.e. , edges and corners ) for complex fonts . Drawing inspiration from multi-channel SDFs [ 4 ] , we observe that complex fonts can be expressed as a composition of multiple simple regions . For example , a local corner can be represented as a suitable composition of two half-planes , each of which can easily be individually encoded as deep implicit functions . We build on this idea by hypothesizing that complex fonts can also be encoded as suitable composition of global implicit functions . We call such a representation to be a multi-implicit neural representation , as each ( global ) implicit function is neurally encoded . Thus , multi-implicits provide a simple representation that is amenable for processing by neural networks and the output fidelity remains comparable , even under resampling , to vector representations without losing edge or corner features . A remaining challenge is how to supervise such a network as there is no dataset with reference multi-implicits that be directly used . In this paper , we present a network structure and training procedure that allow multi-implicts to be trained using only local supervision . We describe how to extract necessary local supervision from vector input and to adaptively obtain training information for the multi-implicits . We extensively evaluated the proposed multi-implicits representation for various tasks including reconstruction , interpolation , and synthesis to demonstrate clear advantages over several existing state-of-the-art alternatives . Additionally , the representation naturally enables glyph completion , wherein a single characteristic font glyph is used to synthesize a whole font family in consistent style . 2 Related Work . Raster-based representation . One of the most intuitive representation of shapes is raster , i.e. , representing a 2D shape by pixels or a 3D shape by voxels , which provides the grid-format data that is perfectly compatible with ( regular ) CNN models . Hence it has acted as a catalyst for many deep learning based methods for semantic editing of raster-based shapes . A general idea is to learn an encoding-decoding model and then manipulate the shape in the latent space . Sharing a similar spirit , many generative models [ 21 , 5 , 15 ] engaged in the raster-based generation and attribute transfer . [ 1 , 25 ] narrowed down the scope to fonts specifically , which focus more on the shape instead of texture . While the raster-based representation has shown strong semantic editability as incorporated with deep models , it is still limited by its intrinsic resolution . Since image super-resolution is an ill-posed problem , details can not be fully recovered after upscaling . Deep learning based vector graphs . Vector graphics ( e.g. , SVG ) have been widely adopted as a scalable representation . Typically , it is constructed by sequences of Bézier curves , which are difficult to be modeled by traditional CNN models since the diverse sequence length across different shapes , as well as diverse types and numbers of Bézier curves for the similar or even the same shape . Therefore , RNN-based models ( e.g. , LSTM ) are commonly adopted in recent works [ 17 , 2 ] to learn the dynamic sequence of curves . Although the sequential modeling could achieve semantic editing and/or interpolation between shapes , it is still lagging behind CNN-based models ( rasterbased methods ) in terms of reconstruction accuracy because it has to model much longer temporal dependency . Also , there is no specific attention given to features like edges and corners . Transformation from raster to vector . Instead of directly modeling raster or vector , an idea of achieving both scalability and editability is to model the transfer from raster to vector [ 20 ] ( we will not consider vectorization methods that purely transfer images to vectors ) . Such an approach inherits the advantages of raster-based methods and supports easy editability . In addition , they output scalable vectors directly . However , this method would be still limited by the raster resolution , i.e. , the output vector graph can not capture enough details since the corresponding raster input may have already lost those finer details . Another challenge for such methods is the high complexity of the shape , e.g. , interpolation between shapes with different topologies leads to undesirable intermediate shapes . Deep implicit representation . Deep implicit functions [ 10 , 3 , 8 , 22 , 18 , 23 ] have achieved great success in shape representation . They take advantage of deep learning techniques to fit an implicit function , which provides a continuous representation breaking the grid limitation of a raster domain . In addition , deep implicit functions model shapes spatially instead of modeling sequentially as aforementioned in deep learning based vector graphs . Therefore , deep implicit functions could preserve the editability like raster-based representation and potentially achieves scalable representation like vectors . Unfortunately , existing works seldom explore the scalability and fidelity of shapes in their representations , especially during editing , interpolation , and upscaling . For instance , sharp corners always suffer from editing and scaling . This is particularly problematic in the domain of fonts , and our work addresses this limitation via the proposed multi-implicit representations . 3 Methods . We represent fonts as the composition of multiple global implicit functions . An implicit representation has two key advantages : it allows for rendering at arbitrary resolution ; it can be locally supervised in characteristic areas such as sharp edges and corners . Unlike a single implicit function , our multi-implicit representation can faithfully reconstruct sharp features with low reconstruction error ( Figure 2 ) . Implicit Model Image Rendering Multi-SDF Differentiable Rasterization Multi-Curve Representation Importance Sampling + z L 2 regularization Corner Template Loss Global Shape Loss Gradient Loss + We train a generative neural network , as shown in the inset , that models fonts using this representation . Instead of directly learning the inside-outside status of the font image [ 3 ] , we predict a set of distance fields of 2D shapes . Their composition is then fed into a differentiable rasterizer to produce the final image . We will show in Section 4 how this generative model enables a diverse set of font reconstruction and editing operations . Section 3.1 details differentiable rasterization of 2D distance fields , and corresponding corner preservation is discussed in Section 3.2 . Finally , training losses and details are given in Section 3.3 . 3.1 Differentiable Rasterization of Distance Field . As aforementioned , we use the signed distance field ( SDF ) to model 2D shapes . The common supervisions for SDF are SDF labels from the ground truth curves or distance transform on the silhouette [ 14 ] . Figure 3 compares the results between training with SDF and training with raster , where training only on the ground truth SDF does not always result in a good raster image in terms of boundary smoothness . In contrast , training with the supervision of raster yields smoother boundary . Therefore , we will draw supervision on rasterized SDF to achieve better shape . Sharing the spirit from vector graphics rasterization works [ 7 , 11 , 16 , 19 , 24 ] , we simplify general vector graphics rendering by analytically approximating the point to curve distance as , I ( x , y ) = K ( min i∈F di ( x , y ) ) g ( x , y ) , ( 1 ) K ( d ) = ⎧⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎩ 1 if d > γ , k ( d γ ) = 1 2 + 1 4 ( ( d γ ) 3 − 3 ( d γ ) ) if − γ ≤ d ≤ γ , 0 if d < −γ , ( 2 ) where di ( x , y ) indicates an SDF of distance from pixel ( x , y ) center to the closest point on the i-th curve corresponding the font F , and K denotes a function that approximates the opacity based on the distance value . There are many works for estimating di ( x , y ) from scene parameters but most of them are constrained by the choice of the parameterization . In this paper , we model each di ( x , y ) using an implicit neural network . The function g models the spatially-varying texture of the shape . For solid fonts , we set g = 1 . In the supplemental we show examples of textured fonts where we make use of spatially varying g. In the function K , γ is the anti-alias range , and the kernel k is a radially symmetric continuous filter that satisfies the constraint k ( 1 ) = 0 and k ( −1 ) = 1 . In our work , we approximate k ( ⋅ ) using a parabolic kernel similar to [ 19 , 13 ] . Note that the rasterization function I ( x , y ) has non-zero gradients only if ( x , y ) falls inside of the anti-aliasing range . We use a progressively decreasing anti-aliasing range strategy for better convergence and fidelity ( see Sec . 3.3 ) . 3.2 Multi-Curve Representation for Sharp Corners . We will lose details like sharp corners when upscaling bitmaps or sign distance functions . Resampling an implicit model that encodes the pixel values or signed distance values of a shape similarly suffers from blurry corners . An brute force solution is to train the implicit model with extremely high-resolution images , but this would drastically increase the burden of training , and still limited by the training resolution . Rather than directly modeling corners , we represent corners as the intersection of multiple curves ( e.g. , lines or parabolas ) , drawing inspiration from traditional representations [ 9 ] . Note that even though these individual sub-curves may be smoothed after encoded by a deep model , the sharpness of their intersection will be preserved . With this insight , we construct sharp corners from multiple smooth curves predicted by the implicit model . More specifically , the implicit model is designed to predict multiple SDFs ( and rasterization of distance fields ) , each of which carries smooth curves/shapes decoupled from corners and edges as illustrated in Figure 4a . Assume a shape F is represented by a set of curves C = { C1 , C2 , ⋯ , Cn } , where Cn is a binary map indicating whether a pixel is inside ( i.e. , 1 ) or outside ( i.e. , 0 ) the n-th curve , like the example in Figure 4a . In our scenario , Cn = K ( dn ) , where dn is the n-th SDF channel estimated by the implicit model . Then , a function over all curves F ( C ) will fuse those curves to reconstruct the shape , preserving sharp corners . As illustrated in Figure 4a , two curves can sufficiently represent a corner , either convex or concave by adopting maximum or minimum as the function F . To represent a shape with arbitrary corners , however , it requires three curves at least . For example , F ( C ) = min ( max ( C1 , C2 ) , C3 ) can model all corners in a shape . Therefore , we set n = 3 , i.e. , C = { C1 , C2 , C3 } . Since deep models are sensitive to the permutation of guidance signals during training , we use the median function as F , which achieves sharp corners and permutational invariance to the order of these curves . Thus , a corner-related loss on F ( C ) could be robust to the ordering ambiguity . Based on F ( C ) ( median function on three curves ) , a typical corner O is presented in Figure 4b , where intersection of two curves divides its local space into four quadrants , i.e. , from Q1 to Q4 . There are always two opposite quadrants that one is inside area ( i.e. , Q1 ) where corresponding values from F ( C ) are 1 , and the other is outside area ( i.e. , Q4 ) where corresponding values from F ( C ) are 0 . The rest two opposite quadrants ( i.e. , Q2 and Q3 ) are equal on F ( C ) but different on C. For example , the values on { C1 , C2 , C3 } corresponding to the Q2 area is ( 1 , 0 , 0 ) , so F ( C ) on Q2 is 0 . Then , F ( C ) on Q3 should be 0 as well . However , the values on { C1 , C2 , C3 } corresponding to the Q3 region must be different from ( 1 , 0 , 0 ) , i.e. , could be ( 0 , 1 , 0 ) or ( 0 , 0 , 1 ) . If F ( C ) on Q2 and Q3 is 1 , O is a concave corner . Otherwise , O is a convex corner . Such distribution of C along the four quadrants around a corner O is referred to as corner template . The shapes are encoded as the multi-curve representation through an implicit model . The implicit model takes sample ( x , y , l ) ( i.e. , 2D spatial location ( x , y ) and glyph label l ) , as well as embedding z that indicates the font style , and outputs three channels of SDF { d1 , d2 , d3 } . Then , the rasterization approach discussed in Differentiable Rasterization of Distance Field converts each di to Ci ( i = 1 , 2 , 3 ) . Finally , the median function F ( C ) renders the final shape . We optimize the global shape using the final render output and ( locally ) supervise each corner to be locally represented as an intersection of two curves . One could use the output of [ 4 ] to train a network . However , the edge coloring approach presented in [ 4 ] has no canonical form , which prevents a neural network from learning a continuous latent space between shapes . Please note that since we focus on corner supervision to ensure sharp corners at higher resolution resampling , it is unnecessary to constraint the model by multi-channel supervision globally . In the optimization of rendered global shape , the median operation F ( C ) would route the gradients to the correspondingly active value only , i.e. , only update a single channel at each location of C. However , at least two of the three channels of C at a certain location need to be updated to approaching the ground truth because of the nature of the median operator . Therefore , in the training stage , we use an approximation F̂ ( C ) defined as the average of the median and the closest value to the median , thus two channels will be updated .

Summary: This paper aims to learn a representation for rasterized fonts that allow for re-scaling, reconstruction, interpolation, and generation by preserving local features such as sharp corners or smooth curves. The proposed method composes multiple implicit representations (i.e., signed distance fields) to model complex details observed in neutral and stylistic fonts. The core idea is to represent corners as an intersection of multiple curves, enabling the construction of sharp corners in arbitrary resolutions. This assumption is further supported by a local supervision objective for the corners. In the experiments, the authors present that their model is able to preserve sharp corners. The proposed model outperforms baselines both qualitatively and quantitatively.