GANet: Glyph-Attention Network for Few-Shot Font Generation

1 Introduction . With the increasing popularity of Chinese characters globally , the demand for Chinese fonts is rising . However , it ’ s very time consuming and expensive to design such font library . As we know , the number of letters in English alphabet is 52 ( upper and lower cases ) , much smaller than that of the Chinese characters which is over 20,000 . It takes about 6 months for an experienced designer to design a Chinese font library . Typically , the designer starts the font design with manually designing a few hundred of Chinese characters which encompass most of the radicals ( i.e. , the components of the characters ) . The rest of the characters will be completed by mixing and matching the radicals based on word vocabulary . However , it is very tedious and time consuming in both the initial design phase and the block building process . If only a few Chinese glyphs ( e.g . 4 or 8 ) are needed by manual design and there is a tool that can automatically complete the font library of the 20,000+ characters , designers ’ efficiency will be greatly improved . Recently , image to image translation based methods Jiang et al . ( 2019 ) have been proposed to generate fonts automatically . Most of these approaches are based on generative adversarial network Goodfellow et al . ( 2014 ) , which was shown to be able to transfer one font style to another successfully . However , they need large scale paired or un-paired source and target domain dataset to train a model . It ’ s inefficient because for every new font style , the pretrained model needs to be finetuned . Moreover , it is difficult and expensive to collect target font dataset such as Chinese historical calligraphic works , whose font glyphs may just have a small percentage left . But these methods typically require hundreds of font glyphs for training as a minimum . In order to address the limitation of lack of dataset , several few-shot font generation methods have been developed Gao et al . ( 2019 ) Park et al . ( 2021 ) , which don ’ t demand a lot of glyphs to train the model . They are capable of producing a complete font library with just few samples . Although these methods can fuse source content and target style representations successfully with few font glyphs unseen style during training , they tend to output characters with either wrong content or weak style . The unsatisfactory results of recent few-shot font generation methods make us re-examine the style and content relationship in the glyphs . A Chinese glyph can be divided into two parts ( i.e . content and style ) . From the content perspective , we recognize a glyph ’ s meaning at a glance . It implies that human understands font ’ s content through vision based on font global information . Inspired by this observation , we propose a content glyph-attention module to capture the global features from the content set of glyphs . On the other side , special stroke and radical details of a glyph determine the its style . In other words , glyph style is closely related to local information . Based on this hypothesis , we propose a style glyph-attention module to encapsulate pattern information from the style set of glyphs . Our contributions are summarized as follows : • We propose a generic few-shot font generation framework that bridges the gap between font content and style . • We propose a content and a style glyph-attention modules to characterize overall structure information from the content set and local stroke details from the style set , respectively . • Experiments demonstrate superior performance in terms of both quantitative and qualitative results compared with other cutting-edge few-shot glyph generation methods . 2 Related Work 2.1 Style Transfer Few-shot Chinese font generation aims to combine the content and reference style . It is a special style transfer problem in essence . However , font design requires high accuracy in content structure , the result of traditional style transfer model for font generation is usually unsatisfactory . Gatys et al . ( 2016 ) proposed neural style transfer in VGG Simonyan & Zisserman ( 2014 ) feature space by reconstructing the gram matrix but with limited inference speed . Johnson et al . ( 2016 ) trained a fast neural style transfer ( FNST ) model , which is able to merge style and content representations in real-time . Although speed-up is achieved in FNST , a new style image is necessary to retrain the model for transfer . In order to solve this problem , Dumoulin et al . ( 2016 ) proposed a n-style transfer method which can embed n style features by using conditional instance normalization . Further improvement was made by Huang & Belongie ( 2017 ) by inventing AdaIN that can achieve arbitrary style transfer in real time without retraining a model . 2.2 Image to Image Translation Although style transfer method has advantage in combining content and style representations , it may damage the content structure . For font generation , it is crucial to keep character semantics so that they are visually distinguishable and recognizable . Image to image translation method ( pix2pix ) Isola et al . ( 2017 ) was devised to transfer from the source domain to the target domain ( e.g. , from a sketch to a real cat drawing ) . Pix2pix differs from style transfer based methods in that it prevents the structure from being destroyed . Nevertheless , the result quality of pix2pix is heavily dependent on paired dataset that is expensive to collect . In order to tackle this problem , Zhu et al . ( 2017 ) moved a step further by introducing unpaired image to image translation method ( CycleGAN ) , which maps the source domain to the target domain by using cyclic loss . Although CycleGAN made break-through by eliminating the necessity of paired dataset , it ’ s inefficient because different ( source , target ) domain tuples require different models for inference . Huang et al . ( 2018 ) extended CycleGAN by implementing multi-domin transfer in a single model . Recently , Liu et al . ( 2019 ) put forward few-shot image to image translation using AdaIN . 2.3 Attention Models Lately , attention mechanisms are widely adopted in several models in order to capture global dependencies Xu et al . ( 2015 ) . In particular , self-attention ( also known as intra-attention ) calculates the response at a position in a sequence by attending to all positions within the same sequence Parikh et al . ( 2016 ) . ( SAGAN ) Zhang et al . ( 2019 ) demonstrated that selfattention could improve image generation quality of the GAN model . ( non-local ) Wang et al . ( 2018 ) used a reformalized self-attention module in spatial-temporal domain between video sequences . To our knowledge , attention module has not yet been explored in font generation tasks . In this study , we apply content and style glyph attention modules to efficiently capture the characteristics of the font content and style sets . 2.4 Few-shot Font Generation Methods The goal of few-shot font generation method is to make the generated glyphs akin to the style references , which are only a few samples , without re-training a new model . At the same time , the generated glyphs ’ semantic is unchanged . Recently , Zhang et al . ( 2018 ) proposed EMD model that leverages a pair of encoders to merge content and style . Gao et al . ( 2019 ) invented AGIS-Net to reach the few-shot font generation goal by transferring both shape and texture with a few reference samples . Unlike other methods , MX-Font Park et al . ( 2021 ) extracts multiple style features not explicitly conditioned on component labels , but automatically by multiple experts to represent different local concepts . 3 METHOD . As mentioned above , our goal is to generate stylized glyph images from a small number of reference samples . We design three encoders , two glyph attention modules and one decoder to form a Glyph-Attention Network ( GANet ) . Details of the model architecture and loss functions are discussed in Sections 3.1 and 3.2 . 3.1 Network architecture As shown in Figure 1 , GANet consists of one query encoder , one style encoder , one content encoder , two glyph-attention modules and one decoder . In order to stablize the training process and improve the synthesized results quality , two discriminators are adopted . 3.1.1 Query , Content and Style Encoders We formulate the content , style and query encoding process as trainable high-dimensional glyph feature extractors . The content encoder Ec aims to extract semantic feature . The input of the content encoder is Xc = { x1 , x2 , ... , xN } , each of them has the same content but different style , the output of the content encoder is Fc = Ec ( Xc ) ∈ RN×H×W×C . Similarly , the style encoder Es is used to characterize style features from the the style set Xs = { y1 , y2 , ... , yN } , which contains N glyphs with the same style but different content . The output of the style encoder is Fs = Es ( Xs ) ∈ RN×H×W×C . The query encoder Eq is devised to obtain a query feature vector Fq that is essential for the glyph-attention modules to identify local style features from the style feature set Fs and generate the most proper global content feature from the content feature set Fc . The input of the query encoder is a glyph image Xq whose content is the same as the content in the content glyph set but with a new style . Output of the query encoder is Fq = Eq ( Xq ) ∈ RH×W×C . The three encoders Es , Eq and Ec have identical architecture but they do not share weights . Symbols H , W and C represent feature map ’ s height , width and channel respectively . 3.1.2 Decoder The style glyph-attention module is intended to query a local style feature tensor from the style feature set . Similarly , the objective of the content glyph-attention module is to acquire the most proper glyph global content from the content feature set . The outputs of the content , style , and query encoders are added element-wise before being fed into the decoder . As defined in ( 1 ) F = SGA ( q , ks , vs ) + CGA ( q , kc , vc ) + Fq ( 1 ) O = Decoder ( F ) ( 2 ) Where SGA and CGA are abbreviations of Style and Content Glyph-Attention respectively , ks and vs are equal to Fs which is from the style encoder . Similar to SGA , kc and vc are equal to Fc which is from the content encoder . q is equal to Fq which is from the query encoder . O is the synthesized result generated by the decoder . 3.1.3 Multi-task discriminators It has been demonstrated that generative adversarial networks ( GAN ) Goodfellow et al . ( 2014 ) are able to generate output with distribution close to that of the actual data . Therefore , we can leverage GAN to develop a model that outputs synthesized glyph images whose content and style comply the distributions of the input content and reference style . In principle , a conventional discriminator , which can discern the fake and real data , can perform the discrimination task independently to complete the generation task . However , the synthesized results are not satisfactory because they are often wrong in content or weak in style . To enhance the model performance , multi-task discriminators are proposed to further disentangle content and style information . They are called content Dc and style Ds discriminators respectively , based on SNGAN Miyato & Koyama ( 2018 ) , which uses a trainable embedding matrix to obtain conditional information . Dc allows closer content distributions between the model output and the real data , and Ds helps the convergence of the output style to that of the real data . 3.2 Glyph attention Most few-shot image generation methodsLiu et al . ( 2019 ) use adaptive instance normalization ( AdaIn ) Huang & Belongie ( 2017 ) to transfer image style such as texture and color to another which provide content like pose and structure . However , it ’ s challenging to transfer font style to another due to the fact that font does not have texture and color , just consists of some white and black strokes . Different from AdaIn , we propose style and content glyph-attention modules to extract style and content features from custom glyph sets . The proposed method Glyph-Attention Network ( GANet ) is named after the glyph-attention modules , which are shown in Figure 2 . 3.2.1 Style glyph-attention As we all know , font style mainly depend on local stroke details , so it means that local features of a glyph dictate styles . Based on this hypothesis , we propose the style glyphattention module . The style encoder provides us a style feature set Fs ∈ RN×H×W×C which contains the desired font style . A style glyph-attention module is utilized to query the style feature from the style set . vs and ks denote the value and key in the attention module . They are equal to Fs . The query vector q ∈ RH×W×C is derived from the query encoder . Key ks and query q are first transformed into two feature spaces fs , gs , where fs ( q ) = Wfsq , gs ( ks ) = Wgsks . We then reshape gs ( ks ) ∈ RNHW×C and transpose and reshape fs ( q ) ∈ RC×HW . Finally , they are used to calculate the attention map β , as defined in ( 3 ) , βij = exp ( ξij ) ∑NHW m=1 exp ( ξmj ) , ξ = gs ( ks ) fs ( q ) ξ ∈ RNHW×HW ( 3 ) The attention map βij indicates the extent to which the model attends to the ith style location when synthesizing jth . Where H , W , C are the height , width and channel of features from previous 1 × 1 conv layer . N is the number of glyphs in the style set . The output of the style glyph-attention layer is Os = ( O1 , O2 , ... , Oj , ... , OHW ) , Oj ∈ R1×C , where , Oij = NHW∑ m=1 τimβmj , τ = hs ( vs ) τ ∈ RC×NHW ( 4 ) where τ is transposed and reshaped from hs ( vs ) = Whsvs . In addition , Whs ∈ RC×C , Wfs ∈ RC×C and Wgs ∈ RC×C are the learned weights in the 1× 1 conv layer of the style glyph-attention module . The final output is reshaped to Os ∈ RH×W×C . 3.2.2 Content glyph-attention Different from the style glyph-attention module , which queries the style features from the local spatial area , the content glyph-attention mainly queries content features globally . It is based on an assumption that content information resides in the overall structure of the skeletons . We define a content feature set Fc ∈ RN×H×W×C that is extracted by the content encoder from the content set Xc = { x1 , x2 , ... , xN } . vc and kc denote the value and key in the attention module , which are equal to Fc . The query vector q ∈ RH×W×C is identical to that in the style glyph-attention module . Similarly , key kc and query q are transformed into two feature spaces fc , gc , where fc ( q ) = Wfcq , gc ( kc ) = Wgckc . gc ( kc ) ∈ RN×HWC and fc ( q ) ∈ RHWC×1 are then reshaped to calculate the attention map γ . As defined in ( 5 ) , γi = exp ( ξi ) ∑N m=1 exp ( ξm ) , ξ = gc ( kc ) fc ( q ) ξ ∈ RN×1 ( 5 ) The attention map γi indicates the extent to which the model attends to the ith global content set location . H , W , and C are the height , width and channel of features from previous 1× 1 conv layer . N is the number of glyphs in the content set . The output of the content glyph-attention layer is Oc = ( O1 , O2 , ... , Oj , ... , OHWC ) , Oj ∈ R , Oj = N∑ m=1 τmjγm , τ = hc ( vc ) τ ∈ RN×HWC ( 6 ) where τ is reshaped from hc ( vc ) = Whcvc . In addition , Whc ∈ RC×C , Wfc ∈ RC×C and Wgc ∈ RC×C are the learned weights in the 1× 1 conv layer of the content glyph-attention module . The final output is reshaped to Oc ∈ RH×W×C . 3.3 Loss function Three individual loss functions are combined during model training : identity loss , feature matching loss , and multi-task adversarial loss . 3.3.1 Identity loss In order to make the output distribution of the model agreeing to that of the target in the pixel and feature space , we employ the identity loss . As we know , perceptual loss Johnson et al . ( 2016 ) is an effective way to measure the distance between images in VGG feature space . Compared to pixel-wise loss functions such as Mean-Squared Error ( MSE ) , perceptual loss can recover more high-frequency details . However , every coin has two sides , due to the downsampling operation ( max-pooling ) in VGG19 , the perceptual loss function will result in partial information loss during reconstruction , especially at low-frequencies . To avoid losing low-frequency details , we add L1 loss as a complementary term with perceptual loss , as defined in ( 7 ) , Lid = ∥Φ3_1 ( T ) − Φ3_1 ( G ( Xc , Xq , Xs ) ) ∥1 + ∥T −G ( Xc , Xq , Xs ) ∥1 ( 7 ) Where Φ3_1 is the feature map of VGG19 in relu3_1 . Xc and Xs are input content set and reference style set respectively , Xq is the query image whose content is the same as the content set . T is the target image whose content is the same as Xc , with style similar to Xs . G is the generator that consists of a content encoder , a query encoder , a style encoder , a decoder and two glyph-attention modules as shown in Figure 1 . 3.3.2 Feature matching loss To stabilize the adversarial training , feature matching ( FM ) loss Salimans et al . ( 2016 ) is also applied in this study . Similar to the perceptual similarity measure , the FM loss uses discriminator network as the feature extractor , which contain a large amount of information about the glyph images . To obtain more disentangled representations in content and style , we use Dc and Ds to extract feature maps , as defined in ( 8 ) , Lfm = l∑ i=1 ∥∥∥D ( i ) c ( T ) −D ( i ) c ( G ( Xc , Xq , Xs ) ) ∥∥∥ 1 + l∑ i=1 ∥∥∥D ( i ) s ( T ) −D ( i ) s ( G ( Xc , Xq , Xs ) ) ∥∥∥ 1 ( 8 ) Where D ( i ) is the feature map of layer i in the discriminator . We use all layers except fully-connected layer to extract features . 3.3.3 Multi-task adversarial loss In the proposed framework , two condition discriminators Dc and Ds are used to improve model performance , each of which has its own mission . Dc and Ds aim at calibrating the model to attain accurate content and stronger style respectively . We utilize hinge version loss Lim & Ye ( 2017 ) as the adversarial objective function , as defined in ( 9-10 ) , LG = −Ds ( G ( Xc , Xq , Xs ) ) −Dc ( G ( Xc , Xq , Xs ) ) ( 9 ) LD =max ( 0 , 1 +Dc ( G ( Xc , Xq , Xs ) ) ) +max ( 0 , 1−Dc ( T ) ) + max ( 0 , 1 +Ds ( G ( Xc , Xq , Xs ) ) ) +max ( 0 , 1−Ds ( T ) ) ( 10 ) 3.3.4 Total loss A simple additive form is used for the total loss , as illustrated in ( 11-12 ) , Lθ ( G ) = λ1Lid + λ2Lfm + LG ( 11 ) Lω ( D ) = LD ( 12 ) Where λ1 and λ2 are the hyper parameters of the generator loss function , θ and ω are the weights for the generator and discriminators respectively . 4 Experiments 4.1 Experimental settings We use the Adam optimizer Kingma & Ba ( 2014 ) to optimize all models with hyperparameters β1 = 0 and β2 = 0.9 . Using the two-time scale learning rates Heusel et al . ( 2017 ) strategy , we set the learning rate of all discriminators to 4e− 4 and the generator to 1e− 4 . The number of both Xc and Xs is set to 8 and the size of all images is 128× 128× 3 . The weights in the loss function are chosen to be λ1 = 1 and λ2 = 10 . We train the model with batch size of 4 for 500,000 iterations in a Nvidia GTX 1080Ti GPU with 12G memory . 4.2 Datasets and evaluation metrics We collected 411 Chinese fonts conforming to the GB2312 standard . Each font contains 6,763 Chinese characters . We randomly selected 405 fonts as the training set and the remaining 6 fonts as the test set . Various metrics are used to evaluate the quality of the synthesized glyphs . To measure the similarity between the generated image and the target , intersection over union ( IOU ) is utilized . We further employ two classifiers with Inception-v3 Szegedy et al . ( 2016 ) backbone to distinguish between content and style labels in the test set . 4.3 Benchmarking We compare our method with four many-shot font generation methods ( i.e. , zi2zi Tian , pix2pix Isola et al . ( 2017 ) , CycleGAN Zhu et al . ( 2017 ) , ZiGAN Wen et al . ( 2021 ) ) and three state-of-the-art few-shot font generation methods ( i.e. , FUNIT Liu et al . ( 2019 ) , MXFont Park et al . ( 2021 ) and AGIS-Net Gao et al . ( 2019 ) ) . The four many-shot methods demand a lot of training data and a different model is required for each font style during inference . As such , for many-shot methods , we further split each test font library into 5763 glyphs for training and 1000 glyphs for testing . Therefore , each of the four manyshot methods generates 6 models . In comparison , few-shot methods including the proposed method GANet need only a few glyphs to transfer font style during inference and no retraining is necessary for different font styles . Different from many-shot method , the few-shot models are trained using the selected 405 font libraries . During inference , few-shot methods take a few glyphs ( i.e . 1 , 2 , 4 , 8 ) as references to synthesize the font library rather than training a new model . To make a fair comparison , we use the 1000 glyphs split from the test set to evaluate all models . 4.4 Quantitative evaluation We assess the quality of the generated images in two aspects . Firstly , Intersection Over Union ( IOU ) is used to measure the similarity between the synthesized and the ground truth . Higher IOU score indicates better result . As shown in Table 1 , we randomly sample four reference images from the 5763 glyphs of one test font , based on which each model generates 1000 glyphs . There are a total number of 6 fonts in the test dataset . The metrics in the table are calculated based on the 1000×6 glyphs for each model and averaged . Table 1 shows that our method outperforms previous state-of-the-art few-shot approaches by a wide margin and generate comparable results to the many-shot approaches . Howerver , our method needs only few reference samples and one-time training . On the contrary , many-shot methods demand re-training using thousands of glyphs for each new font style . IOU can evaluate the similarity in pixel-level , but can not represent the style and content in essence . In addition , pixel-based metrics are often inconsistent with human visual perception . To improve the reliability of the evaluation , we trained two classifiers to identify the content and style classifications respectively and then use them to evaluate model prediction accuracies for the test set . As shown in Figure 4 , MX-Font and FUNIT show comparable performance in terms of style accuracy , but they show lower performance in content preservation . AGIS-Net shows comparable performance in content accuracy , but weak in style . In other words , FUNIT and MX-Font focus only on styling and fails to preserve content structure . AGIS-Net focuses on content and fails to transfer more style . 4.5 Qualitative analysis To demonstrate the model ’ s performance visually , we evaluate the models using the remaining 6 fonts in the test set as reference glyphs . Figure 3 presents the many-shot and few-shot methods generated images from the test dataset . It is observed that many-shot methods except ZiGAN always fail in content preservation as shown in the red box . Among few-shot methods , FUNIT learns the style but the local structure in the generated glyphs is weak as highlighted by the yellow box . Compared with FUNIT , AGIS-Net has better performance in content preservation . However , it has poor performance in capturing global and local styles ( black box ) . MX-Font also succeeds in learning the style of the reference , but the content structure is unsatisfactory ( blue box ) . Overall , our proposed method generated best results qualitatively compared to other methods both in terms of content preservation and style transfer .

Summary: The paper proposes a glyph-attention network for few-shot font generation. They claim that fonts’ content features are basically global features and style features are related to local features. They propose a style glyph attention to capture the global features from the content set of glyphs, and a content glyph attention module to encapsulate pattern information from the style set of glyphs. The motivation is clear and reasonable but there are still some major issues in model construction and the experiment.