GANet: Glyph-Attention Network for Few-Shot Font Generation

1 Introduction . With the increasing popularity of Chinese characters globally , the demand for Chinese fonts is rising . However , it ’ s very time consuming and expensive to design such font library . As we know , the number of letters in English alphabet is 52 ( upper and lower cases ) , much smaller than that of the Chinese characters which is over 20,000 . It takes about 6 months for an experienced designer to design a Chinese font library . Typically , the designer starts the font design with manually designing a few hundred of Chinese characters which encompass most of the radicals ( i.e. , the components of the characters ) . The rest of the characters will be completed by mixing and matching the radicals based on word vocabulary . However , it is very tedious and time consuming in both the initial design phase and the block building process . If only a few Chinese glyphs ( e.g . 4 or 8 ) are needed by manual design and there is a tool that can automatically complete the font library of the 20,000+ characters , designers ’ efficiency will be greatly improved . Recently , image to image translation based methods Jiang et al . ( 2019 ) have been proposed to generate fonts automatically . Most of these approaches are based on generative adversarial network Goodfellow et al . ( 2014 ) , which was shown to be able to transfer one font style to another successfully . However , they need large scale paired or un-paired source and target domain dataset to train a model . It ’ s inefficient because for every new font style , the pretrained model needs to be finetuned . Moreover , it is difficult and expensive to collect target font dataset such as Chinese historical calligraphic works , whose font glyphs may just have a small percentage left . But these methods typically require hundreds of font glyphs for training as a minimum . In order to address the limitation of lack of dataset , several few-shot font generation methods have been developed Gao et al . ( 2019 ) Park et al . ( 2021 ) , which don ’ t demand a lot of glyphs to train the model . They are capable of producing a complete font library with just few samples . Although these methods can fuse source content and target style representations successfully with few font glyphs unseen style during training , they tend to output characters with either wrong content or weak style . The unsatisfactory results of recent few-shot font generation methods make us re-examine the style and content relationship in the glyphs . A Chinese glyph can be divided into two parts ( i.e . content and style ) . From the content perspective , we recognize a glyph ’ s meaning at a glance . It implies that human understands font ’ s content through vision based on font global information . Inspired by this observation , we propose a content glyph-attention module to capture the global features from the content set of glyphs . On the other side , special stroke and radical details of a glyph determine the its style . In other words , glyph style is closely related to local information . Based on this hypothesis , we propose a style glyph-attention module to encapsulate pattern information from the style set of glyphs . Our contributions are summarized as follows : • We propose a generic few-shot font generation framework that bridges the gap between font content and style . • We propose a content and a style glyph-attention modules to characterize overall structure information from the content set and local stroke details from the style set , respectively . • Experiments demonstrate superior performance in terms of both quantitative and qualitative results compared with other cutting-edge few-shot glyph generation methods . 2 Related Work 2.1 Style Transfer Few-shot Chinese font generation aims to combine the content and reference style . It is a special style transfer problem in essence . However , font design requires high accuracy in content structure , the result of traditional style transfer model for font generation is usually unsatisfactory . Gatys et al . ( 2016 ) proposed neural style transfer in VGG Simonyan & Zisserman ( 2014 ) feature space by reconstructing the gram matrix but with limited inference speed . Johnson et al . ( 2016 ) trained a fast neural style transfer ( FNST ) model , which is able to merge style and content representations in real-time . Although speed-up is achieved in FNST , a new style image is necessary to retrain the model for transfer . In order to solve this problem , Dumoulin et al . ( 2016 ) proposed a n-style transfer method which can embed n style features by using conditional instance normalization . Further improvement was made by Huang & Belongie ( 2017 ) by inventing AdaIN that can achieve arbitrary style transfer in real time without retraining a model . 2.2 Image to Image Translation Although style transfer method has advantage in combining content and style representations , it may damage the content structure . For font generation , it is crucial to keep character semantics so that they are visually distinguishable and recognizable . Image to image translation method ( pix2pix ) Isola et al . ( 2017 ) was devised to transfer from the source domain to the target domain ( e.g. , from a sketch to a real cat drawing ) . Pix2pix differs from style transfer based methods in that it prevents the structure from being destroyed . Nevertheless , the result quality of pix2pix is heavily dependent on paired dataset that is expensive to collect . In order to tackle this problem , Zhu et al . ( 2017 ) moved a step further by introducing unpaired image to image translation method ( CycleGAN ) , which maps the source domain to the target domain by using cyclic loss . Although CycleGAN made break-through by eliminating the necessity of paired dataset , it ’ s inefficient because different ( source , target ) domain tuples require different models for inference . Huang et al . ( 2018 ) extended CycleGAN by implementing multi-domin transfer in a single model . Recently , Liu et al . ( 2019 ) put forward few-shot image to image translation using AdaIN . 2.3 Attention Models Lately , attention mechanisms are widely adopted in several models in order to capture global dependencies Xu et al . ( 2015 ) . In particular , self-attention ( also known as intra-attention ) calculates the response at a position in a sequence by attending to all positions within the same sequence Parikh et al . ( 2016 ) . ( SAGAN ) Zhang et al . ( 2019 ) demonstrated that selfattention could improve image generation quality of the GAN model . ( non-local ) Wang et al . ( 2018 ) used a reformalized self-attention module in spatial-temporal domain between video sequences . To our knowledge , attention module has not yet been explored in font generation tasks . In this study , we apply content and style glyph attention modules to efficiently capture the characteristics of the font content and style sets . 2.4 Few-shot Font Generation Methods The goal of few-shot font generation method is to make the generated glyphs akin to the style references , which are only a few samples , without re-training a new model . At the same time , the generated glyphs ’ semantic is unchanged . Recently , Zhang et al . ( 2018 ) proposed EMD model that leverages a pair of encoders to merge content and style . Gao et al . ( 2019 ) invented AGIS-Net to reach the few-shot font generation goal by transferring both shape and texture with a few reference samples . Unlike other methods , MX-Font Park et al . ( 2021 ) extracts multiple style features not explicitly conditioned on component labels , but automatically by multiple experts to represent different local concepts . 3 METHOD . As mentioned above , our goal is to generate stylized glyph images from a small number of reference samples . We design three encoders , two glyph attention modules and one decoder to form a Glyph-Attention Network ( GANet ) . Details of the model architecture and loss functions are discussed in Sections 3.1 and 3.2 . 3.1 Network architecture As shown in Figure 1 , GANet consists of one query encoder , one style encoder , one content encoder , two glyph-attention modules and one decoder . In order to stablize the training process and improve the synthesized results quality , two discriminators are adopted . 3.1.1 Query , Content and Style Encoders We formulate the content , style and query encoding process as trainable high-dimensional glyph feature extractors . The content encoder Ec aims to extract semantic feature . The input of the content encoder is Xc = { x1 , x2 , ... , xN } , each of them has the same content but different style , the output of the content encoder is Fc = Ec ( Xc ) ∈ RN×H×W×C . Similarly , the style encoder Es is used to characterize style features from the the style set Xs = { y1 , y2 , ... , yN } , which contains N glyphs with the same style but different content . The output of the style encoder is Fs = Es ( Xs ) ∈ RN×H×W×C . The query encoder Eq is devised to obtain a query feature vector Fq that is essential for the glyph-attention modules to identify local style features from the style feature set Fs and generate the most proper global content feature from the content feature set Fc . The input of the query encoder is a glyph image Xq whose content is the same as the content in the content glyph set but with a new style . Output of the query encoder is Fq = Eq ( Xq ) ∈ RH×W×C . The three encoders Es , Eq and Ec have identical architecture but they do not share weights . Symbols H , W and C represent feature map ’ s height , width and channel respectively . 3.1.2 Decoder The style glyph-attention module is intended to query a local style feature tensor from the style feature set . Similarly , the objective of the content glyph-attention module is to acquire the most proper glyph global content from the content feature set . The outputs of the content , style , and query encoders are added element-wise before being fed into the decoder . As defined in ( 1 ) F = SGA ( q , ks , vs ) + CGA ( q , kc , vc ) + Fq ( 1 ) O = Decoder ( F ) ( 2 ) Where SGA and CGA are abbreviations of Style and Content Glyph-Attention respectively , ks and vs are equal to Fs which is from the style encoder . Similar to SGA , kc and vc are equal to Fc which is from the content encoder . q is equal to Fq which is from the query encoder . O is the synthesized result generated by the decoder . 3.1.3 Multi-task discriminators It has been demonstrated that generative adversarial networks ( GAN ) Goodfellow et al . ( 2014 ) are able to generate output with distribution close to that of the actual data . Therefore , we can leverage GAN to develop a model that outputs synthesized glyph images whose content and style comply the distributions of the input content and reference style . In principle , a conventional discriminator , which can discern the fake and real data , can perform the discrimination task independently to complete the generation task . However , the synthesized results are not satisfactory because they are often wrong in content or weak in style . To enhance the model performance , multi-task discriminators are proposed to further disentangle content and style information . They are called content Dc and style Ds discriminators respectively , based on SNGAN Miyato & Koyama ( 2018 ) , which uses a trainable embedding matrix to obtain conditional information . Dc allows closer content distributions between the model output and the real data , and Ds helps the convergence of the output style to that of the real data . 3.2 Glyph attention Most few-shot image generation methodsLiu et al . ( 2019 ) use adaptive instance normalization ( AdaIn ) Huang & Belongie ( 2017 ) to transfer image style such as texture and color to another which provide content like pose and structure . However , it ’ s challenging to transfer font style to another due to the fact that font does not have texture and color , just consists of some white and black strokes . Different from AdaIn , we propose style and content glyph-attention modules to extract style and content features from custom glyph sets . The proposed method Glyph-Attention Network ( GANet ) is named after the glyph-attention modules , which are shown in Figure 2 . 3.2.1 Style glyph-attention As we all know , font style mainly depend on local stroke details , so it means that local features of a glyph dictate styles . Based on this hypothesis , we propose the style glyphattention module . The style encoder provides us a style feature set Fs ∈ RN×H×W×C which contains the desired font style . A style glyph-attention module is utilized to query the style feature from the style set . vs and ks denote the value and key in the attention module . They are equal to Fs . The query vector q ∈ RH×W×C is derived from the query encoder . Key ks and query q are first transformed into two feature spaces fs , gs , where fs ( q ) = Wfsq , gs ( ks ) = Wgsks . We then reshape gs ( ks ) ∈ RNHW×C and transpose and reshape fs ( q ) ∈ RC×HW . Finally , they are used to calculate the attention map β , as defined in ( 3 ) , βij = exp ( ξij ) ∑NHW m=1 exp ( ξmj ) , ξ = gs ( ks ) fs ( q ) ξ ∈ RNHW×HW ( 3 ) The attention map βij indicates the extent to which the model attends to the ith style location when synthesizing jth . Where H , W , C are the height , width and channel of features from previous 1 × 1 conv layer . N is the number of glyphs in the style set . The output of the style glyph-attention layer is Os = ( O1 , O2 , ... , Oj , ... , OHW ) , Oj ∈ R1×C , where , Oij = NHW∑ m=1 τimβmj , τ = hs ( vs ) τ ∈ RC×NHW ( 4 ) where τ is transposed and reshaped from hs ( vs ) = Whsvs . In addition , Whs ∈ RC×C , Wfs ∈ RC×C and Wgs ∈ RC×C are the learned weights in the 1× 1 conv layer of the style glyph-attention module . The final output is reshaped to Os ∈ RH×W×C . 3.2.2 Content glyph-attention Different from the style glyph-attention module , which queries the style features from the local spatial area , the content glyph-attention mainly queries content features globally . It is based on an assumption that content information resides in the overall structure of the skeletons . We define a content feature set Fc ∈ RN×H×W×C that is extracted by the content encoder from the content set Xc = { x1 , x2 , ... , xN } . vc and kc denote the value and key in the attention module , which are equal to Fc . The query vector q ∈ RH×W×C is identical to that in the style glyph-attention module . Similarly , key kc and query q are transformed into two feature spaces fc , gc , where fc ( q ) = Wfcq , gc ( kc ) = Wgckc . gc ( kc ) ∈ RN×HWC and fc ( q ) ∈ RHWC×1 are then reshaped to calculate the attention map γ . As defined in ( 5 ) , γi = exp ( ξi ) ∑N m=1 exp ( ξm ) , ξ = gc ( kc ) fc ( q ) ξ ∈ RN×1 ( 5 ) The attention map γi indicates the extent to which the model attends to the ith global content set location . H , W , and C are the height , width and channel of features from previous 1× 1 conv layer . N is the number of glyphs in the content set . The output of the content glyph-attention layer is Oc = ( O1 , O2 , ... , Oj , ... , OHWC ) , Oj ∈ R , Oj = N∑ m=1 τmjγm , τ = hc ( vc ) τ ∈ RN×HWC ( 6 ) where τ is reshaped from hc ( vc ) = Whcvc . In addition , Whc ∈ RC×C , Wfc ∈ RC×C and Wgc ∈ RC×C are the learned weights in the 1× 1 conv layer of the content glyph-attention module . The final output is reshaped to Oc ∈ RH×W×C . 3.3 Loss function Three individual loss functions are combined during model training : identity loss , feature matching loss , and multi-task adversarial loss . 3.3.1 Identity loss In order to make the output distribution of the model agreeing to that of the target in the pixel and feature space , we employ the identity loss . As we know , perceptual loss Johnson et al . ( 2016 ) is an effective way to measure the distance between images in VGG feature space . Compared to pixel-wise loss functions such as Mean-Squared Error ( MSE ) , perceptual loss can recover more high-frequency details . However , every coin has two sides , due to the downsampling operation ( max-pooling ) in VGG19 , the perceptual loss function will result in partial information loss during reconstruction , especially at low-frequencies . To avoid losing low-frequency details , we add L1 loss as a complementary term with perceptual loss , as defined in ( 7 ) , Lid = ∥Φ3_1 ( T ) − Φ3_1 ( G ( Xc , Xq , Xs ) ) ∥1 + ∥T −G ( Xc , Xq , Xs ) ∥1 ( 7 ) Where Φ3_1 is the feature map of VGG19 in relu3_1 . Xc and Xs are input content set and reference style set respectively , Xq is the query image whose content is the same as the content set . T is the target image whose content is the same as Xc , with style similar to Xs . G is the generator that consists of a content encoder , a query encoder , a style encoder , a decoder and two glyph-attention modules as shown in Figure 1 . 3.3.2 Feature matching loss To stabilize the adversarial training , feature matching ( FM ) loss Salimans et al . ( 2016 ) is also applied in this study . Similar to the perceptual similarity measure , the FM loss uses discriminator network as the feature extractor , which contain a large amount of information about the glyph images . To obtain more disentangled representations in content and style , we use Dc and Ds to extract feature maps , as defined in ( 8 ) , Lfm = l∑ i=1 ∥∥∥D ( i ) c ( T ) −D ( i ) c ( G ( Xc , Xq , Xs ) ) ∥∥∥ 1 + l∑ i=1 ∥∥∥D ( i ) s ( T ) −D ( i ) s ( G ( Xc , Xq , Xs ) ) ∥∥∥ 1 ( 8 ) Where D ( i ) is the feature map of layer i in the discriminator . We use all layers except fully-connected layer to extract features . 3.3.3 Multi-task adversarial loss In the proposed framework , two condition discriminators Dc and Ds are used to improve model performance , each of which has its own mission . Dc and Ds aim at calibrating the model to attain accurate content and stronger style respectively . We utilize hinge version loss Lim & Ye ( 2017 ) as the adversarial objective function , as defined in ( 9-10 ) , LG = −Ds ( G ( Xc , Xq , Xs ) ) −Dc ( G ( Xc , Xq , Xs ) ) ( 9 ) LD =max ( 0 , 1 +Dc ( G ( Xc , Xq , Xs ) ) ) +max ( 0 , 1−Dc ( T ) ) + max ( 0 , 1 +Ds ( G ( Xc , Xq , Xs ) ) ) +max ( 0 , 1−Ds ( T ) ) ( 10 ) 3.3.4 Total loss A simple additive form is used for the total loss , as illustrated in ( 11-12 ) , Lθ ( G ) = λ1Lid + λ2Lfm + LG ( 11 ) Lω ( D ) = LD ( 12 ) Where λ1 and λ2 are the hyper parameters of the generator loss function , θ and ω are the weights for the generator and discriminators respectively . 4 Experiments 4.1 Experimental settings We use the Adam optimizer Kingma & Ba ( 2014 ) to optimize all models with hyperparameters β1 = 0 and β2 = 0.9 . Using the two-time scale learning rates Heusel et al . ( 2017 ) strategy , we set the learning rate of all discriminators to 4e− 4 and the generator to 1e− 4 . The number of both Xc and Xs is set to 8 and the size of all images is 128× 128× 3 . The weights in the loss function are chosen to be λ1 = 1 and λ2 = 10 . We train the model with batch size of 4 for 500,000 iterations in a Nvidia GTX 1080Ti GPU with 12G memory . 4.2 Datasets and evaluation metrics We collected 411 Chinese fonts conforming to the GB2312 standard . Each font contains 6,763 Chinese characters . We randomly selected 405 fonts as the training set and the remaining 6 fonts as the test set . Various metrics are used to evaluate the quality of the synthesized glyphs . To measure the similarity between the generated image and the target , intersection over union ( IOU ) is utilized . We further employ two classifiers with Inception-v3 Szegedy et al . ( 2016 ) backbone to distinguish between content and style labels in the test set . 4.3 Benchmarking We compare our method with four many-shot font generation methods ( i.e. , zi2zi Tian , pix2pix Isola et al . ( 2017 ) , CycleGAN Zhu et al . ( 2017 ) , ZiGAN Wen et al . ( 2021 ) ) and three state-of-the-art few-shot font generation methods ( i.e. , FUNIT Liu et al . ( 2019 ) , MXFont Park et al . ( 2021 ) and AGIS-Net Gao et al . ( 2019 ) ) . The four many-shot methods demand a lot of training data and a different model is required for each font style during inference . As such , for many-shot methods , we further split each test font library into 5763 glyphs for training and 1000 glyphs for testing . Therefore , each of the four manyshot methods generates 6 models . In comparison , few-shot methods including the proposed method GANet need only a few glyphs to transfer font style during inference and no retraining is necessary for different font styles . Different from many-shot method , the few-shot models are trained using the selected 405 font libraries . During inference , few-shot methods take a few glyphs ( i.e . 1 , 2 , 4 , 8 ) as references to synthesize the font library rather than training a new model . To make a fair comparison , we use the 1000 glyphs split from the test set to evaluate all models . 4.4 Quantitative evaluation We assess the quality of the generated images in two aspects . Firstly , Intersection Over Union ( IOU ) is used to measure the similarity between the synthesized and the ground truth . Higher IOU score indicates better result . As shown in Table 1 , we randomly sample four reference images from the 5763 glyphs of one test font , based on which each model generates 1000 glyphs . There are a total number of 6 fonts in the test dataset . The metrics in the table are calculated based on the 1000×6 glyphs for each model and averaged . Table 1 shows that our method outperforms previous state-of-the-art few-shot approaches by a wide margin and generate comparable results to the many-shot approaches . Howerver , our method needs only few reference samples and one-time training . On the contrary , many-shot methods demand re-training using thousands of glyphs for each new font style . IOU can evaluate the similarity in pixel-level , but can not represent the style and content in essence . In addition , pixel-based metrics are often inconsistent with human visual perception . To improve the reliability of the evaluation , we trained two classifiers to identify the content and style classifications respectively and then use them to evaluate model prediction accuracies for the test set . As shown in Figure 4 , MX-Font and FUNIT show comparable performance in terms of style accuracy , but they show lower performance in content preservation . AGIS-Net shows comparable performance in content accuracy , but weak in style . In other words , FUNIT and MX-Font focus only on styling and fails to preserve content structure . AGIS-Net focuses on content and fails to transfer more style . 4.5 Qualitative analysis To demonstrate the model ’ s performance visually , we evaluate the models using the remaining 6 fonts in the test set as reference glyphs . Figure 3 presents the many-shot and few-shot methods generated images from the test dataset . It is observed that many-shot methods except ZiGAN always fail in content preservation as shown in the red box . Among few-shot methods , FUNIT learns the style but the local structure in the generated glyphs is weak as highlighted by the yellow box . Compared with FUNIT , AGIS-Net has better performance in content preservation . However , it has poor performance in capturing global and local styles ( black box ) . MX-Font also succeeds in learning the style of the reference , but the content structure is unsatisfactory ( blue box ) . Overall , our proposed method generated best results qualitatively compared to other methods both in terms of content preservation and style transfer .

Summary: This paper considers Chinese glyph font style transfer problem with few reference inputs. The model has three encoders for query, style and content references; one decoder for target generation; and two discriminators for style and content. The main novelty comes with the glyph attention design with both local and global structure for style and content.