IF-Font: Ideographic Description Sequence-Following Font Generation

Authors: xinping  chen
xiao  ke
wenzhong  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:35:06

Abstract: Few-shot font generation (FFG) aims to learn the target style from a limited number of reference glyphs and generate the remaining glyphs in the target font. Previous works focus on disentangling the content and style features of glyphs, combining the content features of the source glyph with the style features of the reference glyph to generate new glyphs. However, the disentanglement is challenging due to the complexity of glyphs, often resulting in glyphs that are influenced by the style of the source glyph and prone to artifacts. We propose IF-Font, a novel paradigm which incorporates Ideographic Description Sequence (IDS) instead of the source glyph to control the semantics of generated glyphs. To achieve this, we quantize the reference glyphs into tokens, and model the token distribution of target glyphs using corresponding IDS and reference tokens. The proposed method excels in synthesizing glyphs with neat and correct strokes, and enables the creation of new glyphs based on provided IDS. Extensive experiments demonstrate that our method greatly outperforms state-of-the-art methods in both one-shot and few-shot settings, particularly when the target styles differ significantly from the training font styles. The code is available at [https://github.com/Stareven233/IF-Font](https://github.com/Stareven233/IF-Font).

Main paper: 
 introduction: 
 At the heart of font generation lies the extraction of styles from some reference glyphs of a certain font, and generate the remaining glyphs of this font. Some languages, such as Chinese, Japanese, and Korean, have a large number of characters and intricate glyph structures. Font generation can significantly reduce the labor intensity of font designers and support tasks like handwriting imitation, ancient book restoration, and internationalization of film and television productions.EMD [58] and SA-VAE [44] are based on the belief that the target glyph can be generated by integrating the content features of the source glyph with the style features of the reference glyph, as illustrated in Fig. 1 (left). The majority of subsequent works [53, 38, 47, 25, 50] continues this paradigm, but this makes font generation a sub-task of image-to-image translation, where the source glyph is morphed to match the style of the reference glyphs, rather than being truly "generated". Due to the complex structure of glyphs, achieving a distinct boundary between style and content features requires considerable effort. Consequently, glyphs produced through the disentangling strategy typically maintain similar stroke thickness to the reference glyph but align more closely with the content font regarding spatial structure, size, and inclination.To this end, DG-Font [54] incorporates deformable convolution. Diff-Font [15] integrate diffusion process to improve the network's learning capabilities. CF-Font [50] proposes content fusion. Additionally, several approaches [38, 47, 25, 30] combine fine-grained prior information such as strokes and components, to further enhance generation quality. These methods essentially follow the content-style disentanglement paradigm, in scenarios where the content font differs substantially from the target font, the resulting glyphs are susceptible to artifacts such as missing strokes, blur, and smudge.We abandon the source glyphs in favor of Ideographic Description Sequence (IDS) to convey content information. It is based on a simple fact: without disentangling features, there is no risk of incomplete disentanglement. Consequently, font generation is reframed as sequence prediction task, where the objective is to generate the tokens of the target glyph based on the given IDS and reference glyphs. This approach mitigates the impact of source glyphs on the outputs and diminishes artifacts by leveraging the prior knowledge embedded in the quantized codebook. The users are allowed to formulate IDSs to create non-existing Chinese characters, such as kokuji 2 (Japanese-invented Chinese characters), provided that the corresponding structures and components have been learned during training. This endows the model with certain cross-linguistic capabilities. We refer to this method as Ideographic Description Sequence-Following Font Generation, or IF-Font. In summary, the key contributions of this paper are as follows:• We propose IF-Font, which abandons the previous content-style disentanglement paradigm and generates glyphs through next-token prediction.• We devise a novel IDS Hierarchical Analysis (IHA) module that analyzes the spatial structures and components of Chinese characters. It allows our decoder flexibly control the generated content with the encoded semantic features.• Leveraging corresponding IDSs, we design the Structure-Style Aggregation (SSA) block to extract and efficiently aggregate the style features of reference glyphs. 
related works: 
 Image-to-image translation Image-to-image translation (I2I) aims to learn a mapping from a source domain to a target domain, requiring the transformation of images in the source domain into those in the reference style's target domain while preserving their content. Pix2Pix [21] is the first I2I method that trains GANs [13] using paired data. CycleGAN [60] achieves unsupervised training through cycle consistency loss, although it is limited to transformations between two domains. UNIT [29] enforces the latent codes of images from two distinct domains to be identical, while employing separate generators for images in each domain. This process embodies the concept of disentanglement. MUNIT [19] further refines UNIT's latent code into content and style codes. Multimodal image translation can be achieved by combining the content code with different style codes.While applying an image-to-image translation framework for font generation is currently the mainstream approach, we believe this to be inappropriate. Unlike ordinary images, the boundaries between content and style in glyphs are ambiguous. For example, although handwriting will certainly differ when the same characters are written by different writers, their semantic meanings remain unchanged.Given that glyph features are challenging to disentangle, we utilize style-neutral IDSs to determine characters, thus avoiding any influence on the styles of results due to insufficient disentanglement of content glyphs.Few-shot font generation Few-shot learning [12] represents the prevailing research focus in font generation, aiming to simulate the target style with just a handful of reference glyphs. Font generation methods fall into two categories based on their utilization of implicit structural information within glyphs. Methods treating glyphs as general images possess broader applicability, enabling generation across various languages. Conversely, methods leveraging structural information typically yield higher quality outputs but are confined to specific language.Among the methods that do not incorporate structural information, EMD [58] stands out as the earliest attempt to disentangle glyphs into content and style features. DG-Font [54] employs deformable convolution [7] to capture the glyph deformations. FontRL [31] uses reinforcement learning [45] to draw the skeleton of Chinese characters. FontDiffuser [55] models the font generation task as a noiseto-denoise paradigm. Shamir et al. [42] explores a parametric representation of oriental alphabets, which can elegantly balance glyph quality and compression. In vector font generation, Deepvecfont [51] exhaustively exploit the dual-modality information (raster images and draw-command sequences) of fonts to synthesize vector glyphs. CLIPFont [43] controls the desired font style through text description rather than relying on style reference images.In terms of methods that incorporate structural information, SA-VAE [44] utilizes the radicals and spatial structures of Chinese characters. CalliGAN [53] adopts the Zi2Zi framework [48] and fully decompose Chinese characters into sequences of components. SC-Font [23] further decomposes Chinese characters into stroke granularity. DM-Font [4] proposes dual memory to update component features continuously. LF-Font [38] represents component-wise style through low-rank matrix factorization [3] . MX-Font [37] automatically extracts the features through multiple localized experts. FS-Font [47] demands that reference glyphs include all components of the target glyph, otherwise may result in a degradation of generation quality. CG-GAN [25] employs GRU [6] and attention mechanism to predict component sequences. XMP-Font [30] performs multimodal pre-training on Chinese character strokes and glyphs data. Most of the above methods are constrained by the content-style disentanglement paradigm. They often neglect the presence of Ideographic Description Character (IDC) which refers to the spatial structure of Chinese characters, suffering from artifacts and inconsistent styles.Vector quantized generative models Vector Quantization (VQ) typically follows a two-stage training scheme. Initially, it employs a codebook to record and update vectors, converting them from a continuous feature space to a discrete latent space. Subsequently, it models the distribution of these quantized vectors with a decoder to predict tokens, which are the codebook indices, and then restores the tokens to a image.VQ-VAE [35] is the first to incorporate quantization into the variational autoencoder (VAE) [24] framework. VQ-VAE2 [41] performs multi-scale quantization and adopts rejection sampling [1] VQGAN [9] acquires the codebook with the help of GAN [13] and employs Transformer [49] to replace the PixelCNN [34] used by VQ-VAE [35] . RQ-VAE [26] proposes a residual quantizer. BEiT [2] performs masked image modeling (MIM) on the patch view with the supervision of visual tokens.MaskGIT [5] directly models visual tokens and proposes parallel decoding. MAGE [27] is similar to MaskGIT [5] , but introduces ViT [8] and contrastive learning [14] . DQ-VAE [18] further encodes images with variable-length codes.Since quantization is equivalent to tokenizing images, many methods attempt to enable multimodal generation. The dVAE proposed by DALL-E [40] relaxes the discrete sampling problem utilizing Gumbel-Softmax [33, 22] , outputing the probability distribution of codebook codes. SEED [11] designed a Causal Q-Former to extract image embeddings and quantize them. LQAE [28] trains VQ-VAE [35] to quantize the image into the frozen LLM codebook space directly. SPAE [56] introduces multi-layer and coarse-to-fine pyramid quantization and semantic guidance with CLIP [39] . V2L [61] further proposes global and local quantizers. Given the absence of a pre-trained model tailored for IDS, we directly concatenate visual tokens with IDS tokens to performed autoregression.3 Method As shown in Fig. 2 , given a target character c y , reference characters C r = {c i r } k i=1 and reference glyphs G r = {g i r } k i=1 , the goal of our framework is to generate a glyph ĝy that conforms to the semantics of c y and has a style consistent with G r . To achieve this objective, we analyze c y with IHA to derive its associated IDS ι y , which is then encoded as a semantic feature f ι . Likewise, we can obtain the IDS I r = {ι i r } k i=1 corresponding to C r . Following this, we employ the similarity module E sim to assess the relationship between ι y and I r . Combined with f ι and the output of E sim , the features Fr corresponding to G r are fused into the final style feature f r in the SSA block. ι y is reshaped as initial tokens t <0 , which is fed into the decoder D along with f r for autoregressive modeling. Finally, the predicted glyph tokens ty are decoded with the pre-trained VQGAN to obtain the generated glyph ĝy . 
ids hierarchical analysis: 
 A simple alternative to using a source glyph as input is to directly employ the character itself to control the semantics of the output. However, considering the vast number of characters in Chinese, this approach proves to be impractical due to its expensive cost. Moreover, it overlooks the structural intricacies of characters.Ideograph Description Sequence is a structural description grammar for Chinese characters defined by the Unicode Standard, which consists of description characters and basic components (mainly Chinese characters) through a prefix notation. Decomposing Chinese characters into their corresponding IDSs can notably streamline the vocabulary, allowing characters with similar structures or components to share common features.However, a Chinese character may have multiple equivalent IDSs. Many Chinese characters have a top-bottom or left-right structure, the IDCs follows a long-tail distribution, presenting challenges for model training. Fortunately, the left-middle-right structure of Chinese characters can be equivalently represented by two left-right structures. Similarly, the top-middle-bottom structure equals two top-bottom structures. The examples can be found in Fig. 3 .Based on the above observation, we employ a IDS Hierarchical Analysis (IHA) module. Instead of rigidly querying the decomposition table when determining the IDS of a character, we examine whether the character follows a left-middle-right or top-middle-bottom structure. Subsequently, we construct multiple equivalent IDSs for the same character through random selection. To summarize, c y and C r are initially decomposed into ι y and I r respectively. In the IDS encoder, ι y is padded to the maximum sequence length l I and encoded into the associated semantic feature f ι ∈ R l×c . 树 = ⼜ ⽊ ⼨ ⿲ 权 ⿰ = 对 ⿰ ⼜⼨ ⽊ ⼜⼨ ⿰⽊⿰ = = ⿰ ⼨ ⿰⽊ 克 = 古 = 兄 = = ⼉ ⼗ ⼗⼝⼉ ⿳ ⿱⿱ ⿱ ⿱ ⿱ ⿱ ⼝⼉ ⼗ ⼝⼉ ⼗Many previous methods [55, 10, 59, 25, 30, 54] overlook interactions between reference and target characters during the extraction of reference styles, resulting in a lack of relevance in the extracted features. The more closely the reference character resembles the target character, the more effortlessly the generation process can preserve the style. Ideally, employing the target glyph itself as the reference, known as self-reconstruction, should yield the most effective output. Although FS-Font [47] endeavors to ensure that the reference characters cover all components of the target character, its implementation hinges on predefined content-reference mapping, which may limit its adaptability. To address this issue, we propose a Structure-Style Aggregation (SSA) block, as shown in Fig. 2 . We convert the reference glyph G r into the latent space of VQGAN and encode it one by one into the corresponding intermediate featuresFr = { f i r ∈ R h•w×c } k i=1 .The similarity module E sim evaluates the resemblance between each reference IDS I r and the target IDS ι y , considering whether they share identical structures or components. It produces a set of similarity weights Sim = {sim i ∈ R 1 } k i=1 , which can guide the subsequent feature fusion process. The fusion module E f use consists of two branches: global and local style feature aggregation, as shown in Fig. 4 . The global features mainly focus on the glyph layout, stroke thickness, and inclination, which can be obtained by merging the coarse style features Fr with the similarity weight Sim obtained in the previous step:EQUATIONWhile local features are more concerned with the strokes, such as stroke length, stroke edge, and other nuances, we adopt cross-attention to gather the required style feature according to the needs of the target character:EQUATIONwhere flatten 2 (•) denotes flattening the first two dimensions of the feature, and L q , L k , L v are linear projections, and LayerNorm(•) denotes layer normalization. In Eq. 3, we obtain the aggregated style feature based on Eq. 1 and Eq. 2, where • denotes concatenation operator:EQUATIONThere are some strategies to maintain style consistency: integrating consistency loss [59, 25] , introducing a discriminator to determine the generated style [25, 47, 36] , or treating the extracted style feature as a variable for further optimization [50] . These approaches are indeed beneficial for improving the generation quality, but they may be inflexible or introduce additional parameters.In this paper, we propose a streamlined approach named the Style Contrast Enhancement (SCE) module, which promotes the proximity of representations for the same style and the distance between representations for different styles. We apply a linear projection to the style feature f r , resulting in a contrastive feature e = M LP (f r ).In one batch, we denote the indices of contrastive features corresponding to all samples asE * = {i ∈ N | 0 ≤ i < 2N },where N represents the batch size. The dimensionality of E * is double the batch size N due to our utilization of a momentum encoder [16] . Each sample x a within the batch undergoes processing by both the encoder and the momentum encoder, yielding two outputs that serve as positive pairs. The negative sample set is defined asE -= {i ∈ E * | s(x i ) ̸ = s(x a )}, while the positive sample set is E + = {i ∈ E * | i ̸ = a, s(x i ) = s(x a )}, where s(•) denotes the operator used to retrieve the corresponding style. The contrastive loss can be calculated as follows:EQUATIONThe decoder D is provided with both semantic feature f ι and style feature f r . It treats f ι as the initial tokens t <0 = f ι , and then predicts the distribution of the next token autoregressively as p(t i | t <i , f r ).Each newly predicted token is appended to the previous tokens for the subsequent iteration, it continues until all tokens are predicted. The likelihood of the entire sequence can be calculated asl T -1 i=0 p(t i | t <i , f r ).There are two ways for incorporating f r . The most straightforward approach involves using f r as initial tokens, represented by t <0 = f ι • f r , akin to f ι . These tokens participate in each forward pass, relying on the self-attention mechanism to extract and integrate features. However, it is only practical for low-resolution scenarios. Viewing f r as tokens may lead to excessively long sequences, requiring a balance between computational efficiency and generation quality. To address this challenge, we incorporate f r into each block of the decoder D via cross-attention. The tokens act as queries to align with the corresponding style features. t <l T denotes all the predicted tokens, from which t <0 is removed to get the glyph tokens ty . The objective in Eq. 5 is to maximize the log-likelihood of the token sequence.EQUATIONFinally, the model can be trained according to the objective in Eq. 6.EQUATIONwhere λ cl controls the weight of contrastive loss, cf. Eq. 4, we set λ cl = 0.5 in our experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the societal impacts in Section 1 and Appendix E. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
dataset and evaluation metrics: 
 Table 1 : 12 IDCs used in this paper. The training set comprises 3,300 randomly selected Chinese characters and 424 fonts, referred to as Seen Fonts and Seen Characters (SFSC).There are two test sets: the first includes the same 3,300 characters but with different 40 fonts, called Unseen Fonts and Seen Characters (UFSC). The second test set consists of the remaining 200 characters and the same 40 fonts, known as Unseen Fonts and Unseen Characters (UFUC). We found a publicly accessible IDS decomposition table 3 . However, it exhibits several redundant entries and circular references, as well as an absence of some characters. Therefore, we performed simplifications and enhancements, reducing the number of IDCs to the 12 depicted in Table 1 , which is sufficient for most frequently used Chinese characters. For convenience, we set the basic component's IDS as itself.We compare all methods in the following metrics, i.e., FID [17] , L1, LPIPS [57] , RMSE, and SSIM [52] . Since aesthetics is inherently subjective, we conduct a user study for all methods to evaluate model performance based on user satisfaction. We observe that the existing font generation methods have differences in data preprocessing and metric selection. Factors such as glyph resolution, the padding around glyphs, the range of pixel values, the number of reference glyphs, and the evaluation function implementation all influence metric values. For example, NTF [10] and CF-Font [50] center the glyph within the canvas, leaving white space around it. However, this leads to inflated metric calculations. To ensure a fair comparison, we adopt consistent test data and metric implementation across all methods under evaluation. Specifically, we eliminate padding around the glyphs, fix the canvas resolution to 128 pixels, scale the data range to [0, 1], utilize SqueezeNet [20] as the network type to calculate LPIPS [57] , and select the inceptionv3 [46] feature layer with 2048 dimensions for FID [17] calculation. We compare the proposed IF-Font with seven SOTA methods on our UFSC and UFUC datasets respectively, including CG-GAN [25] (CVPR 2022), LF-Font [38] (TPAMI 2022), FS-Font [47](CVPR 2022), CF-Font [50] (CVPR 2023), VQ-Font [36] (ICCV 2023), NTF [10] (CVPR 2023) and FontDiffuser [55] (AAAI 2024). All methods are trained from scratch on our SFSC dataset according to their respective official codes and default configurations. We slightly modify the codes of CG-GAN, LF-Font, FS-Font, VQ-Font and FontDiffuser to support varied numbers of reference glyphs.Table 2 compares IF-Font and other SOTA methods. IF-Font significantly surpasses competitors in all reference glyph number settings for both datasets. Notably, IF-Font's performance on FID metric is exceptionally low, reaching a single-digit score, thanks to the high quality and clarity of the samples it generates. FS-Font [47] relies heavily on the predefined content-reference mapping, whereas the reference glyphs in all our experiments are randomly selected. Especially, when only one reference glyph is provided, covering all components of the target character becomes challenging, leading to poor performance of FS-Font, as shown in Fig. 10a . NTF [10] also struggles to imitate the target style, the layout of its generated samples often resembles that of the source font. In cases where there's a significant disparity between the source and target styles, NTF is prone to missing strokes.We attribute CF-Font's performance to its reliance on fusing contents of 10 basic fonts. However, there happen to be a gap between the train dataset and our evaluation dataset. We conduct a user study through Fuxi Youling Crowdsourcing Platform4 to quantify the subjective quality. For each test dataset, 5 characters are randomly selected, and each model is required to generate glyphs corresponding to 40 unseen fonts. A total of 30 participants are asked to select the option that most closely matches the ground truth from the generated results. The outcomes of the user study are presented in the last column of Table 2 . We present the corresponding samples from Table 2 in Fig. 5 . IF-Font stands out by producing the clearest and most style-consistent samples. In contrast, FS-Font [47], LF-Font [38] , CF-Font [50] , and other models exhibit issues such as stroke errors or blur. VQ-Font [36] and NTF [10] are constrained by the source font and struggle with flat or narrow layouts, resulting in incorrect structures. VQ-Font even tends to crop marginal parts of glyphs to fit the target style. While CF-Font generally perserves the correct glyph layout, its outputs exhibit noticeable artifacts, indicating some remaining style inconsistencies. The performance of FontDiffuser [55] is also outstanding, but there is still a slight deficiency in the imitation of font styles. On the other hand, IF-Font maintains the correct character structures and excels in aspects such as the aspect ratio, glyph layouts, and stroke details. 3 . Red, blue and green boxes represent the missing components, style inconsistency and corresponding improvements respectively.Main modules Removing the IHA, SSA, and SCE modules of IF-Font, a baseline model can be obtained. For a input character, it directly looks up the decomposition table to derive the corresponding IDS, and then encode the semantic feature f ι through a embedding layer. The intermediate style features Fr are directly averaged as style features f r , excluding any interactions with the similarity weight Sim and semantic feature f ι . The whole model relies solely on cross-entropy loss for supervision. Building upon the baseline, we incrementally reintegrate three modules to assess their individual contribution. Quantitative results are presented in Table 3 , while Fig. 6 provides visualizations of these results. For further ablation study of the SSA block, please refer to Table 6 in Appendix B.1. Upon integrating our modules, a consistent improvement is observed across most metrics. Fig. 6 illustrates how IHA alleviates the issue of missing components present in the baseline. SSA enhances style consistency, while SCE improves the capability to imitate styles. We further analyze the impact of three different IDS granularities: components, strokes and mixed. Please see Table 5 in the Appendix for the examples of these granularities. Table 4 shows the quantitative results. Stroke granularity results in performance degradation across three metrics. We attribute this decline to the conflict between IDSs, hindering the model's ability to identify the target character. An attempt to concatenate IDSs from both granularities yields performance akin to that of component granularity. While this approach extends the sequence considerably, hence we opt for component granularity. To demonstrate the effectiveness of Structure-Style Aggregation block, we visualize the attention maps corresponding to each IDC and component within the IDSs relative to the reference glyphs, as depicted in Fig. 7 . Specifically, we choose the matrix A ∈ R l×k•h•w in local feature calculation. For each position i of the target IDSι y , there exists a corresponding attention map A i ∈ R k•h•w , which indicates the attention that ι i y pays to the k reference features. We present the attention map A i to visualize the distribution of attention weights directly. Additionally, we apply opacity to this map and overlay it onto the original reference glyph.草 胖 乙 萌 ⿱ ⿰ ⺾ ⽇ 胖 ⽉ 乙 草 草 胖 乙 草 胖 乙 乙 草 胖 乙 草 胖As we can see, when the target IDC or component exists in the reference glyph, more attention will be paid to the corresponding place. For instance, in the first row, the first, second, and fourth columns, and in the second row, the third and fifth columns are distinctly highlighted. Conversely, if the reference glyph lacks the target component, the local branch tends not to engage, as evidenced by the nearly blank third row. This approach stems from a preference to avoid forced attention allocation which might lead to interference. Instead, leveraging the average style captured by the global branch helps maintain a baseline quality of the output. We validate the flexibility of IF-Font by generating the glyphs using IDSs of non-standard Chinese characters. Fig. 8 shows our experimental results. IF-Font demonstrates robust generalization by following given IDSs to produce new glyphs with accurate structure and consistent style. We fixed the font to "Sarasa Gothic" in the experiment, which is a CJK programming font. The last two columns lack ground truths due to their entirely non-existent characters.5 Discussion Failure cases Although our method enables high-quality generation under most circumstances, it still struggles on some hard cases, as illustrated in Fig. 9 . IF-Font encounters difficulties with fancy and irregular font styles, including those with decorations, extremely flat or narrow layouts, excessively curved strokes, and calligraphic writings. Despite these challenges, it continues to preserve the correct character structure. Further discussion on the reasons for the difficulties in generating these fonts can be found in Appendix C.3.Target Output GT 堤 鸿 慧靠粪 窟 魔 簸 染茫 雾 琉磁 辰 虏舍Usability We focus on CJK characters due to their unique spatial structures, which better reflect the characteristics of our method. By expanding the vocabulary and incorporating relevant data for training, IF-Font can also be adapted to handle other character sets.Advantages Conforms to writing habits. We believe that the process of autoregressive modeling with IDS implicitly contains the order of writing. Scalability. Good scalability can be achieved by leveraging the mature experience of LLMs. Robustness. Due to vector quantization, glyphs are represented by a limited number of tokens (only 256 types), which reduces the learning difficulty for the decoder and decreases the likelihood of artifacts and other issues in the generated results. 
conclusion: 
 We have presented IF-Font, a novel font generation paradigm. IF-Font redefines font generation as a sequence prediction task by quantizing glyphs as token sequences and leveraging Ideographic Description Sequence (IDS) to control the semantics of the generated glyphs. This method demonstrates exceptional capability in managing complex styles while preserving the correct structures.Benefiting from the flexibility of IDS, our method enables the creation of glyphs. This is achieved by formulating legal IDSs, which is a salient advantage over other methods that typically require the character to be present in at least one font as a precondition for generation. Refining and improving the IDS decomposition rules is considered future works. Furthermore, exploring the integration of IDS into handwritten font generation may yield interesting insights. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We describe safeguards in Appendix E.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Appendix F for the references and licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 Right: The proposed paradigm. We first autoregressively predict the target tokens and decode them with a VQ decoder. Orange boxes show the main difference between the two paradigms. 
 
 Appendix: A.1 Implementation Details was trained in a server equipped with an Intel Xeon Silver 4110 CPU, 128 GB of RAM, and an NVIDIA Tesla V100 PCIe 16GB GPU. The training takes about 42 hours for 15 epochs with batch size of 128.The VQGAN model in IF-Font loads pre-trained weights. It has a codebook size of 256, and the encoder downsamples the image with a factor of 8. Consequently, the length of the codebook indices corresponding to a single glyph is l T = 256, whereas the IDS have a fixed length of l I = 35. Our decoder consists of 10 Transformer blocks, each integrating a self-attention layer, a cross-attention layer, and a multi-layer perceptron (MLP). We have configured the model with 8 attention heads and a feature dimension of 384. In IF-Font, parameters are optimized using the AdamW optimizer [32] , which employs a learning rate schedule that includes warmup and cosine annealing.We primarily obtain fonts from the Foundertype platform under a personal non-commercial academic research license. We present the examples of three IDS granularities in Table 5 . 10a depicts the one-shot setting, where style extraction poses a considerable challenge, leading to comparable performance across most models. Nevertheless, as the number of reference glyphs increases, the advantage of IF-Font becomes progressively apparent in Fig. 10b and 10c .Since the target glyph is reconstructed from a quantized sequence, the accuracy of this reconstruction imposes a ceiling on IF-Font's potential performance. As shown in Figure 9 , there are slight differences between the target and the ground truth. Given that the glyphs are grayscale images and relatively simple, fine-tuning the VQ-GAN or switching to a superior quantized method is expected to minimize the loss of accuracy. We use original VQ-GAN checkpoints in order to highlight our main contributions.In fact, IDS is still not sufficiently perfect to identify Chinese characters. On the one hand, there are rule conflicts: a few Chinese characters are too similar, and their IDSs of stroke granularity are exactly the same. On the other hand, the spatial descriptions are insufficiently clear. For example, the left-right structure indicates that two components are placed on the left and right, but the distance between them is not specified, which requires the model to distinguish them through sufficient learning. We believe that generating novel fonts presents significant challenges for several reasons. First, they substantially deviate from standard character structures, involving a trade-off between the number of references and the quality of generation. Furthermore, font design involves subjectivity and randomness. For instance, in the first font depicted in Fig. 9 , the position, size, and shape of the auspicious cloud patterns are the result of manual design. It is important to note that this issue is common across all font generation methods. Similar discussions can be found in LF-Font [38] and CF-Font [50] . this limitation has not been well addressed yet. We demonstrate the results of other methods on novel fonts in Fig. 11 to support the above conclusion.To achieve high-quality generation, IF-Font requires training data to cover as many character components and font styles as possible. Due to the utilization of attention mechanisms, IF-Font entails quadratic computational complexity related to sequence length, while inference relies on autoregressive processes, resulting in a slow sampling speed.To be accurate, we pay the Fuxi Youling Crowdsourcing Platform (a third-party platform) to conduct the user study. The participants in the user study are users of that platform, who will be informed of the full details in advance. We limit the number of participants to 30, and they are allowed to choose whether or not to participate and can complete the evaluation at any time and in any location using their own devices (usually a phone or computer).All model names in the evaluation are replaced with numbers, and participants only need to select the option that best matches the ground truth from the model results displayed on the page. After the evaluation, the platform pays participants and provides us with de-identified model effectiveness data.We have no direct contact with participants and are unable to obtain their specific identities.IF-Font could help to improve the productivity and creativity of font designers, and there is also hope for preserving ancient calligraphic works. In addition, IF-Font is capable of imitating the font style from as little as one reference glyph. That makes it easy to reproduce commercial font designs, raising concerns regarding potential copyright infringement. To safeguard the rights of font creators, we urge users to adhere to license requirements and call for the responsible use of our method.We present a complete list of references and licenses in Table 7 for all the existing assets we used in this work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://www.sljfaq.org/afaq/kokuji-list.htmlhttps://babelstone.co.uk/CJK/IDS.TXThttps://fuxi.163.com/solution/data 


Decision: Initially, this paper received mixed ratings: 5, 4, 6 and 5. The comments included more experimental comparison, advantages over some methods, clarification and discussions for the proposed method and experiments, and so on.After post-rebuttal discussion, the ratings were changed to 6, 5, 7, and 5. Three reviewers were satified with the rebuttal and upgraded the ratings. The other reviewer was also satisfied with rating and did not update the rating.  The AC read the comments and the paper carefully.

Reviews: review: paper_summary: paper_summary: IF-Font introduces a novel approach to few-shot font generation by using Ideographic Description Sequence (IDS) instead of traditional source glyphs to control the semantics of generated glyphs. This method quantizes reference glyphs into tokens and models the token distribution of target glyphs using IDS and reference tokens. IF-Font effectively synthesizes glyphs with neat and accurate strokes, significantly outperforming existing methods in both one-shot and few-shot settings, particularly when the target styles differ from the training font styles. The method redefines font generation as a sequence prediction task, enhancing the quality and consistency of the generated glyphs.
strength_weakness: strength_weakness: 1. The author proposes a content-style disentanglement method where style extraction relies on the decomposition of glyphs. However, this setup may not be necessary. In the fields of diffusion models and generative AI, there are many methods that can extract styles from a single image with minimal content interference, such as IP-Adapter and Instant-Style. The work FontDiffuser, which generates text using diffusion models, does not decompose the text but still achieves excellent style and content disentanglement. Therefore, I have doubts about the advancement and necessity of the method proposed in this paper.

2. The compared methods are relatively old and do not include comparisons with the most advanced methods such as FontDiffuser.
Since the target glyph is reconstructed from a quantized sequence, the accuracy of this reconstruction sets a ceiling on IF-Font's potential performance.
questions: questions: 1. Is there any cherry-picking of results?
2. What is the real usability rate of the proposed method?
3. Besides inference speed, what advantages does the VQ-GAN method have compared to diffusion-based methods?
limitations: limitations: see weaknesses
paper_summary: paper_summary: This paper proposed IF-Font handles the task of few-shot font generation via a VQ-GAN based framework. Compared to most existing methods that encode content images, IF-Font only encodes Ideographic Description Sequence (IDS) to convey content information of target characters. Experimental results show the proposed method excels in synthesizing glyphs with neat and correct strokes, and enables the creation of new glyphs based on provided IDS.
strength_weakness: strength_weakness: (1) The idea of utilizing component/stroke information of glyphs has already been exploited in many existing papers, such as LF-Font and XMP-Font. What is the key difference that distinguishes IF-Font from those methods?

(3) If the key difference is only encoding IDS to convey content information, I wonder if there is an ablation study verifying that this is a better design than encoding both modalities (i.e., component sequence and glyph images) in XMP-Font?  The current Section 4.4 (Ablations Studies) is a bit unclear to me.

(3) The authors mentioned that they construct multiple equivalent IDSs for the same character through random selection. Does it mean during training, a random IDS is selected to feed the IDS encoder if there are multiple equivalents? Can the IDS encoder resolve the ambiguities in representing the shape of a character? My guess is this is somewhat similar to many-to-one mapping so it can be done. Correct me if I am wrong.
questions: questions: What is the detailed structure of IDS-Encoder? Is it a Transformer to encode a sequence?
limitations: limitations: The authors discussed the limitations in Section 4.6. Regarding the first limitation (fancy and irregular font styles), I believe the discussion could be expanded further. One key reason might be that novel font styles do not always adhere to the typical topology of characters. For example, the second-to-last font in Figure 5 deviates from the norm, challenging the necessity of using IDS (or pre-defined component sequences). I would like to hear the authors' opinions on this.
paper_summary: paper_summary: This paper approaches the task of Few-shot Font Generation (FFG) for Chinese characters, proposing to model the target glyph with Ideographic Description Sequence (IDS) tokens to achieve style-content disentanglement. Reference font images and the target character’s IDS are fed into a VQGAN-based pipeline to decode the output image autoregressively as quantized codebook tokens. The results show that this method outperforms SOTA in one- and few-shot settings and for significantly differing font styles.
strength_weakness: strength_weakness: The differences in most metrics when ablating components (Table 3) mostly seem quite small, compared to the larger differences in metrics when comparing to competing methods (Table 2). It’s not immediately convincing that the added complexity of these components (particularly IHA) are justified. On the other hand, the significant drop in FID when using I+S without C in Table 2 seems important and is not explained. If the use of these components gives a qualitative advantage that is not fully reflected in the metrics, this should be illustrated or included in the user study.

The design of the SSA (Sec 3.2) includes sub-components (global and local branches) which are not ablated. It would also help to indicate which is which in Figure 4. Claims about these controlling features such as stroke width, edges etc. (L134-147) could be justified. I also found Sec 4.4 hard to follow; how does Figure 7 show that SSA is effective? It’s not obvious where the attention should be focussed for positioning IDC’s or for those that don’t appear in the reference glyphs.

The method is tested on data collected by the authors. It is unclear if the proposed method would outperform the competing methods (Sec 4.2) if trained on datasets used in prior works.
questions: questions: Is this approach specific to CJK characters? If so, it seems like this should be mentioned in the title, abstract, and conclusion.

The use of “SCA” is confusing as in L248 it is mentioned as a “module”, but the acronym “SCA” does not appear before. Assuming this refers to “Style Contrast Augmentation” (Sec 3.3) this seems to be the contrastive learning loss term and not a module in the model architecture. I’m also not sure if “augmentation” is the right term since this is not performing data augmentation.

What does it mean on L111 that directly employing the character would be “impractical”? Isn’t this done in prior works? Additionally, L119 says that the long-tail distribution of IDC’s poses a challenge for training, but isn’t this shown in the first row of Table 3 with overall good results? (relative to the competing methods in Table 2)

Will the curated data and IDS table (L202) be made available?

The paper mentions a user study (L208 etc.). Who were the participants, were they paid (L236 mentions “volunteers”), and does this require IRB approval?

Typos: L19-20 incomplete sentence, L199 L131-136 L217 L311 L325-326 grammar, L139 L176 L250 formatting issues, L497 missing period.
limitations: limitations: Limitations and societal implications are adequately discussed.
paper_summary: paper_summary: This paper presents a method that generate Chinese glyphs using token prediction approach. The core contribution is to leverage the concept called Ideographic Description Sequence and develop a network architecture to generate IDS that represent the target character.
strength_weakness: strength_weakness: The weaknesses of this paper are:
- The exposition is sometimes confusing. I would encourage the authors to explicit show the output of the network before putting the output together as a glyph (I suppose the output sequence is composed of multiple tokens.)
- I understand the proposed method aims to focus on generating Chinese characters, but it seems can not be generlized to other writing systems, including Roman characters?
questions: questions: My questions and concerns are:
- The exposition of the paper is quite unclear to me. In the paper, I did not find an explanation of what each token really represents. Is it a part of the glyph? The decoded results shown in the paper are always a completed glyph.
- It is unclear how to compose the predicted token into a completed glyph.
- I recommend the authors to survey and discuss more about Ideographic in previous font manipulation is not thorough.
    - e.g., Ariel Shamir, A. Rappoport, Compacting oriental fonts by optimizing parametric elements, The Visual Computer, 1999
- It is unclear what does "IDS is not perfect enough to identify Chinese characters" in the limitation section mean?
limitations: limitations: - as mentioned above, I am curious how well the method can generalized to other writing system? If it is hard, then I suggest authors discuss this in the limitation section.

Comments: title: Thank you for the comments
My final recommendation
Result of ablation study requested in Q2
Reviewer-author discussions
comment: Thank you for acknowledging our work. We would be glad to cite the two papers in Section 2 (Related Works) of the final version.

Just a gentle reminder: We have noticed that the score has not been updated yet (the reviewer mentioned that it would be raised to 6), which may be due to forgetting to save. :D
Thank you for the detailed response. Due to the debate regarding the paper's motivation and novelty, I am glad to improve my current score to 6. Nevertheless, I hope this work can cite the following paper.

1. CLIPFont: Text Guided Vector WordArt Generation
2. FontDiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning
Thank you very much for raising the score and for your thoughtful comments. Following your advice, we will add the expanded discussion on limitations to the final version. The mentioned Deepvecfont does indeed take sequences as input and avoid directly morphing the content glyph, we are glad to cite it in our paper.
Thank you to the authors for the detailed responses, which have addressed most of my concerns. My final recommendation is a borderline accept. 

The ablation study on dual-modality inputs effectively demonstrates the superiority of IDS, and I now have a clearer understanding of its approach. 

I would suggest incorporating the expanded discussion on limitations into the final version. 

Additionally, including a reference to the paper [1] on vector font synthesis would be beneficial, as it similarly encodes glyphs using sequential descriptions and avoids morphing the content glyph. 

Ref: "Deepvecfont: synthesizing high-quality vector fonts via dual-modality learning." ACM Transactions on Graphics (TOG), 2021.
Thank you for your time in handling this submission. The term "volunteers" used in the paper is incorrect, and we apologize for any confusion it may have caused.

In fact, our user study was conducted through Fuxi Youling Crowdsourcing Platform (a third-party platform). The participants were users of that platform, who could come from different age groups, genders, occupations, etc., and we had no conflict of interest.

Furthermore, we do not directly contact the participants, nor can we obtain their specific identities. We only receive de-identified results through the platform. Additionally, all model names were replaced with numbers to ensure that participants make judgments based solely on glyph quality. Through these measures, we have attempted to minimize potential biases caused by the composition of the participant group.

We will incorporate the mentioned information into our revision to avoid similar concerns. We hope that our responses have addressed your concerns, and we are happy to discuss and clarify any further questions or doubts.
Thank you for your time in reviewing our work. We have addressed the concerns raised by Reviewer ufc8. As mentioned in our rebuttal to Reviewer ufc8, the term "volunteers" used in the paper is incorrect, and we apologize for any confusion it may have caused.

To be accurate, we paid Fuxi Youling Crowdsourcing Platform (a third-party platform) to conduct the user study. The participants of the user study were users of that platform, who were informed of all the details of the user study. They are allowed to choose whether or not to participate and can complete the evaluation at any time and location using their own devices (usually a phone or computer).

All model names in the evaluation were replaced with numbers, and participants only needed to select the option that best matched the ground truth from the model results displayed on the page. After the evaluation, the platform paid participants and provided us with de-identified model effectiveness data. We have no direct contact with participants and are unable to obtain their specific identities.

This process does not involve any identifiable private information, and the privacy and rights of participants are fully protected, therefore IRB review is not required. We will include the mentioned details in the revised version to avoid confusion. We hope that our responses address your concerns. Please let us know if you have further questions after reading our comments.
Thank you for your thorough response and I think most of my concerns are addressed.
Dear reviewer, we added content glyph as input in addition to the IDS. The experimental results are listed below:
|       UFUC&3shot             | FID↓ | L1↓ | LPIPS↓ | RMSE↓ | SSIM↑ |
|------------|---------|---|---|---|---|
| Both Modalities   | **8.2603**  | **0.1576**  | **0.1316**  | **0.3744**  | **0.4830**  |
| Only IDS           | 8.4922  | 0.1597  | 0.1338  | 0.3775  | 0.4782  |

As expected, the additional information and parameters bring about an improvement in performance. However, please note: 
1. This improvement is not significant compared to the increase in training costs (double the previous amount). 
2. The style of the content font will affect the generated results to some extent.

Unfortunately, it appears that image uploads are not allowed during the author-reviewer discussion period, we are unable to share the visualization.
We sincerely appreciate the reviewer's insightful suggestions and the increased rating. We will incorporate these findings and discussion into the revised paper.
Thank you for your thorough and lucid response. I believe this addresses my concerns so I have updated my rating accordingly. I encourage the authors to incorporate these findings and discussion into the final version of paper.
To the authors:
Thanks a lot for the rebuttal.

To all reviewers: 
Please read the rebuttal and other review comments if necessary, and have a discussion with the authors if you have further comments.

Best,
AC