IF-Font: Ideographic Description Sequence-Following Font Generation
Authors: Xinping Chen
Xiao Ke
Wenzhong Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Font Generation
Vector Quantization
Ideographic Description Sequence
Multimodal
Primary area: machine_vision
TL;DR: IF-Font uses Ideographic Description Sequence as the semantic condition instead of the content image, outperforming all state-of-the-art methods in the few-shot font generation task.

Abstract: Few-shot font generation (FFG) aims to learn the target style from a limited number of reference glyphs and generate the remaining glyphs in the target font. Previous works focus on disentangling the content and style features of glyphs, combining the content features of the source glyph with the style features of the reference glyph to generate new glyphs. However, the disentanglement is challenging due to the complexity of glyphs, often resulting in glyphs that are influenced by the style of the source glyph and prone to artifacts. We propose IF-Font, a novel paradigm which incorporates Ideographic Description Sequence (IDS) instead of the source glyph to control the semantics of generated glyphs. To achieve this, we quantize the reference glyphs into tokens, and model the token distribution of target glyphs using corresponding IDS and reference tokens. The proposed method excels in synthesizing glyphs with neat and correct strokes, and enables the creation of new glyphs based on provided IDS. Extensive experiments demonstrate that our method greatly outperforms state-of-the-art methods in both one-shot and few-shot settings, particularly when the target styles differ significantly from the training font styles. The code is available at [https://github.com/Stareven233/IF-Font](https://github.com/Stareven233/IF-Font).

IF-Font: Ideographic Description Sequence-Following Font Generation
Xinping Chen, Xiao Ke, Wenzhong Guo
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing,
College of Computer and Data Science, Fuzhou University, Fuzhou 3501 16, China Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 3501 16, China {221027017, kex, guovenzhong @fz.uu edu cn
Abstract
Few-shot font generation (FFG) aims learn the target style from limited number of reference glyphs and generate the remaining glyphs in the target font. Previous works focus on disentangling the content and style features of glyphs, combining the content features of the source glyph with the style features of the reference glyph generate new glyphs. However, the disentanglement challenging due the complexity of glyphs, often resulting in glyphs that are influenced by the style of the source glyph and prone artifacts. We propose IF-Font, a novel paradigm which incorporates Ideographic Description Sequence (IDS) instead of the source glyph control the semanties of generated glyphs. To achieve this, we quantize the reference glyphs into tokens, and model the token distribution of target glyphs using corresponding IDS and reference tokens. The proposed method excels synthesizing glyphs with neat and correct strokes, and enables the creation of new glyphs based on provided IDS. Extensive experiments demonstrate that our method greatly outperforms state-of-the-art methods in both one-shot and few-shot settings, particularly when the target styles differ significantly from the training font styles. The code is available at https: //githbb...t/// com/Stareven220 2atee/233/FFFFPP IF-Font.
contert style
Izy Encode
Reinenne
7 +
Tng Esaade
Insofomer Decoker
+
Dacader
bng Encode
Ouggur
D+
Tins Thoools
Astoregrosis
Source
Figure 1: Comparison two font generation paradigms. Left: The style-content disentangling paradigm. It assumes that glyph can be decomposed into two distinct attributes: content and style. Right: The proposed paradigm. We first autoregressively predict the target tokens and decode them with VQ decoder. Orange boxes show the main difference between the two paradigms.
"Coresponniig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
At the heart of font generation lies the extraction of styles from some reference glyphs of a certain font, and generate the remaining glyphs of this font. Some languages, such Chinese, Japanese, and Korean, have a large number of characters and intricate glyph structures. Font generation can significantly reduce the labor intensity of font designers and support tasks like handwriting imitation, ancient book restoration, and internationalization of film and television productions.
EMD [58] and SA-VAE [44] are based on the belief that the target glyph can be generated by integrating the content features of the source glyph with the style features of the reference glyph, as illustrated in Fig. (left). The majority of subsequent works [53, 38, 47, 25, 50] continues this paradigm, but this makes font generation a sub-task of image-to-image translation, where the source glyph is morphed to match the style of the reference glyphs, rather than being truly "generated". Due to the complex structure of glyphs, achieving a distinct boundary between style and content features requires considerable effort. Consequently, glyphs produced through the disentangling strategy typically maintain similar stroke thickness to the reference glyph but align more closely with the content font regarding spatial structure, size, and inclination.
To this end, DG-Font [54] incorporates deformable convolution. Diff-Font [15] integrate diffusion process to improve the network's ' learning capabilities. CF-Font [50] proposes content fusion. Additionally, several approaches [38, 47, 25, 30] combine fine-gratn fine-grained prior information such as strokes and components, to further enhance generation quality. These methods essentially follow the content-stye styyle disentanglement paradigm, in scenarios where the content font differs substantially from the target font, the resulting glyphs susceptible to artifacts such as missing strokes, blur, and smudge.
We abandon the source glyphs in favor of Ideographic Description Sequence (IDS) to convey content information. It is based on a simple fact: without disentangling features, there is no risk of incomplete disentanglement. Consequently, font generation is reframed as sequence prediction task, where the objective to generate the tokens the target glyph based on the given IDS and reference glyphs. This approach mitigates the impact of source glyphs the outputs and diminishes artifacts by leveraging the prior knowledge embedded in the quantized codebook. The users are allowed to formulate IDSs to create non-existing Chinese characters, such kokuji² (Japanese-invented Chinese characters), provided that the corresponding structures and components have been learned during training. This endows the model with certain cross- cross-linguse linguistic capabilities. We refer to this method as Idcographic Description Sequence-Following Font Generation, or IF-Font. In summary, the key contributions of this paper are as follows:
We propose IF-Font, which abandons the previous content-style disentanglement paradigm and generates glyphs through next-token prediction.
We devise a novel IDS Hierarchical Analysis (IHA) module that analyzes the spatial struc- tures and components of Chinese characters. allows our decoder flexibly control the generated content with the encoded semantic features.
Leveraging corresponding IDSs, we design the Structure-Style Aggregation (SSA) block to extract and efficiently aggregate the style features of reference glyphs.
2
Related Works
Image-to-imant translation Image-to-image translation (I2I) aims to learn a mapping from a source domain to a target domain, requiring the transformation of images in the source domain into those in the reference style's 's target domain while preserving their content. Pix2Pix [21] is the first 12I method that trains GANs [13] using paired data. CycleGAN [60] achieves unsupervised training through cycle consistency loss, although it is limited to transformations between two domains. UNIT [29] enforces the latent codes of images from two distinct domains to be identical, while employing separate generators for images in each domain. This process embodies the concept of disentanglement. MUNIT [19] further refines UNIT's latent code into content and style codes. Multimodal image translation can be achieved by combining the content code with different style codes.
hnttps://ww.ssl sl sljfaq. orrg//oggg/aa org/afaq/koku/ju/uaaaag 1ist html

---

While applying an image-to-image translation framework for font generation is currently the main- stream approach, we believe this be inappropriate. Unlike ordinary images, the boundaries between content and style in glyphs are ambiguous. For example, although handwriting will certainly differ when the same characters are written by different writers, their semantic meanings remain unchanged. Given that glyph features are challenging to disentangle, we utilize style-neutral IDSs to determine characters, thus avoiding any influence on the styles of results due to insufficient disentanglement of content glyphs.
Few-shot font generation Few-shot learning [12] represents the prevailing research focus in font generation, aiming to simulate the target style with just handful of reference glyphs. Font generation methods fall into two categories based on their utilization of implicit structural information within glyphs. Methods treating glyphs as general images possess broader applicability, enabling generation across various languages. Conversely, methods leveraging structural information typically yield higher quality outputs but are confined to specific language.
Among the methods that do not incorporate structural information, EMD [58] stands out as the earliest attempt disentangle glyphs into content and style features. DG-Font [54] employs deformable convolution [7] to capture the glyph deformations. FontRL uses reinforcement learning [45] to draw the skeleton of Chinese characters. FontDiffuser [55] models the font generation task as noise- to-denoise paradigm. Shamir et al. [42] explores parametric representation of oriental alphabets, which can elegantly balance glyph quality and compression. In vector font generation, Deepvecfont [S1] exhaustively exploit the dual-modality information (raster images and draw-command sequences) of fonts to synthesize vector glyphs. CLIPFont [43] controls the desired font style through text description rather than relying on style reference images.
In terms of methods that incorporate structural information. LAEVAE/R [44] utilizes the radicals and spatial structures of Chinese characters. CalliGAN [53] adopts the Zi2Zi framework [48] and fully decompose Chinese characters into sequences of components. SC-Font [23] further decomposes Chinese characters into stroke granularity. DM-Font [4] proposes dual memory to update component features continuously. LF-Font [38] represents component- wise style through low-rank matrix factorization [3]. MX-Font [37] automatically extracts the features through multiple localized experts. FS-Font [47] demands that reference glyphs include all components of the target glyph, otherwise may result in a degradation of generation quality. CG-GAN [25] employs GRU [6] and attention mechanism to predict component sequences. XMP-Font [30] performs multimodal pre-training on Chinese character strokes and glyphs data. Most of the above methods are constrained by the sont disentanglement paradigm. They often neglect the presence of Ideographic Description Character (IDC) which refers to the spatial structure of Chinese characters, suffering from artifacts and inconsistent styles.
Vector quantized generative models Vector Quantization (VQ) typically follows a two-stage training scheme. Initially, it employs a codebook to record and update vectors, converting them from continuous feature space to discrete latent space. Subsequently, it models the distribution of these quantized vectors with a decoder to predict tokens, which are the codebook indices, and then restores the tokens to image.
VQ-VAE [35] is the first to incorporate quantization into the variational autoencoder (VAE) [24] framework. VQ-VAE2 [41] performs multi-scale quantization and adopts rejection sampling [1] VQGAN [9] acquires the codebook with the help of GAN [13] and employs Transformer [49] to replace the PixeICNN [34] used by VQ-VAE [35]. RQ-VAE [26) proposes a residual quantizer. BETT
[2] performs masked image modeling (MIM) on the patch view with the supervision of visual tokens. MaskGIT [5] directly models visual tokens and proposes parallel decoding. MAGE [27] is similar to
MaskGIT [5], but introduces ViT [8] and contrastive learning [14]. VAE [18] further encodes
images with variable-Iength codes.
Since quantization is equivalent to tokenizing images, many methods attempt to enable multimodal generation. The dVVAE proposed by DALL-E [40] relaxes the discrete sampling problem utilizing
Gumbel-SSfttax [33, 22], outputing the probability distribution of codebook codes. SEED [11]
designed a Causal Q-Former to extract image embeddings and quantize them. LQAE [28] trains
VQ-VAE [35] to quantize the image into the frozen L.LM codebook space directly. SPAE [56]
introduces multi-layer and coarse-to to-nee pyramid quantization and semantic guidance with CLIP
3

---

[39]. V2L [61] further proposes global and local quantizers. Given the absence of pre-trained model
tailored for IDS, we directly concatenate visual tokens with IDS tokens performed autoregression.
Method
[HA E,
'I8*" Target 4
LRR
Decoder
VOGAN Decoder
+
"HEPERY" Refarener Clars: C,
Decompose
1000
Encoder
Chopptt
B,
Sumilarit Kyir
VOGAN Encoder
Euse
Lig
VQGAN Encoder
+
Encoder
f
Target:
F.
Reforence Imagee: C,
Figure 2: Overview of our proposed method. The overall framework mainly consists of three parts: IDS Hierarchical Analysis module E., Structure-Style Aggregation block E,, and decoder D.
As shown in Fig. 2. given a target character Cy. reference characters C, { and reference glyphs G, {9; === the goal of our framework is to generate glyph ây that conforms to the semantics of c, and has a style consistent with G,. To achieve this objective, we analyze ey with IHA derive its associated IDS ly. which is then encoded obtain the IDS I, [ corresponding to C,. Following this, employ the similarity module Esim to assess the relationship between by and L,.. Combined with f. and the output of Esim- the features F, corresponding to Gp are fused into the final style feature fr in the SSA block. by reshaped as initial tokens t<0 which is fed into the decoder D along with fr for autoregressive modeling. Finally, the predicted glyph tokens ly are decoded with the pre-trained VQGAN obtain the generated glyph õy-
semantic feature fi- Likewise,
3.1 IDS Hierarchical Analysis
simple alternative using source glyph input is to directly employ the character itself to control the semantics of the output. However, considering the vast number of characters in Chinese, this approach proves be impractical due its expensive cost. Moreover, it overlooks the structural intricacies of characters.
Ideograph Description Sequence is structural description grammar for Chinese characters defined by the Unicode Standard, which consists of description characters and basic components (mainly Chinese characters) through prefix notation. Decomposing Chinese characters into their corresponding IDSs can notably streamline the vocabulary, allowing characters with similar structures or components to share common features.
However, a Chinese character may have multiple equivalent IDSs. Many Chinese characters have a top-bottom or left-right structure, the IDCs follows a long-tail distribution, presenting challenges for model training. Fortunately, the seft-middle-rigtt structure of Chinese characters can be equivalently represented by two left-right structures. Similarly, the hop-middle-bottonr structure equals two top-bottom structures. The examples can be found in Fig. 3.
Based on the above observation, we employ IDS Hierarchical Analysis (IHA) module. Instead of rigidly querying the decomposition table when determining the IDS of character, we examine whether the character follows a left-middle-right construct multiple equivalent IDSs for the same character through random selection. summarize, Cy and C,- are initially decomposed into by and I, respectively. In the IDS encoder, by is padded to the maximum sequence length I and encoded into the associated semantic feature E g!xe
top-middle-bottom structure. Subsequently,

---

Tinor
IN
Deppect
-スイュ -
f,
global
R.
Lince
LN
8+
f=
fm
=
Figure 3: The illustration of equivalent IDSs.
Figure 4: The illustration of the fusion module Bfuse of SSA block.
3.2 Structure-Style Aggregation
Many previous methods [55, 10, 59, 25, 30, 54] overlook interactions between reference and target characters during the extraction of reference styles, resulting in a lack of relevance in the extracted features. The more closely the reference character resembles the target character, the more effortlessly the generation process can preserve the style. Ideally, employing the target glyph itself as the reference, known as reofstrconttuuction, should yield the most effective output. Although FS-Font endeavors to ensure that the reference characters cover all components of the target character, its implementation hinges on predefined content-referent mapping, which may limit its adaptability. To address this issue, propose convert the reference glyph G, into the latent space of VQGAN and encode it one by one into the corresponding intermediate features F, { E RR-wxeck Ji=1 The similarity module Esim evaluates the resemblance between each reference IDS 1, and the target IDS ly. considering whether they share identical structures or components. produces a set of similarity weights Sim = (sim² E IR " ===1 which can guide the subsequent feature fusion process. The fusion module Essse consists of two branches: global and local style feature aggregation, as shown in Fig. 4. The global features mainly focus on the glyph layout, stroke thickness, and inclination, which can be obtained by merging the coarse style features F, with the similarity weight Sim obtained in the previous step:
Suuuttur--tyle Aggregation (SSA) block, as shown in Fig. 2. We
(1) While local features are more concerned with the strokes, such stroke length, stroke edge, and other nuances, we adopt cross-attention to gather the required style feature according to the needs of the target character:
frg softmax(Sim) F, R.
F` = flatten? (F,,) E b.-wwxx Q =LayerNorm(I4(O)) E R
K- = LayerNorm(Lx E Rk
V = -L,(F.) E R kxwwcc
QKT
A dropout(
Rikkkw fa Soft e Rlxe
Ve
where flatten?(-) denotes flattening the first two dimensions of the feature, and Lę Lks Ly are linear projections, and LayerNorm..) denotes layer normalization. In Eq. 3, we obtain the aggregated style feature based on Eq. and Eq. 2, where denotes concatenation operator:
fr LayerNorm (Srg o frt).
3.3 Style Contrast Enhancement
There are some strategies to maintain style consistency: integrating consistency loss [59, 25], in- troducing a discriminator determine the generated style [25, 47, 36], or treating the extracted style feature as variable for further optimization [50]. These approaches are indeed beneficial for improving the generation quality, but they may be inflexible or introduce additional parameters.
In this paper, we propose a streamlined approach named the Style Contrast Enhancement (SCE) module, which promotes the proximity of representations for the same style and the distance between representations for different styles. We apply a linear projection to the style feature fr. resulting in a contrastive feature MLP(Jr).
In one batch, we denote the indices of contrastive features corresponding to all samples as F. {i E N 0 < < 3 where N represents the batch size. The dimensionality of E,, is double the
5

---

batch size N due to our utilization of a momentum encoder [16]. Each sample za within the batch undergoes processing by both the encoder and the momentum encoder, yielding two outputs that serve as positive pairs. The negative sample set is defined as E = {i E F., s(x)) # s(xa)): J. while the positive sample set is E+ = {i E F. i + a, 8x;() = s(ra)) where s(-) denotes the operator used to retrieve the corresponding style. The contrastive loss can be calculated as follows:
EpEE, exp(e"'pp/)
DJ
Σ
log
2N
EPEE, expletep/1) + EnEE_ explexee/T)
aEE.
3.4 Generation
The decoder D is provided with both semantic feature f. and style feature fr. It treats f. as the initial tokens yー0 = fc, and then predicts the distribution of the next token autoregressively as p(t' |t fr). Each newly predicted token is appended to the previous tokens for the subsequent iteration, continues until all tokens are predicted. The likelihood of the entire sequence can be calculated as I p(t' | fr). There are two ways for incorporating fr. The most straightforward approach involves using fr as initial tokens, represented by ن<0 f. fr. akin to ft. These tokens participate in each forward pass, relying on the self-attention mechanism extract and integrate features.
However, is only practical for low-resolution scenarios. Viewing fr as tokens may lead to exces- sively long sequences, requiring balance between computational efficiency and generation quality. To address this challenge, we incorporate fr into each block of the decoder D via cross-attention. The tokens act queries to align with the corresponding style features. 1<tr denotes all the predicted tokens, from which e< is removed get the glyph tokens Ey. The objective in Eq. 5 is to maximize the log-likelihood the token sequence.
l--1 Lig log(
II p(L 1 (t),, ==0
(5)
Finally, the model can be trained according to the objective in Eq. 6.
Liotal = Len + Aat
(6)
where det controls the weight of contrastive loss, cf. Eq. 4, we set det 0.5 in experiments.
4 Experiments
4.1 Dataset and Evaluation Metrics
Table 12 IDCs used in this paper.
We gathered 464 fonts from the In- ternet, covering diverse categories like printed, handwritten, and artistic styles. Next, we se- lected 3,500 commonly encountered Chinese characters and rendered them into 128x 128 res- olution images using the collected fonts.
Datasets
IDC
Structure
Example (Char:IDS)
left-right
4
top-bottom
EC,
left-middle-right
18
Is
Mwll
top-middle-botton
5
enclosed-surroundinng
00.
The training set comprises 3,300 randomly se- lected Chinese characters and 424 fonts, referred to as Seen Fonts and Seen Characters (SFSC). There are two test sets: the first includes the same 3,300 characters but with different 40 fonts, called Unseen Fonts and Seen Charac- ters (UFSC). The second test set consists of the remaining 200 characters and the same 40 fonts, known as Unseen Fonts and Unseen Characters (UFUC). We found a publicly accessible IDS decomposition table". However, it exhibits several redundant entries and circular references, as well as an absence of some characters. Therefore, we performed simplifications and enhancements, reducing the number of IDCs to the 12 depicted in Table 1, which is sufficient for most frequently used Chinese characters. For convenience, we set the basic component's IDS itself.
lell-to-rright-surrome
[]
lef--botom-righ---rroomnn
gop-leff-bottom--urroonddd
CF. $:
top-lef--uuroundinng
X F:l T
top-right-surrounding
lefl-bottom-suree surroundin
overlaying
"attps://babelsttom uk/CCK/IDS..T

---

Evaluation metrics We compare all methods in the following metrics, i.e., FID [17], LI, LPIPS [57], RMSE, and SSIM [52]. Since aesthetics is inherently subjective, we conduct a user study for all methods to evaluate model performance based on user satisfaction. We observe that the existing font generation methods have differences in data preprocessing and metric selection. Factors such as glyph resolution, the padding around glyphs, the range of pixel values, the number of reference glyphs, and the evaluation function implementation all influence metric values. For example, NTF [10] and CF-Font [50] center the glyph within the canvas, leaving white space around it. However, this leads to inflated metric calculations. To ensure a fair comparison, we adopt consistent test data and metric implementation across all methods under evaluation. Specifically, we eliminate padding around the glyphs, fix the canvas resolution to 128 pixels, scale the data range to [o. 1), utilize SqueezeNet [20] as the network type to calculate LPIPS [57], and select the inceptionv3 [46] feature layer with 2048 dimensions for FID [17] calculation.
Table 2: Quantitative evaluation on UFSC and UFUC dataset. "User" indicates user study, the samples are generated under 3-shot setting. Bold and underlined numbers denote the best and the second best respectively. Please refer to Fig. 10 Appendix for the corresponding radar plots.
Metbods
FD1
L11
LPIPSi
Libet
RMSEL
SSIM
FID
L1L
LPIPS.
Jahst
RMSE.
553M4
F1D1
1.11
LPIPS]
Sobet
RMSEL
SSIM
(5-1
User
DG--GA
[[-ivel [14
52 9054
0. 17764
1886
0.1977
0.4663
29 191540
0 03996
0.1576
0.999
0.4412
36. 9960
11994
0.155
0.3674
0.4993
11:43
0399
CF Pout 550
FS Flar [47]
132.0971
20.4457
0.2636
0.3839
0.3108
0.1581
0.143
0.0K5
0.2955
0.333
25 535331
30 8436
0.075
0.1767
0,1916
0.0600
0.443
0,3977
0.3865
0.6888
93. 7912
30.9299
0.1900
0.1754
0.2086
0.155
04134
0,399
0.4833
0.4455
12.22
6.55
V0-hum [36]
22.0554
0.1958
13.910
0.1775
0.001
0.1734
0.4413
PontD most
38866
0,1951
0.180
0.337
0.1314
0.3447
0.4420
18.33
FF- (Curn)
5.7645
0.153
0.130
0,368
0.4915
6.539
0.147H
0.151
0.363
0.9031
0.7383
0.143
0.336
0,3553
0.5140
22.78
DG-GAN [2]]
13:4734
0.1801
0.1938
0.182
130142
0.5471
0.000
15,244
0.1750
0.991
0.499
DRCC
CR Funl 500
Foor [67)
22.8931
0.1865
03112
0 01554
05155
0429
0.764
34,245
0.17%
0,0923
0.0600
0,000
01300
0.3785
58.9656
15.3477
0.1809
0.063
04019
0.4131
04599
1200
Vi-Shmil [S6]
78.17 1737
0,1980
0.227
0422
04015
0.100
0.176
0.03
05345
36.666
0.1786
01735
04554
5.25
0.3866
0.753
23,9580
0.175
0450
0.811
155)
IF-Foon (Chen)
5.4544
6,1681
6,3645
0.159
0.756
01366
0.066
21.38
4.2 Comparison with state-of-the-art Methods
We compare the proposed IF-Font with seven SOTA methods on our UFSC and UFUC datasets re- spectively, including CG-GAN [CVPR 2022), LF-Font [38]II 2022), FS-Font [CVPR 2022), CF-Font [SO]CCVPR 2023), VQ-Font [36][CCV 2023), NTF [1O[CVVPR 2023) and FontD- iffuser [55] (AAAI 2024), All methods are trained from scratch on our SESC dataset according to their respective official codes and default configurations. We slightly modify the codes of CG-GAN, LF-Font, FS-Font, VQ-Font and FontDiffuser to support varied numbers of reference glyphs.
4.2.1 Quantitative comparison
Table 2 compares IF-Font and other SOTA methods. IF-Font significantly surpasses competitors in all reference glyph number settings for both datasets. Notably, IF-Font's performance on FID metric is exceptionally low, reaching a single-digit score, thanks to the high quality and clarity of the samples it generates. FS-Font [47] relies heavily the predefined content-reference mapping, whereas the reference glyphs in all our experiments are randomly selected. Especially, when only one reference glyph is provided, covering all components of the target character becomes challenging, leading to poor performance of FS-Font, as shown in Fig. 10a. NTF [10] also struggles to imitate the target style, the layout of its generated samples often resembles that of the source font. In cases where there's a significant disparity between the source and target styles, NTF is prone to missing strokes.
We attribute CF-Font's performance to its reliance on fusing contents of 10 basic fonts. However, there happen to be gap between the train dataset and our evaluation dataset. We conduct user study through Fuxi Youling Crowdsourcing Platform to quantify the subjective quality. For each test dataset, 5 characters are randomly selected, and each model is required to generate glyphs corresponding to 40 unseen fonts. A total of 30 participants are asked to select the option that most closely matches the ground truth from the generated results. The outcomes of the user study are presented in the last column of Table 2.
/nttps://fuxi 163. com com//ooottoo/daaaaaaaaaaaaaaaaaa
7

---

Source
CG-GAN
LF-Font
FS-Font
11
UPC
CF-Fout
*
18
-
VQ-I'ant
X
NIF
#
/
11
FondDiffuser
A
Ours
Target
12)
Source
$
20 (o) 120
CG-GAN
LF-Font
2
R
12
FS-Font
X
17
UFUC
CT-F'ntt
E
D0
13
#
VQ-Font
1ス
77
1111 F2
12
NTF
FontDifflseer
X
Ours
2
X
12
Target
R
K
Figure 5: Qualitative comparison with state-of-the-art methods, in which red boxes outline the artifacts. "Source" denotes the content glyph of other methods, IF-Font only employs the corresponding IDS.
4.2.2 Qualitative comparison
We present the corresponding samples from Table 2 in Fig. 5. IF-Font stands out by producing the clearest and most style-consistent samples. In contrast, FS-Font [47], LF-Font [38], CF-Font [50], and other models exhibit issues such as stroke errors or blur. VQ-Font [36] and NTF [10] are constrained by the source font and struggle with flat or narrow layouts, resulting in incorrect structures. VQ-Font even tends to crop marginal parts of glyphs to fit the target style. While CF-Font generally perserves the correct glyph layout, its outputs exhibit noticeable artifacts, indicating some remaining style inconsistencies. The performance of FontDiffuser [55] is also outstanding, but there is still slight deficiency the imitation of font styles. On the other hand, IF-Font maintains the correct character structures and excels in aspects such the aspect ratio, glyph layouts, and stroke details.
Table 3: Ablation studies on different modules.
IDS
Reference Baseline
IS
1SC
Target
The first row is the results of baseline. I, S and represent IHA, SSA, and SCE respectively.
/x
db
d::l mak
x
lokks
11
#
H
Module
FID
L14
LPIPS
RMSFL
SSIM
BAAE
C
X
X
8,2656
0.1632
0.1383
0.3820
0.4728
1 ETT
#
:r
3
:
X
X
8,3750
0.1638
0.1381
0.3828
0.4764
x
7,5391
0.1614
0.1348
0.3797
0.4780
Figure 6: Visualization of different modules in Table 3. Red, blue and green boxes represent the missing components, style inconsistency and corresponding improvements respectively.
8.4922
0.1597
0.1338
0.3775
0.4782
8

---

4.3 Ablation Studies
Main modules Removing the IHA, SSA, and SCE modules IF-Font, a baseline model can be obtained. For a input character, it directly looks up the decomposition table to derive the corresponding IDS, and then encode the semantic feature f. through a embedding layer. The intermediate style features F, are directly averaged as style features fr. excluding any interactions with the similarity weight Sim and semantic feature fi- The whole model relies solely on cross-entropy loss for supervision. Building upon the baseline, we incrementally reintegrate three modules to assess their individual contribution. Quantitative results are presented in Table 3, while Fig. 6 provides visualizations of these results. For further ablation study of the SSA block, please refer Table 6 in Appendix 1.1. Upon integrating our modules, a consistent improvement is observed across most metrics. Fig. 6 illustrates how IHA alleviates the issue of missing components present in the baseline. SSA enhances style consistency, while SCE improves the capability to imitate styles.
Table 4: The impact of IDS granularity on performance.
Granularity FID..
L.1.
LPIPS
RMSEJ
SSIM
Component
8.4922
0.1597
0.1338
0.3775
0.4782
Stroke
8.4297
0.1616
0.1347
0.3799
0.4888
Mixed
8.5234
0.1598
0.1337
0.3775
0.4782
IDS granularity We further analyze the impact of three different IDS granularities: components, strokes and mixed. Please see Table 5 in the Appendix for the examples of these granularities. Table shows the quantitative results. Stroke granularity results in performance degradation across three metrics. We attribute this decline to the conflict between IDSs, hindering the model's ability to identify the target character. An attempt concatenate IDSs from both granularities yields performance akin to that of component granularity. While this approach extends the sequence considerably, hence we opt for component granularity.
4.4 Visualization of SSA
T
Z
Z
Z
Z
Figure 7: Visualization of attention maps between IDS and reference glyphs. The symbols above are the target character (black) and the corresponding IDS (orange).
To demonstrate the effectiveness of Structure-Style Aggregation block, we visualize the attention maps corresponding to each IDC and component within the IDSs relative to the reference glyphs, as depicted in Fig. 7. Specifically, we choose the matrix A R-xk-b- in local feature calculation. For each position i of the target IDSty, there exists a corresponding attention map A' E & which indicates the attention that d pays to the reference features. We present the attention map A' to visualize the distribution of attention weights directly. Additionally, apply opacity to this map and overlay it onto the original reference glyph.
As we can see, when the target IDC or component exists in the reference glyph, more attention will be paid to the corresponding place. For instance, in the first row, the first, second, and fourth columns, and in the second row, the third and fifth columns are distinctly highlighted. Conversely, if the reference glyph lacks the target component, the local branch tends not to engage, as evidenced by the nearly blank third row. This approach stems from a preference to avoid forced attention allocation which might lead interference. Instead, leveraging the average style captured by the global branch helps maintain a bascline quality of the output.

---

4.5 New Glyph Creation
We validate the flexibility of IF-Font by generating the glyphs using IDSs of
IDS
Outpet
Target
IDS
Output
Target
IDS
Ouiput
IDS
Ouput
F
#
#
ti
s
f
wus
non-standard Chinese characters. Fig. 8 shows experimental results. IF- Font demonstrates robust generaliza- tion by following given IDSs pro- duce new glyphs with accurate structure and consistent style. We fixed the font to "Sarasa Gothic" in the experiment, which is CJK programming font. The last two columns lack ground truths due their entirely non-existent characters.
Am
E~
1
DLX
D
*T
*T
E
USF
Uif
XI 1
EDA
Figure 8: The ability of the IF-Font to create glyphs. The first two columns are kokuji, and the last two columns are completely non-existent Chinese characters.
5 Discussion
GT
*
Target
Output
Figure 9: Failed cases complex fonts of UFUC. Orange boxes highlight reconstruction errors of VQGAN, red outlines the structural errors. GT: the glyphs rendered by fonts; Target: the glyphs reconstructed by VQGAN; Output: the glyphs generated by IF-Font.
Failure cases Although method enables high-quality generation under most circumstances, still struggles on some hard cases, as illustrated in Fig. 9. IF-Font encounters difficulties with fancy and irregular font styles, including those with decorations, extremely flat narrow layouts, excessively curved strokes, and calligraphic writings. Despite these challenges, it continues to preserve the correct character structure. Further discussion on the reasons for the difficulties in generating these fonts can be found in Appendix C.3.
Usability We focus on CJK characters due to their unique spatial structures, which better reflect the characteristics of our method. By expanding the vocabulary and incorporating relevant data for training. IF-Font can also be adapted to handle other character sets.
Ad Advantages Conforms to writing habits. We believe that the process of autoregressive modeling with IDS implicitly contains the order of writing. Scalability. Good scalability can be achieved by leveraging the mature experience of LLMs. Robustness. Due to vector quantization, glyphs are represented by limited number of tokens (only 256 types). which reduces the learning difficulty for the decoder and decreases the likelihood of artifacts and other issues in the generated results.
6 Conclusion
We have presented IF-Font, a novel font generation paradigm. IF-Font redefines font generation as sequence prediction task by quantizing glyphs token sequences and leveraging Ideographic Description Sequence (IDS) to control the semantics of the generated glyphs. This method demon- strates exceptional capability in managing complex styles while preserving the correct structures. Benefiting from the flexibility of IDS, our method enables the creation of glyphs. This is achieved by formulating legal IDSs, which is a salient advantage over other methods that typically require the character to be present in at least one font as a precondition for generation. Refining and improving the IDS decomposition rules considered future works. Furthermore, exploring the integration of IDS into handwritten font generation may yield interesting insights.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Plan of China under Grant 2021 YFB in part by the National Natural Science Foundation of China under Grant 61972097 and U21A20472, in part by the Major Scientific Research Project for Technology Promotes Police under Grant 2024Y2040001, in part by the Natural Science Foundation of Fujian Province under Grant 2021J01612 and 2020J01494.
References
[1] Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian J. Goodfellow, and Augustus Odena. Discriminator Rejection Sampling. In International Conference on Learning Representations (ICLR), 2019.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert Pre-Training of Image Transformers. In International Conference on Learning Representations (ICLR), 2022.
[3] Emmanuel J. Candès and Benjamin Recht. Exact Matrix Completion via Convex Optimization. Foundations of Computational Mathematics, 966:7177-72, 2009.
[4] Junbum Cha, Sanghyuk Chun, Gayoung Lee, Bado Lee, Seonghyeon Kim, and Hwalsuk Lee. Few-Shot Compositional Font Generation with Dual Memory. In European Conference on Computer Vision (ECCV), pages 735-751, 2020.
[5] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked Generative Image Transformer. In Computer Vision and Pattern Recognition (CVPR), pages 1 11305-11315, 2022.
[6] Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, 2014.
[7] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable Convolutional Networks. In IEEE International Conference on Computer Vision (ICCV), pages 764-773, 2017.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), 2021.
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12873-12883, 2021.
[10] Bin Fu, Junjun He, Jianjun Wang, and Yu Qiao. Neural Transformation Fields for Arbitrary- Styled Font Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[11] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a SEED of Vision Large Language Model. ArXiv, 0atp/2307.0804111 2023.
[12] Abel Gonzzlez--Gariia Joost van de Weijer, and Yoshua Bengio. Image-to-image translation for cross-domain disentanglement. In Conference on Neurul Information Processing Systems (NeurIPS), pages 1294-1305, 2018.
[13] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.

---

[14] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality Reduction by Learning an Invariant Mapping. In Computer Vision and Patter Recognition (CVPR), volume 2, pages 1735-1742, 2006.
[15] Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Y. Qiao. Diff-Font: Diffusion Model for Robust One-Shot Font Generation. ArXiv, abs/2212.05895, 2022.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Conference on Neural Information Processing Systems (NeurIPS), pages 6626 2017.
[18) Mengqi Huang. Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization. In Computer Vision and Pattern Recognition (CVPR), pages 22596-22222 22605, 2023.
[19] Xun Huang, Ming- Yu Liu, Serge J. Belongie, and Jan Kautz. Multimodal Unsupervised Image- to- Image Translation. In European Conference on Computer Vision (ECCV), pages 179-196,
2018.
[20] Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size. arXiv, 2016.
[21] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-Imae Imaee Translation with Conditional Adversarial Networks. In Computer Vision and Pattern Recognition (CVPR), pages 5967-5976, 2017.
[22] Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR), 2017.
[23] Yue Jiang, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. Scfont: Structure-Guide Chinese Font Generation via Deep Stacked Networks. Proceedings the AAAI Conference on Artificial Intelligence, (3(01):4015-4022 2019.
[24] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference Learning Representations (ICLR), 2014.
[25] Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look Closer To Supervise Better: One-Shot Font Generation via Component- Based Discriminator. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13482-13491, 2022.
[26] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive Image Generation using Residual Quantization. In Computer Vision and Patlern Recognition (CVPR), pages 11513-11522 1522, 2022.
[27] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked Generative Encoder to Unify Representation Learning and Image Synthesis. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[28] Hao Liu, Wilson Yan, and Pieter Abbeel. Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[29] Ming- Yu Liu, Thomas M. Breuel, and Jan Kautz. Unsupervised Image-to-Image Translation Networks. In Conference on Neural Information Processing Systems (NeurIPS), pages 700-708. 2017.
12

---

[30] Wei Liu, Fangyue Liu, Fei Ding. Qian He, and Zili Yi. Xmp-Font: Self- Supervised Cross- Modality Pre- Fre-Trannng for Few-Shot Font Generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7905-7914, 2022.
[31] Yitian Liu and Zhouhui Lian. Fontrl: Chinese Font Synthesis via Deep Reinforcement Learning. In AAAI Conference on Artificial Intelligence (AAAI), pages 2198-2206, 2021.
[32] Nya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Leurning Representations (ICLR), 2019.
[33] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Con- tinuous Relaxation Discrete Random Variables. In International Conference on Learning Representations (ICLR), 2017.
[34] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel Recurrent Neural Networks. arviv, 2016.
[35] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. Conference on Neural Information Processing Systems (NeurIPS), pages 6306- 6315, 2017.
[36] Wei Pan, Anna Zhu, Xinyu Zhou, Brian Kenji Iwana, and Shilin Li. Few Shot Font Generation Via Transferring Similarity Guided Global Style and Quantization Local Style. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), volume abs/2309,000 000227, pages 19506-19516 2023.
[37] Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Multiple Heads are Better than One: Few-shot Font Generation with Multiple Localized Experts. In IEEE International Conference Computer Vision (ICCV), pages 15880-13888 2021.
[38] Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Few-shot Font Generation with Weakly Supervised Localized Representations. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1-17, 2022.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Iyya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICML), pages 8748-8763, 2021.
[40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. In International Conference on Machine Learning (ICML). pages
2021.
[41] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidellty Images with VQ- 2-E-2.... In Conference on Neural Information Processing Systems (NeurIPS), pages 14837-14847, 2019.
[42] Ariel Shamir and Ari Rappoport. Compacting oriental fonts by optimizing parametric elements. The Visual Computer, I 3566333,,-, 1999.
[43] Yiren Song and Yuxuan Zhang. Clipfont: Text Guided Vector WordArt Generation. In British Machine Vision Conference (BMVC), page 543, 2022.
[44] Danyang Sun, Tongzheng Ren, Chongxuan Li, Hang Su, and Jun Zhu. Learning to Write Stylized Chinese Characters by Reading a Handful of Examples. In International Joint Conference on Artificial Intelligence (IJCAI), volume abs/1712.06422. pages 920-927 2018
[45] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[46] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In Computer Vision and Pattern Recognition (CVPR), pages
2016.
13

---

[47] Licheng Tang, Yiyang Cai, Jiaming Liu, Zhibin Hong, Mingming Gong, Minhu Fan, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. Few-Shot Font Generation by Learning Fine- Grained Local Styles. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7895-7904, 2022.
[48] Yuchen Tian. 7i2zi: Master chinese calligraphy with conditional adversarial networks. https://githb. com/kuaonshhi-yyy/zzz,,, 2017.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Conference Neural Information Processing Systems (NeurIPS), pages
2017.
[50] Chi Wang, Min Zhou, Tiezheng Ge, Yuning Jiang. Hujun Bao, and Weiwei Xu. Cf-Font: Content Fusion for Few-shot Font Generation. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[51] Yizhi Wang and Zhouhui Lian. Deepvecfont: synthesizing high-quality vector fonts via dual- modality learning. ACM Transactions on Graphics (TOG), 4066))2:2266 2021.
[52] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions Image Processing, 13(4):600-612, 2004.
[53] S. Wu, Chih-Yuan Yang, and Jane Yung-jen Hsu. Calligan: Style and Structure-aware Chinese Calligraphy Character Generator. In Al for Content Creation Workshop at CVPR (AI4CC), volume abs/2005.12500, 2020.
[54] Yangchen Xie, Xinyuan Chen, Li Sun, and Yue Lu. Dg-Font: Deformable Generative Networks for Unsupervised Font Generation. In 202/ IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5126-5136, 2021.
[55] Zhenhua Yang, Dezhi Peng, Yuxin Kong, Yuyi Zhang, Cong Yao, and Lianwen Jin. Fontdiffuser: One-Shot Font Generation via Denoising Diffusion with Multi-Scale Content Aggregation and Style Contrastive Learning. In AAA! Conference Artificial Intelligence (AAAI), volume 111/2322.11142, pages 6603-6611, 2024.
[56] Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A. Ross, Irfan Essa, Yonatan Bisk, Ming Yang, K. Murphy, A. Hauptmann, and Lu Jiang. Spae: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs. In seirth- Conference on Neural Information Processing Systems, volume abs/2306. 17842, 2023.
[57] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreason- able Effectiveness of Deep Features as Perceptual Metric. In Computer Vision and Pattern Recognition (CVPR), pages 586-595, 2018
[58] Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating Style and Content for Generalized Style Transfer. In Computer Vision and Puttern Recognition (CVPR), pages 8447-8455. IEEE, 2018.
[59] Yufan Zhang, Junkai Man, and Peng Sun. Mf-Net- A Novel Few-shot Stylized Multilingual Font Generation Method. In Proceedings of the 30th ACM International Conference on Multimedia. ACM, 2022.
[60] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Trans- lation Using Cycle-Consistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251. IEEE, 2017.
[61] Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond Text: Frozen Large Language Models in Visual Signal Comprehension. In JEFÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
14

---

A Experiment Details
A.1 Implementation Details
IF-Font was trained in server equipped with an Intel Xeon Silver 4110 CPU, 128 GB of RAM, and an NVIDIA Tesla V100 PCIe 16GB GPU. The training takes about 42 hours for epochs with batch size of 128.
The VQGAN model in IF-Font loads pre-trained weights. It has a codebook size of 256, and the encoder downsamples the image with factor of 8. Consequently, the length of the codebook indices corresponding to single glyph is lT 256, whereas the IDS have a fixed length of 35. Our decoder consists of 10 Transformer blocks, each integrating self-attention layer, cross-attention layer, and multi-layer perceptron (MLP). We have configured the model with 8 attention heads and feature dimension of 384. In IF-Font, parameters are optimized using the Adam W optimizer [32], which employs learning rate schedule that includes warmup and cosine annealing.
A.2 Training Data
We primarily obtain fonts from the Foundertype platform under a personal non-commercial academic research license. We present the examples of three IDS granularities Table
Table Different IDS granularities. Mixed means that the IDSs of both granularities are used.
Granularity
Length
Componet
m
5
Stroke
21
Mixed
mB
[sep]i]] Ti
7-
27
B Additional Results
B.1
Further ablation of ISA
Table 6: Ablation studies on both branches of SSA.
SSA
FID4
L14
LPIPS.
RMSEJ
SSIM
w/o global
7.9766
0.1607
0.1347
0.3789
0.4775
w/o local
8.2578
0.1620
0.1364
0.3805
0.4756
full
8.4922
0.1597
0.1338
0.3775
0.4782
B.2
Radar plots
Fig. 10a depicts the one-shot setting, where style extraction poses considerable challenge, leading to comparable performance across most models. Nevertheless, as the number of reference glyphs increases, the advantage of IF-Font becomes progressively apparent in Fig. 10b and 10c.
C Further discussion
C.1
Quantization accuracy
Since the target glyph is reconstructed from quantized sequence, the accuracy of this reconstruction imposes ceiling on IF-Font's potential performance. As shown in Figure 9, there are slight differences between the target and the ground truth. Given that the glyphs are grayscale images and
15

---

FOUM
FEOUR
RIDHO
FIZD
RASH
COMUE
SP3I
SEMOO
LBD6
SPSONO
LUN
IMERI
1704
AF"
10- Pyne
10-Te
Pve Row
HIT
Vadee
C-Jee
(8)
bb)
(c)
Figure 10: Compared with the methods based on the content-style disentanglement paradigm, IF-Font achieves state-of-the-ar performance on all metrics under three few-shot settings. The metrics are annotated with brackets in the figure to specify the dataset used for evaluation: (S) represents UFSC, and (U) refers to UFUC. (a) 1-shot setting. (b) 3-shot setting. (c) 8-shot setting.
relatively simple, fine-tuning the VQ-GAN or switching to superior quantized method is expected to minimize the loss accuracy. We use original VQ-GAN checkpoints in order to highlight our main contributions.
C.2 The shortcomings of IDS
In fact, IDS is still not sufficiently perfect to identify Chinese characters. On the hand, there are rule conflicts: a few Chinese characters are too similar, and their IDSs of stroke granularity are exactly the same. On the other hand, the spatial descriptions are insufficiently clear. For example, the left-right structure indicates that two components are placed on the left and right, but the distance between them is not specified, which requires the model to distinguish them through sufficient learning.
C.3 Complete results on novel fonts
24 A TE
CG-GAN
1
I. I.F-Font
FS-Font
CF-Font
#
VQ-Font
# ス
NTF
FontDiflusser
Ours
-
Target
Figure 11: Complete results generated by methods with novel fonts of Fig. 9.
We believe that generating novel fonts presents significant challenges for several reasons. First, they substantially deviate from standard character structures, involving trade-off between the number of references and the quality of generation. Furthermore, font design involves subjectivity and randomness. For instance, in the first font depicted in Fig. 9, the position, size, and shape of the auspicious cloud patterns are the result of manual design. important note that this issue common across all font generation methods. Similar discussions can be found in LF-Font [38] and
16

---

CF-Font [50]. Unfortunately, this limitation has not been well addressed yet. We demonstrate the results of other methods on novel fonts Fig. 11 to support the above conclusion.
C.4 Restrictions
To achieve high-quality generation, IF-Font requires training data to cover as many character com-- ponents and font styles possible. Due the utilization of attention mechanisms, IF-Font entails quadratic computational complexity related to sequence length, while inference relies on autoregres- sive processes, resulting slow sampling speed.
D Details of the user study
To be accurate, we pay the Fuxi Youling Crowdsourcing Platform (a third-party platform) to conduct the user study. The participants the user study are users of that platform, who will be informed of the full details in advance. We limit the number of participants to 30, and they are allowed choose whether or not to participate and can complete the evaluation at any time and any location using their own devices (usually phone computer).
All model names in the evaluation are replaced with numbers, and participants only need select the option that best matches the ground truth from the model results displayed on the page. After the evaluation, the platform pays participants and provides us with de-identified model effectiveness data, We have no direct contact with participants and are unable to obtain their specific identities.
E Broader Impact
IF-Font could help to improve the productivity and creativity of font designers, and there is also hope for preserving ancient calligraphic works. In addition, IF-Font is capable of imitating the font style from as little as one reference glyph. That makes easy to reproduce commercial font designs, raising concerns regarding potential copyright infringement. To safeguard the rights of font creators, we urge users to adhere to license requirements and call for the responsible use of our method.
Licenses
We present a complete list of references and licenses in Table for all the existing assets we used in this work.
Table 7: License information for the existing assets used.
Software Code
URL
License
VQGAN
Link
MIT license
nanoGPT
Link
MIT license
CG-GAN
Link
N/A
LF-Font
Link
MIT license
FS-Font
Link
MIT license
CF-Font
Link
N/A
VQ-Font
Link
N/A
NTF
Link
MIT license
torchmetrics
Link
Apache v2.0
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 3 for detailed information the model.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is available at https: tttps://githhi com/ Swtm/Starreee2331FF5 II-Font. We provide detailed instructions on how to obtain and process the data in Section 4.1 and Appendix A.2.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce cc/
cublic/guides// Saaiidee/ooeSuumisstee htt.amiisssnooiicoy..ooocc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ stips.cc/publice guw/guuiiesseeee /CodeSubmiieeeee hww.simmissioooooooioioooooo onliiicyllll iy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Section 4.1 and Appendix A.I.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Appendix A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nnurrrr..p.. oc hepppuii//Etiisuuiiiitpee ines?
Answer: [Yes]
Justification: Our research conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the socictal impacts in Section and Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe safeguards in Appendix E.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Appendix F for the references and licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper fully describes the details of assets in Section 3, 5 and Appendix A,
F. The assets and documentation can be found in betps://g/i thub. com/Starrven233/
IF- IF-Font.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We include the details of the crowdsourcing experiment in Appendix D.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: We include the details of the crowdsourcing experiment in Appendix D.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Z40Fos1kmV", "reviewer": "Reviewer_MtRY", "summary": "IF-Font introduces a novel approach to few-shot font generation by using Ideographic Description Sequence (IDS) instead of traditional source glyphs to control the semantics of generated glyphs. This method quantizes reference glyphs into tokens and models the token distribution of target glyphs using IDS and reference tokens. IF-Font effectively synthesizes glyphs with neat and accurate strokes, significantly outperforming existing methods in both one-shot and few-shot settings, particularly when the target styles differ from the training font styles. The method redefines font generation as a sequence prediction task, enhancing the quality and consistency of the generated glyphs.", "questions": "1. Is there any cherry-picking of results?\n2. What is the real usability rate of the proposed method?\n3. Besides inference speed, what advantages does the VQ-GAN method have compared to diffusion-based methods?", "limitations": "see weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Paradigm: IF-Font introduces a new approach by using IDS to control glyph semantics, eliminating the need for content-style disentanglement and reducing artifacts.\n\n2. High-Quality Generation: The method excels in producing glyphs with neat and correct strokes, maintaining consistent style even with limited reference glyphs.\n\n3. Cross-Linguistic Capabilities: IF-Font allows for the creation of new and non-existing Chinese characters, demonstrating flexibility and adaptability across different linguistic structures.", "weaknesses": "1. The author proposes a content-style disentanglement method where style extraction relies on the decomposition of glyphs. However, this setup may not be necessary. In the fields of diffusion models and generative AI, there are many methods that can extract styles from a single image with minimal content interference, such as IP-Adapter and Instant-Style. The work FontDiffuser, which generates text using diffusion models, does not decompose the text but still achieves excellent style and content disentanglement. Therefore, I have doubts about the advancement and necessity of the method proposed in this paper.\n\n2. The compared methods are relatively old and do not include comparisons with the most advanced methods such as FontDiffuser.\nSince the target glyph is reconstructed from a quantized sequence, the accuracy of this reconstruction sets a ceiling on IF-Font's potential performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The advancement and necessity of our method\n\nFont generation differs from general image generation tasks. **Our contribution lies in finding a way to describe ideographic characters as \"text\" and successfully applying it to font generation**, solving the problem of low-quality generation in previous methods.\n\nIP-Adapter and Instant-Style are diffusion-based text-to-image methods. They use text to control the generation and aim to generate results that strictly follow the text prompt while having the style of reference images. However, their target is much more complex than glyphs, and the corresponding information can usually be described in natural language. Their generation conditions are relatively loose, and there are multiple reasonable results, but with the problem of content leakage.\n\nThe content of glyphs is just a character, which is difficult to describe with text, so prior works use the glyph image as the input content. Since the glyph used as the input content also has a style, there is a problem of style leakage.\n\nIn summary, these two fields have some intersection but are not identical, and the problems they face are different. We believe that our method has outstanding advantages and contributions in font generation.\n\n### Q2: The comparison with FontDiffuser\nWe have included the comparisons with FontDiffuser, which shows that although FontDiffuser performs well, our method is still at an advantage:\n\n| Model(ufsc)            | FID↓ | L1↓ | LPIPS↓ | RMSE↓ | SSIM↑ |\n|------------|---------|---|---|---|---|\n| FontDiffuser (1shot)  |  3.9969 | 0.1938  |  0.1371 |  0.4180 | 0.4076  |\n| Ours (1shot)            | 6.7695  |  0.1529 | 0.1307  | 0.3688  | 0.4915  |\n| FontDiffuser (3shot)  | **3.6979**  |  0.1774 |  0.1248 | 0.3980  | 0.4370  |\n| Ours (3shot)            | 6.8359  |  0.1478 | 0.1258  | 0.3620  | 0.5021  |\n| FontDiffuser (8shot)  |  4.1017 | 0.1748  | 0.1234  | 0.3947  | 0.4420  |\n| Ours (8shot)            | 6.7383  | **0.1429**  | **0.1216**  | **0.3552**  | **0.5140**  |\n\n| Model(ufuc)            | FID↓ | L1↓ | LPIPS↓ | RMSE↓ | SSIM↑ |\n|------------|---------|---|---|---|---|\n| FontDiffuser (1shot)  | 8.2524  | 0.1914  | 0.1527  | 0.4157  | 0.4163  |\n| Ours (1shot)            | 8.4844  | 0.1651  | 0.1387  | 0.3845  | 0.4676  |\n| FontDiffuser (3shot)  | **7.6444**  | 0.1771  | 0.1413  | 0.3981  | 0.4418  |\n| Ours (3shot)            | 8.4922  | 0.1597  | 0.1338  | 0.3775  | 0.4782  |\n| FontDiffuser (8shot)  | 8.9166  | 0.1702  | 0.1367  | 0.3890  | 0.4543  |\n| Ours (8shot)            | 8.3203  | **0.1561**  | **0.1305**  | **0.3728**  | **0.4864**  |\n\n### Q3: Performance ceiling due to quantization\nYes, our performance is limited by the reconstruction accuracy of VQ-GAN. However, glyph images are binary and relatively simple, fine-tuning VQ-GAN can minimize the precision loss.\n\nWe use original VQ-GAN checkpoints without fine-tuning in order to highlight our main contributions.\n\n### Q4: Cherry-Picking & Real Usability Rate\nIn the paper, we selected representative samples to demonstrate our method's distinguishing features. In fact, almost all generated samples are of high quality, except for those shown in the 4.6 failure cases section.\n\n### Q5: The advantages of the VQ-GAN method compared to diffusion-based methods\n1. **Conforms to writing habits**: Using IDS and quantized tokens together in auto-regressive modeling implicitly incorporates writing order.\n2. **Scalability**: The VQ-GAN method has fewer parameters, and is easier to scale. In addition, it has an advantage in terms of training speed and memory usage, making it suitable for deployment on terminal devices.\n3. **Robustness**: Due to vector quantization, characters are represented by a limited number of tokens (only 256 types), which reduces the difficulty of modeling for the decoder and makes it less likely to produce artifacts.\n\nAs is well known, the diffusion model is excellent, but the focus of this paper is not on a specific model. Indeed, our key idea is model-agnostic, which can be also applied to diffusion or other types of networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRZENyvSc4", "reviewer": "Reviewer_D5bL", "summary": "This paper proposed IF-Font handles the task of few-shot font generation via a VQ-GAN based framework. Compared to most existing methods that encode content images, IF-Font only encodes Ideographic Description Sequence (IDS) to convey content information of target characters. Experimental results show the proposed method excels in synthesizing glyphs with neat and correct strokes, and enables the creation of new glyphs based on provided IDS.", "questions": "What is the detailed structure of IDS-Encoder? Is it a Transformer to encode a sequence?", "limitations": "The authors discussed the limitations in Section 4.6. Regarding the first limitation (fancy and irregular font styles), I believe the discussion could be expanded further. One key reason might be that novel font styles do not always adhere to the typical topology of characters. For example, the second-to-last font in Figure 5 deviates from the norm, challenging the necessity of using IDS (or pre-defined component sequences). I would like to hear the authors' opinions on this.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The presentation is good and this paper is easy to follow. \n\n(2) The proposed method can generate visually pleasing glyph images, from Figure 5.\n\n(3) It is an interesting idea to ONLY encode IDS to model the shape of target characters. However, I have concerns about whether this idea is fully substantiated (see Weaknesses).", "weaknesses": "(1) The idea of utilizing component/stroke information of glyphs has already been exploited in many existing papers, such as LF-Font and XMP-Font. What is the key difference that distinguishes IF-Font from those methods?\n\n(3) If the key difference is only encoding IDS to convey content information, I wonder if there is an ablation study verifying that this is a better design than encoding both modalities (i.e., component sequence and glyph images) in XMP-Font?  The current Section 4.4 (Ablations Studies) is a bit unclear to me.\n\n(3) The authors mentioned that they construct multiple equivalent IDSs for the same character through random selection. Does it mean during training, a random IDS is selected to feed the IDS encoder if there are multiple equivalents? Can the IDS encoder resolve the ambiguities in representing the shape of a character? My guess is this is somewhat similar to many-to-one mapping so it can be done. Correct me if I am wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The key difference between IF-Font and other methods that utilize component/stroke information\nWe believe the key difference lies in **whether style-content disentangling is performed**. Previous methods use component/stroke information but still relied on separating and combining corresponding content and style features to generate glyphs. Our method uses IDS as a condition to directly generate characters, rather than morphing the content glyph.\n\n### Q2: Is the key difference lies in only encoding IDS?\nPerhaps reviewer wonders why our method outperforms previous methods with only one input modality. As mentioned above, **the component/stroke information used in previous methods is not equivalent to IDS**. Hence, it is challenging to attribute the key difference to whether only IDS is used.\n\nWe are happy to include an ablation study that uses both modalities to address the reviewers' concerns. However, the addition of the new modality leads to a substantial increase in training costs. We respectfully request permission from the reviewer to present the experimental results in the subsequent discussion period.\n\n### Q3: Multiple equivalents IDSs for single character\nWe apologize that our descriptions are not clear and cause confusion. As shown in Figure 2, an input character is first broken down into single IDS, then fed into IDS encoder during training. IDS encoder analyzes the input IDS using some rules, and if an IDS has multiple equivalents, it randomly selects one for encoding.\n\nStrictly speaking, IDS encoder is not responsible for resolving ambiguities. Since IDS is similar to text, there are cases where different descriptions refer to the same object. **We designed IDS encoder to present as many diverse inputs as possible to the decoder**, in order to prevent overfitting. However, from the perspective of the decoder, this is indeed similar to a many-to-one mapping, as it needs to generate tokens of the same character for these equivalent IDS.\n\n### Q4: The detailed structure of IDS Encoder\nIDS encoder does not contain a Transformer, in fact its learnable parameters are limited to a simple embedding layer. We apologize for any confusion that may have arisen from the name \"encoder\".\n\nIDS encoder does not perform complex encoding because **we want to preserve the independence of each element within an IDS**. This also allows the decoder to maintain fine-grained attention on each position of the input sequence.\n\nSpecifically, IDS encoder consists of three parts: decomposition, equivalent construction, and embedding.\nFirstly, it recursively breaks down the input IDS into one IDC and corresponding components at each level to figure out whether there is a equivalent. Next, it produces all the equivalents according to certain rules. Finally, it pads to the maximum length and passes through an embedding layer to obtain the final features.\n\n### Q5: More about the first limitation (fancy and irregular font styles)\nThank you for your thorough review of our work, and we are happy to discuss and clarify this issue further. We believe that fancy and irregular fonts are difficult to generate due to several reasons: \n1. As pointed out by the reviewer, one reason is that **novel fonts have a great topological difference from typical ones**, making it challenging for models to learn. \n2. **The subjective and random nature of font design**. For example, in Figure 9's first font, the \"auspicious clouds\" decoration appears on every glyph, but its position, proportion, and shape are carefully designed, making it difficult for the model to grasp the regular pattern. \n3. **Limited reference samples**. More complex styles often require more reference samples to imitate, which requires a trade-off between reference quantity and generation quality.\n\nAlthough we only show the results of our method in Figure 9, please note that **this limitation is common to all font generation methods**. LF-Font[1] has discussed this issue and CF-Font[2] aims to reduce the difference between content and target styles through content fusion. Since the font in Figure 9 is too novel and lacks generality, we did not include it in the comparison in Figure 5. \n\nWe include all methods' results on those novel fonts in Figure R1(in the rebuttal PDF) to support our conclusions. Figure R1 shows that methods based on disentanglement are limited by the content font, and generate worse results when the content style is far away from the target style. Unfortunately, this limitation has not been well addressed yet, we hope to leave it for future works.\n\n[1] Park, S., Chun, S., Cha, J., Lee, B., & Shim, H. (2022). Few-shot Font Generation with Weakly Supervised Localized Representations. In IEEE Transactions on Pattern Analysis and Machine Intelligence: Vol. PP (pp. 1–17). https://doi.org/10.1109/TPAMI.2022.3196675  \n[2] Wang, C., Zhou, M., Ge, T., Jiang, Y., Bao, H., & Xu, W. (2023). CF-Font: Content Fusion for Few-shot Font Generation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JHSl7HE2mk", "reviewer": "Reviewer_ufc8", "summary": "This paper approaches the task of Few-shot Font Generation (FFG) for Chinese characters, proposing to model the target glyph with Ideographic Description Sequence (IDS) tokens to achieve style-content disentanglement. Reference font images and the target character’s IDS are fed into a VQGAN-based pipeline to decode the output image autoregressively as quantized codebook tokens. The results show that this method outperforms SOTA in one- and few-shot settings and for significantly differing font styles.", "questions": "Is this approach specific to CJK characters? If so, it seems like this should be mentioned in the title, abstract, and conclusion.\n\nThe use of “SCA” is confusing as in L248 it is mentioned as a “module”, but the acronym “SCA” does not appear before. Assuming this refers to “Style Contrast Augmentation” (Sec 3.3) this seems to be the contrastive learning loss term and not a module in the model architecture. I’m also not sure if “augmentation” is the right term since this is not performing data augmentation.\n\nWhat does it mean on L111 that directly employing the character would be “impractical”? Isn’t this done in prior works? Additionally, L119 says that the long-tail distribution of IDC’s poses a challenge for training, but isn’t this shown in the first row of Table 3 with overall good results? (relative to the competing methods in Table 2)\n\nWill the curated data and IDS table (L202) be made available?\n\nThe paper mentions a user study (L208 etc.). Who were the participants, were they paid (L236 mentions “volunteers”), and does this require IRB approval?\n\nTypos: L19-20 incomplete sentence, L199 L131-136 L217 L311 L325-326 grammar, L139 L176 L250 formatting issues, L497 missing period.", "limitations": "Limitations and societal implications are adequately discussed.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "The paper approaches an important problem and the idea of using IDS as input for FFG is clever and logical. The qualitative and quantitative results of the essential idea are convincing and the array of comparisons and attention to fair evaluation is appreciated. The presentation is mostly clear.", "weaknesses": "The differences in most metrics when ablating components (Table 3) mostly seem quite small, compared to the larger differences in metrics when comparing to competing methods (Table 2). It’s not immediately convincing that the added complexity of these components (particularly IHA) are justified. On the other hand, the significant drop in FID when using I+S without C in Table 2 seems important and is not explained. If the use of these components gives a qualitative advantage that is not fully reflected in the metrics, this should be illustrated or included in the user study.\n\nThe design of the SSA (Sec 3.2) includes sub-components (global and local branches) which are not ablated. It would also help to indicate which is which in Figure 4. Claims about these controlling features such as stroke width, edges etc. (L134-147) could be justified. I also found Sec 4.4 hard to follow; how does Figure 7 show that SSA is effective? It’s not obvious where the attention should be focussed for positioning IDC’s or for those that don’t appear in the reference glyphs.\n\nThe method is tested on data collected by the authors. It is unclear if the proposed method would outperform the competing methods (Sec 4.2) if trained on datasets used in prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The differences in most metrics seem quite small\nOn one hand, we attribute this to **the advantages brought by the new paradigm**. The usage of IDSs and VQ-token based decoder have already achieved good results, making our baseline strong.\n\nOn the other hand, this is due to **marginal effects of the metrics**. Because the metrics of baseline are already quite excellent, the improvement brought by modules is relatively weak in terms of metrics.\n\nWe show the visualization in Figure 6, and these changes may not be important for all samples, but they are crucial for certain glyphs and effectively improve generation quality.\n\n### Q2: The significant drop in FID when using I+S without C\nBecause FID has limitations and a certain gap with human's perception. We provide corresponding visualization in Figure 6, which shows that our module can effectively improve the quality of glyph.\n\n### Q3: The sub-components of the SSA are not ablated\nThank you for your insightful comments and observations regarding our model. We have added below the ablated results for the two branches of SSA module:\n\n| SSA (3shot, ufuc)        | FID↓ | L1↓ | LPIPS↓ | RMSE↓ | SSIM↑ |\n|------------|---|---|---|---|---|\n| wo/ global | **7.9766**  | 0.1607  | 0.1347  | 0.3789  | 0.4775  |\n| wo/ local  | 8.2578  | 0.1620  | 0.1364  | 0.3805  | 0.4756  |\n| full        | 8.4922  | **0.1597**  | **0.1338**  | **0.3775**  | **0.4782**  |\n\nFigure R2 (in the rebuttal PDF) is our modified version of Figure 4, with the two branches highlighted. If you have any further suggestions, please feel free to continue discussing with us.\n\n### Q4: How does Figure 7 show that SSA is effective?\nFigure 7 shows the attention maps for the local branch in SSA. When there are target IDCs or components present in the reference glyphs, more attention is paid to the corresponding areas. For example, the first, second, fourth attention maps in the first row and third, fifth attention maps in the second row are highlighted accordingly.\n\nWhen a reference glyph does not contain any target elements, the local branch tends to pay less attention to it, which is why the third row appears almost blank.\n\nRather than forcing attention allocation to introduce interference, we think it is better to adopt the overall features extracted by the global branch.\n\n### Q5: Why not test methods on datasets used in prior works?\nThis is a very reasonable concern. However, in the font generation field, due to **copyright protection**, there is no widely recognized open-source dataset. The current mainstream methods, such as FontDiffuser (AAAI 2024), CF-Font (CVPR 2023), VQ-Font (ICCV 2023), LF-Font (TPAMI 2022), are all use private dataset.\n\nOur dataset is large enough and covers a variety of fonts and most commonly used Chinese characters, We believe that the comparison results are relatively fair. If we use other datasets, we are confident that the proposed method can still outperform the competing methods.\n\n### Q6: Is this approach specific to CJK characters?\nOur methods is not specific to CJK characters. We focus on CJK characters because of their spatial structures, which can better reflect the characteristics of our method. If we expand the vocabulary and train with corresponding data, IF-Font can also be suitable for other characters.\n\n### Q7: The use of “SCA” is confusing\nSorry for the confusion: We define SCA as a module because there are some network layers used to extract features in addition to the loss part. \"SCA\" refers to \"Style Contrast Augmentation\", we apologize for not introducing it in Section 3.3, and thank you for pointing it out.\n\nThe term \"augmentation\" might not be suitable, we intended to emphasize the improvement brought by the contrastive loss. Perhaps the term \"enhancement\" is a better choice? We will continue to refine the wording and make revisions to the draft, any further discussions are welcome.\n\n### Q8: Why is it “impractical” to directly employ the character\nSome early font generation works have indeed used characters as input directly, but this approach is only suitable for those with a small vocabulary table, such as the Latin alphabet.\n\nIn this paper, we focus on CJK characters. Due to their large number, if we were to use them as input directly, an **enormous vocabulary table** is needed. At the same time, new characters are constantly being added, low-frequency characters may lack font support, leading to **insufficient training data**. Directly employing the characters makes model difficult to generalize.\n\n### Q9: Why is the problem caused by long-tail distribution not obvious in metrics?\nMost Chinese characters are left-right or top-bottom structures, the improvements made by the IHA module are not significant in terms of metrics. We demonstrate this in the fourth cloumn (the column \"+I\") of Figure 6.\n\n### Q10: Will the curated data and IDS table be made available?\nSure, we will release these data to ensure that our model can be reproduced.\n\n### Q11: Who were the participants in user study, were they paid (L236 mentions “volunteers”), and does this require IRB approval?\nOur user study was conducted through Fuxi Youling Crowdsourcing Platform[1]. We paid the platform to publish a questionnaire with a limited number of 30 slots. After data collection is completed, we received anonymous responses from the platform. Therefore, the term \"volunteers\" might not be accurate, maybe \"platform users\" or \"participants\" would be more suitable? \n\nThe platform users fulfill a task to earn points, which can be redeemed for money. Since we collaborated with the third-party platform, IRB approval is not required.\n\n[1] https://fuxi.163.com/solution/data\n\n### Q12: Typos\nThank you for your detailed reading of our manuscript. We will correct these typos in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYH314fESt", "reviewer": "Reviewer_o7CM", "summary": "This paper presents a method that generate Chinese glyphs using token prediction approach. The core contribution is to leverage the concept called Ideographic Description Sequence and develop a network architecture to generate IDS that represent the target character.", "questions": "My questions and concerns are:\n- The exposition of the paper is quite unclear to me. In the paper, I did not find an explanation of what each token really represents. Is it a part of the glyph? The decoded results shown in the paper are always a completed glyph.\n- It is unclear how to compose the predicted token into a completed glyph.\n- I recommend the authors to survey and discuss more about Ideographic in previous font manipulation is not thorough.\n    - e.g., Ariel Shamir, A. Rappoport, Compacting oriental fonts by optimizing parametric elements, The Visual Computer, 1999\n- It is unclear what does \"IDS is not perfect enough to identify Chinese characters\" in the limitation section mean?", "limitations": "- as mentioned above, I am curious how well the method can generalized to other writing system? If it is hard, then I suggest authors discuss this in the limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper are:\n- The problem formulation using Ideographic Description Sequence is novel and interesting.\n- The evaluation is thorough and the generated Chinese glyphs are convincing.", "weaknesses": "The weaknesses of this paper are:\n- The exposition is sometimes confusing. I would encourage the authors to explicit show the output of the network before putting the output together as a glyph (I suppose the output sequence is composed of multiple tokens.)\n- I understand the proposed method aims to focus on generating Chinese characters, but it seems can not be generlized to other writing systems, including Roman characters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: what is the output of the network before putting the output together as a glyph? Is it composed of multiple tokens?\nYes, the output sequence consists of multiple tokens. Please refer to the right of Figure 1, where the small green squares around Transformer Decoder are tokens (vector quantized tokens), which are **the indexes of codes in the VQ-GAN codebook** (VQ-VAE or other vector quantized methods are also applicable).\n\nSpecifically, after encoding an image into a feature map, each feature vector is replaced by the closest code in the codebook. Since all vectors are selected from the codebook, we can use the corresponding indexes to replace them, yielding a 2D array of integers. This array is then flattened into a sequence where each integer represents a token.\n\n### Q2: How to compose the predicted token into a completed glyph?\nThe token sequence is first restored to a 2D array and replaced by vectors from the codebook, resulting in a quantized feature map. This feature map is then **decoded by a VQ-GAN decoder** to restore the original image. Since our work focuses on modeling the tokens of glyphs, decoding the predicted tokens into a glyph falls beyond the paper’s scope\n\n### Q3: The survey and discussion about Ideographic in previous font manipulation is not thorough.\nThank you for providing a valuable reference, which we carefully read and found that it deals with the parametric representation of glyphs, allowing for a elegant trade-off between glyph quality and the amount of compression. It seems to be more closely related to the VQ-GAN we used, both trying to compress glyph representations. In future work, using this parametric elements as an input/output format may be promising.\n\nFollowing your suggestion, we will include relevant discussions and cite this reference in the final submission. Thanks for your kind response.\n\n### Q4: what does \"IDS is not perfect enough to identify Chinese characters\" in the limitation section mean?\nThe shortcomings of IDS are reflected in two aspects:\n1. **Rule conflicts**. A very small number of characters are too similar, and their IDS of stroke granularity is identical.\n2. **Insufficiently clear spatial descriptions**. For example, the left-right structure puts two components on the left and right, but the distance between them is not precisely specified, which requires the model to learn enough to distinguish.\n\n### Q5: How well the method can generalize to other writing systems?\nWe focused on CJK characters because they have spatial structures, can be used to better demonstrate our main contributions. We have not tested it on other writing systems yet and are uncertain about its generalizability. Thanks for your valuable feedback, we will clearly state this in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We attempted our best to address the questions as time allowed. We believe the comments & revisions have made the paper stronger and thank all the reviewers for their help. Please find individual responses to your questions below. The PDF file for the figures is attached to this general response.

Meta-review: Initially, this paper received mixed ratings: 5, 4, 6 and 5. The comments included more experimental comparison, advantages over some methods, clarification and discussions for the proposed method and experiments, and so on.After post-rebuttal discussion, the ratings were changed to 6, 5, 7, and 5. Three reviewers were satified with the rebuttal and upgraded the ratings. The other reviewer was also satisfied with rating and did not update the rating.  The AC read the comments and the paper carefully.
Decision: Accept (poster)