Decoupling Layout from Glyph in Online Chinese Handwriting Generation
Authors: Minsi Ren
Yan-Ming Zhang
yi chen
Venue: ICLR 2025 Poster (2025)
Keywords: Online handwriting generation; Layout generation; Calligraphy imitation; Conditional diffusion Model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A novel method for Online Chinese Text Line genration.

Abstract: Text plays a crucial role in the transmission of human civilization, and teaching machines to generate online handwritten text in various styles presents an interesting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving complete text line generation largely unexplored. In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. Meanwhile, the font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA-OLHWDB demonstrate that our method is capable of generating structurally correct and indistinguishable imitation samples.

Published as a conference paper at ICLR 2025
DECOUPLING LAYOUT FROM GLYPH IN ONLINE CHINESE HANDWRITING GENERATION
Min-Si Ren 1.2, Yan-Ming Zhangl ; Yi Chen 1.2
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
"State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institution Automation Chinese Academy of Sciences, Beijing 100190, China
renminsil renminsil7@maill 1emaiismmml.s..mmmmmmmmmmmiimllm (ymzhang, (..chene@nlpp aa.ac.cm
wwas accas.ac.... ac.cn on
ABSTRACT
Text plays a crucial role in the transmission of human civilization, and teaching machines generate online handwritten text in various styles presents an inter- esting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving complete text line generation largely unexplored. In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator per- forms in-context-lie learning based the text content and the provided style references to generate positions for each glyph autoregressively. Mcanwhile, the font synthesizer which consists of character embedding dictionary, a multi-scale calligraphy style encoder and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA- OLHWDB demonstrate that our method is capable of generating structurally cor- rect and indistinguishable imitation samples. Our source code will be publicly available at:
INTRODUCTION
Deep generative models, such as GANs (Goodfellow et al. 2014), VAEs (Kingma & Welling 2014 and diffusion models (Sohl-Dickstein al.) 2015 Ho al. 2221). have demonstrated formidable ability to generate a wide array of data types. Handwritten data, which encompasses language char- acters, mathematical symbols, sketches, and diagrams, represents highly personalized form of data with a wide range of application scenarios. Creating handwriting can provide personalized writing service be used for data augmentation for document analysis models (Lai et al. [202]] Kang al. 2021 Xie et al., 2020). In recent years, the emergence of extensive datasets has prompted notable surge in the application of generative modeling techniques to handwritten data (Xu al., 2022]
Aksan et al. 2018).
Among handwritten data, the generation of Chinese handwritten texts has garnered increasing at- tention due to their widespread use and the greater challenges they present compared to Latin scripts (Gao et al., 2019 Liu et al.) (002).. A crucial aspect of this task is ensuring the integrity and accuracy of the structural composition of the characters. The vast diversity and complex ge- ometric structures of these characters render this task particularly challenging. Another important consideration is how to imitate specific personal writing styles while maintaining structural correct- ness ( Yang et al. 2024; Gan & Wang 2021 Xie et al.] 2021 Kong et al.] (022). Compared to individual characters, text lines serve more effective communication tools, capable of conveying
Corresponding Author

---

Published as a conference paper at ICLR 2025
Writer I: 3
B
t/
/
Writer
T
Writer 3:
Figure 1: Handwritten text lines with vastly different styles generated by our methods. It is worth mentioning that the generated online data contains dynamic trajectory information, rather than just static images, enabling more interactive applications. Different colors represent different strokes, showcasing the dynamic process of writing.
more complex information and emotions. However, the majority of prior research has primarily fo- cused on generating individual Chinese fonts, largely overlooking the significantly more challenging task of generating complete text lines.
In this work, we aim to generate stylized and coherent online Chinese handwritten text lines based on specified text contents and style reference samples. A handwritten line of text can be viewed as composed of individual characters and their arrangement. Based on this observation, we propose hierarchical method that disentangles the layout generation from the writing of individual characters during the generation of full lines. We design novel text line layout generator to arrange posi- tions for each character based on their categories and given handwriting style references. For font generation, we construct a ID U-Net (Ronneberger al.) 2015) network and design multi-scale feature [Wang et al. (2023) fusion module to fully mimic the calligraphy characteristics of the given reference sample. We utilize the publicly CASIA- SSSA-OO dataset (Liu et al., 2011) for training and evaluating method, showcasing its effectiveness through combination of quantitative and qualitative experiments. Figure [ shows several visualizations of our generated samples.
Our contributions can be summarized as: 1) We propose hierarchical approach solve the unex- plored online handwritten Chinese text line generation task. 2) We introduce simple but effective layout generator that can generate character positions based the text contents and writing style through in-context-like learning. 3) We construct a 1D U-Net-based network for font generation with a multi-scale contrastive-learing--e style encoder to improve the ability of calligraphy style imitation.
2
RELATED WORK
2.1 ONLINE HANDWRITTEN DATA GENERATION
The most commonly adopted method (Graves) 2013) for generating online handwritten data involves combining neural network models for sequence data processing with mixture of Gaussian distri- butions to model the movement information of the pen. Following this work, SketchRNN (Ha & Eck 2018 adopts RNN as the encoder and decoder of VAE, conducting the task of unconditional sequence data generation. COSE (Aksan et al. 2020) treats drawings as a collection of strokes and projects variable-length strokes into a latent space of fixed dimension and uses MLPs to generate one single stroke based on its latent encoding autoregressively. Subsequently, there has been grow- ing emphasis in academic research on conditional generation, such as producing handwriting with specific contents diverse writing styles (Aksan et al. 2018 Kotani et al. 20200] Tolosana et al.) 2021). They extract calligraphic styles from reference samples and combine them with specific textual content achieve controllable style English handwriting synthesis.
In contrast to alphabetic languages, Chinese encompasses a significantly larger character set and Chinese characters exhibit more intricate shapes and topological structures (Xu et al. 2009 Lin (et al. 2015); Lian et al.) 2018: Radford et al. 2016)) Zhu et al. 2017). For this reason, most methods designed for handwritten English generation do not work well when applied directly to Chinese.
2

---

Published as a conference paper at ICLR 2025
As for the field of online Chinese character generation, both LSTM and GRU models are employed simultaneously in (hhag et al. 2017) to successfully generate readable Chinese characters of spe- cific classes the first time. FontRNN (Tang et al. 2019) focuses stylized font generation, which utilizes a transfer-learning strategy to generate stylized Chinese characters. However, each trained model can only synthesize the same style the train set. By adding style encoder branch the generator, DeepImitator (Zhao et al. 2020) can generate characters of any style given a few reference samples. They utilize CNN as the calligraphy style encoder to extract style codes from offline im- ages and GRU as the generator to synthesize the online trajectories. Similarly, WriteLikeU (Tang Lian 2021), DiffWriter (Ren al. 2023), and SDT (Dai et 2023) are also capable of handling this functionality.
2.2 DIFFUSION MODEL FOR SEQUENCE DATA
Diffusion model, first proposed (Sohl-Dickstein et al. 2015) exhibits a remarkably robust capa- bility in learning the data distribution and generating impressively high quality and diverse images
in recent years Luo 2022 Ho et al., 202 Dhariwal Nichol 2021: Song et al. 2020 Rombach
et al. 202T). In terms of sequence data generation, NaturalSpeech2 (Shen et al. (022)) employs the
latent diffusion framework to synthesize speech content with specific contents by imitating various tones and speaking styles. CHIRODIFF (Das et al., 2023) first utilizes diffusion for unconditional chirographic data generation and later DiffWriter (Ren et al., (023) adopts diffusion models for online Chinese characters generation. However, previous methods only explored how to apply the diffusion process to online handwritten data, lacking fine control over writing details.
METHODS
3.1 PRELIMINARY
Online Handwritten Data. In general, online handwriting is kind of time series data, which composed of a series of trajectory points. Each point (h, v, s] E R2' contains the horizontal movement, vertical movement and the state information of the pen. We use = 1 to represent the pen touching state and = -1 to represent the pen lifting state, making it convenient to distinguish the two states by simply adopting dichotomy with threshold 0 after the addition of zero-mean standard Gaussian noise to the data.
Data Notation. A handwritten font that consists of n points can be represented as x Rax3 Its corresponding category is represented as one-hot vector с E IR where K is the number of character categories. A handwritten text line can be represented as X [x], ... x"] E RR Nx3 where xf E Rn x3 denotes the i-th character, m denotes the number of characters, and N denotes the number of points with N i= n². Its text content be represented C = [ct,..., ,... cm RmxX
Problem Statement. Given the text line content C and style reference samples X (preferably a coherent handwritten text line) from a given author, our objective with the specified content while imitating the author's 's writing habits. The conditional probability can be written as p(X|C, X,e/). Unlike English, Chinese has a large number of character categories, each with its specific stroke order and geometric structure. The combinations to form a text line are endless. Therefore, the most critical challenge in generating handwritten Chinese text lines lies in two aspects: 1) Ensuring the correctness of the structure of each character while maintaining consistency with the chosen writing style. 2) Arranging the relative positions of different characters, especially between Chinese characters and punctuation marks, to achieve natural and fluent layout for the entire handwritten text line.
generate handwritten text lines
3.2 ANALYSIS AND OVERVIEW
Directly modeling p(X C, Xref) is overwhelmingly challenging since the positions of glyphs unknown. In this work, we treat the text line layout as latent variables the text line generation and instead model the joint distribution of the text line and its layout: p(X, Layout C, X,e)))
3

---

Published as a conference paper at ICLR 2025
2
5
[h, W, c
center pos
width
horizontal offawt
Figure 2: The illustration of character bounding box, consists of height, width, vertical center posi- tion and horizontal offset relative to the previous character.
In detail, we define the layout of a handwritten text line as the positions and sizes of all charac- ters in the line. We calculate the [height, width, vertical center position and horizontal offset relative to the previous character] for each character its bounding box, shown in Figure 2 In this way, the layout of X be represented Layoutx = [Box', ... Box" E m=
During the writing process, human beings typically take neighboring glyph positions into account for the current character position to ensure the continuity of the written text lines. However, when the positional information of each character relatively independent. Similarly, we assume that when the bounding box of a character is given, the generation of the character is independent of other characters. By this assumption, we can factorize p(X, Layoutx x C, Xref) as:
provided, the writing process of each character is
p p(X, Layout [C, = p(Layout Xre/ (pee))((XCCCCCCC Xre/. Layoutx)
= p(Layout |C, X f II p(x'lc' Xref, Box').
Equation [ inspires us to decouple the complete generation process into two components: layout generation pLayyuttx C, Xyee) and individual font synthesis p(x'|c', Xref, Box'). Correspond- ingly, our proposed model consists of two parts: text line layout generator and stylized font synthesizer, shown Figure
3.3 IN-CONTEXT LAYOUT GENERATOR
Modeling The Human Writing Process. Considering the consistent and coherent style of the entire text layout when people write a complete text line, is natural and reasonable to use certain length of the prefix of the text line context to predict the subsequent layout. Based this observation, we decompose the generation of layout in an autoregressive manner:
p(Layout C, X,ee)))) p( (Box Box" Xref)
I (Box Box Box c,,, Xres),
which is akin the human writing process. Denote the layout generator model as G, which is an LSTM network in our work. The generation of the i-th character box can be represented as:
Box h²] G(oox (
where h represents the hidden state of LSTM. It is worth noting that different glyphs, punctuation marks, and numbers generally have markedly different shapes. Therefore, it is crucial to consider the character categories during the generation.
Training Objective. During the training phase, we employ the teacher- forching technique and use the
l distance between ground truth and generated layout the loss function. We empirically illustrate that this training method encourages the model to generate layouts that are consistent throughout the whole text line.

---

Published as a conference paper at ICLR 2025
Bounding Boxes Prefix
Layout Generator
Output Layout
In-context-like learning
Multi-scale Style Encoder
Style Reference
CNN
Feature Fusion
Output Text line
Look
Text Content
Xg
concat
Character Embeddings
Character Diffusion
Overview of the proposed method, which consists of layout generator and font synthesizer. Given the text content and style references, the two modules operate simultaneously: the layout generator will arrange the bounding box of each character based on the overall style of the reference, while the font synthesizer will imitate the calligraphic style of the references to produce the corresponding handwritten fonts.
Figure 3:
In-Context Generation. In order to mimic the layout style of the given reference sample, we use the style reference samples as a prompt context, inputting their true bounding boxes as prefix instead of the generator G's outputs. The outputs corresponding to the prefix time steps are inconsequential, but the hidden state of the LSTM network implicitly contains information about the layout style. will plan the subsequent layout according to the writing style of the prefix context.
We experimenaally show that this approach effectively possesses the capability for in-context-lin like learning in this task, and therefore imitate general writing styles unseen in the training set. Ad- ditionally, if no coherent text line provided as the style reference sample, the model will perform unconditional layout planning with prefix length of O.
3.4 STYLIZED DIFFUSION CHARACTER SYNTHESIZER
We adopt the conditional diffusion model as the generator for individual characters. However, pre- vious diffusion based methods for online handwritten data have neglected the multi-scale features of calligraphy style. To address this issue, we design ID U-Net network as the denoiser, and com- bine it with character embedding dictionary and a multi-scale calligraphy style encoder generate characters with specific categories and calligraphy styles.
3.4.1 MODEL ARCHITECTURE
Character Embeddings. Inspired by ÚZhng et al. 2017). dictionary is constructed to encode structural information of characters: E E & where represents number character categories and N, represents the code dimension. Each row in matrix E encodes the structure of one category. To generate a character the k-th category, we first query the embedding dictionary and get the k-th row of matrix E[k] IR N. as the content code. Supposing the shape of noisy data is (n, 3), we duplicate the content code as well as sinusoidal position encoding time emb, E TR No for times and concatenate them with the noisy data x, obtain the input x [x4, emb,, E[k]] (Nxx(NNNN++3) of the denoiser.
Multi-Scale Style Encoder. Compared to the global layout style, the calligraphy styles individual characters are often reflected in various details, such overall neatness, stroke length, and curvature.

---

Published as a conference paper at ICLR 2025
Considering that these characteristics all at different scales, construct 1D convolutional style encoder that employs continuous downsampling layers to extract calligraphy features at different levels and ID U-Net denoiser to leverage these features.
U-Net Denoiser. The U-Net network shares the same number of down-sampling layers and strides as the style encoder. We set the number of down-sampling blocks 3 in this work. This align- ment ensures that during the decoding process, each layer's features in the U-Net can correspond features of the same scale in the style encoder. These features are fused using the QKV cross- attention mechanism (Rombach et al. (20TT), thereby incorporating multi-scale style information into the denoiser. The architectural diagram and further details can found in Appendix A.2.2
3.4.2 TRAINING OBJECTIVE
Denote the font synthesizer as E, the loss function consists of two parts: multi-scale style trastive learning loss le as well as a diffusion process reconstruction loss e,:
Eg la + lr.
We detail the two learning objectives follows.
Multi-Scale Style Contrastive Learning Loss. To incentivize the style encoder to extract discrim- inative calligraphic style features from different authors across diverse scales, we perform trastive learning on distinct feature layers respectively. More concretely, suppose a mini-batch (X Xwy contains characters written by b different authors W {wI,..., wa} and let A(w:) = W Ww:). Recalling section [.4.1] the style encoder SENC contains three downsam- pling blocks corresponding three different scales of features for one character: SENC(x,;) (fre p f0,
For each scale features, take J E R as an sample, where L represents the length of the feature sequence and dl indicates the dimension of each feature vector. We randomly select two non- overlapping segments ea e from f (e E Rixd from the total length L), serving as positive pair from author wi. The contrastive loss for f is formulated as:
exp (s(e /TT) Ew.eA(wr) exp (s(e 34 )/T)
Σ log
(5)
where se e = (e ( T is a temperature parameter, g is linear projection. Similarly, can obtain la and Ea The overall multi-scale contrastive loss is formulated as:
== A1e X22 A
where Å1, A2, As are empirical weights. We set A = 0.01, A2 = A = 0.1 in this work.
Diffusion Reconstruction Loss and Generation Process. Given target character x and style ref- erence samples X,ef which is a piece of handwriten text line written by the same writer. Denoting the style encoder as SENC, and the character embedding dictionary as E, the diffusion reconstruc- tion loss is calculated in Algorithm 1. T, [ T. are hyperparameters of diffusion models, which are introduced in Appendix(A.. I Correspondingly, given the character category & and style reference samples Xyef- we randomly sample standard Gaussian noise as Xy E IR (*3),, where n is the max length, the generation procedure is summarized in Algorithm 2
4 EXPERIMENTS
In the experimental section, we assess our approach in three areas: capability of calligraphy style imitation, capability layout style imitation, and content readability, utilizing both qualitative and quantitative experiments as followings:
4.1 DATASET
We use the CASIA Online Chinese Handwriting Databases (Liu et al. 2011 to train and test our model. For single character generation, following previous work, the CASIA-OLHWDB (1.0-1.2) is

---

Published as a conference paper at ICLR 2025
Algorithm Diffusion Reconstruction Loss E
Algorithm 2 Font Generation Process
1: procedure TRAIN( Xref, k. T. (ox))
procedure GEN(XT, Xress (ast)
2: satyle SENC
Sample t. (1, T), T). ((0,))
3: for = T,T 1,.., do
4: X; Vo + v ThE
4:
Xin -coccat emb, E(M))
5: X concat x;, cmb,, F(A])
5:
{x Sstyle) X1
6: fatyle- = SENC(X,,s)
6:
Ae Mx())
-l-aevar ED
1-a
cp E(xia Jstule)
of (1-4)(17 Th--1)
7:
8: (,(0) = ||e 1col2
-- X;-1 = (o(()) 0,2,2 N\
9: Output: fr
8:
9:
end for
10: Output: Xo
10: end procedure
11: end procedure
adopted as the training set, which contains about 3.7 million online Chinese handwritten characters produced by 1,020 writers. The ICDAR-2013 competition database (Yin et al. 2013b) is adopted the test set, which contains 60 writers, with each contributing the 3.755 most frequently used characters set of GB2312-80. For layout and text line generation, we adopt CASIA-OLHWDD (2.0-2.2) which consists of approximately 52,000 text lines written by 1,200 authors, totaling 1.3 million characters. We take 1,000 writers as the training set and the left 200 writers as the test set. Detailed descriptions of data preprocessing, model architecture, and training process are provided in Appendix A.2
4.2 FONT GENERATION
Evaluation Metrics. To assess the model's ability to imitate calligraphy style and generate accurate character structures, we adopt three core indicators DTW (Normalized Dynamic Time Warping Distance), Content Score, and Style Score following previous work (see Appendix A..2 for details). For each writing style in the test set, we randomly generate 100 characters, totaling 6,000 characters, measure the performance.
Baselines and Results. There are mainly two types of online handwritten Chinese character gen- eration approaches: the first type comprises purely data-driven methods that learn the structure of characters and calligraphy style only from the training set. The second type requires standard printed font image as an auxiliary input, and train models to transform the standard fonts into char- acters with specific writing style. The advantage of the second method is that it is casier generate correct structures, but cannot generate characters without corresponding standard glyphs and the computational cost will be heavier. As shown in Table [ being method of the first type, model achieves state-of-the-an performance in all pure data-driven approaches and is comparable state-of-the-art style transfer-based methods. The qualitative comparison experiment is shown in Appendix A.2.5)
Effects of Contrastive Learning. In this experiment, randomly select writers in the test set, 150 characters for each writer. We first use the style encoder to extract their features and visualize them using the t-SNE method (Van der Maaten & Hinton 2008). As shown in Figure/4 the right side was obtained through our fully trained style encoder, while the left side does not adopt contrastive learning loss. In the right figure, features from the same author usually exhibit a clear clustering trend, while in the left figure, the distribution is more random and scattered. It illustrates that through contrastive learning, calligraphy style encoder can extract discriminative features effectively.
Table 1: Evaluation of Character Structure
Accuracy and Calligraphy Style Imitation.
Purely data-driven
Style transfer
Methods
DeepImitator
DiffWriter
Ours
FontRNN
WriteLikeUU
SDT
DTW (4)
1.062
0.997
0.932
1.045
0.982
0.880
Content Score (t)
0.834
0.887
0.891
0.875
0.938
0.970
Style Score (t)
0.432
0.481
0.918
0.461
0.711
0.945
7

---

Published as a conference paper at ICLR 2025
Figure 5: The ablation of multi-scale contrastive learning.
A
A
As
Style Score
Content Score
0.0
0.0
0.875
0.893
0.0
0.0
0.1
0.900
0.887
wifiout contandee leaning
0.01
0.1
0.1
0.918
0.891
Figure 4: t-SNE visualization of the font style features. Dif- ferent colors represent different writers.
Different Weights for Contrastive Learning. Ta A1, A2. Àg are empirical weights in equa- tion 6 A weight of 0 indicates that the loss is not used in this feature layer. We find that while the Content Score is basically unaffected, the Style Score experiences a certain improvement, particu- larly with the implementation of multi-scale contrastive learning.
4.3 FULL LINE GENERATION
4.3.1
EVALUATION METRICS
Table 2: Binary geometric features for layout gen- eration evaluation, which is calculated between every two adjacent characters.
Layout Assessment. The core of method lies in the in-context layout generation. To better demonstrate the effectiveness of our method, we adopt several binary geometric features from traditional OCR (Zhou et al. 2007 Yin et al. 2013a) that can effectively reflect the characteristics of character spatial relations within text lines, shown in Table 2 We calculate the difference in these features between the generated samples and the real samples as quantitative evaluation indicators, denoted V, to Vs.
No.
Binary geometric features
1
Mean vertical distance between geometric centers
2
Mcan horizontal distance between geometric centers
3
Mean distance between upper bounds
4
Mean distance between lower bounds
5
Mcan distance between left bounds
6
Mean distance between right bounds
7
Mean ratio of heights of hounding boxes
8
Mean ratio of widths of bounding boxes
Readability Assessment. To assess the structural correctness performance, AR (Accurate Rate) and CR (Correct Rate) are two widely used metrics for Chinese string recognition (Su et al. 2009:; Wang al. 2011). We adopt the methods proposed in (Chen et al. (023) as the text line recognizer, which can get 0.962 AR and 0.958 CR on the test set. We report the recognition metrics of the recognizer on our synthesized text lines. where higher values indicate more complete and accurate geometric structure. More details are introduced in Appendix (p.2n
Baselines. 1) In the field of text line recognition, several methods (Peng al. 2019 Yu al. 2024) employs synthesized samples to augment the dataset for training. In these synthesized samples, the bounding box (introduced in Section 3.2p of each glyph category is statistically modeled Gaussian distribution in the training set. When generating, the bounding box for each character sampled based on these Gaussian distributions. We denote this layout generation method as "Gaussian'. 2) Furthermore, generator model can perform unconditional generation, denoted as 'Unconditional" We combine the two layout generation methods with our font synthesizer and compare them with proposed approach quantitatively and qualitatively. The default reference text line length set as 10.
mentioned before, if no reference samples are provided, our layout
4.3.2 QUANTITATIVE RESULTS
As shown in Table 3 we find that: 1) the generated text lines all exhibit AR and CR values exceeding 0.85, demonstrating the complete and correct structures of each character. 2) The layouts generated by the in-context method are closer to real samples in nearly all binary geometric features, which highlights the effectiveness of our proposed layout generative method.
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation of Handwritten Text Line Layout Imitation and Readability.
Baselines
(4(
V2(()
(2(4)
Va(4)
Vs(4)
Vs(U)
Vr(+)
Vs(+)
AR
CR
Gaussian 0.074
0,151
0.092
0.092
0.163
0.160
0.859
1.081
0.851
0.855
Unconditional 0.048
0.132
0,034
0.063
0.160
0.139
0,366
0.408
0.852
0.856
In-Context 0,046
0.122
0,062
0.058
0.129
0.129
0.364
0.419
0.852
0.857
We further investigate the impact of varying reference sample lengths on the performance of lay- out generator as shown in Figure [ Intuitively, with increasing length, most metrics tend to im- prove. Compared to the vertical binary geometric features (Vi, V3. Va, V7), the horizontal features (V2, Vs, V., Vg) exhibit larger deviation from real samples, indicating that learning the spacing size is more challenging when imitating the layout style of text lines. Furthermore, with respect to these horizontal features, the disparity between the generated samples and the authentic samples is markedly decreased as the length of the reference sample extends.
0.35
Differncee Geometic: Seaturee
0.40
0.30
0.35
0.3
91
0.30
93
05
0.20
025
06
0.15
0.29
0.10
0.1
0.5
0.00
Difiorott of Refurence Sampll
Figure 6: The difference between the mean binary geometric features of generated samples and real samples under different reference sample lengths.
4.3.3 QUALITATIVE RESULTS.
Layout Case Study. We specifically select samples with unique layout styles (upward slant) in the test set. 7 shows that although the unconditional layout generation method can produce structurally reasonable layouts, it completely neglects the layout style of the handwritten text line. In comparison, our method effectively mimics characteristics such as glyph spacing and the slant trend of handwriting, verifying the necessity of our layout generator, Additional experimental results can be found in Appendix A.4
Conditional (Ours)
10
Enconditional
10
14
16
Gaussian
0
2
10
12
16
Figure 7: Visualization of character bounding boxes generated by different methods, where green represents the ground truth. For our method, the bounding boxes of the first ten characters are treated as the context prefix.

---

Published as a conference paper at ICLR 2025
Subjective Visualization Experiment. We construct visual subjective test shown in Figure g Fifteen pairs of similar samples are created, and 20 participants are asked to determine whether lines in one block were written by the same person. Over 85% of the responses are "yes." indicating that the distinction between real and synthesized samples is difficult to perceive. It can be observed that our model adeptly imitates the target samples, capturing both the overall layout style, such as spacing and slant, as well as the calligraphic style of individual characters.
Writer 1 2
Writer 2
R408
Pe R 34
B
7p
8 >
&
20
B & B
Fe x to
83,
Writer 3
Writer 4
ERA #*
9 " 10 "
" 4 S
- &
2 #
Writer 5
Writer
# 1/ -
3 & 7
Writer 7
Writer
Figure 8: Examples of subjective experiments. For each author block, one row contains real sam- ples and two rows contain synthesized samples. Participants must judge whether the samples were written by the same person.
CONCLUSIONS AND LIMITATIONS
In this paper, we tackle the stylized online handwritten Chinese text line generation task hierarchi- cally, which is almost unexplored. Our model consists of novel text line layout generator and a stylized diffusion-basse character synthesizer. The text line layout generator can arrange glyph po- sitions based on the text contents and writing habits of the given reference samples while the stylized diffusion-basee character synthesizer can generate characters with specific categorics and calligra- phy styles. However, due to the fact that each character is generated independently, our approach encounters difficulties in mimicking styles with extensive cursive connections between characters. An end-to mnd-to-end text line generation method might be required to address this issue, discussed in Appendix A.3 Furthermore, whether synthesized samples can serve as augmentation data to as- sist in achieving better results for the recognizer is open question worth exploring. Last but not least, how to generate more complex handwritten data, such as mathematical formulas based on the hierarchical approach is also an intriguing topic. We consider these as our future work.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments: This work is supported by the Natural Science Foundation of China (NSFC) Grant No. 62436009, No. 62276258, and U23B2029.
REFERENCES
Emre Aksan, Fabrizio Pece, and Otmar Hilliges. Deepwriting: Making digital ink editable via deep generative modeling. Proceedings of the CHI Conference on Human Factors Computing Systems, pp. 1-14, 2018.
Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. CoSE: Compositional stroke embeddings. Advances in Neural Information Processing Systems, 33:10041-10052, 2020.
Yi Chen, Heng Zhang, and Cheng-Lin Liu. Improved learning for online handwritten chinese text recognition with convolutional prototype network. In International Conference on Document Analysis and Recognition, pp. 38-53. Springer, 2023.
Gang Dai, Yifan Zhang, Qingfeng Wang, Qing Du, Zhuliang Yu, Zhuoman Liu, and Shuangping Huang. Disentangling writer and character styles for handwriting generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- pp- 5977-5986, 2023.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Chirodiff: Modelling chirographic data with diffusion models. International Conference on Learning Representatiins, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-888 8794,794,42 2021.
David Douglas and Thomas K Peucker. Algorithms for the reduction of the number of points required to represent digitized line or its caricature. Cartogruphica: the international journal for geographic information and geovisualization, 10(2):112-122, 1973.
Ji Gan and Weiqiang Wang. HiGAN:
Handwriting imitation conditioned on arbitrary-length texts and disentangled styles. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 7482-7492, 2021.
Yue Gao, Yuan Guo, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. Artistic glyph image synthesis via one-stage few-shot learning. ACM Transactions Graphics, 38(6):1-12, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems, 27, 2014.
Alex Graves. Generating sequences with recurrent neural networks. 1rXiv:1308.0850 2013.
arXiv preprint
David Ha and Douglas Eck. A neural representation sketch drawings. In International Conference on Learning Representutions, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models 2020. Inter-
national Conference on Learning Representations, 2021.
Lei Kang, Pau Riba, Marcal Rusinol, Alicia Fornes, and Mauricio Villegas. Content and style aware generation of text-line images for handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12))8846-8888 8860. 2021.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: one-shot font generation via component-basee discriminator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13482-13491, 2022.
11

---

Published as a conference paper at ICLR 2025
Atsunobu Kotani, Stefanie Tellex, and James Tompkin. Generating handwriting via decoupled style descriptors. In European Conference on Computer Vision, pp. 764-780. Springer, 2020.
Songxuan Lai, Lianwen Jin, Yecheng Zhu, Zhe Li, and Luojun Lin. Synsig2vec: Forgery-fiee learning of dynamic signature representations by sigma lognormal-based synthesis and 1d cnn. IEEE Transactions on Puttern Analysis and Machine Intelligence 44(10):6422-6485 2021.
Zhouhui Lian, Bo Zhao, Xudong Chen, and Jianguo Xiao. Easyfont: style learning-based system to easily build your large-scale handwriting fonts. ACM Transactions on Graphics, 38(1):1-18) 2018.
Jeng- Wei Lin, Chian-Yo Ya Hong, Ray-I Chang. Yu-Chun Wang, Shu- Yu Lin, and Jan-Ming Ho. Com- plete font generation of chinese characters in personal handwriting style. In International Perfor- mance Computing and Communications Conference, pp. 1-5, IEEE, 2015.
Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Casia online and offline chinese handwriting databases. In International Conference on Document Analysis and Recognition, pp- 37-41, IEEE, 2011.
Wei Liu, Fangyue Liu, Fei Ding, Qian He, and Zili Yi. Xmp-font: self-supervised cross- cross-modalle modality pre-training for few-shot font generation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 7905-7914, 2022.
Calvin Luo. Understanding diffusion models: unified perspective. arXiv e-prints, 2022.
Dezhi Peng, Lianwen Jin, Yaqiang Wu, Zhepeng Wang, and Mingxiang Cai. A fast and accurate fully convolutional network for end-to-end handwritten chinese text segmentation and recogni- tion. In International Conference on Document Analysis and Recognition, pp. 25-30. IEEE, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. International Conference on Learning Represen- tations, 2016.
Min-Si Ren, Yan-Ming Zhang, Qiu-Feng Wang, Fei Yin, and Cheng-Lin Liu. Diff-writer: dif- fusion model-based stylized online handwritten chinese character generator. In International Conference on Neural Information Processing, pp. 86-100. Springer, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjm Ommer. High- resolution image synthesis with latent diffusion models. 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241, Springer, 2015.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. International Conference on Learning Representations, 2024.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. JMLR.org. 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Tong-Hua Su, Tian-Wen Wen Zhang, De-Jun Guan, and Hu-Jie Huang. Off-line recognition of realistic chinese handwriting using segmentation-fre strategy. Pattern Recognition, 42(1):167-182, 2009.
Shusen Tang and Zhouhui Lian. Write like you: Synthesizing your cursive online chinese handwrit- ing via metric-based meta learning. Computer Graphics Forum, 40(2):141-15) 2021.
Shusen Tang, Zeqing Xia, Zhouhui Lian, Yingmin Tang, and Jianguo Xiao. FontRNN: Generating largescale chinese fonts via recurrent neural network. Computer Graphics Forum, 38(7)5667-577, 2019.
12

---

Published as a conference paper at ICLR 2025
Ruben Tolosana, Paula Delgado-Santos Andres Perez-Uribe, Ruben Vera-Rodriguez, Julian Fier- rez, and Aythami Morales. Deepwritesyn: On-line handwriting synthesis via deep short-term representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 600-608, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (11), 2008.
Heng Wang. Yiming Wang, and Hongxi Wei. Affganwriting: handwriting image generation method based on multi-feature fusion. In International Conference on Document Analysis and Recognition, pp. 302-312. Springer, 2023.
Qiu-Feng Wang, Fei Yin, and Cheng-Lin Liu. Handwritten chinese text recognition by integrating multiple contexts. IEEE transactions pattern analysis and machine intelligence, 34(8):1469- 1481, 2011.
Canyu Xie, Songxuan Lai, Qianying Liao, and Lianwen Jin. High performance offline handwrit- chinese text recognition with a new data preprocessing and augmentation pipeline. In IAPR International Workshop on Document Analysis Systems, pp. 45-59. Springer, 2020.
Yangehen Xie, Xinyuan Chen, Li Sun, and Yue Lu. DG-Font: Deformable generative networks for unsupervised font generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5130-5140, 2021.
Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. Deep learning for free-hand sketch: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):285-311 2022.
Songhua Xu, Tao Jin, Hao Jiang, and Francis CM Lau. Automatic generation of personal chinese handwriting by capturing the characteristics of personal handwriting. In Twenty-First IAAI Con- ference, 2009.
Zhenhua Yang, Dezhi Peng. Yuxin Kong, Yuyi Zhang, Cong Yao, and Lianwen Jin. Fontdiffuser: One-shot font generation via denoising diffusion with multi-scale content aggregation and style contrastive learning. Association for the Advancement of Artificial Intelligence, 2024.
Fei Yin, Qiu-Feng Wang, and Cheng-Lin Liu. Transcript mapping for handwritten chinese docu- ments by integrating character recognition model and geometric context. Pattern Recognition, 46 (10):2807-2818, 2013a.
Fei Yin, Qiu-Feng Wang, Xu-Yao Zhang, and Cheng-Lin Liu. ICDAR 2013 chinese handwriting recognition competition. In International Conference on Document Analysis and Recognition, pp. 1464-1470. IEEE, 2013b.
Ming-Ming Yu, Heng Zhang, Fei Yin, and Cheng-Lin Liu. An approach for handwritten chinese text recognition unifying character segmentation and recognition. Pattern Recognition, pp. 110373, 2024.
Yao Zhang. Fei Yin. Yan-Ming Zhang, Cheng Lin Liu, and Yoshua Bengio. Drawing and recog- nizing chinese characters with recurrent neural network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):849 862,, 2017.
Bocheng Zhao, Jianhua Tao, Minghao Yang. Zhengkun Tian, Cunhang Fan, and Ye Bai. Deep imitator: handwriting calligraphy imitation via deep attention networks. Patteren Recognition, 2020.
X-D Zhou, J-L Yu, C-L Liu, Takeshi Nagasaki, and Katsumi Marukawa. Online handwritten japanese character string recognition incorporating geometric context. In International Confer- ence on Document Analysis and Recognition, volume pp. 48-52. IEEE, 2007.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired mmage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DENOISING DIFFUSION PROBABILISTIC MODELS
In this paper, adopt the Denoising Diffusion Probabilistic Model (DDPM), which is generative model that operates by iteratively applying a denoising process to noise-corrupted data. This process, known the reverse denoising process aims gradually refine the noisy input towards generating realistic samples. DDPM learns to model the conditional distribution of clean data given noisy inputs, which is derived from the forward diffusion process. Denote the forward process as q and the reverse process as p. the forward process starts from the original data Xo and incrementally adds Gaussian noise to the data:
(x.1X.-1) =
= N(x;; Vor 14)
where {ar)z= are noise schedule hyperparameters, T' is the total number of timesteps. Due to the Markovian nature of the forward transition kernel q(x,\XX-=1, we can directly sample X, (X,, Xo) without reliance on any other t:
((x||Xo) N'(6 Va, (1-0,).), at II 0z i=1
X v1 The ThE, e~,
The set of {a,) and T should guarantee that ãr is almost equal to 0, ensuring that the data distribution (Xxr closely resembles the standard Gaussian distribution. The corresponding normal posterior can be computed as:
(X,,-1/ Xo) -X, ( oo))
M Xo) vac(1 ac- 1), + va-i(11- Xo
(10)
() (1 oz) 0
1-ac a
In training, the denoiser learns to predict the added noise in Equation given noisy data X,. The most commonly used objective function is to minimize the deviation between predicted noise and true noise:
(())- = -o. - ca (X,(X,, s),
(11)
Theoretically, it equivariant to optimizing the evidence lower bound of log p(X)):
L(0)= = FuJUt2.ry DRr (X,,
(X))
(12)
During the reverse process, after substituting Equation into Equation TO and replacing the true noise with the predicted noise (X l). we have:
Po (x,,)) t) = (Xx--i;0 (())) Ho (X,,)) X,
ai
co tx
at
(13)
(t) (1 -1-aai)
1 a₁
which defines the reverse transition kernel (X1-1lXXe,
For the hyperparameters of DDPM, we adopt the linear noising schedule proposed by (Ho et al. : [T' 2021:
1000; ag = 1 le 4; ar -e 22-
A.2
IMPLEMENT DETAILS
A.2.1
DATA PREPROCESSING
We adopt the configuration the previous method: we first use the Ramer -oonner-ouglas--- -Peucker algorithm( Douglas & Peucker 1973) with a parameter of to remove redundant points. Then maintain the aspect ratio and normalize the height of each sample to 1.
14

---

Published as a conference paper at ICLR 2025
A.2.2 MODEL PARAMETERIZATION
For the layout generator model, we adopt 2-layer LSTM with a hidden size of 128 as the layout generator. We also attempted to use a transformer as a replacement and found that the results were nearly the same. Therefore, we chose the simpler and faster LSTM model. For the denoiser model, recall our model consists of ID U-Net network as the denoiser, character embedding dictionary, and a multi-scale calligraphy style encoder. We set the dimension of character embedding as 150 and the dimension of time embedding for diffusion models as 32.
Figure displays the interaction details between 1D U-Net denoiser and style encoder. Each down- sampling block, such as {ʃ", en 1,2.3 consists of three residual dilated convolutional layers and a downsampling layer. The residual dilated convolutional layers have a kernel size and dilation values of 3, and 5, respectively. The downsampling layer has stride of 2 and kernel size of The upsampling blocks of the U-Net are structurally symmetrical to the downsampling block and maintain the same hyperparameters, with the only difference being the substitution of the convolu- tional layers in the downsampling blocks with deconvolutional layers. Table numbers of input and output features for each module. Afterward, we employ simple linear layer to ensure that the output has the same dimensions as the initial data, thus performing the function of noise prediction.
the channel
L.
L
Multi-scale contrastive loss
f
f2
f3
Cross-attention
oxput
610g
deg
de
Skip-connaction
Figure 9: The framework of the multi-scale style encoder and the U-Net Denoiser. The number and stride of downsampling layers of U-Net and the style encoder are aligned, which we set as in this work. The multi-scale style information is injected into U-Net through the cross-attention mechanism.
Table 4: Channel numbers of feature inputs and outputs at different layers..
f
f2
f
enc
en?
en
de
de2
de
input
128
256
512
128
256
512
256
512
1024
output
256
512
1024
256
512
1024
128
256
512
A.2.3 TRAINING DETAILS
Actually, the layout planner module and the character synthesizer can be trained jointly. The size information that the layout planner predicts for each character is used as input to the single-character synthesizer, which is expected to generate characters with specific sizes. However, we find that not normalizing the character sizes will affect the model to learn structural information about the characters, leading to unstable generation results. Our approach is to decouple the training process of the layout planner module and the character synthesizer. When training the character synthesizer, we still first normalize all single characters to fixed height. In this way, the synthesizer only needs to learn to generate characters at standard size. We take full advantage of the nature that online handwriting data has no background noise, allowing us to directly scale the generated standard-size characters and fill them into their corresponding bounding boxes.
We implement our model in Pytorch and run experiments on NVIDIA TITAN RTX 24G GPUs. Both training and testing completed single GPU. For training the layout planner, the optimizer
15

---

Published as a conference paper at ICLR 2025
is Adam with an initial learning rate of 0.01 and the batch size is 32. For training the diffusion character synthesizer, the initial learning rate is 0.001, the gradient clipping 1.0, learning rate decay for each batch is 0.9998. We train the whole model with 400K iterations with the batch size of 64, which takes about days.
A.2.4 IMPLEMENTATION OF EVALUATION METRICS
For individual character evaluation, due to the limited writing styles represented by individual characters, following previous work(Dai et al. 2023 Tang Lian 2021 we combine fifteen characters written by the same author together input. We train a style classifier with the task of distinguishing 60 writers in the test set, achieving an accuracy of 99.5%. The Style Score refers to the accuracy of classifying generated samples that imitate different styles using this classifier. The higher the style score, the better the model's ability to imitate calligra- phy style. Similarly, achieving an accuracy of 98%. Content Score refers to the accuracy of using this classifier to classify the generated samples. The higher the content score, the better the model's s ability generate accu- rate character structures. Both the style classifier and the content classifier adopt a 1D Convolutional Network, which is the same architecture the style encoder introduced before.
Style Score and Content Score:
utilize all individual character data to train the Chinese character classifier,
Accurate Rate and Correctness Rate: Unlike single characters, when recognizing text lines, the number of characters within them is unknown. As a result, there may be discrepancies between the total number of characters and the number of correctly recognized characters in the content parsed by the recognizer. Therefore, the absence of alignment, it is not appropriate simply use the ratio of correctly recognized characters to the total number of characters as measure of content score. Instead, evaluation metries based on edit distance are commonly used:
CR N₂ De Se
100%
(14)
N,
AR N, De - Se Ie x 100%
(15)
N₂
where Nt represents the total number of characters in the real handwritten text line, while Se, De and Le respectively denote substitution errors, deletion errors, and insertion errors.
A.2. MORE VISUALIZATION RESULTS
For character generation, have added visual comparison between our method and the state-of- the-art style transfer method SDT Dai et al. (2023). As shown in Figure I0 our method achieves Dai
comparable performance and effectively mimiss the target calligraphy style.
Writerl
Writer2
Writer3
Font
मे
3
*
t
#
*
3
10
SDT
t
,
#
1
3
マ 12
22
Ours
$ カ
of
3
Target
Figure 10: The visual comparisons with the state-of-the-an method SDT.
In Figure II For full text line generation, select three authors with distinctly different writing styles as imitation targets. We visualized the real samples alongside the generated counterparts. The left columns display the real samples and the right columns display the generated data.
16

---

Published as a conference paper at ICLR 2025
18 Kirt P
4 14,
#bA 3
3094 M
Figure 11: The visualization results. In each block, the left columns display the real samples which are the imitated targets of generated data on the right.
A.3 THE POSSIBILITY OF EXTENDING TO AN END-TO-END APPROACH
We demonstrate the results of directly using our proposed ID U-Net generation module by con- catenating individual character embeddings into full-line embedding for end-to-end generation. As shown in Figure 12 we specifically selected a handwriting style with connected strokes between characters, and the end-to-end model is capable of handling this. This also illustrates the potential of extending our proposed method to an end-to-end generation framework, demonstrating the generalization of our module.
Figure 13 shows the failure case, specifically, as the number of characters increases, structural errors tend to occur. In contrast, the decoupled layout and glyph method offers more stable training and inference. The next challenge will be to enhance the stability and ensure more reliable generative process for end-to-end generation, which will be the focus of our future work.
Targel:
Target:
Generated:
Feiled:
#
A
Figure 12: The end-to-end generation paradigm can produce connective strokes between charac- ters.
Figure 13: The instability of directly using the end-to-end method with longer character sequences, resulting in failure cases.
17

---

Published as a conference paper at ICLR 2025
A.4 EFFECTS OF LAYOUT GENERATOR
We also tested using a portion of another text line written by the same author as a prefix to condition the layout generator for generation, with the results shown in Figure 14 It be observed that the layouts generated by our proposed method still closely approximate the ground truth. This is because different samples from the same author typically share a consistent writing style. The key principle of our method is to ensure that the generated layouts align with the style of the input prefix, rather than requiring the prefix be from the ground truth text lines. In contrast to previous method, our contributions and advantages lie in:
Compared to previous model-free method, our approach is capable of generating layouts that are closer to the real target. The reason is that model-free methods deeply rely on manually designed rules and features, which, especially sizes and shapes of different types of Chinese characters, well the significant differences in punctuation marks. This makes difficult for earlier method to simultancously capture the layout style. In contrast, our method, based on the next-token prediction paradigm, allows the model to naturally generate a layout where not only the positions of characters are reasonable, but the overall style of the layout consistent with the given prefix, in in-context manner.
case of Chinese text lines, are challenged by the varying
2. addition, in terms of generalization, the layout generator can consider both the horizontal and vertical relative positional relationships between characters, therefore is capable of handling simple 2D mathematical expressions, such as e" and without any modifications, while model-free methods are unable achieve.
Gounden
Condioral
20
Coussen
Coundan
Condiiccl
Figure 14: Layout generation visualization, where the prefix is taken from other text lines by the same author. The previous method is denoted as 'Gaussian'.
We have supplemented the quantitative experiments on layout generation presented in Table 5 as follows, where In-Context-gt refers to the case where we use the first 10 bounding boxes from the ground truth text line the prefix, while In-Context-other refers the case where we use 10 bounding boxes from another randomly picked text line written by the same author as the prefix. As can be seen, the performance is almost identical. The result suggests that long the layout style across different lines from the same author remains consistent, our method can perform effectively.
V
V2
V
V,
V
V6
V7
V
In-Context-gt In-Context-other
0.046
0.122
0.062
0.058
0.129
0.129
0.364
0.419
0.047
0.124
0.057
0.058
0.130
0.132
0.363
0.422
Table 5; The performance quantitative evaluation results of the layout generator using other lines from the same author as prefixes.
18


Reviews:
[{"review_id": "10rD81Fq00", "reviewer": "Reviewer_iC1s", "summary": "The paper introduces a novel approach for generating online handwritten Chinese text with specific styles. The authors naturally divide a text line into two components: layout and glyphs, and design a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. The layout generator autoregressively generates the positions for each glyph based on text content and provided style references, while the font synthesizer generates each font at its position while imitating the calligraphy style extracted from the given style references. Experiments on the CASIAOLHWDB demonstrate the method's capability to generate structurally correct and indistinguishable imitation samples.", "questions": "Please see Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The study proposes a hierarchical method to address the under-explored task of online handwritten Chinese text line generation.\n2. By decoupling layout generation from glyph generation, the method offers more flexibility in handling the generation of text lines, which is particularly useful when dealing with complex Chinese characters.\n3.  The experiments conducted on the CASIA-OLHWDB database indicate high performance in imitation sample generation, demonstrating the effectiveness of the method.", "weaknesses": "1. While decoupling layout and glyph generation increases flexibility, it may also add to the model's complexity, potentially affecting training and inference efficiency.\n2. Are there any application scenarios for this task? The author could analyze its practicality.\n3. The paper mentions difficulties in imitating styles with extensive cursive connections between characters due to the independent generation of each character, indicating potential limitations in handling certain calligraphic styles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MqR0803VDs", "reviewer": "Reviewer_kxE9", "summary": "This paper focuses on the generation of online Chinese handwriting text lines. It proposes a hierarchical approach that decouples layout generation from glyph generation. The text line layout generator arranges character positions based on text content and writing style references, while the font synthesizer generates characters with specific styles. The contributions include a novel layout generator, a 1D U-Net network for font generation, and a multi-scale style encoder. Experiments demonstrate the effectiveness of the method in generating structurally correct and stylistically similar samples.", "questions": "(1) Could more details be provided on how the layout-glyph separation specifically enhances performance in comparison to prior models?\n\n(2) Would additional experiments on style consistency across diverse text lines clarify the benefits of this approach?\n\n(3) Could this method can be adapted to non-Chinese scripts or connected handwriting styles?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The hierarchical decomposition into layout and glyph generation is an innovative approach, particularly suited for complex scripts like Chinese. This framework successfully addresses challenges specific to the language, such as the diversity of character structures.\n\n(2) The model is thoroughly tested on both character and line generation, with metrics tailored to layout and stylistic fidelity. The model's success across multiple metrics shows a well-rounded, effective design.\n\n(3) Despite the technical depth, the paper provides a good level of explanation for each module, with helpful visualizations that demonstrate layout and glyph generation separately.\n\n(4) The method has potential applications in handwriting synthesis, digital personalization, and document augmentation, contributing a valuable approach for future research in multilingual handwriting generation.", "weaknesses": "(1) Missing qualitative comparisons with prior methods, limiting insights into this model’s advantages in style fidelity and layout accuracy.\n\n(2) The contributions over previous approaches could be articulated more clearly, especially regarding the effectiveness of the layout-glyph separation.\n\n(3) The organization could be refined for readability, as the methods section contains complex explanations that could benefit from clearer structuring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mrMGfdkonS", "reviewer": "Reviewer_BU7e", "summary": "This paper focuses on the generation of online Chinese handwritten text lines. The core of this method lies in decomposing text line generation into layout generation and character generation, and fill characters into the generated layouts to form complete text lines. Experiments evaluate the proposed method.", "questions": "My main concerns are the novelty of the proposed multi-scale style encoder and style contrastive learning loss, and the effectiveness of the proposed layout generator. For details, please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) This paper proposes a hierarchical online Chinese handwritten text line generation method. The proposed method utilizes a layout generator and a font synthesizer to produce the layouts and characters independently, then arranges the characters within the layouts to create complete text lines.\n\n2) The proposed method achieves the best performance in purely data-driven font generation task.", "weaknesses": "1) The multi-scale style encoder is not a new design in handwriting generation area, as a similar idea has been proposed in [a]. Besides, the proposed style contrastive learning loss is somewhat similar to the style learning loss in [b].\n\n2) The method description is not clear: (1) In lines 233-237, it is mentioned that style reference samples are used as context prefixes, but how they guide the subsequent layout generation is unclear. (2) The paper does not specify the modality of the style references used, online data, or offline images. (3) The paper does not specify the number of style reference samples used, one-shot or few-shot.\n\n3) Section 4.3.2 lacks quantitative experiments in terms of calligraphy styles, raising doubts about whether the proposed Multi-Scale Style Encoder can accurately extract calligraphy styles from entire text lines.\n\n4) In the 'Conditional' row of Figure 7, the generated layouts (red boxes) show significant absences at the beginning of the text line, which raises concerns about the effectiveness of the layout generator.\n\n5) It is recommended to compare the proposed method with style transfer-based approaches, as it can be relatively straightforward to extend this method to a style transfer setting by replacing character embedding with a CNN-based content encoder.\n\n6) The layout generator requires real layouts of style references, which is not directly available in the application, does this limitation affect its applicability? If some simple layout extraction methods are used to extract the pseudo-layouts of style references, what impact would this have on generation performance?\n\n7) The paper provides very few generated visual results and lacks visual comparisons with the baseline.\n\n[a] Wang H, Wang Y, Wei H. Affganwriting: a handwriting image generation method based on multi-feature fusion, ICDAR, 2023.\n\n[b] Dai G, Zhang Y, Wang Q, et al. Disentangling writer and character styles for handwriting generation, CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKdLAyLYgD", "reviewer": "Reviewer_Up7T", "summary": "The paper addresses the task of generating online handwritten Chinese text lines condition on the content and style. It identifies that text lines can be divided into two components: layout and characters. The authors propose a hierarchical approach that includes a text line layout generator and a stylized font synthesizer. The layout generator uses in-context-like learning to determine the positions of each character, while the font synthesizer generates characters that imitate the calligraphic style of the provided references. The method is evaluated using the CASIA-OLHWDB dataset, demonstrating its effectiveness in producing structurally correct and indistinguishable imitation samples.", "questions": "1. If the bounding box generated by the layout model and the bounding box generated by the character model have different shapes, how should this be handled?\n\n2. Since the method can be described as a unified probability distribution according to Equation 1, why not jointly train the two models end-to-end instead of training them separately?\n\n3. The paper does not discuss whether the method can be applied to handwriting generation of other languages.\n\n4. in Line 285, what does L represent? Although the authors write this is the length of the feature sequence, it is not clear what does this sequence represent? \n\n5. In Line 230, is the ground truth the same as the reference input in Figure 3？", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.While some work on English handwritten text line generation exists, as far as I know, no such work has been published for online Chinese text lines. Compared to English characters, Chinese characters have more complex structures and a larger number of categories, making English generation methods unsuitable for direct application to Chinese. This work proposes a method to address this task, representing a noticable contribution.\n\n2.The method decouples text line generation into two steps—layout generation and character generation—under a unified probabilistic framework, providing a good theoretical foundation and considerable novelty.\n\n3.The experimental section includes comprehensive comparative and visualization experiments for both layout generation and character generation, yielding convincing results.\n\n4.The paper is well-organized and clearly written.", "weaknesses": "1. The assumption that character generation is independent given their positions seems too strong. Does text line style only manifest in the relative positions and sizes of individual characters? I hope the authors can give discussion on the reasonableness of this assumption and explain whether it might limit the method's ability in style learning. \n\n2. It is better to add sub-figure index for figure 8 and 9. It seems each of figure 8 and 9 has three sub-figures, but now their boundaries are not clear. In Figure 7, it is also suggested to identify which one is the proposed method in the paper. Of course, this is not a big issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)