Decoupling Layout from Glyph in Online Chinese Handwriting Generation

## Abstract

Abstract Text plays a crucial role in the transmission of human civilization, and teaching machines to generate online handwritten text in various styles presents an interesting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving complete text line generation largely unexplored . In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. Meanwhile, the font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA-OLHWDB demonstrate that our method is capable of generating structurally correct and indistinguishable imitation samples.

## 1 Introduction

Deep generative models, such as GANs (Goodfellow et al., ), VAEs (Kingma & Welling, ), and diffusion models (Sohl-Dickstein et al., ; Ho et al., ), have demonstrated a formidable ability to generate a wide array of data types. Handwritten data, which encompasses language characters, mathematical symbols, sketches, and diagrams, represents a highly personalized form of data with a wide range of application scenarios. Creating handwriting can provide personalized writing service or be used for data augmentation for document analysis models (Lai et al., ; Kang et al., ; Xie et al., ). In recent years, the emergence of extensive datasets has prompted a notable surge in the application of generative modeling techniques to handwritten data (Xu et al., ; Aksan et al., ).

Among handwritten data, the generation of Chinese handwritten texts has garnered increasing attention due to their widespread use and the greater challenges they present compared to Latin scripts (Gao et al., ; Liu et al., ). A crucial aspect of this task is ensuring the integrity and accuracy of the structural composition of the characters. The vast diversity and complex geometric structures of these characters render this task particularly challenging. Another important consideration is how to imitate specific personal writing styles while maintaining structural correctness (Yang et al., ; Gan & Wang, ; Xie et al., ; Kong et al., ). Compared to individual characters, text lines serve as more effective communication tools, capable of conveying more complex information and emotions. However, the majority of prior research has primarily focused on generating individual Chinese fonts, largely overlooking the significantly more challenging task of generating complete text lines.

Figure: Figure 1: Handwritten text lines with vastly different styles generated by our methods. It is worth mentioning that the generated online data contains dynamic trajectory information, rather than just static images, enabling more interactive applications. Different colors represent different strokes, showcasing the dynamic process of writing.
Refer to caption: extracted/5901881/show.jpg

In this work, we aim to generate stylized and coherent online Chinese handwritten text lines based on specified text contents and given style reference samples. A handwritten line of text can be viewed as composed of individual characters and their arrangement. Based on this observation, we propose a hierarchical method that disentangles the layout generation from the writing of individual characters during the generation of full lines. We design a novel text line layout generator to arrange positions for each character based on their categories and given handwriting style references. For font generation, we construct a 1D convolutional U-Net (Ronneberger et al., ) network and design a multi-scale feature fusion module to fully mimic the calligraphy characteristics of the given reference sample. We utilize the publicly available CASIA-OLHWDB dataset (Liu et al., ) for training and evaluating our method, showcasing its effectiveness through a combination of quantitative and qualitative experiments. Figure 1 shows several visualizations of our generated samples.

Our contributions can be summarized as:
1) We propose a hierarchical approach to solve the unexplored online handwritten Chinese text line generation task.
2) We introduce a simple but effective layout generator that can generate character positions based on the text contents and writing style through in-context-like learning. 3) We construct a 1D U-Net network for font generation and design a multi-scale contrastive-learning-based style encoder to improve the ability of calligraphy style imitation.

## 2 Related Work

### 2.1 Online Handwritten Data Generation

The most commonly adopted method (Graves, ) for generating online handwritten data involves combining neural network models for sequence data processing with a mixture of Gaussian distributions to model the movement information of the pen. Following this work, SketchRNN (Ha & Eck, ) adopts RNN as the encoder and decoder of VAE, conducting the task of unconditional sequence data generation. COSE (Aksan et al., ) treats drawings as a collection of strokes and projects variable-length strokes into a latent space of fixed dimension and uses MLPs to generate one single stroke based on its latent encoding autoregressively. Subsequently, there has been a growing emphasis in academic research on conditional generation, such as producing handwriting with specific contents or diverse writing styles (Aksan et al., ; Kotani et al., ; Tolosana et al., ). They extract calligraphic styles from reference samples and combine them with specific textual content to achieve controllable style English handwriting synthesis.

In contrast to alphabetic languages, Chinese encompasses a significantly larger character set and Chinese characters exhibit more intricate shapes and topological structures (Xu et al., ; Lin et al., ; Lian et al., ; Radford et al., ; Zhu et al., ). For this reason, most methods designed for handwritten English generation do not work well when applied directly to Chinese.

As for the field of online Chinese character generation, both LSTM and GRU models are employed simultaneously in (Zhang et al., ) to successfully generate readable Chinese characters of specific classes at the first time.
FontRNN (Tang et al., ) focuses on stylized font generation, which utilizes a transfer-learning strategy to generate stylized Chinese characters. However, each trained model can only synthesize the same style as the train set. By adding a style encoder branch to the generator, DeepImitator (Zhao et al., ) can generate characters of
any style given a few reference samples. They utilize a CNN as the calligraphy style encoder to extract style codes from offline images and a GRU as the generator to synthesize the online trajectories. Similarly, WriteLikeU (Tang & Lian, ), DiffWriter (Ren et al., ), and SDT (Dai et al., ) are also capable of handling this functionality.

### 2.2 Diffusion Model for Sequence Data

Diffusion model, first proposed in (Sohl-Dickstein et al., ) exhibits a remarkably robust capability in learning the data distribution and generating impressively high quality and diverse images in recent years (Luo, ; Ho et al., ; Dhariwal & Nichol, ; Song et al., ; Rombach et al., ). In terms of sequence data generation, NaturalSpeech2 (Shen et al., ) employs the latent diffusion framework to synthesize speech content with specific contents by imitating various tones and speaking styles. CHIRODIFF (Das et al., ) first utilizes diffusion for unconditional chirographic data generation and later DiffWriter (Ren et al., ) adopts diffusion models for online Chinese characters generation. However, previous methods only explored how to apply the diffusion process to online handwritten data, lacking fine control over writing details.

## 3 Methods

### 3.1 Preliminary

Online Handwritten Data.
In general, online handwriting is a kind of time series data, which is composed of a series of trajectory points. Each point ($[h,v,s]\in\mathbb{R}^{3}$) contains the horizontal movement, vertical movement and the state information of the pen. We use $s=1$ to represent the pen touching state and $s=-1$ to represent the pen lifting state, making it convenient to distinguish the two states by simply adopting dichotomy with threshold 0 after the addition of zero-mean standard Gaussian noise to the data.

Data Notation. A handwritten font that consists of $n$ points can be represented as $\mathbf{x}\in\mathbb{R}^{n\times 3}$. Its corresponding category is represented as a one-hot vector $\mathbf{c}\in\mathbb{R}^{K}$, where $K$ is the number of character categories. A handwritten text line can be represented as $\mathbf{X}=[\mathbf{x}^{1},\cdots,\mathbf{x}^{m}]\in\mathbb{R}^{N\times 3}$, where $\mathbf{x}^{i}\in\mathbb{R}^{n^{i}\times 3}$ denotes the $i$-th character, $m$ denotes the number of characters, and $N$ denotes the number of points with $N=\sum_{i=1}^{m}n^{i}$. Its text content can be represented as $\mathbf{C}=[\mathbf{c}^{1},...,\mathbf{c}^{m}]\in\mathbb{R}^{m\times K}$.

Problem Statement.
Given the text line content $\mathbf{C}$ and style reference samples $\mathbf{X}_{ref}$ (preferably a coherent handwritten text line) from a given author, our objective is to generate handwritten text lines with the specified content while imitating the author’s writing habits. The conditional probability can be written as $p(\mathbf{X}|\mathbf{C},\mathbf{X}_{ref})$. Unlike English, Chinese has a large number of character categories, each with its specific stroke order and geometric structure. The combinations to form a text line are endless. Therefore, the most critical challenge in generating handwritten Chinese text lines lies in two aspects:
1) Ensuring the correctness of the structure of each character while maintaining consistency with the chosen writing style. 2) Arranging the relative positions of different characters, especially between Chinese characters and punctuation marks, to achieve a natural and fluent layout for the entire handwritten text line.

Figure: Figure 2: The illustration of character bounding box, consists of height, width, vertical center position and horizontal offset relative to the previous character.
Refer to caption: extracted/5901881/box.png

### 3.2 Analysis and Overview

Directly modeling $p(\mathbf{X}|\mathbf{C},\mathbf{X}_{ref})$ is overwhelmingly challenging since the positions of glyphs are unknown. In this work, we treat the text line layout as latent variables in the text line generation and instead model the joint distribution of the text line and its layout: $p(\mathbf{X},\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})$.

In detail, we define the layout of a handwritten text line as the positions and sizes of all characters in the line. We calculate the [height, width, vertical center position and horizontal offset relative to the previous character] for each character as its bounding box, shown in Figure 2. In this way, the layout of $\mathbf{X}$ can be represented as: $\text{Layout}_{X}=[\text{Box}^{1},\cdots,\text{Box}^{m}]\in\mathbb{R}^{m\times
4}$.

During the writing process, human beings typically take neighboring glyph positions into account for the current character position to ensure the continuity of the written text lines. However, when the positional information of each character is provided, the writing process of each character is relatively independent. Similarly, we assume that when the bounding box of a character is given, the generation of the character is independent of other characters. By this assumption, we can factorize $p(\mathbf{X},\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})$ as:

$$ $\displaystyle p(\mathbf{X},\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})$ $\displaystyle=p(\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})p(\mathbf{X}| \mathbf{C},\mathbf{X}_{ref},\text{Layout}_{X})$ $\displaystyle=p(\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})\prod_{i=1}^{m}p (\mathbf{x}^{i}|\mathbf{c}^{i},\mathbf{X}_{ref},\text{Box}^{i}).$ (1) $$

Equation 1 inspires us to decouple the complete generation process into two components: layout generation $p(\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})$ and individual font synthesis $p(\mathbf{x}^{i}|\mathbf{c}^{i},\mathbf{X}_{ref},\text{Box}^{i})$. Correspondingly, our proposed model consists of two parts: a text line layout generator and a stylized font synthesizer, shown in Figure 3.

Figure: Figure 3: Overview of the proposed method, which consists of a layout generator and a font synthesizer. Given the text content and style references, the two modules operate simultaneously: the layout generator will arrange the bounding box of each character based on the overall style of the reference, while the font synthesizer will imitate the calligraphic style of the references to produce the corresponding handwritten fonts.
Refer to caption: extracted/5901881/method.png

### 3.3 In-Context Layout Generator

Modeling The Human Writing Process.
Considering the consistent and coherent style of the entire text layout when people write a complete text line, it is natural and reasonable to use a certain length of the prefix of the text line as context to predict the subsequent layout. Based on this observation, we decompose the generation of layout in an autoregressive manner:

$$ $\displaystyle p(\text{Layout}_{X}|\mathbf{C},\mathbf{X}_{ref})=p([\text{Box}^{ 1},\cdots,\text{Box}^{m}]|\mathbf{C},\mathbf{X}_{ref})$ $\displaystyle=\prod_{i=1}^{m}p(\text{Box}^{i}|[\text{Box}^{1},\cdots,\text{Box }^{i-1}],\mathbf{c}^{i},\mathbf{X}_{ref}),$ (2) $$

which is akin to the human writing process. Denote the layout generator model as $\mathcal{G}$, which is an LSTM network in our work. The generation of the $i$-th character box can be represented as:

$$ $[\text{Box}^{i},\mathbf{h}^{i}]=\mathcal{G}(\text{Box}^{i-1},\mathbf{c}^{i}, \mathbf{h}^{i-1}),$ (3) $$

where $\mathbf{h}$ represents the hidden state of LSTM. It is worth noting that different glyphs, punctuation marks, and numbers generally have markedly different shapes. Therefore, it is crucial to consider the character categories during the generation.

Training Objective. During the training phase, we employ the teacher-forcing technique and use the $\ell_{1}$ distance between ground truth and generated layout as the loss function. We empirically illustrate that this training method encourages the model to generate layouts that are consistent throughout the whole text line.

In-Context Generation.
In order to mimic the layout style of the given reference sample, we use the style reference samples as a prompt context, inputting their true bounding boxes as a prefix instead of the generator $\mathcal{G}$’s outputs. The outputs corresponding to the prefix time steps are inconsequential, but the hidden state of the LSTM network implicitly contains information about the layout style. It will plan the subsequent layout according to the writing style of the prefix context.

We experimentally show that this approach effectively possesses the capability for in-context-like learning in this task, and therefore can imitate general writing styles unseen in the training set. Additionally, if no coherent text line is provided as the style reference sample, the model will perform unconditional layout planning with a prefix length of 0.

### 3.4 Stylized Diffusion Character Synthesizer

We adopt the conditional diffusion model as the generator for individual characters. However, previous diffusion-based methods for online handwritten data have neglected the multi-scale features of calligraphy style. To address this issue, we design a 1D U-Net network as the denoiser, and combine it with a character embedding dictionary and a multi-scale calligraphy style encoder to generate characters with specific categories and calligraphy styles.

#### 3.4.1 Model Architecture

Character Embeddings. Inspired by (Zhang et al., ), a dictionary is constructed to encode the structural information of characters: $\mathbf{E}\in\mathbb{R}^{K\times N_{z}}$, where $K$ represents the number of character categories and $N_{z}$ represents the code dimension. Each row in matrix E encodes the structure of one category. To generate a character of the $k$-th category, we first query the embedding dictionary and get the $k$-th row of matrix $\mathbf{E}[k]\in\mathbb{R}^{N_{z}}$ as the content code. Supposing the shape of noisy data $\mathbf{x}_{t}$ is $(n,3)$, we duplicate the content code as well as sinusoidal position encoding of time $\text{emb}_{t}\in\mathbb{R}^{N_{t}}$ for $n$ times and concatenate them with the noisy data $\mathbf{x}_{t}$ to obtain the input $\mathbf{x}_{in}=[\mathbf{x}_{t},\text{emb}_{t},\mathbf{E}[k]]\in\mathbb{R}^{n
\times(N_{z}+N_{t}+3)}$ of the denoiser.

Multi-Scale Style Encoder.
Compared to the global layout style, the calligraphy styles of individual characters are often reflected in various details, such as overall neatness, stroke length, and curvature. Considering that these characteristics are all at different scales, we construct a 1D convolutional style encoder that employs continuous downsampling layers to extract calligraphy features at different levels and a 1D U-Net denoiser to leverage these features.

U-Net Denoiser. The U-Net network shares the same number of down-sampling layers and strides as the style encoder. We set the number of down-sampling blocks as 3 in this work. This alignment ensures that during the decoding process, each layer’s features in the U-Net can correspond to features of the same scale in the style encoder. These features are fused using the QKV cross-attention mechanism (Rombach et al., ), thereby incorporating multi-scale style information into the denoiser. The architectural diagram and further details can be found in Appendix A.2.2.

#### 3.4.2 Training Objective

Denote the font synthesizer as $\mathcal{E}$,
the loss function consists of two parts: a multi-scale style contrastive learning loss $\ell_{c}$ as well as a diffusion process reconstruction loss $\ell_{r}$:

$$ $\ell_{\mathcal{E}}=\ell_{c}+\ell_{r}.$ (4) $$

We detail the two learning objectives as follows.

Multi-Scale Style Contrastive Learning Loss. To incentivize the style encoder to extract discriminative calligraphic style features from different authors across diverse scales, we perform contrastive learning on distinct feature layers respectively. More concretely, suppose a mini-batch $\{\mathbf{x}_{w_{1}},\cdots,\mathbf{x}_{w_{b}}\}$ contains $b$ characters written by $b$ different authors $W=\{w_{1},\cdots,w_{b}\}$ and let $A(w_{i})=W\backslash\{w_{i}\}$. Recalling section 3.4.1, the style encoder SENC contains three downsampling blocks corresponding to three different scales of features for one character: SENC$(\mathbf{x}_{w_{i}})=\{{\bm{f}}_{w_{i}}^{1},{\bm{f}}_{w_{i}}^{2},{\bm{f}}_{w_{
i}}^{3}\}$.

For each scale features, take ${\bm{f}}_{w_{i}}^{3}\in\mathbb{R}^{L\times d}$ as an sample, where
$L$ represents the length of the feature sequence and $d$ indicates the dimension of each feature vector. We randomly select two non-overlapping segments ${\bm{e}}_{w_{i}}^{3},{\bm{e}}_{w_{i}}^{3+}$ from ${\bm{f}}_{w_{i}}^{3}$ (${\bm{e}}\in\mathbb{R}^{l\times d}$ from the total length $L$), serving as a positive pair from author $w_{i}$. The contrastive loss for $f^{3}$ is formulated as:

$$ $\ell_{c}^{3}=-\frac{1}{n}\sum_{k=1}^{n}\log\frac{\exp{(s({\bm{e}}_{w_{k}}^{3}, {\bm{e}}_{w_{k}}^{3+})/\tau)}}{\sum_{w_{i}\in A(w_{k})}\exp{(s({\bm{e}}_{w_{k} }^{3},{\bm{e}}_{w_{i}}^{3+})/\tau)}},$ (5) $$

where $s({\bm{e}}_{w_{k}}^{3},{\bm{e}}_{w_{k}}^{3+})=g^{3}({\bm{e}}_{w_{k}}^{3})^{T}g
^{3}({\bm{e}}_{w_{k}}^{3+})$, $\tau$ is a temperature parameter, $g^{3}$ is a linear projection. Similarly, we can obtain $\ell_{c}^{1}$ and $\ell_{c}^{2}$. The overall multi-scale contrastive loss is formulated as:

$$ $\ell_{c}=\lambda_{1}\ell_{c}^{1}+\lambda_{2}\ell_{c}^{2}+\lambda_{3}\ell_{c}^{ 3},$ (6) $$

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are empirical weights. We set $\lambda_{1}=0.01,\lambda_{2}=\lambda_{3}=0.1$ in this work.

Diffusion Reconstruction Loss and Generation Process.
Given a target character $\mathbf{x}$ and style reference samples $\mathbf{X}_{ref}$ which is a piece of handwritten text line written by the same writer. Denoting the style encoder as SENC, and the character embedding dictionary as E, the diffusion reconstruction loss is calculated in Algorithm 1. $T,\{\alpha_{t}\},\overline{\alpha}_{t}$ are hyperparameters of diffusion models, which are introduced in Appendix A.1.
Correspondingly, given the character category $k$ and style reference samples $\mathbf{X}_{ref}$, we randomly sample standard Gaussian noise as $\mathbf{x}_{T}\in\mathbb{R}^{(n\times 3)}$, where $n$ is the max length, the generation procedure is summarized in Algorithm 2.

Figure: Algorithm 1 Diffusion Reconstruction Loss $\ell_{r}$

## 4 Experiments

In the experimental section, we assess our approach in three areas: capability of calligraphy style imitation, capability of layout style imitation, and content readability, utilizing both qualitative and quantitative experiments as followings:

### 4.1 Dataset

We use the CASIA Online Chinese Handwriting Databases (Liu et al., ) to train and test our model. For single character generation, following previous work, the CASIA-OLHWDB (1.0-1.2) is adopted as the training set, which contains about 3.7 million online Chinese handwritten characters produced by 1,020 writers. The ICDAR-2013 competition database (Yin et al., ) is adopted as the test set, which contains 60 writers, with each contributing the 3,755 most frequently used characters set of GB2312-80. For layout and text line generation, we adopt CASIA-OLHWDB (2.0-2.2) which consists of approximately 52,000 text lines written by 1,200 authors, totaling 1.3 million characters. We take 1,000 writers as the training set and the left 200 writers as the test set.
Detailed descriptions of data preprocessing, model architecture, and training process are provided in Appendix A.2.

### 4.2 Font Generation

Evaluation Metrics. To assess the model’s ability to imitate calligraphy style and generate accurate character structures, we adopt three core indicators DTW (Normalized Dynamic Time Warping Distance), Content Score, and Style Score following previous work (see Appendix A.2 for details). For each writing style in the test set, we randomly generate 100 characters, totaling 6,000 characters, to measure the performance.

Baselines and Results. There are mainly two types of online handwritten Chinese character generation approaches: the first type comprises purely data-driven methods that learn the structure of characters and calligraphy style only from the training set. The second type requires a standard printed font image as an auxiliary input, and train models to transform the standard fonts into characters with a specific writing style. The advantage of the second method is that it is easier to generate correct structures, but it cannot generate characters without corresponding standard glyphs and the computational cost will be heavier. As shown in Table 1, being a method of the first type, our model achieves state-of-the-art performance in all pure data-driven approaches and is comparable to state-of-the-art style transfer-based methods.

Effects of Contrastive Learning.
In this experiment, we randomly select 5 writers in the test set, 150 characters for each writer. We first use the style encoder to extract their features and visualize them using the $t$-SNE method (Van der Maaten & Hinton, ). As shown in Figure 5, the right side was obtained through our fully trained style encoder, while the left side does not adopt contrastive learning loss. In the right figure, features from the same author usually exhibit a clear clustering trend, while in the left figure, the distribution is more random and scattered. It illustrates that through contrastive learning, the calligraphy style encoder can extract discriminative features effectively.

Figure: Figure 4: $t$-SNE visualization of the font style features. Different colors represent different writers.
Refer to caption: extracted/5901881/tsne.png

**Table 1: Evaluation of Character Structure Accuracy and Calligraphy Style Imitation.**
| Methods | Purely data-driven | Style transfer |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| DeepImitator | DiffWriter | Ours | FontRNN | WriteLikeU | SDT |  |
| DTW $(\downarrow)$ | 1.062 | 0.997 | 0.932 | 1.045 | 0.982 | 0.880 |
| Content Score $(\uparrow)$ | 0.834 | 0.887 | 0.891 | 0.875 | 0.938 | 0.970 |
| Style Score $(\uparrow)$ | 0.432 | 0.481 | 0.918 | 0.461 | 0.711 | 0.945 |

Different Weights for Contrastive Learning.
In Table 5, $\lambda_{1},\lambda_{2},\lambda_{3}$ are empirical weights in  equation 6. A weight of 0 indicates that the loss is not used in this feature layer. We find that while the Content Score is basically unaffected, the Style Score experiences a certain improvement, particularly with the implementation of multi-scale contrastive learning.

### 4.3 Full Line Generation

#### 4.3.1 Evaluation Metrics

**Table 2: Binary geometric features for layout generation evaluation, which is calculated between every two adjacent characters.**
| No. | Binary geometric features |
| --- | --- |
| 1 | Mean vertical distance between geometric centers |
| 2 | Mean horizontal distance between geometric centers |
| 3 | Mean distance between upper bounds |
| 4 | Mean distance between lower bounds |
| 5 | Mean distance between left bounds |
| 6 | Mean distance between right bounds |
| 7 | Mean ratio of heights of bounding boxes |
| 8 | Mean ratio of widths of bounding boxes |

Readability Assessment.
To assess the structural correctness performance, AR (Accurate Rate) and CR (Correct Rate) are two widely used metrics for Chinese string recognition (Su et al., ; Wang et al., ). We adopt the methods proposed in (Chen et al., ) as the text line recognizer, which can get 0.962 AR and 0.958 CR on the test set. We report the recognition metrics of the recognizer on our synthesized text lines, where higher values indicate more complete and accurate geometric structure. More details are introduced in Appendix A.2.4.

Baselines.
1) In the field of text line recognition, several methods (Peng et al., ; Yu et al., ) employs synthesized samples to augment the dataset for training. In these synthesized samples, the bounding box (introduced in Section 3.2) of each glyph category is statistically modeled as a Gaussian distribution in the training set. When generating, the bounding box for each character is sampled based on these Gaussian distributions. We denote this layout generation method as ‘Gaussian’. 2) Furthermore, as mentioned before, if no reference samples are provided, our layout generator model can perform unconditional generation, denoted as ‘Unconditional’. We combine the two layout generation methods with our font synthesizer and compare them with our proposed approach quantitatively and qualitatively. The default reference text line length is set as 10.

#### 4.3.2 Quantitative Results

As shown in Table 3, we find that: 1) the generated text lines all exhibit AR and CR values exceeding 0.85, demonstrating the complete and correct structures of each character. 2) The layouts generated by the in-context method are closer to real samples in nearly all binary geometric features, which highlights the effectiveness of our proposed layout generative method.

**Table 3: Evaluation of Handwritten Text Line Layout Imitation and Readability.**
| Baselines | $\nabla_{1}(\downarrow)$ | $\nabla_{2}(\downarrow)$ | $\nabla_{3}(\downarrow)$ | $\nabla_{4}(\downarrow)$ | $\nabla_{5}(\downarrow)$ | $\nabla_{6}(\downarrow)$ | $\nabla_{7}(\downarrow)$ | $\nabla_{8}(\downarrow)$ | AR | CR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gaussian | 0.074 | 0.151 | 0.092 | 0.092 | 0.163 | 0.160 | 0.859 | 1.081 | 0.855 | 0.851 |
| Unconditional | 0.048 | 0.132 | 0.034 | 0.063 | 0.160 | 0.139 | 0.366 | 0.408 | 0.856 | 0.852 |
| In-Context | 0.046 | 0.122 | 0.062 | 0.058 | 0.129 | 0.129 | 0.364 | 0.419 | 0.857 | 0.852 |

We further investigate the impact of varying reference sample lengths on the performance of layout generator as shown in Figure 6. Intuitively, with increasing length, most metrics tend to improve. Compared to the vertical binary geometric features ($\nabla_{1},\nabla_{3},\nabla_{4},\nabla_{7}$), the horizontal features ($\nabla_{2},\nabla_{5},\nabla_{6},\nabla_{8}$) exhibit a larger deviation from real samples, indicating that learning the spacing size is more challenging when imitating the layout style of text lines. Furthermore, with respect to these horizontal features, the disparity between the generated samples and the authentic samples is markedly decreased as the length of the reference sample extends.

Figure: Figure 6: The difference between the mean binary geometric features of generated samples and real samples under different reference sample lengths.
Refer to caption: extracted/5901881/ablation.png

### 4.4 Qualitative Results.

Layout Case Study.
We specifically select samples with unique layout styles (upward slant) in the test set. Figure 7 shows that although the unconditional layout generation method can produce structurally reasonable layouts, it completely neglects the layout style of the handwritten text line. In comparison, our method effectively mimics characteristics such as glyph spacing and the slant trend of handwriting, verifying the necessity of our layout generator.

Figure: Figure 7: Visualization of character bounding boxes generated by different methods, where green represents the ground truth.
Refer to caption: extracted/5901881/layout_case.png

Subjective Visualization Experiment. In Figure 8,
we select three authors with distinctly different writing styles as imitation targets. We visualized the real samples alongside the generated counterparts.
The left columns display the real samples and the right columns display the generated data. It can be observed that our model adeptly imitates the target samples, capturing both the overall layout style, such as spacing and slant, as well as the calligraphic style of individual characters.

We take one sample from the real samples and one from the synthesized samples to form a pair, as shown in Figure 9. We create ten similar pairs and ask 20 participants whether the upper and lower lines are written by the same person. More than ninety percent of the responses are ‘yes’, which demonstrates that it is difficult to distinguish between the real samples and the synthesized ones.

## 5 Conclusions and Limitations

In this paper, we tackle the stylized online handwritten Chinese text line generation task hierarchically, which is almost unexplored. Our model consists of a novel text line layout generator and a stylized diffusion-based character synthesizer. The text line layout generator can arrange glyph
positions based on the text contents and writing habits of the given reference
samples while the stylized diffusion-based character synthesizer can generate characters with specific categories and calligraphy styles. However, due to the fact that each character is generated independently, our approach encounters difficulties in mimicking styles with extensive cursive connections between characters. An end-to-end text line generation method might be required to address this issue. Furthermore, whether synthesized samples can serve as augmentation data to assist in achieving better results for the recognizer is an open question worth exploring. Last but not least, how to generate more complex handwritten data, such as mathematical formulas based on the hierarchical approach is also an intriguing topic. We consider these as our future work.

Figure: Figure 8: The visualization results. In each block, the left columns display the real samples which are the imitated targets of generated data on the right.
Refer to caption: extracted/5901881/full_line.jpg

Figure: Figure 9: Examples of subjective experiment. Each pair consists of one row from the synthesized samples and the other row from its corresponding target real samples. Participants need to determine whether one pair was written by the same person.
Refer to caption: extracted/5901881/full_line.png

## Appendix A Appendix

### A.1 Denoising Diffusion Probabilistic Models

In this paper, we adopt the Denoising Diffusion Probabilistic Model (DDPM), which is a generative model that operates by iteratively applying a denoising process to noise-corrupted data. This process, known as the reverse denoising process aims to gradually refine the noisy input towards generating realistic samples. DDPM learns to model the conditional distribution of clean data given noisy inputs, which is derived from the forward diffusion process. Denote the forward process as $q$ and the reverse process as $p$, the forward process starts from the original data $X_{0}$ and incrementally adds Gaussian noise to the data:

$$ $q(X_{t}|X_{t-1})=\mathcal{N}(X_{t};\sqrt{\alpha_{t}}X_{t-1},(1-\alpha_{t})I)$ (7) $$

where $\{\alpha_{t}\}^{T}_{t=0}$ are noise schedule hyperparameters, $T$ is the total number of timesteps. Due to the Markovian nature of the forward transition kernel $q(X_{t}|X_{t-1})$, we can directly sample $X_{t}\sim q(X_{t}|X_{0})$ without reliance on any other $t$:

$$ $\displaystyle q(X_{t}|X_{0})$ $\displaystyle=\mathcal{N}(X_{t};\sqrt{\overline{\alpha}_{t}}X_{0},(1-\overline {\alpha}_{t})I),\quad\overline{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}$ (8) $\displaystyle\Rightarrow\ X_{t}$ $\displaystyle=\sqrt{\overline{\alpha}_{t}}X_{0}+\sqrt{1-\overline{\alpha}_{t}} \epsilon,\quad\epsilon\sim\mathcal{N}(0,I)$ (9) $$

The set of $\{\alpha_{t}\}^{T}_{t=0}$ and $T$ should guarantee that $\overline{\alpha}_{T}$ is almost equal to 0, ensuring that the data distribution $q(X_{T})$ closely resembles the standard Gaussian distribution.
The corresponding normal posterior can be computed as:

$$ $\begin{split}&q(X_{t-1}|X_{t},X_{0})=\mathcal{N}(X_{t-1};\mu_{q}(X_{t},X_{0}), \sigma_{q}^{2}(t)I)\\ &\mu_{q}(X_{t},X_{0})=\frac{\sqrt{\alpha_{t}}(1-\overline{\alpha}_{t-1})X_{t}+ \sqrt{{\overline{\alpha}_{t-1}}}(1-\alpha_{t})X_{0}}{1-\overline{\alpha}_{t}} \\ &\sigma_{q}^{2}(t)=\frac{(1-\alpha_{t})(1-\overline{\alpha}_{t-1})}{1- \overline{\alpha}_{t}}\end{split}$ (10) $$

In training, the denoiser learns to predict the added noise $\epsilon$ in Equation 9 given noisy data $X_{t}$. The most commonly used objective function is to minimize the deviation between predicted noise and true noise:

$$ $L(\theta)=E_{t\sim U\{1,T\},\epsilon\sim N(0,I)}||\epsilon-\epsilon_{\theta}(X _{t}(X_{0},\epsilon),t)||^{2}$ (11) $$

Theoretically, it is equivariant to optimizing the evidence lower bound of $\log p(X_{0})$:

$$ $L(\theta)\equiv E_{t\sim U\{2,T\}}E_{q(X_{t}|X_{0})}D_{KL}(q(X_{t-1}|X_{t},X_{ 0})||p_{\theta}(X_{t-1}|X_{t}))$ (12) $$

During the reverse process, after substituting Equation 9 into Equation 10 and replacing the true noise $\epsilon$ with the predicted noise $\epsilon_{\theta}(X_{t},t)$, we have:

$$ $\begin{split}&p_{\theta}(X_{t-1}|X_{t},t)=\mathcal{N}(X_{t-1};\mu_{\theta}(X_{ t},t),\sigma_{p}^{2}(t)I)\\ &\mu_{\theta}(X_{t},t)=\frac{1}{\alpha_{t}}X_{t}-\frac{1-\alpha_{t}}{\sqrt{1- \overline{\alpha}_{t}}\sqrt{\alpha_{t}}}\epsilon_{\theta}(X_{t},t)\\ &\sigma_{p}^{2}(t)=\frac{(1-\alpha_{t})(1-\overline{\alpha}_{t-1})}{1- \overline{\alpha}_{t}}\end{split}$ (13) $$

which defines the reverse transition kernel $p_{\theta}(X_{t-1}|X_{t},t)$.

For the hyperparameters of DDPM, we adopt the linear noising schedule proposed by (Ho et al., ): {$T=1000;\quad\alpha_{0}=1-1e^{-4};\quad\alpha_{T}=1-2e^{-2}$}.

### A.2 Implement Details

#### A.2.1 Data Preprocessing

We adopt the same configuration as the previous method: we first use the Ramer–Douglas–Peucker algorithm(Douglas & Peucker, ) with a parameter of $\epsilon=2$ to remove redundant points. Then we maintain the aspect ratio and normalize the height of each sample to 1.

#### A.2.2 Model Parameterization

For the layout generator model, we adopt a 2-layer LSTM with a hidden size of 128 as the layout generator. We also attempted to use a transformer as a replacement and found that the results were nearly the same. Therefore, we chose the simpler and faster LSTM model. For the denoiser model, recall our model consists of a 1D U-Net network as the denoiser, a character embedding dictionary, and a multi-scale calligraphy style encoder.
We set the dimension of character embedding as 150 and the dimension of time embedding for diffusion models as 32.

Figure 10 displays the interaction details between 1D U-Net denoiser and style encoder. Each downsampling block, such as $\{f^{i},en^{i}\}_{i=1,2,3}$ consists of three residual dilated convolutional layers and a downsampling layer. The residual dilated convolutional layers have a kernel size of 3 and dilation values of 1, 3, and 5, respectively. The downsampling layer has a stride of 2 and a kernel size of 4. The upsampling blocks of the U-Net are structurally symmetrical to the downsampling block and maintain the same hyperparameters, with the only difference being the substitution of the convolutional layers in the downsampling blocks with deconvolutional layers. Table 4 illustrates the channel numbers of input and output features for each module. Afterward, we employ a simple linear layer to ensure that the output has the same dimensions as the initial data, thus performing the function of noise prediction.

Figure: Figure 10: The framework of the multi-scale style encoder and the U-Net Denoiser. The number and stride of downsampling layers of U-Net and the style encoder are aligned, which we set as 3 in this work. The multi-scale style information is injected into U-Net through the cross-attention mechanism.
Refer to caption: extracted/5901881/cross_attention.jpg

**Table 4: Channel numbers of feature inputs and outputs at different layers..**
|  | $f^{1}$ | $f^{2}$ | $f^{3}$ | $en^{1}$ | $en^{2}$ | $en^{3}$ | $de^{1}$ | $de^{2}$ | $de^{3}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| input | 128 | 256 | 512 | 128 | 256 | 512 | 256 | 512 | 1024 |
| output | 256 | 512 | 1024 | 256 | 512 | 1024 | 128 | 256 | 512 |

#### A.2.3 Training Details

Actually, the layout planner module and the character synthesizer can be trained jointly. The size information that the layout planner predicts for each character is used as input to the single-character synthesizer, which is expected to generate characters with specific sizes. However, we find that not normalizing the character sizes will affect the model to learn structural information about the characters, leading to unstable generation results. Our approach is to decouple the training process of the layout planner module and the character synthesizer. When training the character synthesizer, we still first normalize all single characters to a fixed height. In this way, the synthesizer only needs to learn to generate characters at a standard size. We take full advantage of the nature that online handwriting data has no background noise, allowing us to directly scale the generated standard-sized characters and fill them into their corresponding bounding boxes.

We implement our model in Pytorch and run experiments on
NVIDIA TITAN RTX 24G GPUs. Both training and testing are completed on a single GPU. For training the layout planner, the optimizer is Adam with an initial learning rate of 0.01 and the batch size is 32. For training the diffusion character synthesizer, the initial learning rate is 0.001, the gradient clipping is 1.0, learning rate decay for each batch is 0.9998. We train the whole model with 400K iterations with the batch size of 64, which takes about 4 days.

#### A.2.4 Implementation of Evaluation Metrics

Style Score and Content Score: For individual character evaluation, due to the limited writing styles represented by individual characters, following previous work(Dai et al., ; Tang & Lian, ), we combine fifteen characters written by the same author together as input. We train a style classifier with the task of distinguishing 60 writers in the test set, achieving an accuracy of 99.5%. The Style Score refers to the accuracy of classifying generated samples that imitate different styles using this classifier. The higher the style score, the better the model’s ability to imitate calligraphy style. Similarly, we utilize all individual character data to train the Chinese character classifier, achieving an accuracy of 98%. Content Score refers to the accuracy of using this classifier to classify the generated samples. The higher the content score, the better the model’s ability to generate accurate character structures. Both the style classifier and the content classifier adopt a 1D Convolutional Network, which is the same architecture as the style encoder introduced before.

Accurate Rate and Correctness Rate: Unlike single characters, when recognizing text lines, the number of characters within them is unknown. As a result, there may be discrepancies between the total number of characters and the number of correctly recognized characters in the content parsed by the recognizer. Therefore, in the absence of alignment, it is not appropriate to simply use the ratio of correctly recognized characters to the total number of characters as a measure of content score. Instead, evaluation metrics based on edit distance are commonly used:

$$ $\displaystyle CR$ $\displaystyle=\frac{N_{t}-D_{e}-S_{e}}{N_{t}}\times 100\%$ (14) $\displaystyle AR$ $\displaystyle=\frac{N_{t}-D_{e}-S_{e}-I_{e}}{N_{t}}\times 100\%$ (15) $$

where $N_{t}$ represents the total number of characters in the real handwritten text line, while $S_{e}$, $D_{e}$, and $I_{e}$ respectively denote substitution errors, deletion errors, and insertion errors.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see Weaknesses
soundness: 4
strengths: 1. The study proposes a hierarchical method to address the under-explored task of online handwritten Chinese text line generation.
2. By decoupling layout generation from glyph generation, the method offers more flexibility in handling the generation of text lines, which is particularly useful when dealing with complex Chinese characters.
3.  The experiments conducted on the CASIA-OLHWDB database indicate high performance in imitation sample generation, demonstrating the effectiveness of the method.
summary: The paper introduces a novel approach for generating online handwritten Chinese text with specific styles. The authors naturally divide a text line into two components: layout and glyphs, and design a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. The layout generator autoregressively generates the positions for each glyph based on text content and provided style references, while the font synthesizer generates each font at its position while imitating the calligraphy style extracted from the given style references. Experiments on the CASIAOLHWDB demonstrate the method's capability to generate structurally correct and indistinguishable imitation samples.
weaknesses: 1. While decoupling layout and glyph generation increases flexibility, it may also add to the model's complexity, potentially affecting training and inference efficiency.
2. Are there any application scenarios for this task? The author could analyze its practicality.
3. The paper mentions difficulties in imitating styles with extensive cursive connections between characters due to the independent generation of each character, indicating potential limitations in handling certain calligraphic styles.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: (1) Could more details be provided on how the layout-glyph separation specifically enhances performance in comparison to prior models?

(2) Would additional experiments on style consistency across diverse text lines clarify the benefits of this approach?

(3) Could this method can be adapted to non-Chinese scripts or connected handwriting styles?
soundness: 3
strengths: (1) The hierarchical decomposition into layout and glyph generation is an innovative approach, particularly suited for complex scripts like Chinese. This framework successfully addresses challenges specific to the language, such as the diversity of character structures.

(2) The model is thoroughly tested on both character and line generation, with metrics tailored to layout and stylistic fidelity. The model's success across multiple metrics shows a well-rounded, effective design.

(3) Despite the technical depth, the paper provides a good level of explanation for each module, with helpful visualizations that demonstrate layout and glyph generation separately.

(4) The method has potential applications in handwriting synthesis, digital personalization, and document augmentation, contributing a valuable approach for future research in multilingual handwriting generation.
summary: This paper focuses on the generation of online Chinese handwriting text lines. It proposes a hierarchical approach that decouples layout generation from glyph generation. The text line layout generator arranges character positions based on text content and writing style references, while the font synthesizer generates characters with specific styles. The contributions include a novel layout generator, a 1D U-Net network for font generation, and a multi-scale style encoder. Experiments demonstrate the effectiveness of the method in generating structurally correct and stylistically similar samples.
weaknesses: (1) Missing qualitative comparisons with prior methods, limiting insights into this model’s advantages in style fidelity and layout accuracy.

(2) The contributions over previous approaches could be articulated more clearly, especially regarding the effectiveness of the layout-glyph separation.

(3) The organization could be refined for readability, as the methods section contains complex explanations that could benefit from clearer structuring.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: My main concerns are the novelty of the proposed multi-scale style encoder and style contrastive learning loss, and the effectiveness of the proposed layout generator. For details, please refer to the weaknesses.
soundness: 3
strengths: 1) This paper proposes a hierarchical online Chinese handwritten text line generation method. The proposed method utilizes a layout generator and a font synthesizer to produce the layouts and characters independently, then arranges the characters within the layouts to create complete text lines.

2) The proposed method achieves the best performance in purely data-driven font generation task.
summary: This paper focuses on the generation of online Chinese handwritten text lines. The core of this method lies in decomposing text line generation into layout generation and character generation, and fill characters into the generated layouts to form complete text lines. Experiments evaluate the proposed method.
weaknesses: 1) The multi-scale style encoder is not a new design in handwriting generation area, as a similar idea has been proposed in [a]. Besides, the proposed style contrastive learning loss is somewhat similar to the style learning loss in [b].

2) The method description is not clear: (1) In lines 233-237, it is mentioned that style reference samples are used as context prefixes, but how they guide the subsequent layout generation is unclear. (2) The paper does not specify the modality of the style references used, online data, or offline images. (3) The paper does not specify the number of style reference samples used, one-shot or few-shot.

3) Section 4.3.2 lacks quantitative experiments in terms of calligraphy styles, raising doubts about whether the proposed Multi-Scale Style Encoder can accurately extract calligraphy styles from entire text lines.

4) In the 'Conditional' row of Figure 7, the generated layouts (red boxes) show significant absences at the beginning of the text line, which raises concerns about the effectiveness of the layout generator.

5) It is recommended to compare the proposed method with style transfer-based approaches, as it can be relatively straightforward to extend this method to a style transfer setting by replacing character embedding with a CNN-based content encoder.

6) The layout generator requires real layouts of style references, which is not directly available in the application, does this limitation affect its applicability? If some simple layout extraction methods are used to extract the pseudo-layouts of style references, what impact would this have on generation performance?

7) The paper provides very few generated visual results and lacks visual comparisons with the baseline.

[a] Wang H, Wang Y, Wei H. Affganwriting: a handwriting image generation method based on multi-feature fusion, ICDAR, 2023.

[b] Dai G, Zhang Y, Wang Q, et al. Disentangling writer and character styles for handwriting generation, CVPR, 2023.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. If the bounding box generated by the layout model and the bounding box generated by the character model have different shapes, how should this be handled?

2. Since the method can be described as a unified probability distribution according to Equation 1, why not jointly train the two models end-to-end instead of training them separately?

3. The paper does not discuss whether the method can be applied to handwriting generation of other languages.

4. in Line 285, what does L represent? Although the authors write this is the length of the feature sequence, it is not clear what does this sequence represent? 

5. In Line 230, is the ground truth the same as the reference input in Figure 3？
soundness: 4
strengths: 1.While some work on English handwritten text line generation exists, as far as I know, no such work has been published for online Chinese text lines. Compared to English characters, Chinese characters have more complex structures and a larger number of categories, making English generation methods unsuitable for direct application to Chinese. This work proposes a method to address this task, representing a noticable contribution.

2.The method decouples text line generation into two steps—layout generation and character generation—under a unified probabilistic framework, providing a good theoretical foundation and considerable novelty.

3.The experimental section includes comprehensive comparative and visualization experiments for both layout generation and character generation, yielding convincing results.

4.The paper is well-organized and clearly written.
summary: The paper addresses the task of generating online handwritten Chinese text lines condition on the content and style. It identifies that text lines can be divided into two components: layout and characters. The authors propose a hierarchical approach that includes a text line layout generator and a stylized font synthesizer. The layout generator uses in-context-like learning to determine the positions of each character, while the font synthesizer generates characters that imitate the calligraphic style of the provided references. The method is evaluated using the CASIA-OLHWDB dataset, demonstrating its effectiveness in producing structurally correct and indistinguishable imitation samples.
weaknesses: 1. The assumption that character generation is independent given their positions seems too strong. Does text line style only manifest in the relative positions and sizes of individual characters? I hope the authors can give discussion on the reasonableness of this assumption and explain whether it might limit the method's ability in style learning. 

2. It is better to add sub-figure index for figure 8 and 9. It seems each of figure 8 and 9 has three sub-figures, but now their boundaries are not clear. In Figure 7, it is also suggested to identify which one is the proposed method in the paper. Of course, this is not a big issue.

Meta-review: This paper studies online handwritten Chinese text generation. After rebuttal, the overall rating of this paper is above the marginal acceptance threshold, but with mixed scores of 5,6,6,8. The Area Chair has read the paper, all reviews, and the authors' rebuttal. The main strengths of the paper include 1) the task of the online handwritten Chinese text is considered to be important and the proposed method is well motivated, 2) the proposed decoupling of layout and glyph is reasonable and innovative, 3) the experimental design is thorough, and the results presented are convincing.

During the rebuttal phase, the authors addressed most of the reviewers' concerns, leading Reviewers iC1s, Up7T, and kxE9 to lean toward accepting the paper. However, Reviewer BU7e remained concerned about the novelty of the proposed multi-scale style encoder with contrastive loss, maintaining a rating of 5. Despite this, considering all review comments and the paper's other contributions, the AC agrees that the paper meets the acceptance threshold. Additionally, during the discussion phase, Reviewer BU7e also agrees that the paper could be accepted despite the concerns regarding its novelty.

The recommendation is acceptance. The authors are advised to carefully revise the manuscript by incorporating the reviewers' suggestions to enhance its quality further.

Decision: Accept (Poster)