A Generative Model for Handwritings Based on
Enhanced Feature Desynchronization
Seiichi Uchida, Toru Sasaki, and Feng Yaokai
Kyushu University, Fukuoka, 819-0395, Japan

Abstract—A new generative model of handwriting patterns
is proposed for interpreting their deformations. The modelis based on feature desynchronization, which is a coupling
process of 𝑥 and 𝑦 coordinate features of different timings. Bychanging the timings to be coupled, the model can generatevarious deformed patterns from a single pattern. The model
is further enhanced by incorporating an adaptive rotation ateach timing for increasing the variety of deformed patterns. An
important fact is that this enhanced desynchronization model
can be interpreted intuitively as a deformation process in actual
handwriting. Experimental results showed that the model cangenerate various handwriting patterns close to actual deformed
patterns.
Keywords-feature desynchronization, handwritings, generative model

I. I NTRODUCTION
The purpose of this paper is to introduce a new generativemodel for interpreting deformations in handwritings. That is,
the proposed model imitates a deformation process of humanhandwriting. In fact, the model gives a simple explanation
why we draw a deformed line even if we want to draw
just a straight line. In addition, the model can be embedded
into elastic matching techniques for realizing deformationtolerant online character recognition in future.
The proposed model is inspired by asynchronous pen
movement on drawing a line. For drawing a (perfectly)
straight line, a pen should move synchronously in 𝑥direction and 𝑦-direction, as shown in Fig. 1 (a). In practice,
however, it is impossible to draw a perfectly straight line;
we generally have a curved (i.e., deformed) line. A possible
interpretation of this unexpected deformation is that the
pen usually moves asynchronously in 𝑥-direction and 𝑦direction, as shown in Fig. 1 (b).
This asynchronous pen movement can be modeled byfeature desynchronization [1]. Again, consider a situation of drawing a perfectly straight line from (0, 0)𝑇
to (10, 10)𝑇 . In a discretized manner, the straight line
can be represented as a sequence of pen-tip locations,(0, 0)𝑇 , (1, 1)𝑇 , (2, 2)𝑇 , . . . , (10, 10)𝑇 . Feature desynchronization is the process of coupling 𝑥 and 𝑦 features fromdifferent timings and thus generates a new point. For example, a new point (2, 3)𝑇 is generated from the original points,(2, 2)𝑇 and (3, 3)𝑇 . One important thing is that we canconsider that this new point is generated by an asynchronous
pen movement where the pen movement in 𝑦-direction is a
bit faster than that in 𝑥-direction. The other important thingis that this new point no longer lies on the original straight

y

(a)

x

(b)

(c)

Figure 1. Feature desynchronization for generating deformed handwritings.
(a) No desynchronization. (b) Simple desynchronization. (c) Enhanced
desynchronization.

line; this means that we can generate various deformationsof handwritings by feature desynchronization.
This simple feature desynchronization [1] has a limitationin its ability to generate variable deformation patterns. In
fact, as discussed later, no deformed pattern is generated
from a pattern comprised of line segments parallel to the 𝑥or 𝑦-axis. This fact indicates that the ability of generating
patterns by the simple feature desynchronization heavily
depends on the writing direction and thus the model is
not reasonable to be interpreted as a deformation model ofhuman handwriting.
This paper proposes enhanced feature desynchronization
for a new generative model with more reasonable ability
to generate deformed patterns. In the proposed model, an
adaptive rotation of the 𝑥 − 𝑦 coordinate system is newly
introduced into the feature desynchronization process. Simply speaking, this adaptive rotation can remove the above
limitation because the 𝑥 − 𝑦 coordinate system is no longer
fixed.
A possible application of the proposed model isdeformation-tolerant online character recognition. Specifically, the model can be embedded into some elastic matching
method, such as the dynamic programming (DP) matching
method [2]–[4]. By nonlinear timing alignment by elastic
matching and adaptive deformation by the enhanced featuredesynchronization, it is able to fit a reference pattern toan input pattern closely. This ability will be experimentallyshown in this paper.
It should be emphasized again that the proposed model
can be considered as a new generation model based on a
deformation process of human handwritings. In this point,
the proposed method is related to well-known handwriting
models, such as Flash and Hogan [5] and the log-normal
model [6]. However, the proposed model is far different fromthose models, which focus dynamics or kinematics of thehandwriting process. In contrast, the proposed model focusesstructural fluctuation, i.e., deformations, in a direct manner.
The idea of feature desynchronization is rarely found inliteratures, but several trials in other recognition tasks [7]–[9]. Artières et al. [10] have coupled online and offlinefeatures for handwriting recognition. Their trial does not aim
to model deformations in handwritings and therefore is very
different from ours.
II. G ENERATIVE M ODEL BASED ON S IMPLE F EATURE
D ESYNCHRONIZATION
A. Simple Feature Desynchronization [1]Consider feature desynchronization of a handwriting pattern 𝑹 = 𝒓 1 , 𝒓 2 , . . . , 𝒓 𝑗 , . . . , 𝒓 𝐽 , where 𝒓 𝑗 is the 𝑥-𝑦 coordinate feature vector of the timing 𝑗, i.e., 𝒓 𝑗 = (𝑥𝑗 , 𝑦𝑗 )𝑇 .
As noted in Section I, feature desynchronization is a processof coupling 𝑥 and 𝑦 features of different timings. As shown˜ = (𝑥𝑙 , 𝑦𝑘 )𝑇 is generated by thein Fig. 2 (a), a new point 𝒓
coupling. This fact indicates that feature desynchronization
over the whole pattern will generate a new pattern.
˜ =A generated pattern with 𝐼 couples is represented as 𝑹
˜𝑖 , . . . , 𝒓
˜𝐼 , where 𝒓
˜𝑖 = (𝑥𝑗(𝑖) , 𝑦𝑘(𝑖) )𝑇 . The functions
˜1 , . . . , 𝒓𝒓
𝑗 = 𝑗(𝑖) and 𝑘 = 𝑘(𝑖) specify feature desynchronization. By
controlling those functions, it is possible to generate various
patterns from a single pattern 𝑹.
For regulating the effect of desynchronization, we assume
the following three constraints.
∙ The first constraint is monotonicity and continuity
conditions for preserving the original temporal orderin the both of 𝑥 and 𝑦 feature sequences and avoiding
large temporal jumps:{
𝑗(𝑖) − 𝑗(𝑖 − 1) ∈ {0, 1, 2},(1)𝑘(𝑖) − 𝑘(𝑖 − 1) ∈ {0, 1, 2}.
∙

The second constraint is a constant lag limitation to
avoid coupling 𝑥𝑗 and 𝑦𝑘 of very different timings:∣𝑘(𝑖) − 𝑗(𝑖)∣ ≤ 𝐿,

(2)

where 𝐿 is a positive constant. A simple extension ofthis constraint is an adaptive lag limitation, where thedifferent values 𝐿(𝑖) are used at different 𝑖 instead of
the constant 𝐿.
∙ The third constraint is the boundary condition that atleast one of 𝑗(1) and 𝑘(1) equals to 1 and, similarly,
at least one of 𝑗(𝐼) and 𝑘(𝐼) equals to 𝐽. (Note that
this boundary condition can be modified to be morestrict or more loose. For example, a strict version is thecondition that 𝑗(1) = 𝑘(1) = 1 and 𝑗(𝐼) = 𝑘(𝐼) = 𝐽.)
B. Limitation of Simple Feature Desynchronization
The above simple feature desynchronization has a serious
limitation that the variety of generated patterns depends
on the stroke direction. Consider an extreme case that the
handwriting pattern is just a vertical line. In this case, allof the 𝑥-coordinate features (i.e., 𝑥1 , . . . , 𝑥𝐽 ) have the same~r =(x ,y ) r =(x ,y )
j k
k k k

~
r
rj

y

~
R

R
rj=(xj ,yj)
x

rk

R
~
R

y
x

(a)
(b)
Figure 2. Simple feature desynchronization [1]. (a) A newly generated
point. (b) Limitation in the variety of generated patterns.

value and any feature coupling cannot make a new point.
This is because a generated point (𝑥𝑗 , 𝑦𝑘 ) is equivalent toan original point (𝑥𝑘 , 𝑦𝑘 ) and thus no deformed pattern canbe generated from the vertical line by feature desynchronization.
This limitation appears even in more general cases. Figure 2 (b) illustrates the dependency between the stroke
direction and the variety of generated patterns. If the stroke
direction becomes closer to 0∘ (or 90∘ ), the range of the𝑥 (or 𝑦) feature values is limited. Since the patterns are
generated by using the original feature values, the varietyof the generated patterns is limited for the patterns withthose directions. Although Fig. 2 (b) uses a straight line
as the original pattern 𝑹 for simplicity, any other complex
pattern also has the same limitation depending on its local
directions.
III. G ENERATIVE M ODEL BASED ON E NHANCEDF EATURE D ESYNCHRONIZATIONIn this section, an enhanced feature desynchronization is
introduced for building a new generative model which is freefrom the above limitation. The main idea of the enhanced
feature desynchronization is an adaptive rotation of the 𝑥𝑦 coordinate system. Figure 3 (a) shows the generation ofa new point (𝑥𝜃𝑗 , 𝑦𝑘𝜃 ) under a rotation 𝜃. Comparison to
Fig. 2 (a) shows that a different point can be generated by
the rotation.
The generated point (𝑥𝜃𝑗 , 𝑦𝑘𝜃 ) is described as{ 𝜃
𝑥𝑗 = (𝑦𝑗 − 𝑦𝑘 ) sin 𝜃 cos 𝜃 + 𝑥𝑗 cos2 𝜃 + 𝑥𝑘 sin2 𝜃,(3)𝑦𝑘𝜃 = (𝑥𝑗 −𝑥𝑘 ) sin 𝜃 cos 𝜃 + 𝑦𝑗 sin2 𝜃 + 𝑦𝑘 cos2 𝜃.
By eliminating 𝜃 from these equations, we have(𝑥𝜃𝑗 − (𝑥𝑗 + 𝑥𝑘 )/2)2 + (𝑦𝑘𝜃 − (𝑦𝑗 + 𝑦𝑘 )/2)2)(= (𝑥𝑗 − 𝑥𝑘 )2 + (𝑦𝑗 − 𝑦𝑘 )2 /4.

(4)˜𝜃 = (𝑥𝜃𝑗 , 𝑦𝑘𝜃 ) by theThis indicates that the new point 𝒓
enhanced feature desynchronization lies on the circle whose
diameter is defined by (𝑥𝑗 , 𝑦𝑗 ) and (𝑥𝑘 , 𝑦𝑘 ). Figure 3 (b)illustrates the circle. Note that this circle does not dependon the center of the rotation.

y’

y

~
Rθ

rk=(xk ,yk)~r θ=(xθ, y θ )
j k

x’
R

rj=(xj ,yj)
θ

θmax
−θmax

x

(a)
(b)
(c)
(d)
(e)
Figure 3. Enhanced feature desynchronization. (a) A point generated by enhanced feature desynchronization. (b) The circle representing all points to be˜ 𝜃 from the same 𝑹. (e) Effect of rotation angle limitation.
generated. (c) and (d) Two examples of 𝑹

j/kθ
Figure 4.

( j(i),k(i),θ(i))

θ(i)

i

Optimization of 𝑗(𝑖), 𝑘(𝑖), and 𝜃(𝑖) by DP.

In the enhanced desynchronization, the rotation angle 𝜃is changed adaptively at every 𝑖. Hereafter, 𝜃(𝑖) denotes the˜ 𝜃 denotes a generated pattern. It
rotation angle at 𝑖 and 𝑹
should be noted that 𝜃(𝑖) simulates a fluctuation of handposture as shown in Fig. 1 (c). While we draw a stroke,
our hand posture (especially, wrist angle) varies. Since thedeformation characteristics will depend on the hand posture,
it is natural to introduce 𝜃(𝑖) into the model.
Figure 3 (c) and (d) illustrate two examples of generated
˜ 𝜃 from a single pattern 𝑹. Since there arepatterns 𝑹
many possible couplings of two points (even under a lag
limitation), many different patterns can be generated fromthe same pattern. Furthermore, even from the same point
couplings, many patterns can be generated by moving points
along circles, that is, by changing 𝜃(𝑖), as shown in (d).
According to the introduction of the adaptive rotation, wemust employ two more constraints in addition to the threeconstraints of Section II-A.
∙ The fourth constraint is the rotation angle limitation tolimit 𝜃(𝑖) ∈ [−𝜃max , 𝜃max ] shown in Fig. 3 (e). Under
˜𝑖 is forced to lie on a certain arc instead
this constraint, 𝒓of the whole circle.
∙ The fifth constraint is the continuity of 𝜃(𝑖). From theanalogy of Fig 1 (c), the rotation angle 𝜃(𝑖) should
not change abruptly and thus we employ the continuity
constraint ∣𝜃(𝑖) − 𝜃(𝑖 − 1)∣ ≤ 𝜃max /2.
IV. E MBEDDING THE M ODEL INTO E LASTIC M ATCHING
The proposed generative model has a potential of realizing
deformation-tolerant online character recognition. Assume
that 𝑹 is a reference pattern of a certain class, and 𝑬 =𝒆1 , 𝒆2 , . . . , 𝒆𝑖 , . . . , 𝒆𝐼 is an input pattern to be recognized.
The input pattern 𝑬 will be correctly recognized regardless˜ 𝜃 ∼ 𝑬.
its deformation, if we find 𝑹˜𝜃
The remaining problem is how to generate the optimal 𝑹𝜃˜
for given 𝑬 and 𝑹. In other words, we must find 𝑹 which˜ 𝜃 s. One possible way
is closet to 𝑬 among all possible 𝑹
is to formulate the minimization problem of the following
objective function 𝒥 with respect to {𝑗(𝑖), 𝑘(𝑖), 𝜃(𝑖)∣𝑖 =1, . . . , 𝐼} subject to the five constraints of Section II-A and
Section III:𝐼∑˜𝜃𝑖 ∥,∥𝒆𝑖 − 𝒓
(5)
𝒥 =
𝑖=1

(

𝜃(𝑖)

𝜃(𝑖)

)𝑇

˜𝜃𝑖 = 𝑥𝑗(𝑖) , 𝑦𝑘(𝑖) .
where 𝒓
The optimization problem is an extended version of theconventional elastic matching problem and thus solved efficiently by dynamic programming (DP). Letting 𝑑𝑖 (𝑗, 𝑘, 𝜃) =˜𝜃𝑖 ∥, the DP algorithm is organized as calculation of∥𝒆𝑖 − 𝒓the following recursive equation at all (𝑗, 𝑘, 𝜃) from 𝑖 = 1
to 𝐼:
𝑔𝑖−1 (𝑗 ′ , 𝑘 ′ , 𝜃′ ),𝑔𝑖 (𝑗, 𝑘, 𝜃) = 𝑑𝑖 (𝑗, 𝑘, 𝜃) + ′min′ ′𝑗 ,𝑘 ,𝜃

(6)

where (𝑗 ′ , 𝑘 ′ , 𝜃′ ) should satisfies all the constraints with(𝑗, 𝑘, 𝜃). For example, 𝑗 − 𝑗 ′ ∈ {0, 1, 2} and ∣𝑗 − 𝑘∣ ≤ 𝐿.
Consequently, min𝑗,𝑘,𝜃 𝑔𝐼 (𝑗, 𝑘, 𝜃) is the minimum matching
cost between 𝑬 and 𝑹 and the optimal {𝑗(𝑖), 𝑘(𝑖), 𝜃(𝑖)} is
given as a path of Fig. 4. The class of the reference patterngiving the minimum matching cost becomes the recognition
result of 𝑬.
V. E XPERIMENTAL R ESULTS
A. Qualitative Evaluation
An experiment has conducted to observe the ability of the
proposed model on generating deformation patterns. Specifically, the DP-based elastic matching with the proposedmodel was performed between a pair of digit patterns 𝑹
˜ 𝜃 is close to 𝑬, it proves
and 𝑬 from the same class. If 𝑹
that the proposed model can generate patterns close to actual
deformed patterns (i.e., patterns by human handwriting) in

20°
10°
(enhanced desync.)

θmax = 0°(simple desync.)

L=0(no desync.)

2

E

E

5

~
R

R

E

~
Rθ

E

~θ
R

E

E

E

L=0
(no desync.)

2

5

L=0
(no desync.)

2

5

~
R

~
Rθ

~
Rθ

(a)

(b)

(c)

˜ 𝜃 (blue) optimally fitted the input pattern 𝑬 (red) under different values of the maximum lag 𝐿 and the maximum rotation
Figure 5. Generated pattern 𝑹
angle 𝜃max . The original pattern 𝑹 is shown at 𝐿 = 0.

Figure 6.

E

R

~
E+ R

~
E+ Rθ

˜ and 𝑹˜ 𝜃.
Patterns 𝑬 and 𝑹, and generated patterns 𝑹

the same class. Note that the public online digit databasecalled Ethem Alpaydin Digit was used for the experiment.
Figure 5 shows the results. The patterns under 𝐿 = 0 are
original patterns 𝑹 and 𝑬. Note that conventional elastic
matching methods, such as the DP-based elastic matching,evaluates the geometric difference of those two original
patterns under the optimal point correspondence.
˜ in Fig. 5, which are results byFirst, let us observe 𝑹
elastic matching with the simple feature desynchronization(or, equivalently, the enhanced model under 𝐿 ∕= 0 and𝜃max = 0). A pair of “1” of Fig. 5 (a) clearly shows thelimit of the simple feature desynchronization because thenear-vertical stroke of “1” cannot change its direction largely
˜ andand there is still a significant difference between 𝑹
𝑬. A pair of “4” in Fig. 5 (b) shows that the ending partof “4”, which is originally slanted and thus not parallel tothe 𝑥- or 𝑦-axis, was deformed to be fitted to 𝑬 with thesimple model. A larger 𝐿 provided a better fitting because
it will increase pattern generation ability. In contrast, it ispossible to observe a clear limitation of the simple model atthe beginning part of “4”; simply speaking, no point can begenerated outside of the bounding box of an original stroke
and thus the beginning part is unnaturally deformed like a
vertical stroke.
˜ 𝜃 in Fig. 5, which are results elasSecond, let us observe 𝑹
tic matching with the enhanced feature desynchronization.
It is obvious that the pattern generation ability is increasedby the introduction of the adaptive rotation 𝜃(𝑖). The pattern˜ 𝜃 of “1” no longer remains as a vertical stroke. The pattern𝑹˜ 𝜃 of “4” shows a close fit to 𝑬 (even around its beginning𝑹part) especially under 𝐿 = 5 and 𝜃max = 10. These two
results of Fig. 5 (a) and (b) prove the enhanced model can
˜ 𝜃 close to actual deformed patterns 𝑬.
generate patterns 𝑹
As expected, an overfitting phenomenon happened between patterns from different classes. Figure 5 (c) shows
an example of the overfitting, where a reference pattern 𝑹of “2” is fitted to 𝑬 of “3” by using the higher patterngeneration ability of the proposed model.
One possible remedy to eliminate the overfitting in future
is to learn the optimal constraint parameters using training
patterns. For example, the adaptive lag 𝐿(𝑖) can be designed
for each class. In [1], a trial of training 𝐿(𝑖) have been made

recognition rate (%)

96
94enhanced desync

9290

simple desync

8886

L(i)=∞

L(i)

L(i)/2

L(i)/3

adaptive lag limitation

Figure 7.

Recognition rates.

on the simple desynchronization model and an improvement
on eliminating the overfitting has been reported. The proposed model has another parameter 𝜃(𝑖) to be learned. One
interesting future work is to investigate whether 𝜃(𝑖) is classdependent or writer-dependent. The latter comes from theanalogy between the adaptive rotation and the hand posture
as indicated in Fig. 1 (c).
Figure 6 shows other results between patterns of the same
˜ In
˜ 𝜃 shows a closer fitting than 𝑹.
class. In all results, 𝑹
several cases (e.g., the upper examples of “6” and “8”),˜ 𝜃 shows almost perfect fitting to 𝑬. Again, it should be𝑹emphasized that those close fitting is realized just by featuredesynchronization and rotation.
B. Quantitative Evaluation
An online character recognition experiment has been
conducted, although the main purpose of this paper is to
propose a new generative model for interpreting deformationof human handwriting and observe its deformation ability.
For each of 10 digit classes, 10 reference patterns wereselected from the training set of the database by k-means.
For the rotation angle limitation, 𝜃max was fixed at 20∘ .
The adaptive lag limitation was also used. Its parameter
𝐿(𝑖)(𝑖 = 1, . . . , 𝐼) was determined for each reference
pattern as the maximum lag observed in desynchronization
matching results between training 300 pattern pairs from thesame category.
Figure 7 shows the recognition rates by the DP-basedelastic matching with the simple and the enhanced featuredesynchronization models. As expected, without the laglimitation (i.e., 𝐿(𝑖) = ∞), the enhanced model shows apoor performance because of the overfitting problem. The
performance was improved by limiting the lag by 𝐿(𝑖).
Under more severe lag limitation by 𝐿(𝑖)/2, the performanceby the enhanced model was further improved and slightly
better than the performance by the simple model. This
indicates that the adaptive rotation does not cause severe
overfitting if we set appropriate constraints. In this case, 28
samples turned to be recognized correctly by the enhanced
model, and 22 samples turned to be misrecognized.
VI. C ONCLUSION
A new generative model based on enhanced featuredesynchronization has been proposed for modeling the de-formations of human handwritings. An important point isthat the proposed model is based on a simple structuraland combinatorial framework. In spite of this simplicity,
the model can generate patterns close to actual deformed
patterns just from a single pattern. This fact has been proved
via several experimental results and, consequently, it has also
been proved that the enhanced feature desynchronization canbe treated as a handwriting model.
Future work will focus mainly on the following points.
∙ Class-dependent and writer-dependent characteristics
should be incorporated into the model. This can berealized by learning the constraint parameters (e.g.,
𝐿(𝑖)) via some training steps.
∙ The variations in the resulting adaptive rotation angle
sequences 𝜃(1), . . . , 𝜃(𝑖), . . . , 𝜃(𝐼) should be observed
within a class or within a writer. Then, the relation
between the rotation angle and the hand posture should
be clarified.
∙ The enhanced desynchronization model should be further extended toward the realization of accurate anddeformation-tolerant online character recognition systems. Again, the proposed desynchronization model
is simple and thus there are various ways for theextension.
ACKNOWLEDGMENT
This research was partially supported by JST, CREST.
R EFERENCES[1] S. Uchida, K. Niyagawa, and H. Sakoe, “Feature desynchronization in online character recognition,” ICFHR, 2008.
[2] K. Yoshida and H. Sakoe, “Online handwritten character
recognition for a personal computer system,” IEEE Trans.
CE, 28(3), 1982.
[3] S. D. Connell and A. K. Jain, “Template-based online character recognition,” PR, 34(1), 2001.
[4] C. Bahlmann and H. Burkhardt, “The writer independent
online handwriting recognition system frog on hand andcluster generative statistical dynamic time warping,” PAMI,
26(3), 2004.
[5] T. Flash and N. Hogan, “The coordination of arm movements:
an experimentally confirmed mathematical model,” J. Neuroscience, 5, 1985.
[6] R. Plamondon, “Neuromuscular studies of handwriting generation and representation,” ICFHR, (Invited Talk), 2010.
[7] M. Brand, N. Oliver, and A. Pentland, “Coupled hidden
Markov models for complex action recognition,” CVPR,
1997.
[8] S. Matsuda, M. Nakai, H. Shimodaira, and S. Sagayama,“Asynchronous-transition HMM,” ICASSP, 2000.
[9] A. V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao andK. Murphy, “A coupled HMM for audio-visual speech recognition,” ICASSP, 2002.
[10] T. Artières, N. Gauthier, P. Gallinari, and B. Dorizzi, “A hidden Markov model combination framework for handwriting
recognition,” IJDAR, 5(4), 2002.