2018 16th International Conference on Frontiers in Handwriting Recognition

Introducing Local Distance-Based Features toTemporal Convolutional Neural Networks
Brian Kenji Iwana∗ , Minoru Mori† , Akisato Kimura‡ , Seiichi Uchida∗∗ Department of Advanced Information Technology, Kyushu University, Fukuoka, Japan† Department of Information and Computer Sciences, Kanagawa Institute of Technology, Atsugi, Japan‡ Nippon Telegraph and Telephone, Tokyo, Japan

brian@human.ait.kyushu-u.ac.jp, mmori@ic.kanagawa-it.ac.jp, kimura.akisato@lab.ntt.co.jp, uchida@ait.kyushu-u.ac.jp
Abstract—In this paper, we propose the use of local distancebased features determined by Dynamic Time Warping (DTW) for
temporal Convolutional Neural Networks (CNN). Traditionally,
DTW is used as a robust distance metric for time series patterns.
However, this traditional use of DTW only utilizes the scalardistance metric and discards the local distances between the
dynamically matched sequence elements. This paper proposes
recovering these local distances, or DTW features, and utilizingthem for the input of a CNN. We demonstrate that thesefeatures can provide additional information for the classiﬁcationof isolated handwritten digits and characters. Furthermore, we
demonstrate that the DTW features can be combined with thespatial coordinate features in multi-modal fusion networks toachieve state-of-the-art accuracy on the Unipen online handwritten character datasets.

DTW
Cost Matrix

DTW Features−

Input for CNN

Fig. 1. Extraction of DTW features from a DTW calculation. Given a
prototype sequence p and a sample sequence s, the cost matrix is the pairwise
local distances between each element in the time series with dark boxes
indicating low cost and light boxes indicating high cost. The yellow warping
path on the cost matrix is the estimated minimal path.

I. I NTRODUCTION
Dynamic Time Warping (DTW) [1] is a dynamic programming solution to ﬁnding the optimal match between elements
for the use of a distance metric between time series patterns.
DTW has been shown to be useful as a distance metric due
to it being relatively fast and robust to temporal distortions
and rate differences [2]. DTW has been used for many timeseries applications and been incorporated in many time series
recognition methods. For example, a common usage of DTW
is as a distance metric for k-Nearest Neighbor (k-NN) [3],[4]. Other applications of DTW include using it with Support
Vector Machines (SVM) [5], Dissimilarity Space Embedding (DSE) [6], and Hidden Markov Models (HMM) [7].
To be used as a distance metric, DTW is calculated bysumming the local distance between the optimally matchedsequence elements. The elements are matched by estimating
the minimal path on a cost matrix given constraints. However,when using DTW as a distance metric, only the resulting
summation of local distances is used. While this summation
value is useful for the global distance metric between patterns,
the local distances and the matching information is discarded.
It is possible that there is useful information within theconnections between the matched elements that is not reﬂectedin the summation value.
The discarded information from a DTW can convey the
relationship between the elements of the samples and matchedsequence elements of the prototypes. By capturing the local
distances between matched elements, a sequence of local
distance-based features, or DTW features, is created. The DTW
978-1-5386-5875-8/18/$31.00 ©2018 IEEE
DOI 10.1109/ICFHR-2018.2018.00025

features can then be used as a time series in time series
recognition methods.
Recently, artiﬁcial neural networks (ANN) and neural models have achieved state-of-the-art results in pattern recognition and machine learning [8]. Convolutional Neural Networks (CNN) [9] in particular have had many successes
in the image domain [10], [11]. Recurrent Neural Networks (RNN) [12] and more speciﬁcally Long Short-Term
Memory RNNs (LSTM) [13] have had success in the timeseries and sequence domains [14], [15]. While RNNs traditionally have been used for time series patterns, there havebeen many works which use CNNs for temporal data and it
has been shown that temporal CNNs can be competitive andsometimes even more effective than RNNs [16]. In addition,
the results presented in this paper reﬂect these ﬁndings.
In this paper, we propose the idea of using the sequence ofDTW features as a time series for the input of a temporal CNN,
as shown in Fig. 1. The contribution of this paper is threefold.
First, we show that the novel DTW features recovered fromthe DTW calculation can be used for classiﬁcation by CNN.
Second, we analyze different aspects of using CNNs with timeseries data, such as 1D versus 2D convolutions and convolutional window size. Third, we combine the DTW features andthe original spatial coordinates, or spatial features, in fusion
networks to acquire state-of-the-art results.
92II. R ELATED W ORK
Temporal CNNs are a natural extension of CNNs totime series using a similar function of Time Delay Neural
Networks (TDNN) [17]. The most common application of
temporal CNNs is to use convolutions that stride in one
direction, or 1D convolutions. In this way, the convolution
can encompass multivariate data but only moves across thetime steps. These 1D temporal CNNs have been used many
applications and time series data types [18]–[21] and haveshown to be competitive and at times, more effective thanRNNs, LSTMs, and Gated Recurrent Units (GRU) [16]. In a
recent advancement, WaveNet [22], has produced impressive
results for audio generation by implementing dilated causal
convolutional layers. In addition, CNNs have been used forother non-image structural domains such as graphs [23].

(a)

(b)

Fig. 2. Illustration of two DTW calculations with a similar result but different
matching relationships. The prototype pattern is represented in blue and thesample patterns are represented in (a) red and (b) green. Yellow lines are thepairwise matches determined by DTW.

The primary goal of the proposed method is to reuse thediscarded information from DTW and use it for classiﬁcation
by CNN. In this section, we detail the method of extracting
the DTW features.

where D(i, j) is the cumulative sum of the i-th and j-thelements. The global DTW distance is determined by the
cumulative sum at D(I, J). This asymmetric slope constraint
is used due to the special property that the number of pairwise
matches in M is exactly equal to the number of elements
I in the prototype p. Additionally, i always corresponds tothe same time step as i in the original prototype sequence.
However, j  can correspond non-linearly to the time steps jof the sample. This means that no matter how many elementss contains, the number of matches in M will always equal
the number of elements in p.

A. Dynamic Time Warping

B. DTW Features

III. L OCAL D ISTANCE - BASED F EATURES

Traditionally, only the global distance D(I, J) is used forpattern recognition methods and the individual local distancesin Eq. (1) of the warping path are discarded. However,when strictly using the global distance, information about thestructure of the comparison patterns can be lost. For example,
Fig. 2 illustrates a case where the DTW distance is similar, but
the patterns from Figs. 2a and 2b are clearly different. Thus,
preserving the information about the speciﬁc matches could
be useful to discriminate patterns.
By preserving the information about the speciﬁc matches
in the DTW calculation, new DTW features can be extracted.
Given the slope constraint in Eq. (2) with prototypes of a ﬁxed
length, the number of matches in M would be ﬁxed. In thisway, a sequence of ﬁxed length can be extracted from theDTW calculation by considering the local distances between
matches as a time series. Namely, the time series of DTW
features is deﬁned as:

Discrete time series recognition is not a straightforwardtask due to challenges in overcoming temporal distortions
such as rate differences, translations in time, and variable
lengths. DTW is a popular method used to consider thesechallenges and provide an effective distance metric between
time series [2]. Speciﬁcally, DTW aligns similar features of thetime series by warping the sequences in the time dimension.
In this way, elements of the time series are elastically matched
to minimize the summation of local distances between thosematched elements, while under constraints to maintain thetemporal qualities of the time series.
Namely, DTW can be found between two time series,
sequence p = p1 , . . . , pi , . . . , pI , referred to as the prototype,and sequence s = s1 , . . . , sj , . . . , sJ , referred to as the sample,
where i and j are the indices of sequence elements pi and sj
with length I and J, respectively. For online handwriting, thesequence elements are pen tip captured spatial coordinates,
which we refer to as spatial features.
Given p and s, DTW is deﬁned as the global summationof the local distances between pairwise element matches, or:||pi − sj  ||,(1)
DTW(p, s) =mn = ||pn,1 − s1 ||, . . . , ||pn,i − sj  ||, . . . , ||pn,I  − sJ  ||, (3)where (i , j  ) ∈ M and mn is the time series of DTW features
relating to the n-th prototype pn with a prototype set of N
number of prototypes. The result is a N -dimensional timeseries of a ﬁxed length I based on the local distances ofmatched elements between a sample and each prototype.

(i ,j  )∈M

where (i , j  ) is a pairwise match determined by dynamic
programming over a local cost matrix and M is the set of
all matches. The indices i and j  correspond to the originalindices i and j of p and s, respectively. It should be notedthat the set M may contain duplicate or skipped indices.
In this paper, we utilize the asymmetric slope constraint
deﬁned by the recurrent function:
D(i, j) = ||pi − sj || + minj ∈{j,j−1,j−2}

D(i − 1, j  ),IV. T EMPORAL C ONVOLUTIONAL N EURAL N ETWORKS
CNNs are feedforward neural networks usually contain
three components, convolutional layers, pooling layers, andfully-connected layers. Typically, they are used with structuraldata, most notably images, due to the ability learn structuralqualities while maintaining some shape and location invariance. The core component of a CNN is the convolutional

(2)

93

Conv (3, 1)
1
1

3

DTW Features w.r.t. Prototype 50

50
1

⋯

50
1

3

DTW Features w.r.t. Prototype 1
DTW Features w.r.t. Prototype 2

Conv (3)50

DTW Features w.r.t. Prototype 3

DTW Features w.r.t. Prototype 3
DTW Features w.r.t. Prototype 2
DTW Features w.r.t. Prototype 1t(a) 1D Convolution⋯
3

DTW Features w.r.t. Prototype 50t1(b) 2D Convolution

Fig. 3. Comparison of a (a) 1D convolution and (b) 2D convolution of the
input layer using a sequence created from DTW features. In (a), the 1D
convolution considers each DTW w.r.t. a prototype as a dimensions of the
input and strides in the time dimension t. In (b), the 2D convolution sharesweights and strides both in the time dimension and across prototypes.

Fig. 4. Examples of DTW feature sequences using 50 prototypes. (above)is the sample digit and (below) is the resulting DTW feature sequence. The
prototypes are sorted by class for visualization purposes.

V. C OMPARATIVE R ESULTS BETWEEN S PATIAL F EATURESAND DTW F EATURES
A. Dataset

layer, which applies shared weights, or a convolutional ﬁlter,
across an input. The application of the convolution on the
input is similar to a feature extractor [9] and the result is afeature map. Furthermore, pooling layers are used to downsample the feature maps in order to reduce redundancy andintroduce small translation and size invariance. Finally, thefully-connected layers and the output layers are used to learn
a non-linear function from the high-level features of the highlevel convolutional layer.
The general idea of the proposed method is to constructinput sequences of DTW features and use them with a CNN.
However, using a CNN for time series can be done in twoways, a traditional 1D convolution CNN and a proposed 2D
convolution CNN.

Isolated online handwritten characters serve as a valuable
source of time series data due to having distinct classes yet
still having a high variation within those classes. For theexperiments, we used the Unipen multi-writer 1a (digits),
Unipen 1b (uppercase alphabet), and Unipen 1c (lowercase
alphabet) datasets [24]. The patterns in the datasets consist of
about 15,000 time series made of 2D pen tip coordinates andwas collected by the International Unipen Foundation (iUF).1
This is a well established source of data for time series
recognition.
For the experiments, the dataset was divided into threesets, a training set, a test set, and a prototype set. For the
Unipen 1a evaluation, the test set contained 1,300 patterns andthe training set contained 11,650 patterns with an even class
distribution. The prototype set contained 50 patterns with 5
patterns from each class selected at random. For the Unipen
1b and 1c evaluations, 1,235 patterns were used for the test
set, 11,063 patterns were used for the training set, and 2 ofeach class were used for the prototype set.
Furthermore, the all patterns were preprocessed by scaling
and resampling. The patterns were scaled so that each pattern
could ﬁt within the bounds of a square deﬁned by (−1, −1)and (1, 1). Finally, patterns were resampled to 50 elements inlength to ﬁx the DTW feature sequences to 50 elements andto be able to use the patterns for the spatial feature trials.
To prepare the experiments with DTW features, DTW was
calculated between the patterns in the prototype set and eachpattern in the training and test sets. The DTW features wereextracted as described in Section III-B and N -dimensionaltime series with length I = 50 were created. Figure 4 is a
visualization of example sample patterns encoded into DTW
feature sequences. Lastly, the DTW features were normalized
to be -1 to 1.

A. CNNs with a 1D Convolution
The traditional method of using a CNN with time series
patterns is to use a 1D convolution in the convolutional layers.
Shown in Fig. 3a, the convolution strides across the time series
in the time step dimension and considers the dimensionality ofthe time series as depth (similar to the interaction of channels
for images). This is the equivalent of using a convolution overa signal. In this method, parameters are shared among time
steps and the 1D convolution CNN is able to overcome small
temporal distortions by learning high-level features throughpooling.
B. CNNs with a 2D Convolution
While a 1D convolution CNN shares parameters acrosstime steps, it is possible to share parameters across bothtime steps and prototypes. To accomplish this, we propose
considering the time series as a matrix and employ traditional
2D convolutions for the convolutional layer. Figure 3b is ademonstration of this using the proposed DTW features. Usinga 2D convolution in this manner is possible if the window
size of the 2D convolution has a height of 1. A convolutional
window size with a height of 1 is used because order ofthe DTW features with respect to each prototype is arbitrary.
However, the 2D convolution is still useful because unlikea 1D convolution with different parameters for each DTW
feature channel, parameters are shared with respect to thedifferent prototypes.

B. Settings and Evaluation
1) CNN Evaluations: In order to compare the proposed
method of using DTW features with CNNs to the pen tip
features, we trained four different CNNs architectures using
1 URL: http://www.unipen.org/home.html

94input sequences of one of the two feature types. 1D CNN(3) and 1D CNN (5) are 1D convolutional CNNs using a
convolutional window size of 3 and 5, respectively. Different
sized windows were used to reveal the effect of window sizeon the results. Similarly, 2D CNN (3, 1) and 2D CNN (5,1) are 2D convolution CNNs with window sizes (3, 1) and(5, 1), respectively. The Spatial Features CNNs were trainedusing only the 2D spatial features as a sequence and the DTW
Features CNNs used the DTW feature sequences.
The CNNs used for the experiment are constructed ofthree convolutional layers with accompanying max poolinglayers, two fully-connected layers, and a softmax output. The
convolutional layers have 64, 128, and 256 nodes, respectively,
the fully-connected layers were set to 1024 nodes, and softmax
had an output respective to the number of classes in the task.
Each of the hidden layers use Rectiﬁed Linear Unit (ReLU)
activations and dropout with a keep probability of 0.5 wasused proceeding the fully-connected layers. The networks weretrained using Adaptive Moment Estimation (Adam) [25] withan initial learning rate of 0.0001 and with batches of 100 for
100,000 iterations.
2) LSTM Comparisons: The LSTMs were constructed to be
fair comparisons with the same hyperparameters and structures
as the CNNs. Namely, the LSTMs consisted of three stacked
LSTM layers with an initial forget bias of 1.0, two fullyconnected layers with 1024 nodes, and a softmax output
layer. The fully-connected layers use ReLU activations anduse dropout with a keep probability of 0.5. Similar to the
CNN evaluations, the LSTM evaluations were optimized using
Adam optimizer with batches of 100 for 100,000 iterations.
3) Reported Comparisons: In addition to the LSTM evaluation, we report the state-of-the-art results found in literature.
HMM-CSDTW [7], DAG-SVM-GDTW [5], and OnSNT [26]are established classical baseline methods based on an HMM,
an SVM, and an n-tuple classiﬁer, respectively. DTW-NN [27]and Google [28] are the latest state-of-the-art solutions and are
based on deep neural networks.

TABLE IC OMPARISON OF S PATIAL F EATURES AND DTW F EATURES
Unipen 1a
Acc. (%)

Unipen 1b
Acc. (%)

Unipen 1c
Acc. (%)

Spatial Features
1D CNN (3)1D CNN (5)2D CNN (3, 1)
2D CNN (5, 1)
LSTM

98.00
98.38
98.08
98.08
98.15

97.65
97.81
97.14
97.89
96.36

97.33
97.41
96.76
96.76
95.38

DTW Features
1D CNN (3)1D CNN (5)2D CNN (3, 1)
2D CNN (5, 1)
LSTM

98.15
98.62
98.69
98.77
97.62

97.65
97.25
97.57
97.73
95.71

96.76
96.19
97.65
97.73
94.33

DAG-SVM-GDTW [5]HMM-CSDTW [7]OnSNT [26]DTW-NN [27]Google [28]

96.2
97.1
98.9
96.8
99.2

92.4
92.8
95.7
–
96.9

87.9
90.7
92.1
–
94.9

Method0 as 8

1 as 55 as 7

5 as 36 as 47 as 38 as 98 as 09 as 49 as 1(a) Misclassiﬁcations using Spatial Features Only

1 as 24 as 6

4 as 14 as 6

4 as 18 as 48 as 09 as 0(b) Misclassiﬁcations using DTW Features Only
Fig. 5. Unique misclassiﬁcations of 1D CNN (5) using (a) spatial featuresand (b) DTW features. Shared misclassiﬁcations are not shown. Digits withblue boxes are misclassiﬁcations that were improved by 1D CNN (5) withdecision-level fusion.

feature experiment, and vice versa. The misclassiﬁcations in
Fig. 5a demonstrate that the 1D CNN (5) is susceptible tonoise more so than the misclassiﬁcations in Fig. 5b. This ismost evident in the “7 as 3” and “8 as 9” in Fig. 5a. The
reason for this is due to DTW’s optimal matching which will
avoid elements with large distances. Also, interestingly, theDTW features was able to successfully realize the very small
upper loop in “8 as 0” in Fig. 5a. Again, it was able to do
this because the optimal matching which emphasizes relevant
features. However, this also can cause overﬁtting, such as “1
as 2,” both “4 as 6,” and “8 as 4” in Fig. 5b.

C. Results
Table I reports the result of the experiments as well as theresults from literature. The results show that using the DTW
features with 2D CNN (5, 1) achieved the highest accuracy
for Unipen 1a and 1c with accuracies of 98.77% and 97.73%,
respectively. Conversely, the Unipen 1b experiment had ahigher accuracy for the spatial features. However, all of theCNNs, both DTW features and spatial features, had a higher
accuracy than the LSTM evaluations and all of the reported
results with exception of Google for Unipen 1a. It is alsointeresting to note that the 2D CNN trials generally performed
better than the 1D CNNs despite 1D CNNs being a standardpractice for temporal CNN models. From these results, we cansee that the DTW features successfully encode information
about the underlying sample patterns and sometimes evenmore so than the spatial features.
Figure 5 shows the misclassiﬁcations of 1D CNN (5) thatwere present in the spatial feature experiment but not the DTW

VI. M ULTI - MODAL F USION
The DTW features and the spatial features are different
modalities of data which contain different aspects of the input
sample patterns. Thus, it is possible to combine the two
features to improve the classiﬁcation. Combining different
types of data in one classiﬁer is referred to as multi-modal
classiﬁcation. Using an additional form of data can be used tosupplement the discriminative ability of a classiﬁer using only
one data modality.

95Output
OutputOutput

FC
FC
Conv

FC

FC

FC

FC

FC

FC

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Conv

Input

Input

Input

Input

Input

(a) Data-level Fusion(b) Feature-level Fusion(c) Decision-level Fusion

Fig. 6. Multi-modal data fusion in CNNs. Conv are convolutional layers and FC are fully-connected layers. The blue and red layers indicate the different
data types (e.g. spatial features and DTW features) and purple indicates layers post fusion.

TABLE IIC OMPARISON OF DATA F USION

However, combining modalities is not a straightforwardtask. The data can be combined on a input-level, featurelevel, or a decision-level, shown in Fig. 6. Input-level fusion
concatenates the data modalities and processes them as a single
data source. Feature-level fusion combines the different data
modalities within the internal representation learned by themodel. Finally, decision-level approaches represent the data
separately and combine them in a decision rule. The optimal
point at which data of different modalities is fused is not welldeﬁned.
Fusion neural networks using multi-modal data is an emerging ﬁeld in deep learning-based machine learning. Neural networks with fusion schemes have been used for combining title
and image data [29], and for data from multiple sensors [30],such as multi-source remote sensing [31], activities of daily
life [32], and pedestrian detection [33].

Unipen 1a
Acc. (%)

Unipen 1b
Acc. (%)

Unipen 1c
Acc. (%)

Data-level Fusion
1D CNN (3)1D CNN (5)2D CNN (3, 1)
2D CNN (5, 1)

98.00
98.15
98.69
98.23

97.25
97.17
97.41
97.33

96.28
96.44
97.09
96.84

Feature-level Fusion
1D CNN (3)1D CNN (5)2D CNN (3, 1)
2D CNN (5, 1)

98.77
98.62
98.85
98.85

97.73
97.65
97.89
97.89

96.76
96.92
97.33
97.89

Decision-level Fusion
1D CNN (3)1D CNN (5)2D CNN (3, 1)
2D CNN (5, 1)

99.08
98.85
98.54
98.46

97.73
97.73
97.65
98.22

96.76
97.00
97.41
97.41

MethodA. Settings and Evaluation
In order to demonstrate that the DTW features can be
useful to supplement the raw spatial coordinates of online
handwriting, fusion networks were created from the CNNs
described in Section III. The previous experiments were repeated using three fusion timings, namely, a data-level fusion
CNN, a feature-level fusion CNN, and a decision-level fusion
CNN.
1) Data-level Fusion: Shown in Fig. 6a, the data-level
fusion combined the DTW feature time series and the spatial
feature time series into one time series prior to training of the
CNN. In this way, a time series with 50 time steps of (N +2)dimensions was fed to the CNN, where N is the number ofprototypes and 2 is the extra spatial features. The rest of thehyperparameters are exactly the same as the CNNs describedin Section V-B.
2) Feature-level Fusion: The architecture of the featurelevel fusion network is shown in Fig. 6b. In this fusion
network, the DTW feature data and the spatial feature data is
split for their own convolutional layers, but are combined rightbefore the ﬁrst fully-connected layer. Again the hyperparameters and training are the same as before, however, there aretwice as many total convolutional nodes due to each modality
containing a full set of 64, 128, and 256 nodes.
3) Decision-level Fusion: The third fusion timing,
decision-level fusion, is shown in Fig. 6c. In this model, twoalmost complete CNNs, one for each feature type, is used
and only combined for the softmax output layer.
B. Results
The results of the fusion CNN experiments are shown in
Table II. The decision-level fusion 1D CNN (3) achieved
a near perfect 99.08% accuracy for the Unipen 1a dataset,
which is exceptional compared to the results in Table I. In
addition, the results from decision-level fusion 2D CNN (5,1) for Unipen 1b and feature-level fusion 2D CNN (5, 1) for
Unipen 1c surpassed all of the other results for their respective
datasets. This demonstrates that the proposed method of supplementing time series data with DTW-based distance featuresto prototypes can achieve a dramatic increase in accuracy.

96

Table II also reveals that later fusions tended to be morerobust. From this, it can be inferred that the modalities
interfered with each other when fused early. By fusing at
the data-level, the information from the spatial features isdwarfed due to the higher dimensionality of the DTW features.
Consequently, the data-level fusion results are similar to the
DTW features alone. On the other hand, Fig. 5 provides insight
into the reason for the higher decision-level fusion accuracies.
In the ﬁgure, the decision-level fusion network overcame theweaknesses of the spatial features or the DTW features alone.

[10] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectiﬁers:
Surpassing human-level performance on imagenet classiﬁcation,” inIEEE Int. Conf. Comput. Vision, 2015, pp. 1026–1034.
[11] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate
deep network learning by exponential linear units (elus),” arXiv preprint,
2015, https://arxiv.org/abs/1511.07289.
[12] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Cognitive modeling, vol. 5, no. 3,
pp. 213–220, 1988.
[13] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” NeuralComputation, vol. 9, no. 8, pp. 1735–1780, 1997.
[14] I. Sutskever, J. Martens, and G. E. Hinton, “Generating text withrecurrent neural networks,” in Int. Conf. Mach. Learning, 2011, pp.
1017–1024.
[15] A. Graves and N. Jaitly, “Towards end-to-end speech recognition withrecurrent neural networks,” in Int. Conf. Mach. Learning, 2014, pp.
1764–1772.
[16] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of genericconvolutional and recurrent networks for sequence modeling,” arXiv
preprint arXiv:1803.01271, 2018.
[17] A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang,“Phoneme recognition using time-delay neural networks,” IEEE Trans.
Acoust., Speech, Signal Process., vol. 37, no. 3, pp. 328–339, 1989.
[18] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, andP. Kuksa, “Natural language processing (almost) from scratch,” J. Mach.
Learning Research, vol. 12, no. Aug, pp. 2493–2537, 2011.
[19] Y. Kim, “Convolutional neural networks for sentence classiﬁcation,” inConf. Empirical Methods in Natural Language Process., 2014, pp. 1746–
1751.
[20] N. Razavian and D. Sontag, “Temporal convolutional neural networks
for diagnosis from lab tests,” arXiv preprint, 2015, https://arxiv.org/abs/
1511.07938.
[21] J. Yang, M. N. Nguyen, P. P. San, X. Li, and S. Krishnaswamy, “Deepconvolutional neural networks on multichannel time series for humanactivity recognition.” in Int. Joint Conf. Artiﬁcial Intell., 2015, pp. 3995–
4001.
[22] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves,N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” arXiv preprint, 2016, https://arxiv.org/abs/
1609.03499.
[23] M. Niepert, M. Ahmed, and K. Kutzkov, “Learning convolutional neuralnetworks for graphs,” in Int. Conf. Mach. Learning, 2016, pp. 2014–2023.
[24] I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, and S. Janet,“Unipen project of on-line data exchange and recognizer benchmarks,”
in Int. Conf. Pattern Recognition, vol. 2, 1994, pp. 29–33.
[25] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”in Int. Conf. Learning Representations, 2015.
[26] E. H. Ratzlaff, “Methods, reports and survey for the comparison ofdiverse isolated character recognition results on the unipen database,” inInt. Conf. Document Anal. and Recognition, 2003, pp. 623–628.
[27] B. K. Iwana, V. Frinken, and S. Uchida, “A robust dissimilarity-basedneural network for temporal pattern recognition,” in Int. Conf. Frontiersin Handwriting Recognition, 2016, pp. 265–270.
[28] D. Keysers, T. Deselaers, H. A. Rowley, L.-L. Wang, and V. Carbune,“Multi-language online handwriting recognition,” IEEE Trans. PatternAnal. Mach. Intell., vol. 39, no. 6, pp. 1180–1194, 2017.
[29] T. Zahavy, A. Magnani, A. Krishnan, and S. Mannor, “Is a picture worth
a thousand words? a deep multi-modal fusion architecture for productclassiﬁcation in e-commerce,” arXiv preprint arXiv:1611.09534, 2016.
[30] Z. Ahmad and J. Zhang, “Combination of multiple neural networks
using data fusion techniques for enhanced nonlinear process modelling,”
Comput. & Chemical Engineering, vol. 30, no. 2, pp. 295–308, 2005.
[31] X. Dai and S. Khorram, “Data fusion using artiﬁcial neural networks:a case study on multitemporal change analysis,” Comput., Environ. and
Urban Syst., vol. 23, no. 1, pp. 19–31, 1999.
[32] F. J. Ordóñez and D. Roggen, “Deep convolutional and lstm recurrent
neural networks for multimodal wearable activity recognition,” Sensors,
vol. 16, no. 1, p. 115, 2016.
[33] X. Du, M. El-Khamy, J. Lee, and L. Davis, “Fused dnn: A deep neuralnetwork fusion approach to fast and robust pedestrian detection,” inIEEE Winter Conf. App. Comput. Vision. IEEE, 2017, pp. 953–961.

VII. C ONCLUSION
In this paper, we proposed a novel use of the DTW
calculation by using it as a method to extract distancebased features. These features, or DTW features, are thelocal distances between elements matched by DTW. Throughextensive experimentation with CNNs, we demonstrated thatthe DTW features can be used as time series and are usefulfor classiﬁcation. Furthermore, we demonstrated that the DTW
features can be combined with the original spatial coordinates
within multi-modal fusion networks to achieve state-of-the-art
results. Using the proposed method, we achieved a 99.08%,
98.22%, and 97.89% accuracy for the Unipen 1a, 1b, and 1cdatasets, respectively. These results are dramatic improvements
over the results from a CNN on the spatial coordinates alone,
the reported results from literature, and an evaluated LSTM.
ACKNOWLEDGMENT
This research was partially supported by MEXT-Japan(Grant No. J17H06100) and NTT Communication Science
Laboratories.
R EFERENCES[1] H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization
for spoken word recognition,” IEEE Trans. Acoustics, Speech, and Sig.
Process., vol. 26, no. 1, pp. 43–49, 1978.
[2] P. F. Felzenszwalb and R. Zabih, “Dynamic programming and graphalgorithms in computer vision,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 33, no. 4, pp. 721–740, 2011.
[3] H. Ding, G. Trajcevski, P. Scheuermann, X. Wang, and E. Keogh,“Querying and mining of time series data: experimental comparison ofrepresentations and distance measures,” Proc. Very Large Data BaseEndowment, vol. 1, no. 2, pp. 1542–1552, 2008.
[4] T. Rakthanmanon, B. Campana, A. Mueen, G. Batista, B. Westover,
Q. Zhu, J. Zakaria, and E. Keogh, “Searching and mining trillions oftime series subsequences under dynamic time warping,” in Int. Conf.
Knowledge Discovery and Data Mining, 2012, pp. 262–270.
[5] C. Bahlmann, B. Haasdonk, and H. Burkhardt, “Online handwriting
recognition with support vector machines-a kernel approach,” in Int.
Workshop Frontiers in Handwriting Recognition, 2002, pp. 49–54.
[6] B. K. Iwana, V. Frinken, K. Riesen, and S. Uchida, “Efﬁcient temporal
pattern recognition by means of dissimilarity space embedding withdiscriminative prototypes,” Pattern Recognition, vol. 64, pp. 268–276,
2017.
[7] C. Bahlmann and H. Burkhardt, “The writer independent online handwriting recognition system frog on hand and cluster generative statistical
dynamic time warping,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26,
no. 3, pp. 299–310, 2004.
[8] J. Schmidhuber, “Deep learning in neural networks: An overview,”
Neural Networks, vol. 61, pp. 85–117, 2015.
[9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning
applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–
2324, 1998.

97