Modality Conversion of Handwritten Patterns by Cross Variational Autoencoders
Taichi Sumi∗ , Brian Kenji Iwana∗ , Hideaki Hayashi∗ , Seiichi Uchida∗∗ Advanced Information Technology, Kyushu University, Japan{brian, hideaki.hayashi, uchida}@human.ait.kyushu-u.ac.jp

Keywords-variational autoencoder; handwritten character
recognition; modality conversion

I. I NTRODUCTION
Handwritten characters inherently have two modalities:
image and temporal trajectory. This is because a handwritten
character image is comprised of a single or multiple strokes
and each stroke is originally generated as a temporal trajectory along with the pen movement. This dual-modality
is essential and unique to handwritten characters. Therefore, we can expect unique and more accurate recognition
methods and applications by utilizing the dual-modalityof handwritten characters. This expectation emphasizes thenecessity of the methodologies to convert one modality tothe other.
Modality conversion from a temporal trajectory to an
image is so-called inking. For multi-stroke character recognition, inking is a reasonable strategy to remove stroke-ordervariations. In the past, many hybrid character recognition
methods (e.g., [1]) have been proposed, where two recognition engines are used for the original trajectory pattern andits “inked” image, respectively. In other methods (e.g., [2]),the local direction of the temporal trajectory is embedded
into the inked image as an extra feature channel.
Modality conversion from a handwritten character image
to a temporal trajectory representing the stroke writing orderis so-called stroke recovery [3]. Comparing to the inking
method, stroke recovery is far more difficult because it is theinverse problem of inferring the lost temporal informationfrom a handwritten image.
In this paper, we propose a Cross-Variational Autoencoder(Cross-VAE), a neural network-based modality conversion
method for handwritten characters. Cross-VAE has the ability to convert a handwritten character image into its original

strokerecovery𝑋𝑏inking𝑋𝑡

𝑧𝑏=

Abstract—This research attempts to construct a networkthat can convert online and offline handwritten characters toeach other. The proposed network consists of two Variational
Auto-Encoders (VAEs) with a shared latent space. The VAEs
are trained to generate online and offline handwritten Latincharacters simultaneously. In this way, we create a crossmodal VAE (Cross-VAE). During training, the proposed CrossVAE is trained to minimize the reconstruction loss of the twomodalities, the distribution loss of the two VAEs, and a novel
third loss called the space sharing loss. This third, space sharing
loss is used to encourage the modalities to share the same latent
space by calculating the distance between the latent variables.
Through the proposed method mutual conversion of online
and offline handwritten characters is possible. In this paper,
we demonstrate the performance of the Cross-VAE throughqualitative and quantitative analysis.

𝑌𝑏

𝑧𝑡
𝑌𝑡

Figure 1. Outline of the proposed Cross-VAE for modality conversionof handwritten characters. Two VAEs are prepared for two modalities, i.e.,
bitmap image and temporal trajectory, and co-trained so that their latentvariables become the same for the same handwritten characters in different
modalities. The trained Cross-VAE realizes inking and stroke recovery, as
indicated by orange and purple paths, respectively.

temporal trajectory and vice versa. In other words, theCross-VAE can realize stroke recovery as well as inking
by itself. This means that the Cross-VAE can manage thedual-modality of handwritten characters.
As shown in Fig. 1, the Cross-VAE is compounded fromtwo VAEs. Each VAE [4] is a generation model which
is decomposed into two neural networks: an encoder thatobtains latent variable z from data X and a decoder thatobtains output Y close to X from z, i.e., X ∼ Y . In
general, the dimensionality of z is lower than X and Y andthus the latent variable z represents fundamental informationof X in a compressed manner. One VAE of Cross-VAE
is trained for a handwritten character image (i.e., image
Xb → zb → image Yb (∼ Xb )) and the other VAE is trained
for a temporal writing trajectory (i.e., temporal trajectory
Xt → zt → temporal trajectory Yt (∼ Xt )). Note thatthe suffixes b and t indicate bitmap image and temporal
trajectory, respectively.
The technical highlight of Cross-VAE is that those two
VAEs are trained by considering the dual-modality of handwritten characters. Assume that the input image Xb is generated from a temporal trajectory Xt by inking, then we expectthat their corresponding latent variables can be the same,that is, zb = zt . This is because Xb and Xt are the same
handwritten character in different modalities and thus theirfundamental information should be the same. Consequently,
if we can co-train two VAEs under the condition zb = zt ,we realize, for example, stroke recovery by the following
steps: Xb → zb = zt → Yt (∼ Xt ).
The main contributions of this paper are summarized asfollows:• A cross-modal VAE is proposed for online and offline handwriting conversion. The Cross-VAE is the
combination of two VAEs with different modalitieswith a shared latent space and a dual-modality training
process.
• A novel loss function called the space sharing loss isintroduced. The space sharing loss encourages the latentvariables of the VAEs to use the same latent space.
The shared latent space is what allows for an input
modality to be represented by both output modalities
simultaneously.
• Quantitative and qualitative analyses are performedon the proposed method. We show that the CrossVAE was able to successfully model both online andoffline handwriting as well as be used for cross-modal
conversion.
II. R ELATED W ORK
Recently, there are two common approaches that havebecome popular which use neural networks to learn latent
representations, Encoder-Decoders and Generative Adversarial Networks (GAN) [5]. Encoder-Decoders, such as an
Autoencoder [6], compress data by encoding the inputs intoa latent vector which is then uncompressed by the decoder.
The Autoencoder is trained by minimizes the difference
between the input and the output of the decoder. GANs
take the opposite approach and use a generator, similar
to an encoder, then uses a discriminator to maximize theauthenticity of the generated data. Where Encoder-Decoders
learn the latent representations directly, GANs learn toconstruct data from random latent representations.
As for cross-modal generation applications, X-Shaped
Generative Adversarial Cross-Modal Networks (XGACMN) [7] creates a shared space for text and imagesby crossing GANs. Peng et al. [8] also use GANs for text
and image entanglement, however, they use weight sharing
constraints. Furthermore, a Cross-modal VAE was usedby Spurr et al. [9] for hand pose estimation. However,their model only permits multiple pairs of encoders anddecoders to share the latent space. Our method trains the
VAEs to intertwine with each other and encourages them toshare the same latent space. Multi-modal and cross-modal
VAEs were also used in [10], [11]. Also, image-to-imagetranslation networks can be seen as a modal conversion.
Some examples include CycleGAN [12], StarGAN [13],and UNsupervised Image-to-image Translation (UNIT) [14]networks.
For offline and online handwriting conversion, it has
traditionally been done using classical feature-based methods [15] but there has been some recent work using neural networks. Bhunia et al. [16] used a CNN and RNNbased Encoder-Decoder network for handwriting trajectory
recovery. Attempts were also made using neural networks
to identify graph features [17] and for sequential stroke
prediction using regression CNNs [18].

III. C ROSS - MODAL VARIATIONAL AUTOENCODER(C ROSS -VAE)
VAEs [4] are Autoencoders which use a variational
Bayesian approach to learn the latent representation. VAEs
have been used to generate time series data [19], including
speech synthesis [20] and language generation [21]. They
have also been used for image data [22] and data augmentation [23], [24].
A. Variational Autoencoder (VAE)
A VAE [4] consists of an encoder and a decoder. Given
an input X ∈ RI , the encoder estimates the posterior
distribution of a latent variable z ∈ RJ .1 The decoder,in turn, generates an output Y ∈ RI based on a latentvariable sampled from the estimated posterior distribution.
The VAE is trained end-to-end using a combination of thereconstruction loss LRE and the distribution loss LKL , or:
LVAE = LKL + LRE .

(1)

The reconstruction loss LRE is the cross-entropy between
the input and the output of the decoder. It is determined by:
LRE = −I
X

Xi log Yi +(1−Xi ) log (1−Yi ),

(2)

i=1

assuming that Y follows the multivariate Bernoulli distribution. In Eq. (2), Xi and Yi are the i-th element of X andY , respectively.
The difference between a traditional Autoencoder or
Encoder-Decoder network is that the VAE models the latent space using a Gaussian model and uses a variational
lower bound to infer the posterior distribution of a latentvariable. This is done by including a loss between the latentvariables and the unit Gaussian distribution. Specifically, thedistribution loss LKL is based on the Kullback-Leibler (KL)divergence, or:J

LKL = −

1X
1 + log (σj2 ) − µ2j − σj2 ,
2 j=1

(3)

assuming that the prior distribution of the latent variable z
follows the multivariate Gaussian distribution of N (0, I). In
Eq. (3), and µ and σ 2 are the mean and variance of theposterior distribution of z.
B. Cross-VAE
We propose the use of a Cross-modal VAE (Cross-VAE) to
be used to perform online and offline handwritten character
conversion, as illustrated in Fig. 2. The network in red isa VAE for online handwritten characters and the networkin blue is for offline handwritten characters. The CrossVAE is constructed from the joining of two different single
1 For simplicity, we omit the notation with regard to the number oftraining data. In the actual calculation, all losses described below aresummed over the batch size.

Time Series Encoder 256 32

𝜇t

128

64

32

32 256 Time Series Decoder
128
64ℒRE(t→t)

𝑧t
𝑋t

Conv

Conv

or

or

LSTM

LSTM

ℒKL(t)

Conv
or

LSTM

Conv
or

LSTM

𝜎t

Conv

Conv

or

or

LSTM

LSTM

ℒLS

𝑁(0, 𝑰)ℒRE(t→b)

𝜇b
32

𝑋b

64

Conv
Conv

32
128

Conv

256

ℒKL(b)

Conv

Image Encoder

256

𝜎b

𝑌t→t

𝑌b→t

32

32

𝑧b

ℒRE(b→t)

128

64ℒRE(b→b)

Conv Conv

𝑌t→b

𝑌b→b

Conv

Image Decoder Conv

Figure 2. Details of the proposed Cross-VAE. Xt is a time series input, Xb is an image input. The illustrations of the time series, Xt , Yt→t , and Xb→t ,are colored from pink to yellow according to their sequence order. LKL is the distribution loss, LRE is the reconstruction loss, and LRE is the space
sharing loss. Yt→t and Yb→b are the intra-modal outputs and Yt→b and Yb→t are the cross-modal outputs.

modality VAEs into one multi-modal VAE with a sharedcross-modal latent space. Furthermore, we use a cross-modal
loss function to ensure that the latent space is shared between
the modalities.
During training, the two modalities are trained simultaneously. A time series input Xt and an image input Xb are
entered into the encoders and four outputs are extracted fromthe decoders. For each input Xt and Xb , there are respective
time series outputs, Yt→t and Yb→t , and respective image
outputs Yt→b and Yb→b . The outputs Yt→t and Yb→b are
intra-modal and the outputs Yt→b and Yb→t are cross-modal.
The loss function of the Cross-VAE is:LCross = LKL + LRE + LLS ,(4)

where LKL is the distribution loss and LRE is the reconstruction loss as described in Section III-A. The third loss,
LLS , is the proposed space sharing loss. Due to training withthe two inputs, Xt and Xb , two latent representations are
created zt and zb , respectively. Therefore, the traditional
VAE losses, LKL and LRE , need to be modified for CrossVAE.
Due to the two latent representations, the total distribution
loss LKL is calculated by combining the individual distribution losses, LKL(t) and LKL(b) , or:
LKL = αLKL(t) + βLKL(b) ,(5)where α and β are weights. The distribution loss of theindividual input modalities is calculated using Eq. 3.
Next, the reconstruction loss LRE takes into account thereconstruction of Yt→t and Yb→b , as well as the conversion
of Yt→b and Yb→t . Thus:
LRE = γt→t LRE(t→t) + γb→b LRE(b→b)+γt→b LRE(t→b) + γb→t LRE(b→t) ,(6)

where LRE(t→t) and LRE(b→t) are the losses calculated by
Eq. (2) to input Xt and LRE(b→b) and LRE(t→b) are to(a) Online Handwriting(b) Offline Handwriting

Figure 3. Examples of images created from time series in the experiments.
In (a), pink indicates the beginning of the sequence.

input Xb . Also, γt→t , γb→b , γt→b , γb→t are weight of each
respective loss.
C. Space Sharing Loss
While the Cross-VAE is trained using the combination of
the reconstruction and distribution losses for the different
modalities, we propose the use of a space sharing loss
function to encourage the latent variable to share the samelatent space. The space sharing loss LLS gives the squareerror of the latent variable zt obtained from the online
character VAE and the latent variable zb of the offline
character VAE. Specifically:1
LLS = δ kzt − zb k2 ,2(7)where δ is a weight and k · k is the Euclidean norm.
IV. O NLINE AND O FFLINE C ONVERSION OF
H ANDWRITTEN C HARACTERS USING C ROSS -VAE
A. Dataset
For the experiment, we used handwritten uppercase
characters from the Unipen online handwritten character
dataset [25]. The online handwritten characters consist oftime series made of (x, y) coordinates. The online characters
were normalized to fit within a square bound by (0, 0)
and (1, 1). In order to use a second modality, the online
characters were rendered into images. The images were(a) Result using LSTM layers for the online encoder and decoder(b) Result using convolutional layers for the online encoder and decoder
Figure 4. Result of the Cross-VAE. Xb is the original image and Xt is the original time series. Yb→b and Yt→t are outputs of the Cross-VAE whichcorrespond to the same modalities and Yb→t and Yt→b are between different modalities. The illustrations of the time series, Xt , Yt→t , and Xb→t arecolored from pink to yellow according to their sequence order.

32 × 32 pixels with 0 as the background and 1 as theforeground. Examples of the image renderings can be foundin Fig. 3.
B. Architecture Details
The image-based encoder and decoder were constructedfrom a Convolutional Neural Network (CNN) with a similar
structure as a ConvDeconv network [26]. The image encoder
consists of four 3 × 3 convolutional layers with Rectified
Linear Unit (ReLU) activations and corresponding 2 × 2
maxpooling layers. The number of nodes are detailed inFig. 2. The decoder is a reflection of the encoder whichuses unpooling and deconvolutions. Between the convolutional layers, there exist three fully-connected layers. One
belonging to each, the encoder and decoder, and one for thelatent variable.
For the time series-based encoder and decoder, there
were two architectures chosen. The first is a CNN-based
approach with 1D convolutions and no pooling. The second
is a Recurrent Neural Network (RNN) approach using Long
Short Term Memory (LSTM) [27] layers. Both the CNNbased approach and the LSTM-based approach have threefully-connected layers, one for the encoder, one for thelatent variable, and one for the decoder. The two layer types
were chosen to compare the difference between the LSTM
layers which were designed specifically for time series andconvolutional layers which are traditionally used for images.

The Cross-VAE was optimized with RMSProp [28] for
200 epochs. The weighting factors of each loss function
were determined through experiments. Specifically, they are
α = 0.5, β = 0.5, γt→t = 0.4, γb→b = 0.5, γt→b = 0.4,γb→t = 0.2, δ = 1.0. The number of dimensions of thelatent variable was 32 in all experiments.
C. Conversion Result
The results of the Cross-VAE are shown in Fig. 4.
Fig. 4 (a) is from using LSTM layers for the online encoderand decoder and Fig. 4 (b) is from using convolutional layers
in the online encoder and decoder. The results Yb→b andYt→b are the images generated by the inputs Xt and Xb ,respectively. The results Yt→t and Yb→t are renderings ofthe time series colored from pink to yellow in chronologicalorder. Notably, the output Yb→t is the trajectory prediction
based on the image input Xb .
By examining Fig. 4, it can be seen that the mutual
conversion of the modalities was accurately performed.
This shows that the shared latent space learned by thesimultaneous encoding of Xb and Xt is able to accurately
represent both image data and time series data. In addition,
not only was the stroke trajectory inferred, the results showthat the shared latent space was able to encode temporal
information about what is expected from the characters. For
example, the “B” in Fig. 4 (a) is missing information, yet
the time series results Yb→t and Yt→t were able to restore
the character. The results from Fig. 4 qualitatively confirm

Table I
C ROSS -C ONVERSION E VALUATIONS

Cross-VAE (LSTM)
Cross-VAE (Conv)
Class Average

Figure 5. Multiple example results for the letter “A” using convolutional
layers for the online encoder and decoderthat the Cross-VAE is able to do mutual modality conversion
between the online and offline handwritten characters.
The letter “A” is another character that would normallybe difficult to recover lost time series information due tohaving multiple variations. In some cases, the left-moststroke is drawn downwards and in some, it is drawn upwards
depending on the author. Fig. 5 are examples of manydifferent “A”s generated by the Cross-VAE. The figure
shows that the Cross-VAE was able to correctly estimate
most of the strokes of the “A”s. In particular, the resultsfrom Yb→t was able to not only correctly predict the strokeorder but also was able to replicate the stroke velocity. Notethe stroke that crosses the center of the “A.” This further
enforces the success of the proposed Cross-VAE.
D. Quantitative Evaluation of Conversion
In order to evaluate the method quantitatively, we constructed the following three measures to determine thequality of the generated characters:
PSNR: Peak signal-to-noise ratio (PSNR) calculates thesimilarity between the input images and the generated output
images. PSNR is the ratio between the maximum luminance
MAX and the amount of noise, or:
MAX2
PSNR = 10 log10
,
(8)
MSE
where MSE is the mean squared error between Xb and Yt→b .
PSNR is measured in decibels (dB) with a larger value being
better.
SSIM: Structural Similarity (SSIM) predicts the perceived difference between images. Similar to PSNR, this acts
as a similarity measure between Xb and Yt→b . The equationfor SSIM is:(2µX µY+ C1 ) + (2σXb Yt→b + C2 )
 , (9)SSIM = 2 b 2t→b
2 + σ2µXb + µYt→b + C1 )(σX
Yt→b + C2bwhere C1 and C2 are stabilizing constants set to C1 =(0.01 × 255)2 and C2 = (0.03 × 255)2 . µ is the averageluminance, σ 2 is the variance, and σ is the covariance. SSIM
is a value from 0 to 1 with a larger value meaning moresimilar.

Yt→b
PSNR
SSIM
15.26
0.617
15.99
0.707
9.197
0.159

Yb→t
DTW
0.04110.0361
0.206

DTW: Dynamic time warping (DTW) was used as anevaluation for the time series generation as a method of
measuring the stroke trajectory estimation. DTW is a robust
distance measure between time series which uses dynamic
programming to optimally match sequence elements. In thiscase, we use the average DTW-distance between the input
time series Xt and the cross-modality output Xt→b . Smaller
the DTW-distances between Xt and Xt→b means that thepatterns are more similar and the Cross-VAE was able toreplicate the original input time series. Thus, a smaller value
is better.
The results of quantitative evaluations are shown in Table
I. In the table, we evaluate the difference between using
LSTM layers and convolutional layers in the time series
encoder and decoder. The results are compared to the images
and time series of the average pattern in each respectiveclass. PSNR and SSIM are used for the cross-modal conversion from Xt to Yt→b and DTW is used for the evaluationof the cross-modal conversion from Xb to Yb→t .
For online to offline handwritten character conversion, or
inking, the Cross-VAE did much better than the class average. In addition, the time series encoder and decoder withconvolutional layers performed better than the LSTM. Thisshows that, despite being time series data, the convolutional
layers were able to encode the information into the latent
space better than the LSTM layers.
Similarly, for the offline to online handwritten character
conversion, the Cross-VAE performed better than the average and the convolutional layer based time series encoderand decoder did better in reconstructing the time series.
The DTW results specifically demonstrate that the CrossVAE is able to predict the trajectories of the strokes. This
information is normally lost during the rendering, however,
the Cross-VAE is able to infer the stroke trajectory from theshared latent space.
Both evaluations found that using convolutional layers
was better than using LSTM layers. This is justified forthis data target because handwritten characters are spatial
coordinates where the relevance of every element depends
on its neighbors. Structured data such as this is well suited to
convolutional layers, whereas the advantages of maintaining
long-term dependencies in LSTMs is lost. We believe that
due to this, the convolutional layer based encoder anddecoder for the time series modality produces better results.

V. C ONCLUSION
In this paper, we proposed a VAE for mutual modality
conversion called a Cross-VAE. The Cross-VAE is madefrom the merging of two VAEs of different modalities by
enforcing a shared latent space. To train the Cross-VAE, we
propose using the combination of reconstruction loss anddistribution loss from the original VAE and an additional
space sharing loss. The space sharing loss encourages thedifferent modalities of the Cross-VAE to use the same latent
space embedding. In the experiments, we used online andoffline handwritten characters to verify the ability of the
Cross-VAE. The results show that the mutual conversion was
possible and that the proposed Cross-VAE could accurately
reconstruct the images and time series.
In the future, we will continue to improve the model andapply it to other applications. The Cross-VAE can be used
for other types of data and tackle other tasks. Furthermore,this work opens the way for embedding different modalities
into one shared latent space which can be used as a tool for
representing those modalities in one space.
ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant
Number JP17H06100.
R EFERENCES[1] H. Tanaka, K. Nakajima, K. Ishigaki, K. Akiyama, and
M. Nakagawa, “Hybrid pen-input character recognition system based on integration of online-offline recognition,” in
IAPR Int. Conf. Document Analysis and Recognition, 1999,
pp. 209–212.
[2] M. Hamanaka, K. Yamada, and J. Tsukumo, “On-line
japanese character recognition experiments by an off-line
method based on normalization-cooperated feature extraction,” in IAPR Int. Conf. Document Analysis and Recognition,
1993, pp. 204–207.
[3] V. Nguyen and M. Blumenstein, “Techniques for static handwriting trajectory recovery: A survey,” in IAPR Int. Workshop
on Document Analysis Systems, 2010.
[4] D. P. Kingma and M. Welling, “Auto-encoding variational
bayes,” in Int. Conf. Learning Representations, 2013.
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing
Systems, 2014, pp. 2672–2680.
[6] P. Baldi, “Autoencoders, unsupervised learning, and deeparchitectures,” in ICML Workshop Unsupervised and Transfer
Learning, 2012, pp. 37–49.
[7] W. Guo, J. Liang, X. Kong, L. Song, and R. He, “X-gacmn:
An x-shaped generative adversarial cross-modal network withhypersphere embedding,” in Asian Conf. Computer Vision,
2018.
[8] Y. Peng and J. Qi, “CM-GANs,” ACM Trans. Multimedia Computing, Communications, and Applications, vol. 15,
no. 1, pp. 1–24, feb 2019.
[9] A. Spurr, J. Song, S. Park, and O. Hilliges, “Cross-modal deep
variational hand pose estimation,” in IEEE Conf. Computer
Vision and Pattern Recognition, 2018.

[10] F. Huang, X. Zhang, C. Li, Z. Li, Y. He, and Z. Zhao,“Multimodal network embedding via attention based multiview variational autoencoder,” in ACM Int. Conf. Multimedia
Retrieval, 2018.
[11] I. V. Serban, A. G. Ororbia II, J. Pineau, and A. Courville,“Multi-modal variational encoder-decoders,” in Int. Conf.
Learning Representations, 2016.
[12] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired
image-to-image translation using cycle-consistent adversarial
networks,” in IEEE Int. Conf. on Computer Vision, 2017.
[13] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo,“StarGAN: Unified generative adversarial networks for multidomain image-to-image translation,” in IEEE Conf. on Computer Vision and Pattern Recognition, 2018.
[14] M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-toimage translation networks,” in Advances in Neural Information Processing Systems, 2017, pp. 700–708.
[15] V. Nguyen and M. Blumenstein, “Techniques for static handwriting trajectory recovery,” in IAPR Int. Workshop Document
Analysis Systems. ACM Press, 2010.
[16] A. K. Bhunia, A. Bhowmick, A. K. Bhunia, A. Konwer,
P. Banerjee, P. P. Roy, and U. Pal, “Handwriting trajectory
recovery using end-to-end deep encoder-decoder network,” inInt. Conf. Pattern Recognition, 2018.
[17] Y. Qiao, M. Nishiara, and M. Yasuhara, “A framework toward
restoration of writing order from single-stroked handwriting
image,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1724–1737, nov 2006.
[18] B. Zhao, M. Yang, and J. Tao, “Pen tip motion prediction
for handwriting drawing order recovery using deep neuralnetwork,” in Int. Conf. Pattern Recognition, 2018.
[19] O. Fabius and J. R. van Amersfoort, “Variational recurrent
auto-encoders,” in ICLR Workshop, 2014.
[20] K. Akuzawa, Y. Iwasawa, and Y. Matsuo, “Expressive speechsynthesis via modeling expressions with variational autoencoder,” in Interspeech, 2018.
[21] S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz,
and S. Bengio, “Generating sentences from a continuous
space,” in SIGNLL Conf. Computational Natural Language
Learning. Association for Computational Linguistics, 2016.
[22] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and
L. Carin, “Variational autoencoder for deep learning of images, labels and captions,” in Advances in Neural Information
Processing Systems, 2016, pp. 2352–2360.
[23] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling,“Semi-supervised learning with deep generative models,” inAdvances in Neural Information Processing Systems, 2014,
pp. 3581–3589.
[24] Z. Wan, Y. Zhang, and H. He, “Variational autoencoder based
synthetic data generation for imbalanced learning,” in IEEE
Symposium Series Computational Intelligence, 2017, pp. 1–7.
[25] I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, andS. Janet, “UNIPEN project of on-line data exchange andrecognizer benchmarks,” in Int. Conf. on Pattern Recognition,
vol. 2, 1994, pp. 29–33.
[26] H. Noh, S. Hong, and B. Han, “Learning deconvolution
network for semantic segmentation,” in IEEE Int. Conf.
Computer Vision, 2015.
[27] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[28] G. Hinton, N. Srivastava, and K. Swersky, “Neural networks
for machine learning lecture 6a overview of mini-batch gradient descent.”