# An Integrated Deep Generative Model for Text Classification and Generation

Authors: Zheng Wang; Qingbiao Wu
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Text classification and generation are two important tasks in the field of natural language processing. In this paper, we deal with both tasks via Variational Autoencoder, which is a powerful deep generative model. The self-attention mechanism is introduced to the encoder. The modified encoder extracts the global feature of the input text to produce the hidden code, and we train a neural network classifier based on the hidden code to perform the classification. On the other hand, the label of the text is fed into the decoder explicitly to enhance the categorization information, which could help with text generation. The experiments have shown that our model could achieve competitive classification results and the generated text is realistic. Thus the proposed integrated deep generative model could be an alternative for both tasks.


---

## Body

## 1. Introduction

Text classification is one of the most basic and important tasks in the field of natural language processing, in which one should assign predefined categories to the text. The result of classification is often used as the input for other tasks; thus an efficient and accurate classification algorithm is of great benefit. Traditional classification models combined with traditional representation of the text such as support vector machine base on the bag-of-word vectors or other classical methods have been able to achieve good results in some simple application scenarios [1–4].In recent years, deep learning models based on neural networks have achieved remarkable results in various tasks, such as computer vision [5] and speech recognition [6]. Obviously, the above models have also played a significant role in the field of natural language processing. A considerable part of the task is based on the word vector representations, which are learned through neural language models [7–10]. These word vector representations, also known as word embeddings, are transformed by a deep neural network such as convolutional neural network (CNN) or recurrent neural network (RNN) to obtain more abstract features about the text [11–14]. Thus we could perform the text classification based on these features. Such methods often lead to more advanced results on larger dataset[15].On the other hand, text generation is also an important task of concern. Many works about neural language models or sequence to sequence models have been made to improve the quality of the generated text for a variety of purposes [16–18]. Compared with other models, deep generative models have stronger expressive power, have the potential to handle more types of data, and have the natural advantage of being able to generate sample from the models, indicating their potential to deal with the above task.Variational Autoencoder (VAE), which is a powerful deep generative model, has attracted the attention of many researchers in recent years [19, 20]. It consists of a probability encoder and a probability decoder and takes advantages of the variational inference. A variational lower bound is optimized instead of the traditional log-likelihood.Variational Autoencoder has proved its ability, both in theory and in practice, thus could be chosen to perform the text generation task. What is more, the encoder of the VAE could be regarded as a feature extractor, which could be utilized to perform the classification task. In this paper, we propose an integrated model based on VAE to handle both generation and classification tasks. Although VAE has shown impressive advances in visual domain, such as image classification and image generation, its application in natural language processing has been relatively less studied [21]. In our work, we use a modified version of VAE to extract global features about the text, while using the text label to facilitate the generation of text during the decoding stage. The influence of the label information is enhanced by explicitly feeding label to the decoder at each time step. For text classification, we train a classifier based on neural network and integrate it with the VAE. Experiments have shown that our model can handle text classification and text generation problems at the same time and could achieve ideal results.The remainder of the paper is structured as follows. In Section2 we review the important architecture of VAE, which is the foundation of our method and experiments. In Section 3 we introduce the basic RNN structure. In Section 4 we elaborate the details of the proposed method. In Section 5 we introduce some related works. We provide the experimental evaluation and show the results in Section 6. Finally we make a conclusion in Section 7.


## 2. Review of VAE

In this section, we review the basic VAE model as the foundation of our work.Variational methods provide an optimization-based alternative to the sampling-based Monte Carlo methods in learning the complex latent variables models. Variational methods try to approximate the true posterior distribution by minimizing the Kullback-Leibler divergence between the true posterior distribution and a simple probability distribution which may be predefined and tractable. For instance, mean-field variational method [22] approximate the true posterior distribution with a fully factorized set of distributions. Recently, some stochastic variational inference methods have been proposed to update the variational parameters directly by sampling from the variational posterior distribution [23–25].Variational Autoencoder (VAE)[19, 26] has proved to be one of the most successful probabilistic generative models, which combines the variational learning framework and deep neural network. It could be regarded as a generative model that is based on a regularized version of the standard autoencoder.In VAE, the deterministic encoderφenc is replaced by a learned posterior recognition model qϕ(z∣x). Similarly, there is a probabilistic decoder pθ(x∣z).The objective function is the same as the Evidence Lower Bound (ELBO) in variational learning, which takes the following form:(1)
log
⁡
p
θ
x
=
log
⁡
∑
z
p
θ
x
,
z
=
log
⁡
∑
z
q
ϕ
z
∣
x
p
θ
x
,
z
q
ϕ
z
∣
x
≥
∑
z
q
ϕ
z
∣
x
log
⁡
p
θ
x
,
z
q
ϕ
z
∣
x
=
∑
z
q
ϕ
z
∣
x
log
⁡
p
θ
x
,
z
+
H
q
=
E
L
B
O where H(·)is the entropy.The ELBO could be reformulated as(2)-DKLqϕz∣x∥pθx+Eqϕz∣xlog⁡pθx∣zIn practice, the prior over the latent variables is usually chosen to be the centered isotropic multivariate Gaussianpθ(z)=N(z;0,I). The variational approximate posterior is chosen to be a multivariate Gaussian with a diagonal covariance structure whose distribution parameters are computed from z with a fully connected neural network with a single hidden layer: (3)log⁡qϕz∣xi=log⁡Nz;μi,σ2iIwhereμ(i) and σ(i) are outputs of the encoding MLP.In the traditional stochastic variational learning framework, the Evidence Lower Bound is optimized with respect to both the generative parametersθ and the variational parameters ϕ. The gradient with respect to θ could be calculated by (4)∇θLx=Eqϕz∣x∇θlog⁡pθx,z(5)∇θLx≈1n∑i=1n∇θlog⁡pθx,ziwhile the gradient with respect to ϕ could be calculated by (6)∇ϕLx=Eqϕz∣x∇θlog⁡pθx,z-log⁡qϕz∣x=Eqϕz∣xlog⁡pθx,z-log⁡qϕz∣x∇ϕlog⁡qϕz∣x(7)∇ϕLx≈1n∑i=1nlog⁡pθx,zi-log⁡qϕzi∣x×∇ϕlog⁡qϕzi∣xUnfortunately, this gradient estimator exhibits high variance, and some other methods should be used to alleviate it.In VAE, the so-called reparameterization trick is used to solve the above problem. Assume thatz is a continuous random variable and could be sampled by z~qϕ(z∣x). Then z sometimes could be expressed as a deterministic variable z=gϕ(ϵ,x), where ϵ is an auxiliary variable with independent marginal p(ϵ) and gϕ(·) is some vector valued function parameterized by ϕ.Givenz=gϕ(ϵ,x), we have (8)qϕz∣x∏idzi=pϵ∏idϵi(9)∫qϕz∣xfzdz=∫pϵfzdϵ=∫pϵfgϕϵ,xdϵ;thus we could construct the estimator by (10)∫qϕz∣xfzdz≃1L∑l=1Lfgϕϵl,x where ϵ(l)~p(ϵ).For instance, assume thatz~p(z∣x)=N(μ,σ2). Then z could be reparameterized by z=μ+σϵ, where ϵ~N(0,1). We have (11)ENz;μ,σ2fz=ENϵ;0,1fμ+σϵ≃1L∑l=1Lfμ+σϵl where ϵ(l)~N(0,1)


## 3. Recurrent Neural Network

Neural networks have been applied to a variety of tasks in the field of natural language processing. In particular, recurrent neural networks with long short-term memory (LSTM) [27, 28] cells or gated recurrent units (GRU)[29] have proven successful at tasks including machine translation [16–18], machine comprehension, and many others. These models are especially suitable for handling sequential data.Due to the vanishing and exploding gradient problems, it is widely believed that learning long-range dependencies with recurrent neural networks is challenging. To deal with this issue, LSTM introduces a more complex interaction structure. The recurrent computations in LSTM can be represented by(12)ft=σWf·ht-1,xt+bf(13)it=σWi·ht-1,xt+bi(14)C~t=tanh⁡WC·ht-1,xt+bC(15)Ct=ft∗Ct-1+it∗C~t(16)ot=σWo·ht-1,xt+bo(17)ht=ot∗tanh⁡Ctwhere xt is the input, ht is the cell’s state, Ct is the cell’s memory, C~t is the cell’s candidate memory, ft,it,ot are the state of the corresponding gate, and Wf,Wi,WC,Wo,bf,bi,bC,bo are parameters of the cell.It should be noted that there are many variants of LSTM, but throughout this paper, we always use the standard LSTM.


## 4. Details of the Model

In this section, we introduce the structure of the model in detail.


### 4.1. The Probabilistic Encoder

In VAE, the encoder would encode the input into the hidden code. To handle text data, we use LSTM as the main part of the encoder. Given a sequence consisting of N words{x1,x2,…,xN}, we first convert each word into an embedding vector wi∈Rdemb. The embeddings are encoded by column vectors in an embedding matrix Wemb∈Rdemb×V, where demb is the dimension of the embedding and V is the size of the vocabulary. Each column corresponds to the embedding of the i-th word in the vocabulary. The matrix Wemb is a parameter to be learned and the dimension of the embedding is a hyperparameter to be chosen by the user.The embeddings of the words are then fed into the LSTM cell step by step, and finally we get the hidden stateh1,h2,…,hN of all the N timesteps. Unlike the vanilla sequential VAE[21], here we prefer to use bidirectional LSTM and introduce the self-attention mechanism to more efficiently extract the information from the entire text. The variational posterior represented by the encoder would be a multivariate Gaussian with a diagonal covariance structure, where the mean and the standard deviation of the posterior are set to be the outputs of the MLP. Formally, we have (18)Hfw=LSTMencx1,x2,…,xN;ϕenc(19)Hbw=LSTMencxN,xN-1,…,x1;ϕenc(20)Hfull=Hfw;Hbw(21)a=softmaxws2tanh⁡Ws1HfullT(22)hC=aTHfull(23)μ=sigmoidW0hC+b0(24)σ=sigmoidW1hC+b1(25)qz∣x=Nz;μ,σ2Iwhere ϕenc,Ws1,ws2,W0,W1,b0,b1 are parameters to be learned. In the above procedure, Hfw,Hbw∈RN×hdim represent all the hidden state of the forward and backward directions of LSTM. Hfull∈RN×2hdim is the concatenation of Hfw and Hbw. a∈RN is the weight vector used in self-attention mechanism. hC∈R2hdim is the weighted average of all the hidden state, also called the context state vector.


### 4.2. The Text Classifier

Since text classification is one of our aims, we can use the hidden codez as the extracted abstract feature of text. Ideally, we assume that the hidden code contains important information about the input text, such as semantics, length, and sentiment. Based on these features, we can use traditional classifiers, such as SVM and AdaBoost to classify texts. It is important to note that the categorization of text is only one of our goals and our other goal is the generation of text, while the class information about the text helps the latter. Therefore, we consider integrating the classifier into the VAE model and train the neural network as classifier.The hidden code is fed into a fully connected layer followed by a softmax layer, and the output is a vector indicating the probability of each class. Formally, we have(26)qcy=softmaxWcz+bc where Wc,bc are parameters to be learned.


### 4.3. The Probabilistic Decoder

The decoder would generate the word sequentially, conditioned on the information provided by the encoder. We again use LSTM as the main part of the decoder. The procedure of word embedding remains unchanged. Decoding of timestept could be expressed as (27)ht=LSTMdecx1,x2,…,xt-1,z;θdec(28)pxt=softmaxWoht+bo(29)xt^~pxtwhere θdec,Wo,bo are parameters to be learned.Recall that we have the categorization information for the text. Thus we may use the label to help controlling the generation progress. A natural idea is to condition on both the hidden code and the label at the beginning of the decoding procedure. Unfortunately, this often does not achieve good performance in practice. As the sequence gets longer, the signal provided by the class label gets weaker rapidly. To make full use of the label, we concatenate on the word embedding and the one-hot label vector at each timestep, and the hidden state of each timestep could be represented by(30)ht=LSTMdecx1,x2,…,xt-1,z,y;θdec The remaining steps are the same as before.


### 4.4. The Objective Function

As our model needs to take into account both text classification and generation, the objective function we need to optimize also consists of two parts. Given a labeled data pair(x,y), we have (31)pθx∣z,y=∏tpθxt∣x<t,z,y and the Evidence Lower Bound could be expressed as (32)log⁡pθx≥Eqϕz∣xlog⁡pθx∣z,y-DKLqϕz∣x∥pθz=-Lgx where DKL(q∥p) is the Kullback-Leibler divergence between the distribution q and p.The loss of the classifier could be expressed as(33)Lcx,y=-∑ypeylog⁡qcy∣x where pe(y) represents the empirical distribution of the ground truth label.The final objective function for the entire dataset is(34)Ltotal=∑x,yLgx+αLcx,y where α is the hyperparameter controlling the trade-off between these two parts.The reparameterization trick and sampling method could be used to calculate the gradient with respect to each parameter. Then the optimization would be applied.The framework of our model is shown in Figure1.Figure 1
The framework of the proposed model.


## 4.1. The Probabilistic Encoder

In VAE, the encoder would encode the input into the hidden code. To handle text data, we use LSTM as the main part of the encoder. Given a sequence consisting of N words{x1,x2,…,xN}, we first convert each word into an embedding vector wi∈Rdemb. The embeddings are encoded by column vectors in an embedding matrix Wemb∈Rdemb×V, where demb is the dimension of the embedding and V is the size of the vocabulary. Each column corresponds to the embedding of the i-th word in the vocabulary. The matrix Wemb is a parameter to be learned and the dimension of the embedding is a hyperparameter to be chosen by the user.The embeddings of the words are then fed into the LSTM cell step by step, and finally we get the hidden stateh1,h2,…,hN of all the N timesteps. Unlike the vanilla sequential VAE[21], here we prefer to use bidirectional LSTM and introduce the self-attention mechanism to more efficiently extract the information from the entire text. The variational posterior represented by the encoder would be a multivariate Gaussian with a diagonal covariance structure, where the mean and the standard deviation of the posterior are set to be the outputs of the MLP. Formally, we have (18)Hfw=LSTMencx1,x2,…,xN;ϕenc(19)Hbw=LSTMencxN,xN-1,…,x1;ϕenc(20)Hfull=Hfw;Hbw(21)a=softmaxws2tanh⁡Ws1HfullT(22)hC=aTHfull(23)μ=sigmoidW0hC+b0(24)σ=sigmoidW1hC+b1(25)qz∣x=Nz;μ,σ2Iwhere ϕenc,Ws1,ws2,W0,W1,b0,b1 are parameters to be learned. In the above procedure, Hfw,Hbw∈RN×hdim represent all the hidden state of the forward and backward directions of LSTM. Hfull∈RN×2hdim is the concatenation of Hfw and Hbw. a∈RN is the weight vector used in self-attention mechanism. hC∈R2hdim is the weighted average of all the hidden state, also called the context state vector.


## 4.2. The Text Classifier

Since text classification is one of our aims, we can use the hidden codez as the extracted abstract feature of text. Ideally, we assume that the hidden code contains important information about the input text, such as semantics, length, and sentiment. Based on these features, we can use traditional classifiers, such as SVM and AdaBoost to classify texts. It is important to note that the categorization of text is only one of our goals and our other goal is the generation of text, while the class information about the text helps the latter. Therefore, we consider integrating the classifier into the VAE model and train the neural network as classifier.The hidden code is fed into a fully connected layer followed by a softmax layer, and the output is a vector indicating the probability of each class. Formally, we have(26)qcy=softmaxWcz+bc where Wc,bc are parameters to be learned.


## 4.3. The Probabilistic Decoder

The decoder would generate the word sequentially, conditioned on the information provided by the encoder. We again use LSTM as the main part of the decoder. The procedure of word embedding remains unchanged. Decoding of timestept could be expressed as (27)ht=LSTMdecx1,x2,…,xt-1,z;θdec(28)pxt=softmaxWoht+bo(29)xt^~pxtwhere θdec,Wo,bo are parameters to be learned.Recall that we have the categorization information for the text. Thus we may use the label to help controlling the generation progress. A natural idea is to condition on both the hidden code and the label at the beginning of the decoding procedure. Unfortunately, this often does not achieve good performance in practice. As the sequence gets longer, the signal provided by the class label gets weaker rapidly. To make full use of the label, we concatenate on the word embedding and the one-hot label vector at each timestep, and the hidden state of each timestep could be represented by(30)ht=LSTMdecx1,x2,…,xt-1,z,y;θdec The remaining steps are the same as before.


## 4.4. The Objective Function

As our model needs to take into account both text classification and generation, the objective function we need to optimize also consists of two parts. Given a labeled data pair(x,y), we have (31)pθx∣z,y=∏tpθxt∣x<t,z,y and the Evidence Lower Bound could be expressed as (32)log⁡pθx≥Eqϕz∣xlog⁡pθx∣z,y-DKLqϕz∣x∥pθz=-Lgx where DKL(q∥p) is the Kullback-Leibler divergence between the distribution q and p.The loss of the classifier could be expressed as(33)Lcx,y=-∑ypeylog⁡qcy∣x where pe(y) represents the empirical distribution of the ground truth label.The final objective function for the entire dataset is(34)Ltotal=∑x,yLgx+αLcx,y where α is the hyperparameter controlling the trade-off between these two parts.The reparameterization trick and sampling method could be used to calculate the gradient with respect to each parameter. Then the optimization would be applied.The framework of our model is shown in Figure1.Figure 1
The framework of the proposed model.


## 5. Related Works

In recent years, VAE has proved to be a powerful deep generative model. The first work to combine VAE with text generation is [21]. They use the vanilla LSTM as both the encoder and the decoder. They introduced some tricks for training VAE such as word drop and KL annealing, which we also used in our work. But their work do not include the classification task. At the same time, their model cannot generate text for a given category by controlling the label.There are some similar works on the same topic such as [36, 37]. In [36], they use the vanilla LSTM as the encoder of VAE, and the hidden code of the VAE is transformed from the hidden state of the last timestep in the LSTM. This means that the LSTM should compress the information into the last hidden state as much as possible, which is hard due to the sequential structure of the vanilla LSTM. In fact, the information of the previous timesteps is often lost. In contrast, our model use bidirectional LSTM as the encoder, which could capture more contextual information than the vanilla one. More importantly, we introduced the self-attention mechanisms [38, 39]; the final context state vector would be derived from all the hidden state in the LSTM rather than just the last hidden state. The extracted feature would contain more global information and could be used to improve the performance of both the classification and generation tasks. What is more, although they use a modified version of CNN-based decoder, they do not explicitly take advantage of the label during decoding stage, and the categorical signal may not be strong enough to guide the generation procedure, while our method explicitly uses the label information during decoding and it would help controlling the generation progress. In [37], they also use the vanilla LSTM as encoder, which would be less robust and less efficient.


## 6. Experiments

In this section we show the experimental results on several datasets to demonstrate the performance of the proposed method. First we would introduce the dataset we used in the experiments; then we would describe the details of the experiments; finally we would show the results of the experiment from two perspectives, text classification and text generation.


### 6.1. Datasets

We performed the model on three different benchmarks: IMDB dataset, SST1, and SST2.The IMDB dataset is a benchmark for sentiment classification [30]. The task is to determine if the movie reviews are positive or negative. The Stanford Sentiment Treebank (SST) dataset consists of movie reviews with one sentence per review [35]. In SST1, the fine-grained labels are provided, including very positive, positive, neutral, negative, and very negative. The SST2 dataset is the same as SST1 but with neutral reviews removed and binary labels. Details about the dataset are shown in Table 1.Table 1
Details about the used datasets.
Dataset number of classes training validation testing IMDB 2 25000 10000 15000 SST1 5 8544 1101 2210 SST2 2 6920 872 1821


### 6.2. Experimental Setup

For all the experiments, the vocabulary size is set to 15000; the word out of the dictionary is replaced by the<unk> token. In practice, it is unlikely to handle sentences of arbitrary length; thus we would set the max length of the sentence to be 35. We would truncate the sentence if the length exceeds the maximum, and we would pad the sentence with the <pad> token to make the length of sentences in each minibatch consistent. During the decoding procedure, we would add the <bos> and <eos> token at the beginning and end of the sentence, respectively.We use the pretrained GloVe vectors [40] with 300 dimensions as the initialization of the word embedding. During training, these embeddings are optimized as part of the parameters of the model. The hidden size of the LSTM cell is set to 128. The size of the hidden code z is also set to 128. The batch size is set to 64. The classifier is a neural network with two hidden layers, and the size of each layer is set to 200 and 400.We implemented the model with Tensorflow. The model is trained end-to-end using the ADAM optimizer, with learning rate of4e-3.In order to better train the VAE model, the cost annealing trick is adopted to smooth the training by gradually increasing the weight of the KL divergence from zero to one. With this trick, we would avoid vanishingly small KL term in the VAE module. We also use word dropout to regularize the model. In other words, the tokens fed into the decoder would be replaced by the<unk> token with a certain probability. We set this ratio to 0.25 during training, while in the decoding stage, this trick would not be used.All the other hyperparameters, such as the trade-off between the two terms in the cost function, are chosen based on the performance on the development set.


### 6.3. Classification Performance

To evaluate the proposed model, we first train a SVM classifier based on the bag of words feature as a baseline model on each dataset. These models are implemented by LIBSVM. We also compared our model with a few previous best methods.The classification results on IMDB dataset are shown in Table2. Some previous best supervised results are contained. For instance, the NB-LM bow3 method first generates binary bag of n-gram vectors, multiplies the component for each n-gram with the NB weight, and then trains a logistic regression classifier.Table 2
The classification error rate (%) on the IMDB dataset of each method. Lower is better.
Methods Error rate on IMDB SVM + bow(baseline) 11.55 WRRBM+bow[30] 10.77 SVM+bow2+NB[31] 8.78 NB-LM + bow3[32] 8.13 LSTM 13.50 VAE+NN(ours) 8.86We found that although our model does not outperform some best methods, the proposed model could still get a competitive result, which indicates that the feature extracted by the encoder is meaningful. We emphasize that the best models are specially optimized for classification task, while our model also needs to consider the task of text generation. Therefore, simply using the result of classification as the basis of judgement does not fully demonstrate the ability of our model.We also carried out a series of similar experiments on the SST dataset. The results are shown on Table3. This time we compared some best methods based on neural networks, such as the Recursive Autoencoders (RAE) with pretrained word vectors from Wikipedia, Matrix-Vector Recursive Neural Network (MV-RNN) with parse trees, Recursive Neural Tensor Network (RNTN) with tensor based feature function and parse trees, and Dynamic Convolutional Neural Network (DCNN) with k-max pooling. Once again, we found that our model far outperforms the baseline and could achieve competitive results.Table 3
The classification error rate (%) on the SST dataset of each method. Lower is better.
Methods Error rate on SST1 Error rate on SST2 SVM + bow(baseline) 61.4 22.8 RAE[33] 56.8 17.6 MV-RNN[34] 55.6 17.1 RNTN[35] 54.3 14.6 DCNN[13] 51.5 12.2 VAE+NN(ours) 55.3 16.9


### 6.4. Generation Performance

We then conducted additional experiments to demonstrate the ability of our model on text generation. In order to make full use of the label information to generate the corresponding category of text, we explicitly feed the label into the decoder. In other words, the word embedding and the one-hot label vector are concatenated at each timestep as the input.We used the model trained on the IMDB dataset to generate text corresponding to different categories. The results are shown on Table4. We found that the generated text is realistic while corresponding the correct label, that is, including different positive and negative emotions.Table 4
Samples of the text learned from the IMDB dataset.
Positive samples:  Lynch has a similar sense of humor in the movie The music score itself is somewhat touching memorable He left a deep impression on us and more elements would describe the movie This movie did get serious as well and had some really emotional moments as well It was entertaining and I still had great fun in watching it  Negative samples:  Imagination is not enough Honestly those fans would not like to see some aspects of movies It didn’t really stick out that much The CG is not advanced enough to make the animated dogs real This is only a disturbing, superficial entertainmentTo make the experiments more complete, we also evaluated the performance of text generation on SST dataset. The samples are shown on Table5. We found that the generated text is shorter and simpler compared with the text generated from the IMDB dataset. This is because the IMDB dataset is larger and our model would learn more complex structures. The results have demonstrated that our model could consistently achieve good generative performance.Table 5
Samples of the text learned from the SST dataset.
Positive samples:  Emotional movie Theatrical foreign love The film is the very good story She was definitely a unique film He does a wonderful performance as well  Negative samples:  This movie is not worth the fare It is very disappointing Empowerment life into a depression era There are many meaningless scenes There was too much of what we have seen before


## 6.1. Datasets

We performed the model on three different benchmarks: IMDB dataset, SST1, and SST2.The IMDB dataset is a benchmark for sentiment classification [30]. The task is to determine if the movie reviews are positive or negative. The Stanford Sentiment Treebank (SST) dataset consists of movie reviews with one sentence per review [35]. In SST1, the fine-grained labels are provided, including very positive, positive, neutral, negative, and very negative. The SST2 dataset is the same as SST1 but with neutral reviews removed and binary labels. Details about the dataset are shown in Table 1.Table 1
Details about the used datasets.
Dataset number of classes training validation testing IMDB 2 25000 10000 15000 SST1 5 8544 1101 2210 SST2 2 6920 872 1821


## 6.2. Experimental Setup

For all the experiments, the vocabulary size is set to 15000; the word out of the dictionary is replaced by the<unk> token. In practice, it is unlikely to handle sentences of arbitrary length; thus we would set the max length of the sentence to be 35. We would truncate the sentence if the length exceeds the maximum, and we would pad the sentence with the <pad> token to make the length of sentences in each minibatch consistent. During the decoding procedure, we would add the <bos> and <eos> token at the beginning and end of the sentence, respectively.We use the pretrained GloVe vectors [40] with 300 dimensions as the initialization of the word embedding. During training, these embeddings are optimized as part of the parameters of the model. The hidden size of the LSTM cell is set to 128. The size of the hidden code z is also set to 128. The batch size is set to 64. The classifier is a neural network with two hidden layers, and the size of each layer is set to 200 and 400.We implemented the model with Tensorflow. The model is trained end-to-end using the ADAM optimizer, with learning rate of4e-3.In order to better train the VAE model, the cost annealing trick is adopted to smooth the training by gradually increasing the weight of the KL divergence from zero to one. With this trick, we would avoid vanishingly small KL term in the VAE module. We also use word dropout to regularize the model. In other words, the tokens fed into the decoder would be replaced by the<unk> token with a certain probability. We set this ratio to 0.25 during training, while in the decoding stage, this trick would not be used.All the other hyperparameters, such as the trade-off between the two terms in the cost function, are chosen based on the performance on the development set.


## 6.3. Classification Performance

To evaluate the proposed model, we first train a SVM classifier based on the bag of words feature as a baseline model on each dataset. These models are implemented by LIBSVM. We also compared our model with a few previous best methods.The classification results on IMDB dataset are shown in Table2. Some previous best supervised results are contained. For instance, the NB-LM bow3 method first generates binary bag of n-gram vectors, multiplies the component for each n-gram with the NB weight, and then trains a logistic regression classifier.Table 2
The classification error rate (%) on the IMDB dataset of each method. Lower is better.
Methods Error rate on IMDB SVM + bow(baseline) 11.55 WRRBM+bow[30] 10.77 SVM+bow2+NB[31] 8.78 NB-LM + bow3[32] 8.13 LSTM 13.50 VAE+NN(ours) 8.86We found that although our model does not outperform some best methods, the proposed model could still get a competitive result, which indicates that the feature extracted by the encoder is meaningful. We emphasize that the best models are specially optimized for classification task, while our model also needs to consider the task of text generation. Therefore, simply using the result of classification as the basis of judgement does not fully demonstrate the ability of our model.We also carried out a series of similar experiments on the SST dataset. The results are shown on Table3. This time we compared some best methods based on neural networks, such as the Recursive Autoencoders (RAE) with pretrained word vectors from Wikipedia, Matrix-Vector Recursive Neural Network (MV-RNN) with parse trees, Recursive Neural Tensor Network (RNTN) with tensor based feature function and parse trees, and Dynamic Convolutional Neural Network (DCNN) with k-max pooling. Once again, we found that our model far outperforms the baseline and could achieve competitive results.Table 3
The classification error rate (%) on the SST dataset of each method. Lower is better.
Methods Error rate on SST1 Error rate on SST2 SVM + bow(baseline) 61.4 22.8 RAE[33] 56.8 17.6 MV-RNN[34] 55.6 17.1 RNTN[35] 54.3 14.6 DCNN[13] 51.5 12.2 VAE+NN(ours) 55.3 16.9


## 6.4. Generation Performance

We then conducted additional experiments to demonstrate the ability of our model on text generation. In order to make full use of the label information to generate the corresponding category of text, we explicitly feed the label into the decoder. In other words, the word embedding and the one-hot label vector are concatenated at each timestep as the input.We used the model trained on the IMDB dataset to generate text corresponding to different categories. The results are shown on Table4. We found that the generated text is realistic while corresponding the correct label, that is, including different positive and negative emotions.Table 4
Samples of the text learned from the IMDB dataset.
Positive samples:  Lynch has a similar sense of humor in the movie The music score itself is somewhat touching memorable He left a deep impression on us and more elements would describe the movie This movie did get serious as well and had some really emotional moments as well It was entertaining and I still had great fun in watching it  Negative samples:  Imagination is not enough Honestly those fans would not like to see some aspects of movies It didn’t really stick out that much The CG is not advanced enough to make the animated dogs real This is only a disturbing, superficial entertainmentTo make the experiments more complete, we also evaluated the performance of text generation on SST dataset. The samples are shown on Table5. We found that the generated text is shorter and simpler compared with the text generated from the IMDB dataset. This is because the IMDB dataset is larger and our model would learn more complex structures. The results have demonstrated that our model could consistently achieve good generative performance.Table 5
Samples of the text learned from the SST dataset.
Positive samples:  Emotional movie Theatrical foreign love The film is the very good story She was definitely a unique film He does a wonderful performance as well  Negative samples:  This movie is not worth the fare It is very disappointing Empowerment life into a depression era There are many meaningless scenes There was too much of what we have seen before


## 7. Conclusion

In this paper, we have proposed an integrated deep generative model to deal with both the text classification and generation. We use bidirectional LSTM and introduce the self-attention mechanism to enhance the encoder of VAE. We then extract the feature of the text and use the neural network classifier to perform the classification based on the global feature. What is more, the categorization information is explicitly fed into the decoder to help controlling the generation progress. The experiments have shown that our model could achieve competitive results on both tasks. While our work has enhanced the encoder of VAE for better performance, we have not modified the main structure of the decoder. In fact, some new mechanisms can be introduced to enhance the decoder, such as coverage probability. What is more, some constraint functions could be introduced to produce more controllable text after modifying the objective function. We leave them as our future work.



---

*Source: 7529286-2018-08-19.xml*