SlowGuess/ACL_2020_OCR huggingface dataset:
Yu Duan $^{1*}$ , Canwen Xu $^{2*}$ , Jiaxin Pei $^{3*}$ , Jialong Han $^{4\dagger}$ , Chenliang Li $^{2‡}$
<sup>1</sup> Alibaba Group, China <sup>2</sup> Wuhan University, China
<sup>3</sup> University of Michigan, United States <sup>4</sup> Amazon, United States
<sup>1</sup>derrick.dy@alibaba-inc.com, <sup>2</sup>xucanwen, cllee}@whu.edu.cn
<sup>3</sup> pedropei@umich.edu, <sup>4</sup> jialonghan@gmail.com
Conditional Text Generation has drawn much attention as a topic of Natural Language Generation (NLG) which provides the possibility for humans to control the properties of generated contents. Current conditional generation models cannot handle emerging conditions due to their joint end-to-end learning fashion. When a new condition added, these techniques require full retraining. In this paper, we present a new framework named Pre-train and Plug-in Variational Auto-Encoder (PPVAE) towards flexible conditional text generation. PPVAE decouples the text generation module from the condition representation module to allow "one-to-many" conditional generation. When a fresh condition emerges, only a lightweight network needs to be trained and works as a plug-in for PPVAE, which is efficient and desirable for real-world applications. Extensive experiments demonstrate the superiority of PPVAE against the existing alternatives with better conditionality and diversity but less training effort. $^{1}$
Currently, neural generation techniques have powered many inspiring applications, e.g., poem generation (Yang et al., 2018), neural machine translation (NMT) (Bahdanau et al., 2015) and chatbot (Zhao et al., 2017). Conditional (also known as controllable) text generation is an important task of text generation, aiming to generate realistic text that carries a specific attribute (e.g., positive or negative sentiment). A common solution is to encode the condition into a vector representation and then integrate it with the text generation process (Kingma
et al., 2014; Hu et al., 2017; Mirza and Osindero, 2014). These existing neural models have achieved encouraging results. However, when a new condition is added (e.g., a new topic for categorical generation), they require a full retraining or finetuning. This process is both time-consuming and computationally inefficient (Houlsby et al., 2019). Both fine-tuning and retraining are not desirable in real-world applications since the delivery (e.g., transmitting updated weights through the Internet) and client-side re-deployment (e.g., distribute updated weights to users) of large-scale weights are often difficult.
Inspired by the recent success of Variational Auto-Encoder (VAE) (Kingma and Welling, 2014) based post-hoc conditional image generation strategy (Engel et al., 2018), we provide a new perspective for flexible conditional text generation. We propose Pre-train and Plug-in Variational AutoEncoder (PPVAE), which decouples the text generation module from the condition representation module. PPVAE is a hierarchical framework composed of two VAEs: (1) PRETRAINVAE, which derives the global latent space of text with its encoder (pre-trained global encoder) and learns to generate text based on an easily-accessible large unlabeled dataset with its decoder (pre-trained global decoder); (2) PLGINVAE, which is a lightweight neural network that learns to transform vectors from the conditional latent space to the global latent space, and vice versa. This mapping function can be easily learned with only a few conditional training samples. In this sense, once we transform a latent variable (also known as latent code) randomly sampled from the conditional space distribution to the global space, the pre-trained global decoder is directly adopted for generation. In other words, whenever a new condition emerges, we only need to train a PLGINVAE and directly plug it into the framework.
* The first three authors contribute equally to this paper.  
† Work done when Jialong Han was with Tencent AI Lab.  
‡ Chenliang Li is the corresponding author.
<sup>1</sup>The code is available at https://github.com/WHUIR/PPVAE.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 253-262 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Different from the existing end-to-end neural models (Mirza and Osindero, 2014; Sohn et al., 2015; Kingma et al., 2014), PPVAE focuses on the learning of pure transformation between the continuous latent spaces, instead of the tricky discrete text generation. Once trained, PRETRAINVAE is fixed for text representation and generation under all conditions. Our proposed framework decouples the conditional space learning from the text generation, endowing PPVAE with more flexibility when handling emerging conditions. Also, training only a small conditional network for latent space transformation is much more efficient than co-training with the text generation. Additionally, we can easily increase the capability of generation using a larger corpus or deeper neural networks for text encoding and decoding. Our main contributions can be summarized as follows: (1) We propose a novel framework, PPVAE, for conditional text generation, which allows a separate training for a new condition without retraining the whole network. (2) We conduct extensive experiments and analysis to verify the effectiveness of our proposed PPVAE. Our framework achieves state-of-the-art performance on conditionality in both automatic and human evaluations.
Boosted by the recent success of deep learning technology, Natural Language Generation (NLG) has recently become popular in the NLP community. Many great works have attempted to solve various subtasks like dialogue generation (Li et al., 2016), poetry generation (Yi et al., 2018) and story generation (Fan et al., 2018) and new techniques keep emerging (Bowman et al., 2016; Yu et al., 2017; Zhou et al., 2020). However, due to the black-box nature of neural networks, the recent proposed generic models suffer the problem of lacking interpretability and controllability.
To handle this problem and support generating plausible text with a specified condition, conditional text generation (Kikuchi et al., 2016; Ficler and Goldberg, 2017; Hu et al., 2017) has recently attracted extensive attention. Current research in this direction mainly falls into two fashions: the supervised methods and semi-supervised methods. For supervised methods, Mirza and Osindero (2014); Sohn et al. (2015) first converted the condition information to one-hot vectors, then integrated them into a generator and a discriminator. To enhance
the correlation between structured conditional code and generated samples, Chen et al. (2016) adopted an extra adversarial classifier to infer the structured code from generated samples. Wang and Wan (2018) used multiple generators for multiple conditions and a multi-class classifier to provide training signals for the learning of generators.
However, given only a limited number of conditional samples, semi-supervised methods are compulsory. To utilize the implicit conditional distribution behind the unlabeled text, Kingma et al. (2014) introduced a classifier into the VAE architecture. Hu et al. (2017) further involved two additional independent regularization terms in enhancing the disentanglement between structured code and unstructured code. Very recently, Keskar et al. (2019) used human-defined "control code" to pre-trained Language Model in an unsupervised manner.
Our work falls in the category of semi-supervised learning yet differs from the existing works in the following ways: (1) Our model decouples the text generation module from the condition representation module which two are tightly fused as a single one in previous studies, enabling possible exploitation for pre-trained Language Models (e.g., GPT-2 (Radford et al., 2019)). (2) Our model allows single-condition generation, which could inspire new applications like polite speech generator (Niu and Bansal, 2018) and data augmentation (Guo et al., 2018). (3) Our model can handle emerging conditions while achieving state-of-the-art performance with fewer parameters and less training time.
Variational Auto-Encoder (VAE). VAE (Kingma and Ba, 2015) is widely used in continuous generation (e.g., image generation). Bowman et al. (2016) introduced VAE to NLG to solve the "one-to-many" generation problem (i.e., generating multiple feasible samples for the same input). Given a latent variable  $z$  randomly sampled from a prior distribution, VAE comprises an encoder  $\operatorname{enc}(x) = q_{\phi}(z|x)$  and a decoder  $\operatorname{dec}(z) = p_{\theta}(x|z)$ . The encoder aims to encode input data  $x$  into latent space  $Z \in \mathbb{R}^d$ . The decoder is used to reconstruct the original input  $x$ , given the corresponding  $z$ . Thus, the loss function of VAE is formulated as:
$$
\begin{array}{r l} \mathcal {L} _ {V A E} (x) & = - \mathbb {E} _ {q _ {\phi} (z | x)} [ \log p _ {\theta} (x | z) ] \\ & + \mathrm {K L} (q _ {\phi} (z | x) \| p (z)) \end{array} \tag {1}
$$
where  $\mathrm{KL}(\cdot ||\cdot)$  is the Kullback-Leibler (KL) divergence,  $p(z) = \mathcal{N}(0,1)$  is the prior distribution. The first term ensures that VAE can distill compact variable  $z$  in latent space for reconstruction. The second term pushes posterior distribution to be close to the prior distribution, securing the mutual information between original data and the latent space (Dupont, 2018).
Conditional Text Generation with VAE. Conditional text generation has drawn much attention recently. By controlling the properties of generated contents, we can apply the generative models to many real-world scenarios. We follow the problem setting in (Hu et al., 2017). Given a set of  $k$  conditions  $C = \{c_1, c_2, \dots, c_k\}$ , an unlabeled corpus  $X$ , and conditional text samples  $Y = Y_{1} \cup Y_{2} \cup \dots \cup Y_{k}$  where each  $Y_{i}$  is a set of text samples that carries the condition  $c_{i}$ . The goal of a VAE model is to learn a decoder  $p_{\theta}(\hat{y} | z, c_{i})$  that takes the latent variable  $z$  and the condition  $c_{i}$  to calculate the distribution over the text samples  $Y_{i}$ . Thus, when the condition  $c_{i}$  and a randomly sampled latent variable  $z \sim p(z)$  specified, the model could generate realistic text samples matching the given condition.
4 Pre-train and Plug-in Variational Auto-Encoder
As a basis for semi-supervised learning, a large unlabeled corpus should include diverse text which covers a vast spectrum of conditions. Thus, text under each condition forms a conditional latent space, which could be mapped from a larger global latent space. Based on this, we propose a PRETRAIN-VAE and a PLUGINVAE to derive the global and conditional latent space, respectively.
PRETRAINVAE is composed of a pre-trained global encoder for text representation and a pretrained global decoder for text generation.
PRETRAINVAE. The encoder and decoder of PRETRAINVAE are used to encode and generate text, respectively. As discussed above, PRETRAINVAE is trained on a large amount of unlabeled text to derive the global latent space  $Z_{g}$  for the latent variable  $z_{g}$ , where  $Z_{g} \in \mathbb{R}^{d_{g}}$  and  $d_{g}$  is the space dimension. Previous studies usually use a common VAE for text representation and generation. However, as pointed out in (Bowman et al., 2016), VAE suffers the notorious "posterior collapse" problem. To address this, we utilize Wasserstein Autoen
coder (WAE) (Tolstikhin et al., 2018) for PRE-TRAINVAE. Different from the original VAE, WAE encourages aggregated posterior distribution to be close to the prior, which is effective in alleviating the reconstruction problem of VAE (Tolstikhin et al., 2018). Specifically, we adopt WAE-GAN, a variant of WAE, which incorporates the merits of adversarial learning. During training, the encoder  $\text{enc}_g(x) = q_g(z_g|x)$  encodes the text to the latent space and the decoder  $\text{dec}_g(z_g) = p_g(x|z_g)$  reconstruct the text with the latent variable  $z_g$ . Thus, the loss function of PRE-TRAINVAE is formulated as:
$$
\begin{array}{l} \mathcal {L} _ {\text {P R E T R A I N V A E}} (x) = - \mathbb {E} _ {q _ {g} \left(z _ {g} \mid x\right)} \left[ \log p _ {g} \left(x \mid z _ {g}\right) \right] \\ + \lambda D \left(Q \left(z _ {g}\right), p \left(z _ {g}\right)\right) \tag {2} \\ \end{array}
$$
where  $Q(z_{g}) = \int q_{g}(z_{g}|x)p(x)dx$  is the aggregated posterior distribution;  $p(z_{g})$  is the prior normal distribution;  $D$  is the adversarial discriminator;  $\lambda$  is the coefficient hyper-parameter  $(\lambda >0)$ .
PLUGINVAE. For each condition, we use a condition-specific PLUGINVAE to derive the conditional space. That is, PLUGINVAE is proposed to learn the transformation between the conditional and global latent space for each condition. Specifically, for each condition  $c_{i}$ , we use a limited number of conditional samples  $y_{i}$  and utilize the global encoder  $enc_{g}$  to encode them into  $v_{y_{i}}$ . Note that normally, the encoded text samples under a single condition are not likely to densely clustered in the global text space  $Z_{g}$ , since the learning process of  $Z_{g}$  is condition-independent and the unlabeled corpus contains diverse text samples. PLUGINVAE for condition  $c_{i}$  consists of an encoder  $enc_{c_{i}}(v_{y_{i}}) = q_{c_{i}}(z_{c_{i}}|v_{y_{i}})$  and a decoder  $dec_{c_{i}}(z_{c_{i}}) = p_{c_{i}}(v_{y_{i}}|z_{c_{i}})$ . The learned condition-dependent latent space is  $Z_{c_{i}} \in \mathbb{R}^{d_{c}}$ , where  $d_{c}$  is the space dimension. Thus, PLUGINVAE is capable of mapping the samples in the global latent space to and from a denser conditional latent space (i.e.,  $d_{c} < d_{g}$ ). During training, the loss function of PLUGINVAE for a single condition is defined as:
$$
\begin{array}{l} \mathcal {L} _ {s i n g l e} (v _ {y _ {i}}) = - \mathbb {E} _ {q (z _ {c _ {i}} | v _ {y _ {i}})} [ \log p _ {c _ {i}} (v _ {y _ {i}} | z _ {c _ {i}}) ] \\ + | \left(\operatorname {K L} \left(q _ {c _ {i}} \left(z _ {c _ {i}} \mid v _ {y _ {i}}\right) \| p \left(z _ {c _ {i}}\right)\right) - \beta \right| \tag {3} \\ \end{array}
$$
where  $p(z_{c_i})$  is the prior normal distribution of the conditional latent space;  $z_{c_i}$  is the latent variable;  $v_{y_i} = \operatorname{enc}_g(y_i)$  is encoded text samples from  $Y_i$ . To enhance the diversity of generated text, we introduce an extra constant term  $\beta$  to control the amount
(c)
Figure 1: The whole workflow of our proposed framework.
of encoded information in VAE (Dupont, 2018; Chen et al., 2018; Kim and Mnih, 2018). By setting  $\beta$  to an appropriate value, PLGINVAE could extract compact conditional information without sacrificing the fluency or accuracy.
Although we can already generate conditional text under a single condition by Equation 3, it is possible to even further improve the conditionality by introducing negative samples. We construct the negative samples  $y_{i}^{\prime}$  from  $Y_{i}^{\prime}$  and encode them:
$$
Y _ {i} ^ {\prime} = Y - Y _ {i} \tag {4}
$$
$$
v _ {y _ {i}} ^ {'} = e n c _ {g} (y _ {i} ^ {'})
$$
Thus, the loss function of PLGINVAE with negative samples is defined as:
$$
\begin{array}{l} \mathcal {L} _ {\text {P L U G I N V A E}} \left(v _ {y _ {i}}, v _ {y _ {i}} ^ {\prime}\right) \tag {5} \\ = \mathcal {L} _ {s i n g l e} (v _ {y _ {i}}) - \gamma \mathcal {L} _ {s i n g l e} (v _ {y _ {i}} ^ {'}) \\ \end{array}
$$
where  $v_{y_i}$  is a batch of encoded samples under condition  $c_i$ , and  $v_{y_i}'$  is a batch of encoded negative samples;  $\gamma$  is a hyper-parameter balancing the positive and negative samples. For different tasks, the best setting for  $\gamma$  may vary. Intuitively, the larger the difference between the conditions is, the smaller  $\gamma$  should be.
4.2 Workflow
In this section, we provide the details of training and generation procedures. As illustrated in Figure 1, the workflow is composed of three steps.
Pre-train once, infer everywhere. First, as shown in Figure 1(a), using the unlabeled corpus  $X$ , we pre-train PRETRAINVAE to learn the global latent space  $Z_{g}$  by reconstruction with Equation 2. Once pre-trained, the weights of both  $\text{enc}_{g}$  and  $\text{dec}_{g}$  are fixed. As an unsupervised VAE model, PRETRAINVAE is capable of generating diverse but unconditional text.
Train it when you need it. Previous methods (Kingma et al., 2014; Hu et al., 2017) learn
the joint conditional space by jointly considering all conditions. However, once the model is trained, it is not possible to add a new condition without a full retraining. Different from those approaches, PPVAE is totally flexible that allows adding new conditions. Shown in Figure 1(b), once a condition is added, we only need to train a PLUGINVAE specifically for this condition with Equation 3 (or Equation 5, if provided with samples of other conditions). Since PLUGINVAE is text-irrelevant and only learns to map between two latent spaces, the training number of parameters is only  $0.34\%$  (see Section 6.3) of fine-tuning PRETRAINVAE or retraining other models. Additionally, although we need to train  $k$  PLUGINVAE for  $k$  conditions, the total number of trained parameters is still much smaller than existing methods (unless  $k > 1/0.34\% \approx 294$ , which is impossible in actual applications). Plus, we can parallel the conditional training to speed up the process easily. Plug it in and generate. Shown in Figure 1(c), once PLUGINVAE for the condition  $c_i$  is trained, we can plug it into the PPVAE framework and generate text together with PRETRAINVAE.
First, we randomly sample a latent variable  $z_{c_i}$  from the prior distribution  $p(z_{c_i}) = \mathcal{N}(0,1)$ . Then we use PLUGINVAE's decoder  $dec_{c_i}$  to map  $z_{c_i}$  to the global latent space  $Z_g$  and obtain  $z_{c_i}'$ :
$$
z _ {c _ {i}} ^ {\prime} = d e c _ {c _ {i}} (z _ {c _ {i}}). \tag {6}
$$
Since  $z_{c_i}' \in Z_g$ , we can directly use the global decoder  $dec_g$  to generate text:
$$
\hat {y} _ {i} = \operatorname {d e c} _ {g} \left(z _ {c _ {i}} ^ {\prime}\right) \tag {7}
$$
where  $\hat{y}_i$  is the generated text under condition  $c_i$ .
Following the setting of (Hu et al., 2017), we mainly focus on short text generation (no longer
Dataset	#Train	#Dev	#Test	Avg-len
Yelp	444,101	63,483	126,670	8.93
News Titles	249,043	23,949	20,000	9.85
Table 1: The statistics of Yelp and News Titles.
than 15 tokens), which is easier for both automatic and human evaluations. We use Yelp (Shen et al., 2017) and News Titles (Fu et al., 2018) for experiments. Yelp is a collection of restaurant reviews. We use the pre-processed version used in (Shen et al., 2017), where two polarity sentiment labels are provided. For News Titles, we choose the titles belong to Business, Entertainment and Health categories for our experiments.
Both Yelp and News Titles are datasets with relatively short text. We filter out text longer than 15 words, then choose the top 8,900 and 10,000 words as the vocabulary for Yelp and News Titles, respectively. The statistics of the two datasets are listed in Table 1. We discard the labels in the original training and validation splits. We use the original training split as the unlabeled corpus; the validation split to select the best unsupervised models, and the test split as the labeled conditional text.
Based on the Yelp dataset, we define two tasks: (1) Sentiment. This task aims at generating text samples, either positive or negative. The ratio of positive/negative text in Yelp is roughly  $0.6:0.4$ . We randomly sample 200 positive and 200 negative text for supervised training. (2) Length. This task aims at generating text samples with a specific length. We define  $(len \leq 3)$  as short text,  $(len \geq 12)$  as long text and  $(3 < len < 12)$  as medium text. We respectively sample 200 text for short, medium, and long text for supervised training.
Based on the News Titles dataset, we define the categorical text generation task called Topic. This task aims at generating text samples on a certain topic. The ratio of business/health/entertainment in News Title is  $0.38:0.15:0.47$ , which is more imbalanced than Yelp. We randomly sample 200 text for each category for supervised learning.
We use two semi-supervised methods, S-VAE (Kingma et al., 2014) and CTRL-GEN (Hu et al., 2017) as our baselines. S-VAE incorporates a classifier to provide conditional distribution for unlabeled data. Note that S-VAE is originally
proposed for image generation but adapted to text generation as a baseline by Hu et al. (2017). CTRL-GEN further exploits several regularization terms to enhance the disentanglement between the structured code and the unstructured code. For a fair comparison, both the text encoder and decoder of the two baselines are the same as PRETRAINVAE. Furthermore, the baseline methods also exploit the same unlabeled corpus  $X$  and labeled corpus  $Y$  as described in the original papers.
5.3 Models
PPVAE is a model-agnostic approach, which means that both the encoders and encoders of PRE-TRAINVAE and PLUGINVAE can be modified to work under different settings. Here, we describe the model architecture used in our experiments.
PRETRAINVAE. For the encoder, we use a one-layer Bidirectional Gated Recurrent Unit (Bi-GRU) with 256 hidden units in each direction as its encoder. Two linear Fully-Connected (FC) layers are used for re-parameteristic trick (Kingma and Welling, 2014). For the decoder, we use a Transformer (Vaswani et al., 2017) (3 layers, 8 heads). Additionally, we add extra positional embedding after each block, and the linearly transformed encoded vector is provided as input for each block (Brock et al., 2019). For a fair comparison, we use the same encoder-decoder architecture for both S-VAE and CTRL-GEN.
PLUGINVAE. The encoder is a two-layer FC network of 64/32 hidden units taking input in  $d_{g}$  dimensions with an additional linear output layer of  $d_{c}$  units. The decoder is a two-layer FC network of 32/64 hidden units taking the latent variable in  $d_{c}$  dimensions as input with a linear output layer of  $d_{g}$  units. The activation function used in the FC networks is LeakyRelu (Maas et al., 2013).
5.4 Hyper-Parameters
PRETRAINVAE. The size of latent space  $d_g$  is set to 128. The word embedding is in 256 dimensions and randomly initialized. The output softmax matrix is tied with the embedding layer. For the adversarial classifier, we adopt two 128D hidden FC layers with LeakyRelu activation and one 1D output linear layer without bias. The balance coefficient  $\lambda$  is 20 for Yelp and 15 for News Titles. We train the WAE-GAN with Wasserstein Divergence (Wu et al., 2018) to smooth the training process. The coefficient  $k$  and power  $p$  of Wasserstein Divergence
Task	Conditions	Method	Accuracy (↑ better)	Log-Variance (↓ better)	Distinct-1 (↑ better)	Distinct-2 (↑ better)
Sentiment	{Positive, Negative}	S-VAE	0.7194	-5.38	0.0198	0.2520
CTRL-GEN	0.6998	-2.78	0.0026	0.0164
PPVAE-single (ours)	0.7832	-11.12	0.0350	0.2568
PPVAE (ours)	0.8484	-11.90	0.0356	0.2627
Length	{Short, Medium, Long}	S-VAE	0.8598	-4.82	0.0187	0.1795
CTRL-GEN	0.3957	-1.96	0.0021	0.0146
PPVAE-single (ours)	0.9640	-6.96	0.0375	0.2549
PPVAE (ours)	0.9722	-7.64	0.0372	0.2538
Topic	{Business, Health, Entmt.}	S-VAE	0.6930	-2.32	0.0360	0.2162
CTRL-GEN	0.5335	-3.39	0.0107	0.0431
PPVAE-single (ours)	0.7725	-3.82	0.0497	0.3152
PPVAE (ours)	0.8024	-3.68	0.0478	0.3056
Table 2: The results of conditional text generation tasks. We use **boldface** and **underline** to indicate the best and the second-best performance. PPVAE-single indicates PPVAE with a PLUGINVAE trained under the single condition setting, as described in Section 5.5. We show the natural logarithm (ln) of variance, since the original scale is too small for demonstration.
are set to 2 and 6, respectively. During pre-training, the batch size is set to 512. Adam (Kingma and Ba, 2015) with  $\text{beta}_1 = 0$  is used as the optimizer. The learning rate is set to  $5 \times 10^{-4}$ .