SlowGuess/ACL_2020_OCR huggingface dataset:
PLUGINVAE. We set the size of latent space  $d_{c} = 20$ .  $\gamma$  is set to 0.1 for sentiment tasks, 0.05 for categorical tasks, and  $3 \times 10^{-3}$  for length tasks. The batch size is set to 128. Adam (Kingma and Ba, 2015) with  $beta_{1} = 0.5$  is used as the optimizer, learning rate is  $3 \times 10^{-4}$  for 20K iterations.  $\beta$  linearly increases from 0 to 5 in first 10K iterations.
5.5 Evaluation Settings
Metrics. We evaluate the results with two metrics, accuracy and diversity. For accuracy, we train a sentiment classifier and categorical classifier (Kim, 2014), which could achieve accuracy of  $90\%$  and  $97\%$  on the validation set, respectively. The accuracy of length task can be directly calculated with the word count of generated text. Plus, a model that performs well on only one condition but poorly on others is not practically useful. Thus, to measure the robustness among conditions, we calculate the variance of accuracy under all conditions in a task. For diversity, we adopt Distinct-1 and Distinct-2 (Li et al., 2016) metrics. Distinct-1/Distinct-2 are the ratios of unique 1-gram/2-gram, respectively. A higher value indicates better diversity. For all tasks and models, we randomly generate 10K text for each condition by greedy decoding and report the averaged results.
Single Condition Generation. In a real-world scenario, the full set of conditions is not always avail-
able. When provided only a labeled set of target text (i.e.,  $k = 1$ ), it is not possible to learn the joint conditional space for S-VAE and CTRL-GEN any more. However, PPVAE can deal with that by training without negative samples using Equation 3.
6.1 Overall Comparisons
Accuracy. The results of conditional text generation are listed in Table 2. On sentiment task, our model outperforms CTRL-GEN and S-VAE by 0.1486 and 0.129, respectively. On length task, the accuracy of our model exceeds  $95\%$ , dramatically outperforming S-VAE and CTRL-GEN by 0.1124 and 0.5765 on accuracy. Notably, the performance of CTRL-GEN (0.3957) is extremely low, demonstrating the limitation of its generator-discriminator (Goodfellow et al., 2014) training process and its token-based discriminator, which is unable to discriminate text with different lengths. On topic task, our model scores higher on accuracy than S-VAE and CTRL-GEN by 0.1094 and 0.2689, respectively. On all three tasks, PPVAE-single performs slightly poorer than PPVAE with negative samples, verifying the effectiveness of negative sampling. Furthermore, our models achieve the lowest variance on all three tasks, indicating that PPVAE is robust and achieves a good balance among conditions.
Diversity. Diversity is a long-lasting issue lying in the field of generative models. Recent works (Wang et al., 2017; Razavi et al., 2019) reveal the capability of the diverse content generation with
Task	Method	Fluency	Conditionality
Sentiment	S-VAE	3.10	3.04
CTRL-GEN	3.65	3.23
PPVAE-single	3.54	3.23
PPVAE	3.30	3.29
Length	S-VAE	3.64	0.8598
CTRL-GEN	2.53	0.3597
PPVAE-single	3.43	0.9640
PPVAE	3.50	0.9722
Topic	S-VAE	3.31	2.78
CTRL-GEN	3.09	2.51
PPVAE-single	3.38	3.33
PPVAE	3.45	3.57
VAE-based methods. These works also conclude that VAE-based methods have better output diversity than GAN-based models. Our experimental results support this conclusion well. Particularly, CTRL-GEN suffers poor diversity, which indicates the generation of "dull text" (Li et al., 2016). Both S-VAE and PPVAE show prominently better diversity than GAN-based model, CTRL-GEN. Note that the relation between the usage of negative examples and text diversity of PPVAE is not statistically prominent  $(p > 0.05)$ .
We conduct human annotations as a complementary evaluation beyond automatic metrics. Specifically, eight individual judges are asked to rate over 200 conditional samples generated from each model and each condition. That is, for each model, a total of 4,800 text samples are annotated. A judge needs to rate fluency and conditionality in the standard 1 to 5 scale. Fluency measures whether the text samples are natural and fluent as real (i.e., human-written) ones. Conditionality indicates whether the generated text adheres to the given condition. Shown in Table 3, PPVAE achieves the best conditionality in both automatic and human evaluations on all three tasks. Meanwhile, PPVAE retains a satisfying fluency on sentiment and length tasks and obtains the best fluency on the topic task.
6.3 Training Costs
To measure the efficiency of proposed methods, we report the training time and the number of parameters of S-VAE, CTRL-GEN and PPVAE in Table 4. We train the models on a single Nvidia
Table 3: Human evaluation results. Note that since the length task is objectively defined, we copy the accuracy results from Table 2.
Method	# Training Params	Training Time
S-VAE	6.5M	1.4h
CTRL-GEN	8.5M	3.5h
PRETRAINVAE	6.5M	1.2h (only once)
PLUGGINVAE	22K	64s
Table 4: Average numbers of parameters and time costs for training.
Task	Method	Acc.	Distinct-1/2
Sentiment	Fine-tuning	0.5319	0.0281 / 0.2845
PPVAE-single	0.7832	0.0350 / 0.2568
PPVAE	0.8484	0.0356 / 0.2627
Length	Fine-tuning	0.9456	0.0340 / 0.2923
PPVAE-single	0.9640	0.0375 / 0.2549
PPVAE	0.9722	0.0372 / 0.2538
Table 5: The comparisons of fine-tuned PRETRAIN-VAE with the full PPVAE on the two tasks of Yelp dataset.
β	Accuracy	Distinct-1	Distinct-2
0.0	1.0000	0.0001	0.0001
2.0	0.9938	0.0256	0.1629
5.0	0.9908	0.0301	0.2112
10.0	0.9875	0.0324	0.2370
Table 6: The impact of different  $\beta$  on long text generation task.
GTX 1080 GPU and report the training time until the convergence of each model. PRETRAINVAE has the same size of S-VAE but only needs to be trained once and does not require a full retraining when a new condition added. Also, PLUGINVAE, which learns to transform between the global latent space and the conditional latent space, only has  $22\mathrm{K}$  parameters and can be trained within about one minute.
6.4 PLUGINVAE vs. Fine-Tuning
As a natural baseline, the conditional generation can also be done by directly fine-tuning PRETRAINVAE on each condition. Shown in Table 5, despite the fact that it is not computationally efficient and saving the full weights is undesirable for industrial applications when the model is large (e.g., GPT-2 (Radford et al., 2019)), both PLGINVAE trained with and without negative samples significantly outperform a directly fine-tuned PRETRAINVAE on accuracy.
Task	Condition	Generated Examples
Sentiment	Positive
Negative	The services are friendly, fast.
The egg drop soup was old and tasted like feet.
Length	Short
Medium
Long	Great pricing!
I refused to work with you and this place.
And this made me feel uncomfortable and the prices aren’t right.
Topic	Business
Health
Entertainment	FDA Approves New Case of E-cigarettes
Ebola : Virus Spreads in the US
Surprise Birthday: The Guys of the Cast of Disney Parks
Table 7: Some conditional examples generated by PPVAE for qualitative analysis (cherry-picked).
Generated Examples
S-VAE
Chinese State Media: 17 Miners Trapped Underground
Huge Increases in Obamarcare Premiums Are Coming
Herbalife Ltd. (HLF) Probe Earns Bill Ackman Back Millions
CTRL-GEN
Pfizer's AstraZeneca's AstraZeneca Bid for AstraZeneca
FDA's New Drug to Treat Migraines
Pfizer to Acquire Seragon in $42.9B
PPVAE
Despite Highway Crisis, Many Worries Remain on US Oil Exports
Lululemon: Digital Sales Surge in 1Q Net Income, Revenue
Crisis of Market: US Stocks Climb; Nike Jumps
Table 8: Some generated conditional examples under condition Business (randomly sampled).
Failed Examples
Grammatical
Eat the service!
In addition, this location sucks it is.
Star Wars 7 will include US production on set
Conditional
(Negative) I was shocked that this is what I needed.
(Long) Are you actually drunk outside?
(Business) Michael Jackson's New Album ‘Xscape’
Table 9: Some failed examples (cherry-picked).
6.5 Effect of Hyper-parameter  $\beta$
Since  $\beta$  is an important hyper-parameter for PPVAE, we test  $\beta \in \{0,2,5,10\}$  on the long text generation task. From the results in Table 6, we find that  $\beta$  controls the balance between diversity and accuracy. Specifically, when  $\beta$  is too large, more diverse samples could be generated, but the accuracy may be sacrificed slightly. On the contrary, when  $\beta$  is too small, the accuracy could climb to a higher value, but meanwhile, the diversity drops drastically. Empirically, we find that  $\beta = 5$  is an appropriate value for all tasks.
We select some generated conditional text of each condition in Table 7. As shown in the table, our proposed PPVAE is capable of generating realistic conditional text. Also, shown in Table 8, on topic task, we randomly select some examples from the output of each model. The output of S-VAE seems to be diverse but is poorly conditioned. CTRLGEN suffers an obvious diversity issue, which makes it repeatedly output similar text.
For the error analysis, we pick some failed examples of PPVAE in Table 9. We categorize the errors into two main classes. (1) Grammatical. Grammatical problems are common in NLG. As we analyze, this kind of errors can be mitigated with a deeper encoder and decoder with even more unlabeled data for pre-training. (2) Conditional. Conditional errors are of great interest to us since they lie in our focus. We choose three typical errors and list them in Table 9. In the first sentence, "shocked" is a subtle word which may indicate either positive or negative sentiment depending on the context. Thus, with a greedy decoding strategy, it may be incorrectly decoded into the other polarity. We believe this kind of errors could be fixed with more elaborate decoding strategies (e.g., Weighted Decoding (See et al., 2019)). In the second sentence, the length is limited by the nature of an interrogative sentence. As a linguistic fact, an interrogative sentence often has fewer words than a declarative sentence. In the third sentence, we remark an overlapping problem between classes. Some topics (e.g., music album) may appear in both business and entertainment news. In some way, these samples can also be considered as correctly conditioned ones, which highlights the importance of a fine-grained human evaluation on this task.
In this paper, we present a novel PPVAE framework for flexible conditional text generation, which decouples the text generation module from the condition representation module. The extensive experiments demonstrate the superiority of the proposed PPVAE against the existing alternatives on conditionality and diversity while allowing new conditions to be added without a full retraining.
We are grateful for the insightful comments from the anonymous reviewers. We would like to especially thank Daya Guo for his help and suggestions. This research was supported by National Natural Science Foundation of China (No. 61872278). Chenliang Li is the corresponding author.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL.
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large scale GAN training for high fidelity natural image synthesis. In ICLR.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David K. Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In NeurIPS.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS.
Emilien Dupont. 2018. Learning disentangled joint continuous and discrete representations. In NeurIPS.
Jesse H. Engel, Matthew Hoffman, and Adam Roberts. 2018. Latent constraints: Learning to generate conditionally from unconditional generative models. In ICLR.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In ACL.
Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. CoRR, abs/1707.02633.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In AAAI.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NeurIPS.
Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin, Hong Chi, James Cao, Peng Chen, and Ming Zhou. 2018. Question generation from SQL queries improves neural semantic parsing. In EMNLP.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzkebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In ICML.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In EMNLP.
Hyunjik Kim and Andriy Mnih. 2018. Disentangling by factorising. In ICML.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In NeurIPS.
Diederik P. Kingma and Max Welling. 2014. Auto-encoding variational bayes. In ICLR.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In NAACL-HLT.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In ICML.
Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. CoRR, abs/1411.1784.
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Trans. Assoc. Comput. Linguistics, 6:373-389.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In NAACL-HLT.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NeurIPS.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In NeurIPS.
Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Scholkopf. 2018. Wasserstein autoencoders. In ICLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Ke Wang and Xiaojun Wan. 2018. Sentigan: Generating sentimental texts via mixture adversarial networks. In *IJCAI*.
Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. 2017. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In NeurIPS.
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. 2018. Wasserstein divergence for gans. In ECCV.
Xiaopeng Yang, Xiaowen Lin, Shunda Suo, and Ming Li. 2018. Generating thematic chinese poetry using conditional variational autoencoders with hybrid decoders. In *IJCAI*.
Xiaoyuan Yi, Maosong Sun, Ruoyu Li, and Zonghan Yang. 2018. Chinese poetry generation with a working memory model. In *IJCAI*.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL.
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2020. Self-adversarial learning with comparative discrimination for text generation. In ICLR.