SlowGuess/ACL_2020_OCR huggingface dataset:
Method	Test-BLEU-2	3	4	5	Self-BLEU-2	3	4
SeqGAN (Yu et al., 2017)	0.820	0.604	0.361	0.211	0.807	0.577	0.278
RankGAN (Lin et al., 2017)	0.852	0.637	0.389	0.248	0.822	0.592	0.230
GSGAN (Kusner and Miguel, 2016)	0.810	0.566	0.335	0.197	0.785	0.522	0.230
TextGAN (Zhang et al., 2017)	0.910	0.728	0.484	0.306	0.806	0.548	0.217
LeakGAN (Guo et al., 2017)	0.922	0.797	0.602	0.416	0.912	0.825	0.689
MLE (Caccia et al., 2018)	0.902	0.706	0.470	0.392	0.787	0.646	0.485
GMGAN (ours)	0.949	0.823	0.635	0.421	0.746	0.511	0.319
Table 1: Test-BLEU (↑) and Self-BLEU (↓) scores on Image COCO.
Method	Test-BLEU-2	3	4	5	Self-BLEU-2	3	4
SeqGAN (Yu et al., 2017)	0.630	0.354	0.164	0.087	0.728	0.411	0.139
RankGAN (Lin et al., 2017)	0.723	0.440	0.210	0.107	0.672	0.346	0.119
GSGAN (Kusner and Miguel, 2016)	0.723	0.440	0.210	0.107	0.807	0.680	0.450
TextGAN (Zhang et al., 2017)	0.777	0.529	0.305	0.161	0.806	0.662	0.448
LeakGAN (Guo et al., 2017)	0.923	0.757	0.546	0.335	0.837	0.683	0.513
MLE (Caccia et al., 2018)	0.902	0.706	0.470	0.392	0.787	0.646	0.485
GMGAN (ours)	0.923	0.727	0.491	0.303	0.814	0.576	0.328
Table 2: Test-BLEU (↑) and Self-BLEU (↓) scores on EMNLP2017 WMT News.
We test the proposed framework on unconditional and conditional text generation tasks, and analyze the results to understand the performance gained by the guider network. We also perform an ablation investigation on the improvements brought by each part of our proposed method, and consider non-parallel style transfer. All experiments are conducted on a single Tesla P100 GPU and implemented with TensorFlow and Theano. Details of the datasets, the experimental setup and model architectures are provided in the Appendix.
Encoder as the feature extractor For unconditional generation, the feature extractor that generates inputs for the guider network shares the CNN part of the encoder. We stop gradients from the guider network to the encoder CNN in the training process. For conditional generation, we use a pretrained feature extractor, trained similarly to the unconditional generation.
Training procedure As with many imitation-learning models (Bahdanau et al., 2017; Rennie et al., 2016; Sutskever et al., 2014), we first train the encoder-decoder part based on the off-policy data with an MLE loss. Then we use RL training to fine-tune the trained generator. We adaptively transfer the training from MLE loss to RL loss, similar to (Paulus et al., 2017; Ranzato et al., 2016).
Initial states We use the same initial state for both the generator and the guider networks. For conditional generation, the initial state is the encoded latent code of the conditional information for both training and testing. For unconditional generation, the initial state is the encoded latent code of a target sentence in training and random noise in testing.
6.2 Adversarial Text Generation
We focus on adversarial text generation, and compare our approach with a number of related works (Guo et al., 2017; Lin et al., 2017; Yu et al., 2017; Zhang et al., 2017; Zhu et al., 2018). In this setting, a discriminator in the GAN framework is added to the model in Figure 1 to guide the generator to generate high-quality sentences. This is implemented by defining the final reward to be the output of the discriminator. All baseline experiments are implemented on the texygen platform (Zhu et al., 2018). We adopt the BLEU score, referenced by the test set (test-BLEU, higher value implies better quality) and itself (self-BLEU, lower value implies better diversity) (Zhu et al., 2018) to evaluate quality of generated samples, where test-BLEU evaluates the reality of generated samples, and self-BLEU measures the diversity. A good generator should achieve both a high test-BLEU score and a low self-BLEU score. In practice, we use  $\triangle t = c = 4$  and  $\gamma = 0.25$ . We call the proposed method guider-matching GAN (GMGAN) for unconditional text
generation. More details of GMGAN are provided in Appendix D.
Short Text Generation: COCO Image Captions We use the COCO Image Captions Dataset, in which most sentences have a length of about 10 words. Since we consider unconditional text generation, only image captions are used as the training data. After preprocessing, we use 120,000 random sample sentences as the training set, and 10,000 as the test set. The BLEU scores with different methods are listed in Table 1. We observe that GMGAN performs significantly better than the baseline models. Specifically, besides achieving higher test-BLEU scores, the proposed method also generates samples with very good diversity in terms of self-BLEU scores. LeakGAN represents the state-of-the-art in adversarial text generation, however, its diversity measurement is relatively poor (Zhu et al., 2018). We suspect that the high BLEU score achieved by LeakGAN is due to its mode collapse on some good samples, resulting in high self-BLEU scores. Other baselines achieve lower self-BLEU scores since they cannot generate reasonable sentences.
Long Text Generation: EMNLP2017 WMT Following (Zhu et al., 2018), we use the News section in the EMNLP2017 WMT4 Dataset as our training data. The dataset consists of 646,459 words and 397,726 sentences. After preprocessing, the training dataset contains 5,728 words and 278,686 sentences. The BLEU scores with different methods are provided in Table 2. Compared with other methods, LeakGAN and GMGAN achieve comparable test-BLEU scores, demonstrating high-quality generated sentences. Again, LeakGAN tends to over-fit on training data, leading to much higher (worse) self-BLEU scores. Our proposed GMGAN shows good diversity of long text generation with lower self-BLEU scores. Other baselines obtain both low self-BLEU and test-BLEU scores, leading to more random generations.
Human Evaluation Simply relying on the above metrics is not sufficient to evaluate the proposed method (Caccia et al., 2018). Following previous work (Guo et al., 2017), we perform human evaluations using Amazon Mechanical Turk, evaluating the text quality based on readability and meaningfulness (whether sentences make sense) on the EMNNLP2017 WMT News dataset. We ask the worker to rate the input sentence with scores scal
Scores	Criteria
5 (Best)	It is consistent, informative, grammatically correct.
4	It is grammatically correct and makes sense.
3	It is mostly meaningful and with small grammatical error.
2	It needs some time to understand and has grammatical errors.
1 (Worst)	Meaningless, not readable.
Table 3: Human evaluation rating criteria.
Methods	MLE	SeqGAN	RankGAN	GSGAN
Human scores	2.45±0.14	2.57±0.15	2.91±0.17	2.48±0.14
Methods	textGAN	LeakGAN	GMGAN	Real
Human scores	3.11±0.16	3.47±0.15	3.89±0.15	4.21±0.14
Table 4: Results of human evaluation with different methods on EMNLP2017 WMT dataset.
ing from 1 to 5, with 1 as the worst score and 5 as the best. The detailed criteria is listed in Table 3. We require all the workers to be native English speakers, with approval rate higher than  $90\%$  and at least 100 assignments completed.
We randomly sample 100 sentences generated by each model. Ten native English speakers on Amazon Mechanical Turk are asked to rate each sentence. The average human rating scores are shown in Table 4, indicating GMGAN achieves higher human scores compared to other methods. As examples, Table 5 illustrates some generated samples by GMGAN and its baselines. The performance on the two datasets indicates that the generated sentences of GMGAN are of higher global consistency and better readability than SeqGAN and LeakGAN. More generated examples are provided in the Appendix.
Ablation Study We conduct ablation studies on long text generation to investigate the improvements brought by each part of our proposed method. We first test the benefits of using the guider network. Among the methods compared, Guider is the standard MLE model with the guider network. We further compare RL training with  $i$  only final rewards,  $ii$  only feature-matching rewards, and  $iii$  combining both rewards, namely GMGAN. The results are shown in Table 6. We observe that guider network plays an important role in improving the performance. RL training with final rewards given by a discriminator typically damages the generation quality, but feature-matching reward produces sentences with much better diversity due to the ability of exploration.
Method	COCO Image Captions	EMNLP2017 WMT News
SeqGAN	(1) A person and black wooden ta-ble.	(1) She added on a page where it was made clear more old but public got said.
(2) A closeup of a window at night.	(2) I think she're guys in four years , and more after it played well enough.
LeakGAN	(1) A bathroom with a black sink and a white toilet next to a tub.	(1)"I'm a fan of all the game, I think if that's something that I've not," she said, adding that he would not be decided.
(2) A man throws a Frisbee across the grass covered yard.	(2) The UK is Google's largest non-US market, he has added "20, before the best team is amount of fewer than one or the closest home or two years ago.
GMGAN	(1) Bicycles are parked near a row of large trees near a sidewalk.	(1) "Sometimes decisions are big, but they're easy to make," he told The Sunday Times in the New Year.
(2) A married couple posing in front of a piece of birthday cake.	(2) A BBC star has been questioned by police on suspicion of sexual assault against a 23-year-old man , it was reported last night.
Table 5: Examples of generated samples with different methods on COCO and EMNLP datasets.
Methods	MLE	Guider	Final	Stepwise	GMGAN
Test-BLEU-2	0.761	0.920	0.843	0.914	0.923
BLEU-3	0.468	0.723	0.623	0.704	0.727
BLEU-4	0.230	0.489	0.390	0.457	0.491
BLEU-5	0.116	0.289	0.221	0.276	0.303
Self-BLEU-2	0.664	0.812	0.778	0.798	0.814
BLEU-3	0.338	0.589	0.525	0.563	0.576
BLEU-4	0.113	0.360	0.273	0.331	0.328
Figure 3: Guider-Matching Rewards Illustrations.
Case Study of Guider-Matching Rewards Figure 3(a) illustrates the feature-matching rewards in the generation. Figure 3(a) shows an example of failure generation in the training stage, when two sentences are combined by the word 'was'. It is grammatically wrong to select 'was' for the generator, thus the guider network gives a small reward. We can see that the rewards become lower with more time steps, which is consistent with the exposure bias. Figure 3(b) shows a successful generation, where the rewards given by the guider are relatively high (larger than 0.5). These observations validate that: (i) exposure bias exists in MLE training. (ii) RL training with exploration can help reduce the effects of exposure bias. (iii) Our proposed feature-matching rewards can provide meaningful guidance to maintain sentence structure and fluency.
Table 6: Ablation study on EMNLP2017 WMT.
Model	Acc(%)	BLEU	BLEU-ref
CVAE (Shen et al., 2017)	73.9	20.7	7.8
Controllable (Hu et al., 2017)	86.7	58.4	-
BackTrans (Prabhumoye et al., 2018)	91.2	2.8	2.0
DeleteAndRetrieval (Li et al., 2018a)	88.9	36.8	14.7
Guider (Ours)	92.7	52.1	25.4
Table 7: Non-parallel text style transfer results on the test set with human references.
6.3 Non-parallel Text-style Transfer
We test the proposed framework on the non-parallel text-style-transfer task, where the goal is to transfer one sentence in one style (e.g., positive) to a similar sentence but with a different style (e.g., negative). Pair-wise information should be inferred from the training data, which becomes more challenging. For a fair comparison, we use the same data and its split method as in (Shen et al., 2017). Specifically, there are 444,000, 63,500, and 127,000 sentences with either positive or negative sentiments in the training, validation and test sets, respectively.
To measure whether the original sentences (in the test set) have been transferred to the desired sentiment, we follow the settings of (Shen et al., 2017) and employ a pretrained CNN classifier, which achieves an accuracy of  $97.4\%$  on the validation set, to evaluate the transferred sentences. We also report the BLEU scores with original sentences (BLEU) and human references (BLEU-ref) (Li et al., 2018a), to evaluate the content preservation of transferred sentences. Results are summarized in Table 7. Our proposed model exhibits higher transfer accuracy and better content preservation, indicating the guider network provides good sentiment guidance to better preserve the content information.
2523
From positive to negative
Original: all the employees are friendly and helpful
Transferred: all the employees are rude and unfriendly.
Original: i'm so lucky to have found this place!
Transferred: i'm so embarrassed that i picked this place.
From negative to positive
Original: the service was slow.
Transferred: the service was fast and friendly.
Original: i would never eat there again and would probably not stay there either.
Transferred: i would definitely eat this place and i would recommend them
Table 8: Generated samples of guided style transfer.
We have proposed a model-based imitation-learning framework for adversarial text generation, by introducing a guider network to model the generation environment. The guider network provides a plan-ahead mechanism for next-word selection. Furthermore, this framework can alleviate the sparse-reward issue, as the intermediate rewards are used to optimize the generator. Our proposed models are validated on both unconditional and conditional text generation, including adversarial text generation and non-parallel style transfer. We achieve improved performance in terms of generation quality and diversity for unconditional and conditional generation tasks.
Acknowledgement The authors would like to thank the anonymous reviewers for their insightful comments. The research was supported in part by DARPA, DOE, NIH, NSF and ONR.
Philip Bachman and Doina Precup. 2015. Data generation as sequential decision making. In NIPS.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. In ICLR.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshop.
Nir Baram, Oron Anschel, and Shie Mannor. 2017. Model-based adversarial imitation learning. In ICML.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. arXiv:1811.02549.
Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua
Bengio. 2017. Maximum-likelihood augmented discrete generative adversarial networks. In arXiv:1702.07983.
Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. 2018. Adversarial text generation via feature-mover's distance. In NeurIPS.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Improving sequence-to-sequence learning via optimal transport. In ICLR.
Ching-An Cheng, Xinyan Yan, Evangelos Theodorou, and Byron Boots. 2019. Accelerating imitation learning with predictive models. In AISTATS.
Pengyu Cheng, Renqiang Min, Dinghan Shen, Yizhe Zhang, Yitong Li, and Lawrence Carin. 2020. Improving disentangled text representation learning with information theoretical guidance. In ACL.
William Fedus, Ian Goodfellow, and Andrew M Dai. 2018. Maskgan: Better text generation via filling in the _. ICLR.
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic compositional networks for visual captioning. In CVPR.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. 2016. Continuous deep q-learning with model-based acceleration. In ICML.
Caglar Gulcehre, Francis Dutil, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Character-level neural machine translation with planning. In NIPS.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2017. Long text generation via adversarial training with leaked information. In AAAI.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Controllable text generation. In ICML.
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In *ICLR*.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. 2018. Model-ensemble trust-region policy optimization. In ICLR Workshop.
Matt J Kusner and Hernández-Lobato José Miguel. 2016. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018a. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In *NAACL*.
Piji Li, Lidong Bing, and Wai Lam. 2018b. Actor-critic based training framework for abstractive summarization. In arXiv:1803.11070.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun. 2017. Adversarial ranking for language generation. In NIPS.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. 2017. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In ICRA.
Weili Nie, Nina Narodytska, and Ankit Patel. 2019. *Relgan: Relational generative adversarial networks for text generation*. In *ICLR*.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In ICML.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. In ICLR.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In ACL.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In ICLR.
Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. 2017. Deep reinforcement learning-based image captioning with embedding reward. In CVPR.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2016. Self-critical sequence training for image captioning. In CVPR.
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv:1509.00685.
Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. 2018. Twin networks: Matching the future for sequence generation. In ICLR.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NIPS.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS.
Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT Press.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In NeurIPS.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI.
Ruiyi Zhang, Changyou Chen, Zhe Gan, Zheng Wen, Wenlin Wang, and Lawrence Carin. 2020. Nestedwasserstein self-imitation learning for sequence generation. In AISTATS.
Ruiyi Zhang, Changyou Chen, Chunyuan Li, and Lawrence Carin. 2018. Policy optimization as Wasserstein gradient flows. In ICML.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial feature matching for text generation. In ICML.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. In SIGIR.
More Generated Samples of Text Generation Table 13 lists more generated samples on the proposed GMGAN and its baselines. From the experiments, we can see, (i) SeqGAN tends to generate shorter sentences, and the readability and fluency is very poor. (ii) LeakGAN tends to generate very long sentences, and usually longer than the original sentences. However, even with good locality fluency, its sentences usually are not semantically consistent. By contrast, our proposed GMGAN can generate sentences with similar length to the original sentences, and has good readability and fluency. This is also validated in the Human evaluation experiment.
Image Captioning We conduct experiments on image captioning (Karpathy and Fei-Fei, 2015), investigating benefits brought by the Guider network. In image captioning, instead of using a discriminator to define final rewards for generated sentence, we adopt evaluation metrics computed based on human references. The final rewards appear more important as they contain reference (ground-truth) information. Feature-matching rewards work as a regularizer of the final rewards. We call our model in this setting a guider-matching sequence training (GMST) model. An overview of GMST is provided in the Appendix. We test our proposed model on the MS COCO dataset (Karpathy and Fei-Fei, 2015), containing 123,287 images in total. Each image is annotated with at least 5 captions. Following Karpathy's split (Karpathy and Fei-Fei, 2015), 5,000 images are used for both validation and testing. We report BLEU- $k$  ( $k$  from 1 to 4), CIDEr (Vedantam et al., 2015), and ME-TEOR (Banerjee and Lavie, 2005) scores. We consider two settings:  $(i)$  using a pre-trained 152-layer ResNet (He et al., 2016) for feature extraction, where we take the output of the 2048-way pool5 layer from ResNet-152, pretrained on the ImageNet dataset; and  $(ii)$  using semantic tags detected from the image as features (Gan et al., 2017). We use an LSTM with 512 hidden units with mini-batches of size 64. Adam (Kingma and Ba, 2014) is used for optimization, with learning rate  $2 \times 10^{-4}$ . We pretrain the captioning model for the maximum 20 epochs, then use the reinforcement learning to train it for 20 epochs and test on the best model on the validation set.
The results are summarized in Table 9. When
Method	BLEU-3	BLEU-4	METEOR	CIDEr
No attention, Greedy, Resnet-152
MLE	37.2	26.5	23.1	83.9
Guider	38.0	27.3	23.9	85.4
MIXER (BLEU)	39.1	29.3	22.3	79.7
SCST (BLEU)	41.6	31.6	23.1	87.5
GMST (BLEU)	41.8	32.1	23.4	87.9
MIXER (CIDEr)	39.1	27.7	23.0	90.9
SCST (CIDEr)	41.2	30.0	24.3	98.6
GMST (CIDEr)	41.3	30.3	24.4	100.1
No attention, Greedy, Tag
MLE	39.4	28.8	24.4	91.3
Guider	39.6	29.0	24.6	92.7
MIXER (BLEU)	42.4	32.2	23.7	90.4
SCST (BLEU)	43.9	33.6	24.5	95.9
GMST (BLEU)	44.3	33.9	24.5	97.1
MIXER (CIDEr)	42.1	30.8	24.7	101.2
SCST (CIDEr)	43.6	32.1	25.4	105.5
GMST (CIDEr)	44.1	32.6	25.5	107.4
Table 9: Results for image captioning on the MS COCO dataset; the higher the better for all metrics.
comparing an AutoEncoder (AE) with a variant implemented by adding a guider network (Guider), improvements are observed. We compare the proposed GMST with SCST. Note the main difference between GMST and SCST is that the former employs our proposed feature-matching reward, while the latter only considers the final reward provided by evaluation metrics. GMST achieves higher scores compared with SCST on its optimized metrics. The gain of GMST compared with SCST comes from the immediate rewards, which can maintain the semantic consistency and sentence structure, preventing language-fluency damage caused by only focusing on evaluation metrics. Specifically, the average length of generated sentence with a Guider is 15.7, and 12.9 for traditional generator.
Comparison with MLE The guider network models the long-term dependency and overcome the issue of sparse reward inspired by model predictive control (MPC). The experiments aim to quantify the gain when incorporating MPC for imitation learning, i.e., MLE and RL finetune.
We provide an additional comparison with Caccia et al. (2018) and evaluate the diversity and quality with BLEU scores. We also report the F1-BLEU which considers both diversity and quality in Table 10.
B Discussions of the Guider Network