SlowGuess/ACL_2020_OCR huggingface dataset:
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 169-214.
Tong Che, Yanran Li, Ruixiang Zhang, R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. 2017. Maximum-likelihood augmented discrete generative adversarial networks. CoRR, abs/1702.07983.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2016. A learned representation for artistic style. CoRR, abs/1610.07629.
William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. MaskGAN: Better text generation via filling in the _____. In International Conference on Learning Representations.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Proceedings of AAAI.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proc. of NIPS, page 2672.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR, abs/1706.08500.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167.
Matt J. Kusner and José Miguel Hernández-Lobato. 2016. GANS for sequences of discrete elements with the gumbel-softmax distribution. CoRR, abs/1611.04051.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. 2017. Adversarial ranking for language generation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3155-3165. Curran Associates, Inc.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision - ECCV 2014, pages 740-755, Cham. Springer International Publishing.
Cyprien de Masson d'Autume, Shakir Mohamed, Michaela Rosca, and Jack Rae. 2019. Training language gans from scratch. In Advances in Neural Information Processing Systems, pages 4302-4313.
Igor Melnyk, Cicero Nogueira dos Santos, Kahini Wadhawan, Inkit Padhi, and Abhishek Kumar. 2017. Improved neural text attribute transfer with non-parallel data. In NIPS Workshop on Learning Disentangled Representations: from Perception to Control.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. arXiv preprint arXiv:1804.09000.
Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 189-194, Melbourne, Australia. Association for Computational Linguistics.
Cicero Nogueira dos Santos, Youssef Mroueh, Inkit Padhi, and Pierre Dognin. 2019. Learning implicit generative models by matching perceptual features. In International Conference on Computer Vision (ICCV).
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30, pages 6830-6841.
Nina Narodytska Weili Nie and Ankit Patel. 2019. *Relgan: Relational generative adversarial networks for text generation*. In *ICLR*.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, pages 2852-2858.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017a. Adversarial feature matching for text generation. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 4006-4015.
Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, and Lawrence Carin. 2017b. Deconvolutional paragraph representation learning. In NIPS, pages 4172-4182.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. SIGIR.
SeqGFMN Generator: We use a deconvolutional generator that extends the decoder architecture proposed in (Zhang et al., 2017). It consists of three strided deconvolutional layers followed by cosine similarity between the generated token embeddings and an embedding matrix. Our adaptations are as follows: (1) we added two convolutional layers after the second deconvolution; (2) we added a self-attention layer before the last deconvolutional layer; (3) we added a convolutional layer after the last deconvolutional layer; (4) after the final convolution, we multiply the resulting token embeddings by the embedding matrix and apply the softmax function to generate a probability distribution over the vocabulary. We use the embedding matrix from BERT model and this matrix is not updated during the training of seqGFMN. The number of convolutional filters used is 400 with kernel size of 5.
SeqGFMN Training: SeqGFMNs are trained with an ADAM optimizer for which most hyperparameters are kept fixed across datasets. We use  $n_z = 100$  and minibatch size of 128. We use learning rates of  $10^{-4}$  and  $10^{-3}$  for updating  $G$ , and ADAM Moving Averages (AMA), respectively. The generator is trained for about 100K iterations.
Feature Extractor Details: In the experiments with GloVe and FastText, we used their default 300 dimension vectors pre-trained on 6 billion tokens from Wikipedia 2014 & Gigaword 5, and English Wikipedia, respectively. In the experiments with BERT, we use  $\mathrm{BERT}_{\mathrm{BASE}}$  model, which contains 12 layers and produces 768 features per token per layer. When using a maximum sequence of 32, that leads to a total 294,912 features.
B Unconditional Text Generation
An interesting comparison would be between SeqGFMN and GANs that use BERT as a pre-trained discriminator. However, GANs fail to train when a very deep network is used as the discriminator. Moreover, SeqGFMN also outperforms GAN generators even when shallow word embeddings (Glove / FastText) are used to perform feature matching. Pretrained word embeddings are normally used in GANs for text.
In Tab. 4, we present randomly selected samples that were generated by SeqGFMN and RelGAN. These samples corroborate the quantitative results
and show that SeqGFMN can generate good text when trained from scratch. At the same time, the state-of-the-art method RelGAN is unable to generate reasonable text without pretraining.
C Class-Conditional Generation
In Tab. 5, we present cherry-picked examples of generated text. Interestingly, since our input noise  $z$  is transformed according to sentiment  $c$ , we implicitly have a pairing between  $z_0$  and  $z_1$ . Text generated from  $z_0$  and  $z_1$  are related to the same  $z$ . The effect of this implicit pairing can be seen in the examples where sentences seem somehow related, but of the opposite sentiment. Qualitatively, conditional SeqGFMN models can leverage class information to improve generation.
In Table 6, we present samples of original and sentiment transferred sentences. For each original sentence, we show the reference transferred sentence from the test set (done by a human) and the sentence that was transferred by SeqGFMN. Similar to other recently proposed UTST methods, the most successful cases of sentiment transfer are the ones where the transfer can be done by removing and replacing a few words of the sentence. In Table 6, the last example of each block are cases where SeqGFMN does not do a good job when significant changes in the original sentence are required to perform a more fluent sentiment transfer.
D Unsupervised Text Style Transfer
The baselines are calculated with the data collected by (Luo et al., 2019)  ${}^{1}$  and using Unsupervised NMT methods (Zhang et al., 2018).
E Interpolation
We interpolate in the latent space of SeqGFMN  $z$  and check whether the sentences generated by the interpolation are syntactically and/or semantically related. In detail, we sample two vectors  $z_0$  and  $z_1$  from the prior distribution  $p_z$  and build intermediate points  $z_{\lambda} = \lambda z_{1} + (1 - \lambda)z_{0}$ . In Tab. 7, we show samples from two interpolations, on models trained on COCO and WMT news dataset. In both these cases, we notice that there exists some syntactic and/or semantic relationship between the sentences along the interpolating path. This is supporting evidence that the latent space induced by SeqGFMN is meaningful, and related sentences are close together in this latent space.
$^{1}$ https://github.com/luofuli/DualRL/
3862
Model	COCO
SeqGFMN	a 747 aircraft plane flying on a runway .
a kitchen with a kitchen sink and a microwave on the counters .
a bike flag showcasing a person sitting near a street sign .
a bathroom with a toilet on the counter .
RelGAN	fry up on a nuts cargo black tonic rocks kept cruising basket adorable graveyard .
border itl washer table a an green with bmw suit heater down . his pushed
docked sofas wave messy nursing , triple black school a continue plane siking BBQ pickup .
quadruple several lots a loft buckets vines a bullhorn the appliances sidewalk sidewalk . uniforms
Model	WMT News
SeqGFMN	the ban did nothing but say voters were illegally investing their time at college and to take on your calls at
[CONT.] court , ” ross . announced .
in addition , 32 typical economies in this period are reportedly pledged to have trillion pledged in another
[CONT.] time , typically , tens to millions in million in feed .
RelGAN	should should children about about about states .
inquiry matthew his s a about am . .
appeal only over a ve about found .
Table 4: Randomly sampled sentences from generators trained from scratch on COCO and WMT News datasets.
Positive Sentiment generated z1	Negative Sentiment generated from z0
full of good food	everything is bad food
love this place	avoid this place
good job	horrible !
just perfect because my entire menu was fabulous	completely upset with the salon
everything is good !	disgusting
the service staff is extremely welcoming - and my mom loved it	the salon itself is very poor , and my mom admitted it
Table 5: Sentences generated using conditional SeqGFMN trained on Yelp Reviews dataset.
Positive Sentiment (Original)	Negative Sentiment (Transferred)
place was clean and well kept , drinks were reasonably priced .	place was dirty and drinks were expensive and watered down . (GT)
place was dirty and horribly kept , drinks were horribly priced . (SeqGFMN)
food is very fresh and amazing !	food was old and stale . (GT)
food was ridiculous , too . (SeqGFMN)
this place reminds me of home !	this place reminds me why i want to go home . (GT)
this jerk reminds me of trash . (SeqGFMN)
Negative Sentiment (Original)	Positive Sentiment (Transferred)
the decor was seriously lacking .	the decor was nice . (GT)
the decor was superb . (SeqGFMN)
now the food : not horrible , but below average .	now the food : not bad , above average . (GT)
now the food is fantastic ! (SeqGFMN)
i wish i could give less than one star .	i wish there were more stars to give . (GT)
i love getting them ! (SeqGFMN)
Table 6: Examples of sentiment transferred texts using SeqGFMN. (GT) = ground truth produced by a human.
COCO
a group of people sleeps in the street
a group of people standing in the street
a toy of people warming a street sidewalk
an automobile car lies on an short parking road
an automobile car lies on an green parking road
an automobile car lies on an green bike field
the automobile car lies on an green parking field
the automobile car is on an green parking field
WMT News
“although that might do nothing -i admit it- and i’ve invested time time at work,” i tend to say it doesn’t do nothing.
“although the odds do it -i get it- and ross hasn always conceded his chance at it,” i tend to say our odds are there.
reportedly upon the call to court, i get it, while romney has promised that his ban did nothing but say voters had better announce...
reportedly upon the call at court and i get it, while voters didn ##rem realize the ban was there.
the said pledge would take on one another day, sexually claiming to top the worst in your period at the academy.
the us has to feed two-thirds in one month, typically in the best ##quest best ##gist at the in & in millions in.
this will cover two-thirds billion trillion in this period, possibly two-thirds - 630 in one months.
in addition, regulators selected millions in one years, potentially billions in another decade, possibly the bottom-profile economies ...
Table 7: Interpolation in the latent space  $z$  of SeqGFMN models trained on COCO Captions and WMT News.