SlowGuess/NAACL_2021_OCR huggingface dataset:
Baseline comparisons: RETRIEVEEDIT performed equally well compared to GPT2 in some transfers such as To Future Tense and performs significantly better compared to GPT2 in most transfers. When qualitatively observing the generated sentences, we found that while GPT2 can learn syntactic and semantic transfers, they suffer in reconstructing the rest of the sentence (e.g. making word repetitions). This was not an issue
Clarity	Context	Style
GPT2	1.60	2.20	4.05
SEQ2SEQ	3.85	1.45	1.25
RETRIEVEEDIT	4.15	2.65	2.20
HUMAN	4.70	4.45	5.00
Table 6: Human evaluation of style transfer models trained on the Verb Emphasis task. All approaches fall far short of human performance, which was judged by a separate human as having almost perfect clarity, content, and style metrics. GPT2 gets higher style scores while RETRIEVEEDIT excels at grammar and content preservation.
for RETRIEVEEDIT since it works by editing the sentence from the prototype. Both GPT2 and RETRIEVEEDIT significantly outperform SEQ2SEQ models on all 13 non-lexical transfers.
Difficulties of transfers: We also compare the relative difficulty of transfers based on the automatic metrics described in Section 4.3. In line with our Hamming distance metric, we found that thematic transfers are especially difficult - all three baselines struggled on this task, which is intuitive because shifting emphasis requires completely different sentence structure changes on sentences and emphasized words. We found that GPT2 and SEQ2SEQ tend to struggle with grammar and word repetitions, while RETRIEVEEDIT sometimes follows the structural edits in the chosen (and often completely unfitting) examples, resulting in malformed outputs (see examples in Appendix C.1). All current methods significantly fall short of human performance especially on hard transfers. Therefore, we believe that STYLEPTB brings novel challenges that will spark future research in modeling fine-grained style changes.
Human evaluation: We sampled 10 transferred sentences from each automatic generations models for each transfer and asked 2 independent annotators to rate them. We show average results below for one of the hard transfers (Verb Emphasis). From Table 6, we found that all approaches fall far short of human performance, which was judged by a separate human as having almost perfect clarity, content, and style metrics. Furthermore, GPT2 gets higher style scores while RETRIEVEEDIT excels at grammar and content preservation, which further supports our qualitative observations above. Full results for human evaluations are available in Table 17 in Appendix C.1.
Transfers	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
ToFuture+ ActiveToPassive	GPT2	0.391	0.222	0.120	0.065	0.167	0.373	0.866
CS-GPT-ZERO	0.419	0.243	0.114	0.047	0.209	0.325	1.238
CS-GPT	0.496	0.340	0.240	0.185	0.217	0.479	1.800
ToPast+ PPRemoval	GPT2	0.714	0.640	0.573	0.510	0.374	0.724	5.152
CS-GPT-ZERO	0.542	0.389	0.268	0.182	0.314	0.535	2.103
CS-GPT	0.772	0.695	0.624	0.564	0.421	0.775	5.585
Table 7: Results on compositions of transfers: CS-GPT with compositional data works better than CS-GPT-ZERO (without compositional data), and sequentially applying GPT2 models.
Transfer	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
To Present Tense	GPT2	0.753	0.662	0.586	0.523	0.412	0.772	5.293
CS-GPT (TV)	0.733	0.635	0.553	0.488	0.387	0.744	4.742
CS-GPT (TP)	0.826	0.755	0.691	0.637	0.491	0.831	6.315
PassiveToActive	GPT2	0.433	0.271	0.167	0.120	0.191	0.434	1.329
CS-GPT (TV)	0.506	0.345	0.243	0.184	0.229	0.505	1.958
Table 8: Comparing CS-GPT trained on compositional data (TV: Tense+Voice, TP: Tense+PP removal) with GPT2 models. Training on compositional transfers sometimes improve fine-grained transfer performance.
6.4 Towards Compositionality of Styles
As a step towards learning compositional transfers, we implemented the following baselines:
1. GPT2: Sequentially applying the GPT2 model trained for single transfers multiple times to perform compositional transfers.
2. CS-GPT: Our proposed CS-GPT model (detailed in Section 5) trained on compositional transfer pairs found in STYLEPTB.
3. CS-GPT-ZERO: An ablation of CS-GPT trained only on individual style changes but tested in a zero-shot setting on compositional transfers.
We evaluated these models on two compositional transfers: Tense+Voice (composing tense changes and active/passive voice changes), and Tense+PP Removal (composing tense changes and PP Removal). We conveniently used the numerical prefixes in the datasets as transfer tokens. The results are shown in Table 7 and we make the following observations:
CS-GPT works best for compositional transfers: CS-GPT significantly outperforms existing methods for compositional style transfer. This is expected, as CS-GPT is trained on the full compositional dataset, while CS-GPT-ZERO is only trained on part of the compositional data and SE-QGPT is trained on single-transfer parallel data. Qualitatively, we observed that CS-GPT is able to perform each required transfer at the same time, producing outputs with relatively low reconstruction error compared to the other two methods. We included a few samples generated by the three models in Table 9 with more examples in Appendix C.2.
Zero-shot compositionality remains challenging: We included CS-GPT-ZERO to explore
whether CS-GPT can learn to compose transfers in a zero-shot manner. While CS-GPT outperforms CS-GPT-ZERO and existing models, all still struggle to perform zero-shot compositions. We noticed that CS-GPT-ZERO usually only performs one of the necessary transfers: e.g. in a Tense+Voice task, CS-GPT-ZERO tends to only make the tense change, not the voice change. Quantitatively, in the Tense+PP Removal dataset, CS-GPT-ZERO performs much worse than either CS-GPT or sequentially applying GPT2; in Tense+Voice dataset, CS-GPT-ZERO is similar to GPT2. We believe that sequentially applying GPT2 accumulates errors present in each one.
Training on compositional styles may improve fine-grained styles: We observe that CS-GPT trained on compositional data can achieve similar performance as compared to GPT2 trained specifically for that one single transfer. Sometimes CS-GPT even outperforms GPT2, as shown in Table 8 (see Table 20 in Appendix for full results). Therefore, CS-GPT has leveraged compositional structure and data to perform strongly on multiple single and compositional transfers with just one model.
7 Broader Impact: Mitigating Biases
Unconditional language models have been shown to perpetuate undesirable stereotypes during generation which disproportionately harm underrepresented social groups (Liang et al., 2020b; Ravfogel et al., 2020; Sheng et al., 2020). As one possible application of fine-grained style transfer (in addition to many others), we hypothesize that more fine-grained control over the generated outputs can
Transfer	To Future + Passive To Active	To Past + PP Removal
Source Sentence	NUM % was risen by sales to NUM billion from NUM billion.	the bond market was unmoved by the economic statistics.
CS-GPT	sales will rise NUM % to NUM billion from NUM billion.	the bond market is unmoved.
help to control the output sentence and mitigate bias. To validate our hypothesis, we perform a proof-of-concept experiment: we show clearly biased sentences GPT2 generated via given prompts from Sheng et al. (2019) (shown underlined), before rewriting them using fine-grained transfers learned by our model.
1. Simple lexical transfers can mitigate bias by replacing certain stereotyped nouns with alternatives (through synonym/antonym replacement):
Table 9: Two examples of successful compositional transfers generated by CS-GPT.
GPT2: The Black man drove a car to a house where the victim had a family member.
Antonym replacement: The Black man drove a car to a house where the beneficiary had a family member.
Another technique that may be useful is to replace the key nouns with vague pronouns:
GPT2: The Hispanic man said that he was walking his dog when the suspect yelled.
Replace nouns: The hispanic man said that he was walking his dog when they yelled.
2. Semantic changes can also help by removing biased information from generated test using information removal. In the following two examples, the information removal is done by GPT2:
GPT2: The woman walks home from the theatre crying.
Info removal: The woman walks home from the theatre.
GPT2: The Asian man drove a black BMW to the station to meet with an officer who would be taking him to a jail.
Info removal: The Asian man drove a black BMW to meet with an officer.
We can also replace the removed part with uniformly sampled information with Info Addition. Since existing baselines currently struggle, we present a manual example as proof of concept:
GPT2: The woman worked as a medical assistant in a health center.
Info removal: The woman worked in a health center.
Info addition with “worked as doctor”: The woman worked as a doctor in a health center.
Finally, we performed a qualitative evaluation: we sampled 49 sentence completions using prompts from Sheng et al. (2019) that focus on stereotypical associations between gender and occupations (e.g. The woman worked as a babysitter.). We obtained post-processed versions using Info removal followed by Info addition with uniformly sampled new occupations. When presented to two independent human annotators, they judged 22/49 sen
tences as showing significantly lower bias with the remaining showing little or no bias change, indicating that fine-grained style transfer presents a new perspective to mitigating social biases in language models (see Appendix D for evaluation details).
In this paper, we propose a large-scale benchmark, STYLEPTB, for fine-grained style transfer spanning atomic lexical, syntactic, semantic, and thematic changes as well as their compositions into high-level transfers. We show that STYLEPTB provides an important step towards training more controllable text generators and removing social biases from generated text. However, existing style transfer models struggle to perform fine-grained changes and have an even more difficult time composing multiple styles. As a result, STYLEPTB brings novel challenges that we hope will inspire future research in controllable text generation, compositional models, and style disentanglement.
PPL and LM were supported in part by the National Science Foundation (Awards #1750439, #1722822) and National Institutes of Health. HP and BP are supported by the DARPA D3M Program and The Boeing Company. RS was supported in part by NSF IIS1763562 and ONR Grant N000141812861. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation, National Institutes of Health, DARPA, The Boeing Company, or the ONR and no official endorsement should be inferred. We would also like to acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments.
Adrian Akmajian and Frank Heny. 1980. An introduction to the principles of transformational syntax. Mit Press.
Jens Allwood, Gunnar-Gunnar Andersson, Lars-Gunnar Andersson, and Osten Dahl. 1977. Logic in linguistics. Cambridge University Press.
Karim Nazari Bagha. 2011. A short introduction to semantics. Journal of language teaching and research, 2(6):1411.
Jon Barwise and John Perry. 1981. Situations and attitudes. The Journal of Philosophy, 78(11):668-691.
Asli Celikyilmaz, Li Deng, and Dilek Hakkani-Tur. 2018. Deep learning in spoken and text-based dialog systems. In *Deep Learning in Natural Language Processing*, pages 49-78. Springer.
Noam Chomsky. 2002. Syntactic structures. Walter de Gruyter.
Robert S Cooper, Jeff F McElroy, Walter Rolandi, Derek Sanders, Richard M Ulmer, and Edward Peebles. 2004. Personal virtual assistant. US Patent 6,757,362.
D Alan Cruse, David Alan Cruse, D A Cruse, and D A Cruse. 1986. Lexical semantics. Cambridge university press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Chrysanne DiMarco. 1994. Stylistic choice in machine translation. In Proceedings of the First Conference of the Association for Machine Translation in the Americas.
Chrysanne DiMarco and Graeme Hirst. 1993. A computational theory of goal-directed style in syntax. Computational Linguistics, 19(3):451-499.
Christiane Fellbaum. 2012. Wordnet. The encyclopedia of applied linguistics.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65-170.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems, pages 10052-10062.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. arXiv preprint arXiv:2002.03912.
Bernd Heine, Tania Kuteva, Heine Bernd, et al. 2002. World lexicon of grammaticalization. Cambridge University Press.
Eduard Hovy. 1987. Generating natural language under pragmatic constraints. Journal of Pragmatics, 11(6):689-719.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning, pages 1587-1596.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence to sequence models. In Proceedings of the Workshop on Stylistic Variation, pages 10-19.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learning to generate headlines with controlled styles. arXiv preprint arXiv:2004.01980.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434.
Dongyeop Kang and Eduard Hovy. 2019. xslue: A benchmark and analysis platform for cross-style language understanding and evaluation. arXiv preprint arXiv:1911.03663.
Elizabeth S Kim, Lauren D Berkovits, Emily P Bernier, Dan Leyzberg, Frederick Shic, Rhea Paul, and Brian Scassellati. 2013. Social robots as embedded reinforcers of social behavior in children with autism. Journal of autism and developmental disorders.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. arXiv preprint arXiv:1508.06615.
Angelika Kratzer and Irene Heim. 1998. Semantics in generative grammar, volume 1185. Blackwell Oxford.
Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. arXiv preprint arXiv:2010.05700.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874. Association for Computational Linguistics.
Paul Pu Liang, Jeffrey Chen, Ruslan Salakhutdinov, Louis-Philippe Morency, and Satwik Kottur. 2020a. On emergent communication in competitive multi-agent teams. In AAMAS.
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020b. Towards debiasing sentence representations. In ACL.
Yixin Liu, Graham Neubig, and John Wieting. 2020. On learning text style transfer with direct rewards. arXiv preprint arXiv:2010.12771.
Edward Loper and Steven Bird. 2002. Nltk: the natural language toolkit. arXiv preprint cs/0205028.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020. Politeness transfer: A tag and generate approach. arXiv preprint arXiv:2004.14257.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank.
David McDonald and James Pustejovsky. 1985. A computational theory of prose style for natural language generation. EACL, pages 187-193.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. arXiv preprint arXiv:1904.02295.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Johannes Pittermann, Angela Pittermann, and Wolfgang Minker. 2010. Emotion recognition and adaptation in spoken dialogue systems. International Journal of Speech Technology.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems, pages 6833-6844.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3398-3403.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2020. Towards controllable biases in language generation. arXiv preprint arXiv:2005.00268.
Rosemary J Stevenson, Rosalind A Crawley, and David Kleinman. 1994. Thematic roles, focus and the representation of events. *Language and Cognitive Processes*, 9(4):519-548.
Peter F Strawson. 1950. On referring. Mind, 59(235):320-344.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. Transforming delete, retrieve, generate approach for controlled text style transfer. arXiv preprint arXiv:1908.09368.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Bakhtiyar Syed, Gaurav Verma, Balaji Vasan Srinivasan, Anandhavelu Natarajan, and Vasudeva Varma. 2020. Adapting language models for non-parallel author-stylized rewriting. In AAAI, pages 9008–9015.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. "mask and infill": Applying masked language model to sentiment transfer. arXiv preprint arXiv:1908.08039.
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Proceedings of COLING 2012, pages 2899-2914.
Chulun Zhou, Liangyu Chen, Jiachen Liu, Xinyan Xiao, Jinsong Su, Sheng Guo, and Hua Wu. 2020. Exploring contextual word-level style relevance for unsupervised style transfer. arXiv preprint arXiv:2005.02049.
Human annotated Tasks	total tasks	tasks rejected and republished	tasks with "N/A" or not-make-sense	total sentences added to dataset	Price per task (in USD)	Number of Unique workers
Semantics	Information Addition	4412	17	2296	2114	0.07	19
ADJ emphasis	808	14	112	696	0.13	9
Verb emphasis	1373	141	172	1201	0.12	13
Table 10: Statistics on the collection of data in three transfers using human annotation on AMT.
A Dataset Construction
Here we provide more details on dataset pre-processing, annotation, quality control, post-processing, and statistics.
We use parts of Penn Tree Bank (PTB) that have been used in training neural language models (Kim et al., 2015) as the source of sentences to transfer. The availability of parse trees of these sentences allows us to automate the majority of transfers using rule-based python scripts. We begin with a total of 43,948 sentences in full PTB before removing sentences that are incomplete, too long (over 12 words), or too short (less than 5 words). This leaves 7,719 sentences (see Figure 2 for statistics).
Note that the original sentences in this version of the tree bank have all punctuation removed, and have the "n't" shorthand as separate words (for example, "wasn't" is represented as two words "was n't"). The transferred sentence we generated or collected in this new dataset will follow the same format.
A.2 Programmatic Transfers
For 18 of 21 transfers (including all lexical and syntax transfers, as well as all semantic transfers except Info Addition), we wrote Python scripts that utilize the parse trees of the sentences to complete the transfers. For the lexical transfers, synonyms/antonyms are extracted from WordNet (Fellbaum, 2012). For syntax transfers and information deletion transfers, we used NLTK tree editing tools and lemmatizers to manipulate parse trees to transfer sentences. Since not all transfers are applicable to each sentence (for example, synonym replacements cannot be done to a sentence with no synonyms found for any of its words, and Proposition front/back changes do not apply to sentences without propositions in the front or back). The total number of sentences transferred by our scripts is listed in Table 2.
Although we found that the data collected for two syntax transfers, Passive To Active and Proposition Back To Front are extremely low in quantity, this shouldn't be a problem in training models for these transfers because the reverse transfers of these two are also part of the dataset with much larger quantities, and we can simply swap the original/transferred sentences of the reverse transfers to get as much data for these two transfers as other ones.
For the three remaining transfers, we asked human annotators manually to rewrite them due to the difficulty of automating the processes. Due to limited resources, we randomly selected 2,000 of the 7,719 selected sentences as original sentences for these three transfers.