SlowGuess/NAACL_2021_OCR huggingface dataset:
Overall We collect overall judgments of the system outputs using relative ranking: given the informal reference and a formal human rewrite, workers are asked to rank system outputs in the order of their overall formality, taking into account both fluency and meaning preservation. An overall score is then computed for each model via averaging results across annotating instances.
$^{13}$ https://github.com/xingniu/multitask-ft-fsmt  
 $^{14}$ https://github.com/cindyxinyiwang/deep-latent-sequence-model  
 $^{15}$ See A.A for discussion on this assumption.
Table 7: Automatic and human evaluation results for multilingual FoST.  $\star$  denotes statistical significance differences compared to RULE-BASED ( $p < 0.05$ ). NMT-based results are ensembles of 4 systems; unsupervised models are average results across 4 reruns. Darker colors denote higher rankings, bold numbers denote best systems.
Table 7 shows automatic results for all models across the four dimensions as well as human ratings for selected top models.
NMT-based model evaluation Concretely, the RULE-BASED baselines are significantly  $(p < 0.05)$  the best performing models in terms of meaning preservation across languages. This result is intuitive as the RULE-BASED models act at the surface level and are unlikely to change the informal sentence's meaning. The BACKTRANSLATE ensemble systems are the second-best performing models in terms of meaning preservation, while the ROUND-TRIP MT outputs diverge semantically from the informal sentences the most. Those results are consistent across languages and human/automatic evaluations. On the other hand, when we compare systems in terms of their formality, we observe the opposite pattern: the RULE-BASED and BACKTRANSLATE outputs are the most informal compared to the other ensemble NMT-based approaches across languages. Interestingly, the ROUND-TRIP MT outputs exhibit the largest formality shift for BR-PT and FR as measured by human evaluation. The trade-off between meaning preservation and formality among models was also observed in EN (RT18). Moreover, when we move to fluency, we notice similar results across systems. Specifically, human evaluation assigns almost all models an average score of  $>4$ , denoting that system outputs are comprehensible on average, with small differences
between systems not being statistically significant. Notably, perplexity tells a different story: all system outputs are significantly better compared to the RULE-BASED systems across configurations and languages. This result denotes that perplexity might not be a reliable metric to measure fluency in this setting, as noticed in Mir et al. (2019) and Krishna et al. (2020). When it comes to the overall ranking of systems, we observe that the NMT-based ensembles are better than the RULE-BASED baselines for BR-PT and FR, yet by a small margin as denoted by both multi-BLEU and human evaluation. However, the corresponding results for IT denote that there is no clear win, and the NMT-based ensembles still fail to surpass the naive RULE-BASED models, yet by a small margin. Finally, all ensembles outperform the trivial COPY baseline. Table 8 presents examples of system outputs. As a side note, we followed the recommendation of Tikhonov and Yamshchikov (2018) to show the performance of ST models of individual runs and visualize trade-offs between metrics better. Unlike their work which found that reruns of the same model showed wide performance discrepancies, we found that most of our NMT-based models did not vary in performance on XFORMAL. The results can be visualized in A.M.
Unsupervised model evaluation We also benchmark the unsupervised models but focus solely on automatic metrics since they lag behind their supervised counterparts. As shown in Table 7, when
(BR-PT)	INFORMAL	n precise pedir bois q ela vai vir atras!!
XFORMAL REWRITE	Não é precise pedir, bois sei que ela virá em busca!
RULE-BASED	não precisio pedir bois quèela vai vir atras!
ROUND-TRIP MT	Não precisio perguntar porque bois que ela virá atrás de mim!
TRANSLATE TRAIN TAG	Eu não precisio pedir que eu saiba que ela vai vir atras!
MULTI-TASK TAG-STYLE	Eu não precisio pedir que eu saiba que ela vai vir atras!
TRANSLATE TRAIN BACK.	Não precisio pedir porque bois que ela vai vir atras!
(FR)	INFORMAL	drolhe heinnnnnnnnn s étais ma femme de menage!
XFORMAL REWRITE	Le plus amusant est le fait qu'il s'agissait de mon ancienne employée de maison.
RULE-BASED	Drolhe hein s étais ma femme de menage!
ROUND-TRIP MT	C'était drôle ma bonne!
TRANSLATE TRAIN TAG	C'est drôle, mais j'étais ma femme de menage.
MULTI-TASK TAG-STYLE	C'était drôle, c'était ma femme de menage!
TRANSLATE TRAIN BACK.	C'était drôle, mais j'étais ma femme de menage!
(IT)	INFORMAL	un po'di raffreddore ma todo ok!!!
XFORMAL REWRITE	Sono affetta da un leggero raffreddore ma per il resto va tutto bene.
RULE-BASED	Un po'di raffreddore ma todo ok!
ROUND-TRIP MT	Un po'freddo ma va bene!
TRANSLATE TRAIN TAG	Un po'di raffreddore, ma tutto va bene.
MULTI-TASK TAG-STYLE	Un po'di raffreddore, ma va bene.
TRANSLATE TRAIN BACK.	Un po'di raffreddore, ma tutto va bene.
Table 8: Example system outputs on random sentences of XFORMAL.
Figure 3: Heatmaps of LeD scores showing the lexical difference between pairs of systems.
using out-of-domain data (e.g., OpenSubtitles) for training, the models perform worse than their NMT counterparts across all three languages. The difference is most stark when considering self-BLEU and multi-BLEU scores. However, given access to large in-domain corpora (e.g., L26 Yahoo! French Answers) the gap between the two model classes closes with DLSM achieving a multi-BLEU score of 42.1 compared to 48.3 for the best performing NMT model BACKTRANSLATE. This shows the promise of unsupervised methods, assuming a large amount of in-domain data, on multilingual ST tasks.
Lexical differences of system outputs Finally, in Figure 3 we analyze the diversity of outputs by leveraging LeD scores resulting from pair-wise comparisons of different NMT systems. A larger LeD score denotes a larger difference between the lexical choices of the two systems under comparison. First, we observe that the ROUND-TRIP MT outputs have the smallest lexical overlap with the informal input sentences. However, when this observation is examined together with human evaluation results, we conclude that the large number of lexical edits happens at the cost of diverging se
mantically from the input sentences. Moreover, we observe that the average lexical differences within NMT-based systems are small. This indicates that different systems perform similar edit operations that do not deviate a lot from the input sentence in terms of their lexical choices. This is unfortunate given that multilingual FoST requires systems to perform more phrase-based operations, as shown in the analysis in §4.
Evaluation Metric While evaluating evaluation metrics is not a goal of this work (though the data can be used for that purpose), we observe that the top models identified by the automatic metrics generally align with the top models identified by humans. While promising, further work is required to confirm if the automatic measures really do correlate with human judgments.
6 Conclusions & Future Directions
This work extends the task of formality style transfer to a multilingual setting. Specifically, we contribute XFORMAL, an evaluation testbed consisting of informal sentences and multiple formal rewrites spanning three languages: BR-PT, FR, and IT. As in Rao and Tetreault (2018) Turkers can be effective in creating high quality ST corpora. In contrast to the aforementioned EN corpus, we find that the rewrites in XFORMAL tend to be more diverse, making it a more challenging task.
Additionally, inspired by work on cross-lingual transfer and EN FoST, we benchmark several methods and perform automatic and human evaluations on their outputs. We found that NMT-based ensembles are the best performing models for FR and BR-PT—a result consistent with EN—however, they perform comparably to a naive RULE-BASED baseline for IT. To further facilitate reproducibility of our evaluations and corpus creation processes, as well as drive future work, we will release our scripts, rule-based baselines, source data, and annotation templates, on top of the release of XFORMAL.
Our results open several avenues for future work in terms of benchmarking and evaluating FoST in a more language inclusive direction. Notably, current supervised and unsupervised approaches for EN FoST rely on parallel in-domain data—with the latter treating the parallel set as two unpaired corpora—that are not available in most languages. We suggest that benchmarking FoST models in multilingual settings will help understand their ability to generalize and lead to safer conclusions when
comparing approaches. At the same time, multilingual FoST calls for more language-inclusive consideration for automatic evaluation metrics. Model-based approaches have been recently proposed for evaluating different aspects of ST. However, most of them rely heavily on English resources or pretrained models. How those methods can be extended to multilingual settings and how we evaluate their performance remain open questions.
We thank Sudha Rao for providing references and materials of the GYAFC dataset, Chris Callison-Burch and Courtney Naples for discussions on MTurk annotations, Svebor Karaman for helping with data collection, our colleagues at Dataminr, and the NAACL reviewers for their helpful and constructive comments.
Finally, we address ethical considerations for dataset papers given that our work proposes a new corpus XFORMAL. We reply to the relevant questions posed in the NAACL 2021 Ethics FAQ. $^{16}$
7.1 Dataset Rights
The underlying data for our dataset as well as training our FoST models and formality classifiers are from Yahoo! Answers L6 dataset. We were granted written permission by Yahoo (now Verizon) to make the resulting dataset public for academic use.
7.2 Dataset Collection Process
Turkers are paid over 10 USD an hour. We targeted a rate higher than the US national minimum wage of 7.50 USD given discussions with other researchers who use crowdsourcing. We include more information on collection procedures in §3.
7.3 IRB Approval
This question is not applicable for our work.
7.4 Dataset Characteristics
We follow Bender and Friedman (2018) and Gebru et al. (2018) and report characteristics in §3 and §4.
$^{16}$ https://2021.naacl.org/ethics/faq/
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497-511, San Diego, California. Association for Computational Linguistics.
Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1549-1564, Hong Kong, China. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604.
Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communication between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061-1071, Online. Association for Computational Linguistics.
Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6455-6469, Online. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Weston Feely, Eva Hasler, and Adrià de Gispert. 2019. Controlling Japanese honorifics in English-to-Japanese neural machine translation. In Proceedings of the 6th Workshop on Asian Translation, pages 45-53, Hong Kong, China. Association for Computational Linguistics.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daume, and Kate Crawford. 2018. Datasheets for datasets. ArXiv, abs/1803.09010.
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3168-3180, Minneapolis, Minnesota. Association for Computational Linguistics.
Eva Hasler, Adrià de Gispert, Gonzalo Iglesias, and Bill Byrne. 2018. Neural machine translation decoding with terminology constraints. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 506-512, New Orleans, Louisiana. Association for Computational Linguistics.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In Proceedings of International Conference on Learning Representations.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, WWW '16, page 507-517, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 174-180, Baltimore, Maryland. Association for Computational Linguistics.
F. Heylighen, J. Dewaele, and L. Apostel. 1999. Formality of language: definition, measurement and behavioral determinants.
Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2018. The sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 200-207, Boston, MA. Association for Machine Translation in the Americas.
Eduard Hovy. 1987. Generating natural language under pragmatic constraint. Journal of Pragmatics, 11(6):689-719.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596, International Convention Centre, Sydney, Australia. PMLR.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Imat: Unsupervised text attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3095-3107. Association for Computational Linguistics.
Dongyeop Kang, Varun Gangal, and Eduard Hovy. 2019. (male, bachelor) and (female, ph.d) have different connotations: Parallely annotated stylistic language dataset with multiple personas. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Hong Kong.
Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In ICASSP, pages 181-184. IEEE Computer Society.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Elizaveta Korotkova, Agnes Luhtaru, Maksym Del, Krista Liin, Daiga Deksne, and Mark Fishel. 2019. Grammatical error correction and style transfer via zero-shot monolingual translation. CoRR, abs/1903.11283.
Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online. Association for Computational Linguistics.
Shibamouli Lahiri. 2015. Squinky! A corpus of sentence-level formality, informativeness, and implicature. CoRR, abs/1506.02306.
Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Bill Dolan, and Ming-Ting Sun. 2019. Domain adaptive text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3304-3313, Hong Kong, China. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. 2018. Content preserving text generation with attribute controls. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5103-5113. Curran Associates, Inc.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJ-CAI 2019.
Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8671-8680, Online. Association for Computational Linguistics.
Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level of machine translation output. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 193-203, Dublin, Ireland. European Association for Machine Translation.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 495-504, Minneapolis, Minnesota. Association for Computational Linguistics.
Shachar Mirkin and Jean-Luc Meunier. 2015. Personalized machine translation: Predicting translational preferences. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 49-54, Florence, Italy. Association for Computational Linguistics.
Xing Niu and Marine Carpuat. 2020. Controlling neural machine translation formality with synthetic supervision. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 8568-8575.
Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2814-2819, Copenhagen, Denmark. Association for Computational Linguistics.
Xing Niu, Sudha Rao, and Marine Carpuat. 2018. Multi-task neural models for translating between styles within and across languages. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1008-1021, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Richard Yuanzhe Pang. 2019. Towards actual (not operational) textual style transfer auto-evaluation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 444-445, Hong Kong, China. Association for Computational Linguistics.
Richard Yuanzhe Pang and Kevin Gimpel. 2019. Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 138–147, Hong Kong. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.