SlowGuess/NAACL_2021_OCR huggingface dataset:
Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. Transactions of the Association for Computational Linguistics, 4:61-74.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074-1084, Valencia, Spain. Association for Computational Linguistics.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez-Sánchez. Promp-sit's submission to wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35-40, San Diego, California. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Mingyue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019.
Semi-supervised text style transfer: Cross projection in latent space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4937-4946, Hong Kong, China. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6830-6841. Curran Associates, Inc.
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2018. Multiple-attribute text style transfer. CoRR, abs/1811.00552.
Aleksey Tikhonov and Ivan Yamshchikov. 2018. Sounds wilde. phonetically extended embeddings for author-stylized poetry generation. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117-124, Brussels, Belgium. Association for Computational Linguistics.
Alexey Tikhonov, Viacheslav Shibaev, Aleksander Nagaev, Aigul Nugmanova, and Ivan P. Yamshchikov. 2019. Style transfer for texts: Retrain, report errors, compare with rewrites. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3936-3945, Hong Kong, China. Association for Computational Linguistics.
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003-3008, Brussels, Belgium. Association for Computational Linguistics.
Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3573-3578, Hong Kong, China. Association for Computational Linguistics.
Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4873-4883.
Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In ACL.
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Proceedings of COLING 2012, pages 2899-2914, Mumbai, India. The COLING 2012 Organizing Committee.
Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. 2016. Controlling the voice of a sentence in Japanese-to-English neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 203–210, Osaka, Japan. The COLING 2016 Organizing Committee.
Ivan P. Yamshchikov, Viacheslav Shibaev, Nikolay Khlebnikov, and Alexey Tikhonov. 2020. Styletransfer and paraphrase: Looking for a sensible semantic similarity metric. ArXiv, abs/2004.05001.
Yi Zhang, Tao Ge, and Xu SUN. 2020. Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3221-3228, Online. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texy- gen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Con- ference on Research; Development in Information Retrieval, SIGIR '18, page 1097-1100, New York, NY, USA. Association for Computing Machinery.
A Does translation preserve formality?
We examine the extend to which machine translation—through the AWS service—affects the formality level of an input sentence: starting from a set of English sentences we have formality judgments for (ORIGINAL-EN), we perform a round-trip translation via pivoting through an auxiliary language (PIVOT-X). We then compare the formality prediction scores of the English Formality regression model for the two versions of the English input. In terms of Spearman correlation, the model's performance drops by 7.5 points on average when tested on round-trip translations. To better understand what causes this drop in performance, we present a per formality bin analysis in Table 9. On average we observe that translation preserves the formality level of formal sentences considerably well, while at the same time it tends to shift the formality level of informal sentences towards formal values—by a margin smaller than 1 point—most of the times. To account for the formalization effect of translation, we draw the line between formal and informal sentences at the value of  $-1$  for scores predicted by multilingual regression models. This decision is based on the following intuition: if the formality shift of machine translated informal sentences is around  $+1$  value, the propagation of English formality labels imposes a negative shift of formal sentences in the model's predictions.
B Amazon Web Service details
We compute the performance of the AWS system on 2.5K randomly sentences from OpenSubtitles, as a sanity check of translation performance. We report BLEU of 37.16 (BR-PT), 33.79 (FR), and 32.67 (IT).
C Rule-based baselines
We develop a set of rules to automatically make an informal sentence more formal via performing surface-level edits similar to the EN rule-based system of Rao and Tetreault (2018). The set of extracted rules are shared across languages with the only difference being the list of abbreviations:
Normalize punctuation We remove punctuation symbols that are repeated  $> = 2$  times.
Character repetition We trailed characters repeated  $> = 3$  times (e.g., ciaooo  $\rightarrow$  ciao (IT)).
Normalize casing Several sentences might consist of words that are written in upper case. We lower case all characters apart from the first characters of the first word.
Normalize abbreviations Informal text might contain slang words that are be abreviated. We hand-craft a list of expansions for each language. For example, we replace  $kra \rightarrow cara$  (BR-PT),  $ta \rightarrow ti$  amo (IT), and  $bjr \rightarrow bonjour$  (FR), using publicly available resources.[17, 18, 19] The resulting lists sizes are: 64 (BR-PT), 21 (IT), and 48 (FR).
D NMT architecture details
We used the NMT implementations of Niu et al. (2018) that are publicly available: https://github.com/xingniu/multitask-ft-fsmt. The NMT models are implemented as bi-directional LSTMs on Sockeye (Hieber et al., 2018), using the same configurations across languages to establish fair comparisons. We use single LSTMs consisting of a single of size 512, multilayer perceptron attention with a layer size of 512, and word representations of size 512. We apply layer normalization and tie the source and target embeddings as well as the output layer's weight matrix. We add dropout with probability 0.2 (for the embeddings and LSMT cells in both the encoder and the decoder). For training, we use the Adam optimizer with a batch size of 64 sentences and checkpoint the model every 1000 updates. Training stops after 8 checkpoints without improvement of validation perplexity. For decoding, we use a beam size of 5.
E Inter-annotator agreement on human evaluation
To quantify inter-annotator agreement for the tasks of formality, meaning preservation, and fluency we measure the correlation of their ordinal ratings using inter-class correlation (ICC) and their categorical agreement using a variations of Cohen's  $\kappa$  coefficient. For the latter, given that we collect human evaluation judgments through crowd-sourcing, we follow the simulation framework of Pavlick and Tetreault (2016) to quantify agreement. Concretely, we simulate two annotators (Annotator 1,
17https://braziliangringo.com/   
brazilianinternetslang/ 18https://www.dummies.com/languages/   
italian/texting-and-chatting-in-italian/ 19https://frencttogether.com/   
french-texting-slang/
Informal bins	Formal bins
[-3,-2]	[-2,-1]	[-1,0]	[0,1]	[1,2]	[2,3]
GOLD-STANDARD	-2.35 ± 0.31	-1.49 ± 0.36	-0.57 ± 0.33	0.46 ± 0.32	1.33 ± 0.34	2.27 ± 0.25
ORIGINAL-EN	-1.81 ± 0.59	-1.18 ± 0.73	-0.53 ± 0.71	0.30 ± 0.82	1.17 ± 0.68	1.50 ± 0.45
PIVOT-IT	-1.58 ± 0.67	-0.95 ± 0.74	-0.35 ± 0.68	0.34 ± 0.86	1.12 ± 0.73	1.41 ± 0.54
Δ	+0.23 ± 0.45*	+0.23 ± 0.54*	+0.18 ± 0.53*	+0.04 ± 0.39	-0.05 ± 0.23	-0.09 ± 0.24
PIVOT-FR	-1.44 ± 0.77	-0.93 ± 0.75	-0.37 ± 0.72	0.38 ± 0.80	1.12 ± 0.82	1.50 ± 0.47
Δ	+0.37 ± 0.54*	+0.25 ± 0.50*	+0.15 ± 0.48	+0.08 ± 0.42	-0.05 ± 0.49	-0.00 ± 0.17
PIVOT-PT	-1.46 ± 0.71	-0.87 ± 0.80	-0.31 ± 0.70	0.34 ± 0.83	1.14 ± 0.71	1.47 ± 0.47
Δ	+0.35 ± 0.49*	+0.31 ± 0.57*	+0.21 ± 0.55*	+0.03 ± 0.39	-0.03 ± 0.31	-0.03 ± 0.14
Table 9: Average formality scores of human annotations (GOLD-STANDARD) and model's predictions on the original and round-trip translated PT16 test set grouped in 6 bins of varying formality.  $\Delta$  gives the average formality shift of the English sentences resulting from round-trip translation. Scores in blue and red indicate that mean is above and below zero, respectively. * denotes statistical significant formality shifts with  $p < 0.05$ . Translation preserves formality of formal sentences while informal sentences exhibit a shift towards formal values.
Annotator 2) via randomly choosing one annotator's judgment for a given instance as the rating of Annotator 1 and taking the mean rating of the rest judgments as the rating of Annotator 2. We then compute Cohen's  $\kappa$  for these two simulated annotators. We repeat this process 1,000 times, and report the median and standard deviation of results. For measuring agreement of the overall ranking evaluation task we use the same simulated framework and report results of Kendall's  $\tau$ . Table 10 presents Inter-annotator agreement results on human evaluation across evaluation aspects and languages.
F Yahoo! L6 language statistics
Table 11 presents the total number of questions included in L6 Yahoo! Corpus, for each language. Although almost  $90\%$  of questions are in English, the corpus contains a non-negligible number of informal sentences in Spanish, French, Portuguese, Italian, and German, with a long tail of few questions for other languages.
G Instructions for XFORMAL
Summary of the task Given an informal sentence in French (or Portuguese/Italian), generate its formal rewrite, without changing its meaning.
Detailed Instructions Given an informal sentence, provide us with its informal rewrite. The informal rewrite should only change the formality attribute of the original sentence and preserve its meaning. Each sentence should be treated independently while rewrites should only rely on the information available in the sentences. There is no need to guess what additional information might
be available in the documents the sentences come from.
Examples Following we include examples of good and bad rewrites given to Turkers:
INFORMALWow, I am very dumb in my observation skills......
GOOD FORMAL I do not have good observation skills.
REASONING Formality is properly transferred and meaning is preserved.
BAD FORMAL Wow, I am very foolish in my observation skills..
REASONING Formality is not properly transferred.
BAD FORMAL I am very unintelligent and I don't have good observation skills.
REASONING Meaning has changed.
H Qualitative analysis of XFORMAL
Following, we include the instructions given to native speakers for the qualitative analysis of XFORMAL, as described in  $\S 4$
Background context In this task you will be asked to judge the quality of formal rewrites of a set of informal sentences. To give more background context, your work would serve as a quality check over annotations obtained from the Amazon Mechanical Turk (AMT) platform. AMT workers were given an informal sentence, e.g., "I'd say it is punk though", and then asked to provide us with its formal rewrite while maintaining the original
TASK	BR-PT	FR	IT
Weighted κ	ICC	Weighted κ	ICC	Weighted κ	ICC
Formality	0.57 ± 0.02	0.72 ± 0.01	0.56 ± 0.02	0.71 ± 0.01	0.67 ± 0.02	0.79 ± 0.02
Fluency	0.43 ± 0.02	0.62 ± 0.02	0.58 ± 0.02	0.73 ± 0.01	0.46 ± 0.02	0.64 ± 0.02
Meaning Preservation	0.53 ± 0.02	0.70 ± 0.02	0.71 ± 0.02	0.83 ± 0.01	0.55 ± 0.03	0.71 ± 0.02
Kendall's τ
Overall	0.41 ± 0.03	0.44 ± 0.03	0.41 ± 0.03
Table 10: Inter-annotator agreement for human evaluation results.
LANGUAGE	CODE	#QUESTIONS
English	EN	3,895,407
Spanish	ES	258,086
French	FR	125,393
Portuguese	PT	105,813
Italian	IT	55,027
German	DE	43,149
Persian	FA	30
Catalan	CA	27
Dutch	NL	13
Danish	DA	10
Arabic	AR	8
Romania	RO	8
Norwegian	NO	8
Swedish	SV	5
Estonian	ET	5
Finnish	FI	4
Malay	MS	4
Turkish	TR	4
Slovak	SK	3
Latvian	LV	3
Albania	SQ	3
Croatian	HR	3
Tagalog	TL	3
Chech	CS	3
Slovenian	SL	3
Vietnamese	VI	2
Icelandic	IS	2
Hungarian	HU	2
Polish	PL	2
Table 11: Number of question per language included in L6 Yahoo! Answers.
content and being grammatical/fluent, e.g., "However, I do believe it to be punk". In our work, workers were presented with sentences in French, Italian and Brazilian Portuguese. For our analysis we want to know a) whether the quality of the collected annotations is good (Task 1), b) what are the types of edits workers performed when formalizing the input sentence (Task 2). Both tasks consist of the same 200 informal-formal sentences-pairs and could be performed either in parallel (e.g., judging a single informal-formal sentence-pair both in terms of quality and types of edit at the same time; Task 1 and 2 in Google sheet), or individually (Task 1, Task 2 in Google sheet). More information and
examples for the two tasks are included below.
Task 1 Given an informal sentence you are asked to assess the quality of its formal rewrite. For each sentence pair type excellent, acceptable, or poor under the rate column. Read the instructions below before performing the task!
What constitutes a good formal rewrite?
- The style of the rewrite should be formal
- The overall meaning of the informal sentence should be preserved
- The rewrite should be fluent
How should I interpret the provided options? Below we include detailed instructions on how to interpret the provided options.
- Excellent the rewrite is formal, fluent and the original meaning is maintained. There is very little that could be done to make a better rewrite.
- Acceptable the rewrite is generally an improvement upon the original but there are some minor issues (such as the rewrite contains a typo or missed transforming some informal parts of the sentence, for example).
- Poor the rewrite offers a marginal or no improvement over the original. There are many aspects that could be improved.
Task 2 In this task the goal is to characterize the types of edits workers made while formalizing the informal input sentence. For each informal-formal sentence pair you should check each of the provided boxes. Note that: a) Multiple types of edits might hold at the same type (e.g., in the informal—formal “Lexus cars are awesome!”)—“Lexus cars
are very nice." you should check both the paraphrase and the punctuation boxes.); b) At the end of the Google sheet there is a column named 'Other' you are welcome to write down in plain text any additional type that you observed and it is not covered by the existing classes. The provided classes are: capitalization, punctuation, paraphrase, delete fillers, completion, add context, contractions, spelling, normalization, slang/idioms, politeness, split sentences, and relativizers.
I Quantitative analysis of XFORMAL
Following, we include more details on the qualitative analysis procedures.
Capitalization A rewrite performs a capitalization edit if it contains tokens that appear in capital letters in the informal text but in lowercase letters in the formal text.
Punctuation A rewrite contains a punctuation edits if any of the punctuation of the informal-formal texts differs.
Spelling We identify spelling errors based on the character-based Levenshtein distance between informal-formal tokens.
Normalization We identify normalization edits based on a hand-crafted list of abbreviations for each language.
Split sentences We split sentences using the NLTK toolkit.
Paraphrase A formal rewrite is considered to contain a paraphrase edit if the token level Levenshtein distance between the informal-formal text is greater than 3 tokens.
Lowercase A rewrite performs a capitalization edit if it contains tokens that appear in lower case in the informal text but in capital letters in the formal text.
Repetition We identify repetition tokens using regular expressions (a token that appears more than 3 times in a row is considered a repetition.)
J XFORMAL: Data Quality
We ask a native speaker to judge the quality of a sample of 200 rewrites for each language via choosing one of the following three options: "excellent", "acceptable", and "poor". The details of this analysis are included in A.H (Task 1). Results indicate
that on average less than  $10\%$  of the rewrites were identified as of poor quality across languages while more than  $40\%$  as of excellent quality. The small number of rewrites identified as "poor" consists mostly of edits where humans add context not appearing in the original sentence. We choose not to exclude any of the rewrites from the dataset as we provide multiple reformulations of each informal instance.
K OpenSubtitles data
OpeSubtiles	Preprocessing	BR-PT	IT	FR
All	X	61.3M	35.2M	41.8M
✓	29.8M	16.8M	19.0M
Informal	✓	25.7M	14.7M	16.3M
Table 12: OpenSubtitles statistics in three languages.
L Qualification tests
Table 13 presents questions and answers used in QC2 of our annotation protocol. Turkers have to score 80 and above to participate in the task. To compute an average score for each test, we assume that an answer is incorrect if it deviates more than 1 point from the gold-standard scores given in the second column of Table 13.
M Trade-off plots
Trade-off plots Figure 4 presents trade-off plots between multi-BLEU vs. formality score and fluency, across different reruns as proposed by Tikhonov and Yamshchikov (2018). First, we observe that models exhibit small variations in terms of formality and fluency scores across different reruns, and larger variations across BLEU for most cases. Notably, the single seed BACKTRANSLATE systems are the most consistent across reruns for all metrics and languages. Furthermore, in almost all cases, models trained on 2M data, perform better that the naive COPY baseline, across metrics. However, single seed models fail to consistently outperform the RULE-BASED baselines in almost all cases, with the exception of BACKTRANSLATE for BR-PT and FR which report an improvement of about 1 BLEU score.
N Compute time & Infrastructure
All experiments for benchmarking both NMT-based and unsupervised approaches use Amazon EC2 P3 instances: https://aws.amazon.com/
(c) IT
Figure 4: BLEU vs. STYLE SCORE vs. PERPLEXITY trade-off plots across 4 reruns.
FRENCH
Heureux ceux qui ont la chance de pouvoir faire la sieste devant Derrick! -2je kiffe trop ce film il déchire carément!! -3Moi je les données à des vieilles personnes qui ne comprend pas le fonctionnement d'un lecteur DVD! -1Le producteur de ce morceau, c'est Scott Torch. 2Il s'agit bien du groupe America et le titre "A horse with no name". 2La France ne gagnera jamais si elle continue avec des chansons qui ont l'air de dater des années 70. 0Le but n'est pas de se faire du profit dessus! -1Il existeURTOUT sur les chaînes du cable et du satellite, TL9, TPS foot, par exemple. 2
ITALIAN
TI do questo site qui puoi trovare qualunque casa su Tru Calling buonagettura! -1Parla proprio di amore in chat. -2ma parla cm mangi)!!!!!!!!!!io ho un programma si kiama evil lyrics. -1Se non ci si da una svegliata "dove andremo a finire?" -1Alla fine del Festival ci saranno due vincitori, uno per categoria. 2è quello che stò passando ora. -1Il montaggio risulta quando ridotto al minimo. 2
BRAZILIAN-PORTUGUESE
Me parece que você usa BASTANTEessaPALAVRA e seušerivados,né?!-2Mas vem Cá, você não tem nada melhor pra fazer do que escatar RBD ao contrário?! -2Então na minha opinião é babaquice daquele que tem preconceitos esse tipo. -1Vi a propaganda na localadora, na minha idade. 0Na minha opinião, o minimo existencial subentende a PALAVRA opportunitadé. 1AúnicaDIFFENÇAessa mulher para um assassino comum, é que ela nem chega a conhecer sua vítima. 1Dentre as hipóteses menos consideradas por historiadores sérios está o famoso mito celta. 1Dado um ambiente qualquer, uma sala ou um quarto, por exemplo, as ondas sonoras "respondem" de formas différentes. -1
Table 13: Qualification test (questions and answers) for QC2
ec2/instance-types/p3/, on Tesla V100 GPUs. Concretely, NMT-based experiments are run on 4 GPUs (~1-2 hours), while unsupervised models are run on single GPUs, with average run time spanning from  $\sim 5$  hours (for out-of-domain data—100K), to  $\sim 72$  hours (for in domain data—2M).