Venue: NAACL 2021

# Ola, Bonjour, Salve!

XFORMAL: A Benchmark for Multilingual Formality Style Transfer

Eleftheria Briakou1

University of Maryland

ebriakou@cs.umd.edu &Di Lu Dataminr, Inc. dlu@dataminr.com &Ke Zhang Dataminr, Inc. kzhang@dataminr.com &Joel Tetreault Dataminr, Inc. jettreault@dataminr.com

Footnote 1: Work done as a Research Intern at Datamir, Inc.

###### Abstract

We take the first step towards multilingual style transfer by creating and releasing xformal, a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on xformal suggest that state-of-the-art style transfer approaches perform close to simple baselines, indicating that style transfer is even more challenging when moving multilingual.2

Footnote 2: Code and data: [https://github.com/Elbria/xformal-FoST](https://github.com/Elbria/xformal-FoST)

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3199–3216 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Style Transfer (st) is the task of automatically transforming text in one style into another (for example, making an impolite request more polite). Most work in this growing field has focused primarily on style transfer within English, while _covering different languages_ has received disproportional interest. Concretely, out of \(35\) st papers we reviewed, all of them report results for st within English text, while there is just a single work covering each of the following languages: Chinese, Russian, Latvian, Estonian, and French [4, 7, 8]; Notokova et al., 2019; Niu et al., 2018). Notably, even though some efforts have been made towards multilingual st, researchers are limited to providing system outputs as a means of evaluation, and progress is hampered by the scarcity of resources for most languages. At the same time, st lies at the core of human communication: when humans produce language, they condition their choice of grammatical and lexical transformations to a target audience and a specific situation. Among the many possible stylistic variations, Heylighen et al. (1999) argue that _"a dimension similar to **formality** appears as the most **important** and **universal** feature distinguishing styles, registers or genres in **different languages**"_. Consider the informal excerpts and their formal reformulations in French (fr) and Brazilian Portuguese (br-pt) in Table 1. Both informal-formal pairs share the same content. However, the informal language conveys more information than is contained in the literal meaning of the words [8]. These examples relate to the notion of _deep formality_[1], where the ultimate goal is that of adding the context needed to disambiguate an expression. On the other hand, variations in formality might just reflect different situational and personal factors, as shown in the Italian (it) example.

This work takes the first step towards a more language-inclusive direction for the field of st by building the first corpus of style transfer for non-English languages. In particular, we make the following contributions: **1.** Building upon prior work on Formality Style Transfer (fost) [1], we contribute an **evaluation dataset**, xformal that consists of multiple formal rewrites of informal sentences in three Romance languages: Brazilian Portuguese (br-pt), French (fr), and Italian (it); **2.** Without assuming access to any gold-standard training data for the languages at hand, we **benchmark a myriad

\begin{table}
\begin{tabular}{l l}  & BrazilIAN-PORTUGuese \\  & saiam disso, forca de vontade!! \\  & _get out of it, willpower!_ \\  & Abandonem essa situacao, tenham força _de ventade._ \\  & _Abandon this situation, have willpower!_ \\  & \\

of leading st baselines** through automatic and human evaluation methods. Our results show that fost in non-English languages is particularly challenging as complex neural models perform on par with a simple rule-based system consisting of handcrafted transformations. We make xformal, our annotations protocols, and analysis code publicly available and hope that this study facilitates and encourages more research towards Multilingual st.

## 2 Related Work

Controlling style aspects in generation tasks is studied in monolingual settings with an English-centric focus (intra-language) and cross-lingual settings together with Machine Translation (mt) (inter-language). Our work rests in intra-language st with a multilingual focus, in contrast to prior work.

st datasetsthat consist of parallel pairs in different styles include: gyafc for formality Rao and Tetreault (2018), Yelp Shen et al. (2017) and Amazon Product Reviews for sentiment He and McAuley (2016), political slant and gender controlled datasets Prabhumoye et al. (2018), Expert Style Transfer Cao et al. (2020), pastel for imitating personal Kang et al. (2019), simile for simile generation Chakrabarty et al. (2020), and others.

Intra-language stwas first cast as generation task by Xu et al. (2012) and is addressed through methods that use either parallel data or unpaired corpora of different styles. **Parallel corpora designed for the task at hand** are used to train traditional encoder-decoder architectures Rao and Tetreault (2018), learn mappings between latent representation of different styles Shang et al. (2019), or fine-tune pre-trained models Wang et al. (2019). Other approaches use **parallel data from similar tasks** to facilitate transfer in the target style via domain adaptation Li et al. (2019), multi-task learning Niu et al. (2018); Niu and Carpuat (2020), and zero-shot transfer Korotkova et al. (2019) or create **pseudo-parallel data** via data augmentation techniques Zhang et al. (2020); Krishna et al. (2020). Approaches that rely on **non-parallel data** include _disentanglement methods_ based on the idea of learning style-agnostic latent representations e.g., Shen et al. (2017); Hu et al. (2017)). However, they are recently criticized for resulting in poor content preservation Xu et al. (2018); Jin et al. (2019); Luo et al. (2019); Subramanian et al. (2018) and alternatively, _translation-based models_ are proposed that use reconstruction and back-translation losses (e.g., Logeswaran et al. (2018); Prabhumoye et al. (2018)). Another line of work, focuses on _manipulation methods_ that remove the style-specific attribute of text (e.g., Li et al. (2018); Xu et al. (2018)), while recent approaches use _reinforcement learning_ (e.g., Wu et al. (2019); Gong et al. (2019), probabilistic formulations He et al. (2020), and masked language models Malmi et al. (2020).

Inter-language stis introduced by Mirkin and Meunier (2015) who proposed personalized mt for en-French and en-German. Subsequent mt works control for politeness Sennrich et al. (2016a), voice Yamagishi et al. (2016), personality traits Rabinovich et al. (2017), user-provided terminology Hasler et al. (2018), gender Vanmassenhove et al. (2018), formality Niu et al. (2017); Feely et al. (2019), morphological variations Moryossef et al. (2019), complexity Agrawal and Carpuat (2019) and reading level Marchisio et al. (2019).

## 3 xformal Collection

We describe the process of collecting formal rewrites using data statements protocols Bender and Friedman (2018); Gebru et al. (2018).

Curation rationalTo collect xformal, we firstly curate informal excerpts in multiple languages. To this end, we follow the procedures described in Rao and Tetreault (2018) (henceforth rt\(18\)) who create a corpus of informal sentence-pairs in English (en) entitled Grammarly's Yahoo Answers Formality Corpus (gyafc).

Conrectely, we use the L\(6\) Yahoo! Answers corpus that consists of questions and answers posted to the Yahoo! Answers platform.2 The corpus contains a large number of informal text and allows control for different languages and different domains.3 Similar to the collection of gyafc, we extract all answers from the Family & Relationship (F&R) topic that correspond to the three languages of interest: _Famlta e Relacionaments_ (br-pt), _Relazioni e famiglia_ (it ), and _Amour et relations_ (fr) (**Step 1**). We follow the same pre-processing steps as described in rt\(18\) for consistency (**Step 2**). We filter out answers that: a) consist of questions; b) include urls; c) have fewer 

than five or more than \(25\) tokens; or d) constitute duplicates.4 We automatically extract informal candidate sentences, as described in SS5.3 (**Step 3**). Finally, we randomly sample \(1{,}000\) sentences from the pool of informal candidates for each language. Table 2 presents statistics of the curation steps.

Footnote 4: We tokenize with nltk: [https://www.nltk.org/api/nltk.tokenize.html](https://www.nltk.org/api/nltk.tokenize.html)

ProceduresWe use the Amazon Mechanical Turk (mturk) platform to collect formal rewrites for our informal sentences. For each language, we split the annotation into \(20\) batches of \(50\) Human Intelligence Tasks (hits). In each hit, Turkers are given an informal excerpt and asked to _generate its formal rewrite in the same language without changing its meaning_. We collect \(4\) rewrites per excerpt and release detailed instructions under A.G.

Annotation Workflow & Quality ControlOur annotation protocol consists of multiple Quality Control (qc) steps to ensure the recruitment of high-quality annotators. As a first step, we use location restrictions (qc1) to limit the pool of workers to countries where native speakers are most likely to be found. Next, we run several small _pilot studies_ (of \(10\) hints) to recruit potential workers. To participate in the pilot study, Turkers have to pass a **qualification test (qc2)** consisting of multiple-choice questions that test workers' understanding of formality (see A.L). The pilot study results are **reviewed by a native speaker (qc3)** of each language to exclude workers who performed consistently poorly. We find that the two main reasons for poor quality are: a) rewrites of minimum-level edits, or b) rewrites that change the input's meaning. Table 3 presents the number of workers at each qc step. Only workers passing all quality control steps (last row of Table 3) contribute to the final task. Finally, we **post-process** the collected rewrites by a) removing instances consisting of normalization-based edits only and b) correcting minor spelling errors using an off-the-shelf tool.5

Footnote 5: [https://languagetool.org/](https://languagetool.org/)

Turkers' demographicsWe recruit Turkers from Brazil, France/Canada, and Italy for br-qt, fr, and it, respectively. Beyond their country of residence, no further information is available.

CompensationWe compensate at a rate of $\(0.10\) per hit with additional one-time bonuses that bumps them up to a target rate of over $10/hour.

After this entire process, we have constructed a high-quality corpus of formality rewrites of \(1{,}000\) sentences for three languages. In the next section, we provide statistics and an analysis of xformal.

## 4 xformal Statistics & Analysis

Types of formality editsFollowing Pavlick and Tetreault [2016], we analyze the most frequent **edit** operations Turkers perform when formalizing the informal sentences. We conduct both an **automatic analysis** (details in A.I) of the whole set of rewrites, and a **human analysis** (details in A.H) of a random sample of \(200\) rewrites per language (we recruited a native speaker for each language). Table 4 presents both analyses' results, where we also include the corresponding statistics for the English language (gyafc). In general, we observe similar trends across languages: humans make edits covering both the "noisy-text" sense of formality (e.g., fixing punctuation, spelling errors, capitalization) and the more situational sense (paraphrase-based edits). Although cross-language trends are similar, we also observe differences: _deleting fillers_ and word _completion_ seems to be more prominent in the English rewrites than in other languages; _normalizing_ abbreviations is a considerably frequent edit type for Brazilian Portuguese; _paraphrasing_ is more frequent in the three non-English languages.

Surface differences of informal-formal pairsWe quantify surface-level differences between the informal sentences and formal rewrites via computing their character-level Levenshtein distance (Figure 1) and their pairwise Lexical Difference (LeD) based on the percentages of **tokens** that are _not_ found in both sentences (Table 5). Both analyses show that Italian rewrites have the most edits com

\begin{table}
\begin{tabular}{l r r r} \hline Corpus & br-pt & fr & it \\ \hline L6 Yahoo! & \(230{,}302\) & \(225{,}476\) & \(101{,}894\) \\  + **Step 1** & \(37{,}356\) & \(34{,}849\) & \(13{,}443\) \\  + **Step 2** & \(14{,}448\) & \(14{,}856\) & \(4{,}095\) \\  + **Step 3** & \(8{,}617\) & \(11{,}118\) & \(2{,}864\) \\ \end{tabular}
\end{table}
Table 2: Number of sentences in filtered versions of the L6 Yahoo! Corpus across curation steps and languages.

\begin{table}
\begin{tabular}{l l r r r} \hline Step & Description & br-pt & fr & it \\ \hline \hline   **(qc1)** & Location restriction & \(151\) & \(78\) & \(59\) \\   **(qc2)** & Qualification test & \(54\) & \(40\) & \(33\) \\   **(qc3)** & Rewrites review & \(\mathbf{9}\) & \(\mathbf{16}\) & \(\mathbf{11}\) \\ \end{tabular}
\end{table}
Table 3: Number of Turkers **after** each qc step.

 

pared to their corresponding informal sentences. French and Brazilian Portuguese follow, with English rewrites being closer to the informal inputs.

Diversity of formal rewritesAre Turkers making similar choices when formalizing text? Since a large number of reformulations consist of paraphrase-based edits (more than \(50\)%), we want to quantify the extent to which the formal rewrites of each sentence are **diverse**, in terms of their lexical choices. To that end, we quantify diversity via measuring self-bleu[23]: considering one set of formal sentences as the hypothesis set and the others as references, we compute bleu for each formal set and define the average bleu score as a measure of the dataset's diversity. Higher scores imply less diversity of the set. Results (last row of Table 4) show that xformal consists of more diverse rewrites compared to gyafc.

Formality shift of rewritesWe analyze the formality distribution of the original informal sentences with their formal rewrites in gyafc and xformal, as predicted by formality mbert models (SS5.3). The distributions of formal rewrites are skewed towards positive values (Figure 2).

## 5 Multilingual fost Experiments

We benchmark eight st models on xformal to serve as baseline scores for future research. We describe the models (SS5.1), the experimental setting (SS5.2), the human and automatic evaluation methods (SS5.3 and SS5.4), and results (SS5.5).

\begin{table}
\begin{tabular}{l r r r r}
**EDIT types** & **BR-PT** & **FR** & **it** & **EN** \\ \hline capitalization & \(0.24\) & \(0.33\) & \(0.32\) & \(0.43\) \\  & \(0.43\) & \(0.56\) & \(0.60\) & \(0.46\) \\ punctuation & \(0.64\) & \(0.87\) & \(0.79\) & \(0.74\) \\  & \(0.76\) & \(0.64\) & \(0.66\) & \(0.40\) \\ spelling & \(0.36\) & \(0.38\) & \(0.32\) & \(0.29\) \\  & \(0.09\) & \(0.12\) & \(0.10\) & \(0.14\) \\ normalization & \(0.17\) & \(0.01\) & \(0.04\) & \(0.07\) \\  & \(0.27\) & \(0.02\) & \(0.06\) & \(0.10\) \\ split sentences & \(0.07\) & \(0.06\) & \(0.11\) & \(0.08\) \\  & \(0.07\) & \(0.07\) & \(0.07\) & \(0.04\) \\ paraphrase & \(0.68\) & \(0.75\) & \(0.86\) & \(0.68\) \\  & \(0.74\) & \(0.75\) & \(0.58\) & \(0.47\) \\ unchanged & \(0.00\) & \(0.00\) & \(0.00\) & \(0.01\) \\ lowercase & \(0.13\) & \(0.06\) & \(0.07\) & \(0.13\) \\ repetition & \(0.01\) & \(0.01\) & \(0.02\) & \(0.09\) \\ delete fillers & \(0.02\) & \(0.05\) & \(0.11\) & \(0.26\) \\ completion & \(0.07\) & \(0.04\) & \(0.04\) & \(0.15\) \\ add context & \(0.02\) & \(0.00\) & \(0.07\) \\ contractions & \(0.02\) & \(0.01\) & \(0.01\) \\ slang/idioms & \(0.19\) & \(0.10\) & \(0.15\) \\ politeness & \(0.82\) & \(0.00\) & \(0.10\) \\ relativizers & \(0.01\) & \(0.00\) & \(0.00\) \\ \end{tabular}
\end{table}
Table 4: Percentage categories of frequent edits calculated **automatically** (not highlighted) and **manually** (highlighted). Categories are not mutually exclusive.

Figure 1: Distribution of character-based Levenshtein distance between informal sentences and formal rewrites.

Figure 2: Number of informal sentences and formal rewrites binned according to formality score.

 



### Models

Simple baselinesWe define three baselines: 1. copy Motivated by Pang and Gimpel (2019) who notice that untransferred sentences with no alterations have the highest bleu score by a large margin for st tasks, we use this simple baseline as a lower bound; 2. rule-based Based on the quantitative analysis of SS4 and similarly to rt18, we develop a rule-based approach that performs a set of predefined edits-based operations defined by handcrafted rules. Example transformations include fix casing, remove repeated punctuation, handcraft a list of contraction expansions--a detailed description is found at A.C; 3. round-trip mt Inspired by Zhang et al. (2020) who identify useful training pairs from the paraphrases generated by round-trip translations of millions of sentences, we devise a simpler baseline that starts from a text in language \(x\), pivots to en and then backtranslates to \(x\), using the aws translation service.6

Footnote 6: [https://aws.amazon.com/translate/](https://aws.amazon.com/translate/)

**NMT-based models with synthetic parallel data** We follow the translate trainConneau et al. (2018); Artetxe et al. (2020) approach to collect data in multilingual settings: we obtain _pseudo-parallel_ corpora in each language via machine translating anen resource of informal-formal pairs (SS5.2).7 Then, starting with translate train we benchmark the following nmt-based models: 1. translate train tag extends a leading en fost approachNiu et al. (2018) and trains a unified model that handles either formality direction via attaching a source tag that denotes the desired target formality; 2. multi-task tag-style Niu et al. (2018) augments the previous approach with bilingual data that is automatically identified as formal (SS5.3). The models are then trained in a multi-task fashion; 3. backtranslate augments the translate train data with backtranslated sentences of automatically detected informal textSennrich et al. (2016b), using \(1\). as the base model. We exclude backtranslated pairs consisting of copies. The output of the rule-based system is given as input to each model at inference time. For all three models, we run each system with \(4\) random seeds, and combine them in a linear ensemble for decoding.

Footnote 7: Details on the aws performance are found in A.B.

Unsupervised approachesWe benchmark two unsupervised methods that are used for en st: I. unpsupervised neural machine translation (unmt)Subramanian et al. (2018) defines a pseudo-supervised setting and combines denoising auto-encoding and back-translation losses; 2. deep latent sequence model (dlsm)He et al. (2020) defines a probabilistic generative story that treats two unpaired corpora of separate styles as a partially observed parallel corpus and learns a mapping between them, using variational inference.

### Experimental setting

Training dataFor translate train tag we use gyafc, a large set of \(110\text{K}~{}\texttt{en}\) informal-formal parallel sentence-pairs obtained through crowdsourcing. Additionally, we augment the translated resource with OpenSubtitlesLison and Tiedemann (2016) bilingual data used for training mt models.8 Given that bilingual sentence-pairs can be noisy, we perform a filtering step to extract noisy bitexts using the Bicleaner toolkit Sanchez-Cartagena et al. (2019). Furthermore, we apply the same filtering steps as in SS3 (Curation rational). Finally, each of the remaining sentences is assigned a formality score (SS5.3), resulting in two pools of informal and formal text. Training instances are then randomly sampled from those pools: formal parallel pairs are used for multi-task tag-style; informal target side sentences are backtranslated for backtranslate; both informal and formal target-side texts are independently sampled from the two pools for training unsupervised models. Finally, for unsupervised fost in fr, we additionally experiment with in-domain data from the L\(26\) French Yahoo! Answer Corpus that consists of \(1.7\text{M}~{}\texttt{fr}\) questions. 10, 11 Table 6 includes statistics on training sizes.12

Footnote 8: Data are available at: [http://opus.nlpl.eu/](http://opus.nlpl.eu/).

[TABLE

PreprocessingWe preprocess data consistently across languages using moses(Koehn et al., 2007). Our pipeline consists of three steps: a) normalization; b) tokenization; c) true-casing. For nmt-based approaches, we also learn joint source-target bpe with \(32\text{K}\) operations (Sennrich et al., 2016b).

Model ImplementationsFor NMT-based and unsupervised models we use the open-sourced implementations of Niu et al. (2018) and He et al. (2020), respectively.14[13, 14] We include more details on model architectures in A.D.

Footnote 14: [https://github.com/cindyxinyiwang/deep-latent-sequence-model](https://github.com/cindyxinyiwang/deep-latent-sequence-model)

### Automatic Evaluation

Recent work on st evaluation highlights the **lack of standard evaluation practices**Yamshchikov et al. (2020); Pang (2019); Pang and Gimpel (2019); Mir et al. (2019). We follow the most frequent evaluation metrics used in en tasks and measure the quality of the system's outputs with respect to four dimensions, while we leave an extensive evaluation of automatic metrics for future work.

Meaning PreservationWe compute selfbleuPapineni et al. (2002) which compares system outputs with the informal sentences.

FormalityWe average the style transfer score of transferred sentences computed by a formality regression model. We fine-tune mbertDevlin et al. (2019) pre-trained language models on the machine-translated answers genre from Pavlick and Tetreault (2016) that consists of about \(4\text{K}\) human-annotated sentences rated on a \(7\)-point formality scale. To acquire an annotated corpus in the languages of interest, we follow the translate train transfer approach: we propagate the original en training data's human ratings to their corresponding translations, assuming that translation preserves formality.15 To evaluate the multilingual formality regression models' performance, we crowdsourced human judgments of \(5\) Turkers for \(200\) sentences per language. We report Spearman correlations of \(64\) (br-pt), \(70\) (it), \(71\) (fr), and \(81\) (en).

Footnote 15: See A.A for discussion on this assumption.

FluencyWe compute the logarithm of each sentence's probability--computed by a \(5\)-gram Kneser-Ney language model (Kneser and Ney, 1995)--and normalize it by the sequence length. We train each language model on \(2\text{M}\) random sample of the non-English side of OpenSubtitles formal data.

OverallWe compute multi-bleuPost (2018) via comparing with multiple formal rewrites on xformalFreitag et al. (2020) shows that correlation with human judgments improves when considering multiple references for mt evaluation.

### Human evaluation

Given that automatic evaluation of st lacks standard evaluation practices--even in cases when en is considered--we turn to human evaluation to reliably assess our baselines following the protocols of rt18. We sample a subset of \(100\) sentences from xformal per language, evaluate outputs of \(5\) systems, and collect \(5\) judgments per instance.We open the task to all workers passing qc2 in Table 3. We include inter-annotator agreement results in A.E.

FormalityWe collect formality ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a \(7\)-point discrete scale of \(-3\) to \(3\), following Lahiri (2015) (_Very informal_\(\rightarrow\)_Informal_\(\rightarrow\)_Somewhat Informal_\(\rightarrow\)_Neutral_\(\rightarrow\)_Somewhat Formal_\(\rightarrow\)_Formal_).

FluencyWe collect fluency ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a discrete scale of \(1\) to \(5\), following Heilman et al. (2014) (_Other_\(\rightarrow\)_Incomprehensible_\(\rightarrow\)_Somewhat Comprehensible_\(\rightarrow\)_Comprehensible_\(\rightarrow\)_Perfect_).

Meaning PreservationWe adopt the annotation scheme of Semantic Textual Similarity Agirre et al. (2016): given the informal reference and formal human rewrite or the formal system outputs, Turkers rate the two sentences' similarity on a \(1\) to \(6\) scale (_Completely dissimilar_\(\rightarrow\)_Not equivalent but on same topic_\(\rightarrow\)_Not equivalent but share some details_\(\rightarrow\)_Roughly equivalent_\(\rightarrow\)_Mostly equivalent_\(\rightarrow\)_Completely equivalent_).

OverallWe collect overall judgments of the system outputs using _relative ranking_: given the informal reference and a formal human rewrite, workers are asked to rank system outputs in the order of their overall formality, taking into account both fluency and meaning preservation. An overall score is then computed for each model via averaging results across annotating instances 



### Results

Table 7 shows automatic results for all models across the four dimensions as well as human ratings for selected top models.

nmt-based model evaluationConcretely, the rule-based baselines are significantly (\(p<0.05\)) the best performing models in terms of **meaning preservation** across languages. This result is intuitive as the rule-based models act at the surface level and are unlikely to change the informal sentence's meaning. The backtranslate ensemble systems are the second-best performing models in terms of meaning preservation, while the round-trip mt outputs diverge semantically from the informal sentences the most. Those results are consistent across languages and human/automatic evaluations. On the other hand, when we compare systems in terms of their **formality**, we observe the opposite pattern: the rule-based and backtranslate outputs are the most informal compared to the other ensemble nmt-based approaches across languages. Interestingly, the round-trip mt outputs exhibit the largest formality shift for br-pt and fr as measured by human evaluation. The trade-off between meaning preservation and formality among models was also observed in en (rt18). Moreover, when we move to **fluency**, we notice similar results across systems. Specifically, human evaluation assigns almost all models an average score of \(>4\), denoting that system outputs are comprehensible on average, with small differences between systems not being statistically significant. Notably, perplexity tells a different story: all system outputs are significantly better compared to the rule-based systems across configurations and languages. This result denotes that perplexity might not be a reliable metric to measure fluency in this setting, as noticed in Mir et al. (2019) and Krishna et al. (2020). When it comes to the **overall ranking** of systems, we observe that the nmt-based ensembles are better than the rule-based baselines for br-pt and fr, yet by a small margin as denoted by both multi-bleu and human evaluation. However, the corresponding results for it denote that there is no clear win, and the nmt-based ensembles still fail to surpass the naive rule-based models, yet by a small margin. Finally, all ensembles outperform the trivial copy baseline. Table 8 presents examples of system outputs. As a side note, we followed the recommendation of Tikhonov and Yamshchikov (2018) to show the performance of st models of individual runs and visualize trade-offs between metrics better. Unlike their work which found that reruns of the same model showed wide performance discrepancies, we found that most of our nmt-based models did not vary in performance on xformal. The results can be visualized in A.M.

Unsupervised model evaluationWe also benchmark the unsupervised models but focus solely on automatic metrics since they lag behind their supervised counterparts. As shown in Table 7, when



using out-of-domain data (e.g., OpenSubtitles) for training, the models perform worse than their nmt counterparts across all three languages. The difference is most stark when considering self-bleu and multi-bleu scores. However, given access to large _in-domain_ corpora (e.g., L26 Yahoo! French Answers) the gap between the two model classes closes with dlsm achieving a multi-bleu score of 42.1 compared to 48.3 for the best performing nmt model backtranslate. This shows the promise of unsupervised methods, assuming a large amount of in-domain data, on multilingual st tasks.

Lexical differences of system outputsFinally, in Figure 3 we analyze the diversity of outputs by leveraging LeD scores resulting from pair-wise comparisons of different nmt systems. A larger LeD score denotes a larger difference between the lexical choices of the two systems under comparison. First, we observe that the round-trip mt outputs have the smallest lexical overlap with the informal input sentences. However, when this observation is examined together with human evaluation results, we conclude that the large number of lexical edits happens at the cost of diverging se

Figure 3: Heatmaps of LeD scores showing the lexical difference between pairs of systems.

 

mantically from the input sentences. Moreover, we observe that the average lexical differences within nmt-based systems are small. This indicates that different systems perform similar edit operations that do not deviate a lot from the input sentence in terms of their lexical choices. This is unfortunate given that multilingual fost requires systems to perform more phrase-based operations, as shown in the analysis in SS4.

Evaluation MetricWhile evaluating evaluation metrics is not a goal of this work (though the data can be used for that purpose), we observe that the top models identified by the automatic metrics generally align with the top models identified by humans. While promising, further work is required to confirm if the automatic measures really do correlate with human judgments.

## 6 Conclusions & Future Directions

This work extends the task of formality style transfer to a multilingual setting. Specifically, we contribute xformal, an evaluation testbed consisting of informal sentences and multiple formal rewrites spanning three languages: br-pt, fr, and it. As in Rao and Tetreault (2018) Turkers can be effective in creating high quality st corpora. In contrast to the aforementioned en corpus, we find that the rewrites in xformal tend to be more diverse, making it a more challenging task.

Additionally, inspired by work on cross-lingual transfer and en fost, we benchmark several methods and perform automatic and human evaluations on their outputs. We found that nmt-based ensembles are the best performing models for fr and br-pt--a result consistent with en--however, they perform comparably to a naive rule-based baseline for it. To further facilitate reproducibility of our evaluations and corpus creation processes, as well as drive future work, we will release our scripts, rule-based baselines, source data, and annotation templates, on top of the release of xformal.

Our results open several avenues for future work in terms of benchmarking and evaluating fost in a more language inclusive direction. Notably, current supervised **and** unsupervised approaches for en fost rely on parallel in-domain data--with the latter treating the parallel set as two unpaired corpora--that are not available in most languages. We suggest that **benchmarking**fost models in multilingual settings will help understand their ability to generalize and lead to safer conclusions when comparing approaches. At the same time, multilingual fost calls for more language-inclusive consideration for **automatic evaluation** metrics. Model-based approaches have been recently proposed for evaluating different aspects of st. However, most of them rely heavily on English resources or pre-trained models. How those methods can be extended to multilingual settings and how we evaluate their performance remain open questions.

### Acknowledgements

We thank Sudha Rao for providing references and materials of the gyafc dataset, Chris Callison-Burch and Courtney Napoles for discussions on MTurk annotations, Svebor Karaman for helping with data collection, our colleagues at Dataminr, and the naacl reviewers for their helpful and constructive comments.

## 7 Ethical Considerations

Finally, we address ethical considerations for **dataset papers** given that our work proposes a new corpus xformal. We reply to the relevant questions posed in the naacl\(2021\) Ethics faq.16

Footnote 16: [https://2021.naacl.org/ethics/faq/](https://2021.naacl.org/ethics/faq/) 



## References

* Agrirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_, pages 497-511, San Diego, California. Association for Computational Linguistics.
* Agrawal and Carpuat (2019) Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1549-1564, Hong Kong, China. Association for Computational Linguistics.
* Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 88th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. _Transactions of the Association for Computational Linguistics_, 6:587-604.
* Cao et al. (2020) Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communication between experts and laymen. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1061-1071, Online. Association for Computational Linguistics.
* Chakrabarty et al. (2020) Tuhin Chakrabarty, Smaranda Muresan, and Nanyun Peng. 2020. Generating similes effortlessly like a pro: A style transfer approach for simile generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6455-6469, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Feely et al. (2019) Weston Feely, Eva Hasler, and Adria de Gispert. 2019. Controlling Japanese honorifics in English-to-Japanese neural machine translation. In _Proceedings of the 6th Workshop on Asian Translation_, pages 45-53, Hong Kong, China. Association for Computational Linguistics.
* Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 61-71, Online. Association for Computational Linguistics.
* Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daume, and Kate Crawford. 2018. Datasheets for datasets. _ArXiv_, abs/1803.09010.
* Gong et al. (2019) Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3168-3180, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hasler et al. (2018) Eva Hasler, Adria de Gispert, Gonzalo Iglesias, and Bill Byrne. 2018. Neural machine translation decoding with terminology constraints. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 506-512, New Orleans, Louisiana. Association for Computational Linguistics.
* He et al. (2020) Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In _Proceedings of International Conference on Learning Representations_.
* He and McAuley (2016) Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _Proceedings of the 25th International Conference on World Wide Web_, WWW '16, page 507-517, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Heilman et al. (2014) Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 174-180, Baltimore, Maryland. Association for Computational Linguistics

F. Heylighen, J. Dewaele, and L. Apostel. 1999. Formality of language: definition, measurement and behavioral determinants.
* Hieber et al. (2018) Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2018. The sockeye neural machine translation toolkit at AMTA 2018. In _Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers)_, pages 200-207, Boston, MA. Association for Machine Translation in the Americas.
* Hovy (1987) Eduard Hovy. 1987. Generating natural language under pragmatic constraint. _Journal of Pragmatics_, 11(6):689-719.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70 of _Proceedings of Machine Learning Research_, pages 1587-1596, International Convention Centre, Sydney, Australia. PMLR.
* Jin et al. (2019) Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Imat: Unsupervised text attribute transfer via iterative matching and translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 3095-3107. Association for Computational Linguistics.
* Kang et al. (2019) Dongyeop Kang, Varun Gangal, and Eduard Hovy. 2019. (male, bachelor) and (female, ph.d) have different connotations: Parallelly annotated stylistic language dataset with multiple personas. In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_, Hong Kong.
* Kneser and Ney (1995) Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In _ICASSP_, pages 181-184. IEEE Computer Society.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zengs, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions_, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
* Korotkova et al. (2019) Elizaveta Korotkova, Agnes Luhtaru, Maksym Del, Krista Lin, Daiga Deksne, and Mark Fishel. 2019. Grammatical error correction and style transfer via zero-shot monolingual translation. _CoRR_, abs/1903.11283.
* Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 737-762, Online. Association for Computational Linguistics.
* Lahiri (2015) Shibamouli Lahiri. 2015. Squinky! A corpus of sentence-level formality, informativeness, and implicature. _CoRR_, abs/1506.02306.
* Li et al. (2019) Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Bill Dolan, and Ming-Ting Sun. 2019. Domain adaptive text style transfer. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3304-3313, Hong Kong, China. Association for Computational Linguistics.
* Li et al. (2018) Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
* Lison and Tiedemann (2016) Pierre Lison and Jorg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 923-929, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Logeswaran et al. (2018) Lajanuggen Logeswaran, Honglak Lee, and Samy Bengio. 2018. Content preserving text generation with attribute controls. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems 31_, pages 5103-5113. Curran Associates, Inc.
* Luo et al. (2019) Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In _Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019_.
* Malmi et al. (2020) Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8671-8680, Online. Association for Computational Linguistics.
* Marchisio et al. (2019) Kelly Marchisio, Jiialiang Guo, Cheng-I Lai, and Philippo Koehn. 2019. Controlling the reading level of machine translation output. In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_, pages 193-203, Dublin, Ireland. European Association for Machine Translation 

Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 495-504, Minneapolis, Minnesota. Association for Computational Linguistics.
* Mirkin and Meunier (2015) Shachar Mirkin and Jean-Luc Meunier. 2015. Personalized machine translation: Predicting translational preferences. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Moryssof et al. (2019) Amit Moryssof, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_, pages 49-54, Florence, Italy. Association for Computational Linguistics.
* Niu and Carpuat (2020) Xing Niu and Marine Carpuat. 2020. Controlling neural machine translation formality with synthetic supervision. In _Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence_, pages 8568-8575.
* Niu et al. (2017) Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2814-2819, Copenhagen, Denmark. Association for Computational Linguistics.
* Niu et al. (2018) Xing Niu, Sudha Rao, and Marine Carpuat. 2018. Multi-task neural models for translating between styles within and across languages. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1008-1021, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Pang (2019) Richard Yuanzhe Pang. 2019. Towards actual (not operational) textual style transfer auto-evaluation. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_, pages 444-445, Hong Kong, China. Association for Computational Linguistics.
* Pang and Gimpel (2019) Richard Yuanzhe Pang and Kevin Gimpel. 2019. Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_, pages 138-147, Hong Kong. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, ACL '02, page 311-318, USA. Association for Computational Linguistics.
* Pavlick and Tetreault (2016) Ellie Pavlick and Joel Tetreault. 2016. An empirical analysis of formality in online communication. _Transactions of the Association for Computational Linguistics_, 4:61-74.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
* Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
* Rabinovich et al. (2017) Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Winther. 2017. Personalized machine translation: Preserving original author traits. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 1074-1084, Valencia, Spain. Association for Computational Linguistics.
* Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
* Sanchez-Cartagena et al. (2018) Victor M. Sanchez-Cartagena, Marta Banon, Sergio Ortiz-Rojas, and Gema Ramirez-Sanchez. Prompsit's submission to wmt 2018 parallel corpus filtering shared task. In _Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers_, Brussels, Belgium. Association for Computational Linguistics.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 35-40, San Diego, California. Association for Computational Linguistics.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Shang et al. (2019) Mingyuue Shang, Piji Li, Zhenxin Fu, Lidong Bing, Dongyan Zhao, Shuming Shi, and Rui Yan. 2019.

 

Semi-supervised text style transfer: Cross projection in latent space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4937-4946. External Links: Link Cited by: SS5.
* T. Shen, T. Lei, R. Barzilay, and T. Jaakkola (2017)Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. , pp. 6830-6841. External Links: Document, ISSN 1574-469X Cited by: SS1, SS2.1.
* S. Subramanian, G. Lample, E. M. Smith, L. Denoyer, M. Ramazato, and Y. Boureau (2018)Multiple-attribute text style transfer. CoRRabs/1811.00552. External Links: Link, 1811.00552 Cited by: SS1, SS2.1.
* A. Tikhonov and I. Yamshchikov (2018)Sounds wilde phonetically extended embeddings for author-stylized poetry generation. In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology, Brussels, Belgium, pp. 117-124. External Links: ISBN 978-3-319-5033-3, Link, Document Cited by: SS1, SS2.1.
* A. Tikhonov, V. Shibaev, A. Nagaev, A. Nugmanova, and I. P. Yamshchikov (2019)Style transfer for texts: retrain, report errors, compare with rewrites. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3936-3945. External Links: Link, Document Cited by: SS1, SS2.1.
* E. Vanmassenhove, C. Hardmeier, and A. Way (2018)Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 3003-3008. External Links: Link, Document Cited by: SS1, SS2.1.
* Y. Wang, Y. Wu, L. Mou, Z. Li, and W. Chao (2019)Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3573-3578. External Links: Link, Document Cited by: SS1, SS2.1.
* C. Wu, X. Ren, F. Luo, and X. Sun (2019)A hierarchical reinforced sequence operation method for unsupervised text style transfer. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Vol. , pp. 4873-4883. External Links: Link, Document Cited by: SS1, SS2.1.
* J. Xu, X. Sun, Q. Zeng, X. Ren, X. Zhang, H. Wang, and W. Li (2018)Unpaired sentiment-to-sentiment translation: a cycled reinforcement learning approach. In ACL, Cited by: SS1, SS2.1.
* W. Xu, A. Ritter, B. Dolan, R. Grishman, and C. Cherry (2012)Paraphrasing for style. In Proceedings of COLING 2012, Mumbai, India, pp. 2899-2914. External Links: Link, Document Cited by: SS1, SS2.1.
* H. Yamagishi, S. Kanouchi, T. Sato, and M. Komachi (2016)Controlling the voice of a sentence in Japanese-to-English neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), Osaka, Japan, pp. 203-210. External Links: Link, Document Cited by: SS1, SS2.1.
* I. P. Yamashchikov, V. Shibaev, N. Khlebnikov, and A. Tikhonov (2020)Style-transfer and paraphrase: looking for a sensible semantic similarity metric. ArXivabs/2004.05001. External Links: Link, 2004.05001 Cited by: SS1, SS2.1.
* Y. Zhang, T. Ge, and X. SUN (2020)Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 3221-3228. External Links: Link, Document Cited by: SS1, SS2.1.
* Yaoming Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y. Yu (2018)Texygen: a benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research; Development in Information Retrieval, SIGIR '18, New York, NY, USA, pp. 1097-1100. External Links: Link, Document Cited by: SS1, SS2.1.

 

Does translation preserve formality?

We examine the extend to which machine translation--through the aws service--affects the formality level of an input sentence: starting from a set of English sentences we have formality judgments for (ORIGINAL-**EN**), we perform a round-trip translation via pivoting through an auxiliary language (PIVOT-**X**). We then compare the formality prediction scores of the English Formality regression model for the two versions of the English input. In terms of Spearman correlation, the model's performance drops by \(7.5\) points on average when tested on round-trip translations. To better understand what causes this drop in performance, we present a per formality bin analysis in Table 9. On average we observe that translation preserves the formality level of formal sentences considerably well, while at the same time it tends to shift the formality level of informal sentences towards formal values--by a margin smaller than \(1\) point--most of the times. To account for the formalization effect of translation, we draw the line between formal and informal sentences at the value of ~\(1\) for scores predicted by multilingual regression models. This decision is based on the following intuition: if the formality shift of machine translated informal sentences is around \(+1\) value, the propagation of English formality labels imposes a negative shift of formal sentences in the model's predictions.

## Appendix B Amazon Web Service details

We compute the performance of the aws system on \(2.5\)K randomly sentences from OpenSubtitles, as a sanity check of translation performance. We report bleu of \(37.16\) (br-pt), \(33.79\) (fr), and \(32.67\) (it).

## Appendix C Rule-based baselines

We develop a set of rules to automatically make an informal sentence more formal via performing surface-level edits similar to the en rule-based system of Rao and Tetreault (2018). The set of extracted rules are shared across languages with the only difference being the list of abbreviations:

Normalize punctuationWe remove punctuation symbols that are repeated >= \(2\) times.

Character repetitionWe trailed characters repeated >= \(3\) times (_e.g., ciaooo_\(\rightarrow\)_ciao_ (IT)).

Normalize casingSeveral sentences might consist of words that are written in upper case. We lower case all characters apart from the first characters of the first word.

Normalize abbreviationsInformal text might contain slang words that are be abbreviated. We hand-craft a list of expansions for each language. For example, we replace _kra_\(\rightarrow\)_cara_ (br-pt), _ta_\(\rightarrow\)_ti amo_ (it), and _bjr_\(\rightarrow\)_bonjour_ (fr), using publicly available resources. 17, 18, 19 The resulting lists sizes are: \(64\) (br-pt), \(21\) (it), and \(48\) (fr).

Footnote 17: [https://braziliangringo.com/brazilianinternetslang/](https://braziliangringo.com/brazilianinternetslang/)

Footnote 18: [https://www.dummies.com/languages/italian/texting-and-chatting-in-italian/](https://www.dummies.com/languages/italian/texting-and-chatting-in-italian/)

Footnote 19: [https://frenchtogether.com/french-testing-slang/](https://frenchtogether.com/french-testing-slang/) 

_Annotator \(2)\)_ via randomly choosing one annotator's judgment for a given instance as the rating of _Annotator \(1\)_ and taking the mean rating of the rest judgments as the rating of _Annotator \(2\)_. We then compute Cohen's \(\kappa\) for these two simulated annotators. We repeat this process \(1{,}000\) times, and report the median and standard deviation of results. For measuring agreement of the _overall_ ranking evaluation task we use the same simulated framework and report results of Kendall's \(\tau\). Table 10 presents Inter-annotator agreement results on human evaluation across evaluation aspects and languages.

## Appendix F Yahoo! L6 language statistics

Table 11 presents the total number of questions included in L6 Yahoo! Corpus, for each language. Although almost \(90\%\) of questions are in English, the corpus contains a non-neglibible number of informal sentences in Spanish, French, Portuguese, Italian, and German, with a long tail of few questions for other languages.

## Appendix G Instructions for xformal

Summary of the taskGiven an **informal sentence** in French (or Portuguese/Italian), generate its **formal rewrite**, _without changing its meaning_.

Detailed InstructionsGiven an informal sentence, provide us with its informal rewrite. The informal rewrite should only change the formality attribute of the original sentence and preserve its meaning. Each sentence should be treated independently while rewrites should only rely on the information available in the sentences. There is no need to guess what additional information might be available in the documents the sentences come from.

ExamplesFollowing we include examples of good and bad rewrites given to Turkers: informal _Wow, I am very dumb in my observation skills......._good formal _I do not have good observation skills._reasoning _Formality is properly transferred and meaning is preserved._bad formal _Wow, I am very foolish in my observation skills.._reasoning _Formality is not properly transferred._bad formal _I am very unintligent and I don't have good observation skills._reasoning _Meaning has changed._

## Appendix H Qualitative analysis of xformal

Following, we include the instructions given to **native speakers** for the qualitative analysis of xformal, as described in SS4.

Background contextIn this task you will be asked to judge the quality of formal rewrites of a set of informal sentences. To give more background context, your work would serve as a quality check over annotations obtained from the Amazon Mechanical Turk (AMT) platform. AMT workers were given an informal sentence, e.g., "I'd say it is punk though", and then asked to provide us with its formal rewrite while maintaining the original

\begin{table}
\begin{tabular}{l r r r r r r}  & \multicolumn{3}{c}{_Informal bins_} & \multicolumn{3}{c}{_Formal bins_} \\  & [-3;2] & [-2;-1] & [-1;0] & [0;1] & [1,2] & [2,3] \\ \hline GOLD-STANDARD & \(-2.35\pm 0.31\) & \(-1.49\pm 0.36\) & \(-0.57\pm 0.33\) & \(0.46\pm 0.32\) & \(1.33\pm 0.34\) & \(2.27\pm 0.25\) \\ \hline ORIGINL–**EN** & \(-1.81\pm 0.59\) & \(-1.18\pm 0.73\) & \(-0.53\pm 0.71\) & \(0.30\pm 0.82\) & \(1.17\pm 0.68\) & \(1.50\pm 0.45\) \\ \hline PIVOT-**IT** & \(-1.58\pm 0.67\) & \(-0.95\pm 0.74\) & \(-0.35\pm 0.68\) & \(0.34\pm 0.86\) & \(1.12\pm 0.73\) & \(1.41\pm 0.54\) \\ \(\Delta\) & \(+0.23\pm 0.45^{*}\) & \(+0.23\pm 0.54^{*}\) & \(+0.18\pm 0.53^{*}\) & \(+0.04\pm 0.39\) & \(-0.05\pm 0.23\) & \(-0.09\pm 0.24\) \\ PIVOT-**FR** & \(-1.44\pm 0.77\) & \(-0.93\pm 0.75\) & \(-0.37\pm 0.72\) & \(0.38\pm 0.80\) & \(1.12\pm 0.82\) & \(1.50\pm 0.47\) \\ \(\Delta\) & \(+0.37\pm 0.54^{*}\) & \(+0.25\pm 0.50^{*}\) & \(+0.15\pm 0.48\) & \(+0.08\pm 0.42\) & \(-0.05\pm 0.49\) & \(-0.00\pm 0.17\) \\ PIVOT-**PT** & \(-1.46\pm 0.71\) & \(-0.87\pm 0.80\) & \(-0.31\pm 0.70\) & \(0.34\pm 0.83\) & \(1.14\pm 0.71\) & \(1.47\pm 0.47\) \\ \(\Delta\) & \(+0.35\pm 0.49^{*}\) & \(+0.31\pm 0.57^{*}\) & \(+0.21\pm 0.55^{*}\) & \(+0.03\pm 0.39\) & \(-0.03\pm 0.31\) & \(-0.03\pm 0.14\) \\ \end{tabular}
\end{table}
Table 9: Average formality scores of human annotations (GOLD-STANDARD) and model’s predictions on the original and round-trip translated PT16 test set grouped in 6 bins of varying formality. \(\Delta\) gives the average formality shift of the English sentences resulting from round-trip translation. Scores in blue and red indicate that mean is above and below zero, respectively. \(*\) denotes statistical significant formality shifts with \(p<0.05\). Translation preserves formality of formal sentences while informal sentences exhibit a shift towards formal values.

 

content and being grammatical/fluent, e.g., "However, I do believe it to be punk". In our work, workers were presented with sentences in French, Italian and Brazilian Portuguese. For our analysis we want to know a) whether the quality of the collected annotations is good (Task 1), b) what are the types of edits workers performed when formalizing the input sentence (Task 2). Both tasks consist of the same \(200\) informal-formal sentences-pairs and could be performed either in parallel (e.g., judging a single informal-formal sentence-pair both in terms of quality and types of edit at the same time; Task 1 and 2 in Google sheet), or individually (Task 1, Task 2 in Google sheet). More information and examples for the two tasks are included below.

**Task 1** Given an informal sentence you are asked to assess the quality of its formal rewrite. For each sentence pair type excellent, acceptable, or poor under the rate column. Read the instructions below before performing the task!

**What constitutes a good formal rewrite?**

* The style of the rewrite should be formal
* The overall meaning of the informal sentence should be preserved
* The rewrite should be fluent

**How should I interpret the provided options?** Below we include detailed instructions on how to interpret the provided options.

* **Excellent** the rewrite is formal, fluent and the original meaning is maintained. There is very little that could be done to make a better rewrite.
* **Acceptable** the rewrite is generally an improvement upon the original but there are some minor issues (such as the rewrite contains a typo or missed transforming some informal parts of the sentence, for example).
* **Poor** the rewrite offers a marginal or no improvement over the original. There are many aspects that could be improved.

**Task 2** In this task the goal is to characterize the types of edits workers made while formalizing the informal input sentence. For each informal-formal sentence pair you should check each of the provided boxes. Note that: a) Multiple types of edits might hold at the same type (e.g., in the informal--formal "Lexus cars are awesome!"---"Lexus cars

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**task** & \multicolumn{3}{c}{br-pt} & \multicolumn{3}{c}{fr} & \multicolumn{3}{c}{tr} \\ \cline{4-7}  & Weighted \(\kappa\) & ICC & Weighted \(\kappa\) & ICC & Weighted \(\kappa\) & ICC \\ Formality & \(0.57\pm 0.02\) & \(0.72\pm 0.01\) & \(0.56\pm 0.02\) & \(0.71\pm 0.01\) & \(0.67\pm 0.02\) & \(0.79\pm 0.02\) \\ Fluency & \(0.43\pm 0.02\) & \(0.62\pm 0.02\) & \(0.58\pm 0.02\) & \(0.73\pm 0.01\) & \(0.46\pm 0.02\) & \(0.64\pm 0.02\) \\ Meaning Preservation & \(0.53\pm 0.02\) & \(0.70\pm 0.02\) & \(0.71\pm 0.02\) & \(0.83\pm 0.01\) & \(0.55\pm 0.03\) & \(0.71\pm 0.02\) \\  & & & & Kendall’s \(\tau\) & & \\ Overall & & \(0.41\pm 0.03\) & \(0.44\pm 0.03\) & \(0.41\pm 0.03\) & \\ \hline \hline \end{tabular}
\end{table}
Table 10: Inter-annotator agreement for human evaluation results.

\begin{table}
\begin{tabular}{l r r} language & code & \# questions \\ \hline English & en & 3,895,407 \\ Spanish & es & 258,086 \\ French & fr & 125,393 \\ Portuguese & pt & 105,813 \\ Italian & tt & 55,027 \\ German & de & 43,149 \\ Persian & fa & 30 \\ Catalan & ca & 27 \\ Dutch & nl & 13 \\ Danish & da & 10 \\ Arabic & ar & 8 \\ Romania & bo & 8 \\ Norwegian & so & 8 \\ Swedish & sv & 5 \\ Estonian & et & 5 \\ Finnish & hi & 4 \\ Malay & ms & 4 \\ Turkish & tr & 4 \\ Slovak & sk & 3 \\ Latvian & lv & 3 \\ Albania & sq & 3 \\ Croatian & hr & 3 \\ Tagalog & tl & 3 \\ Chech & cs & 3 \\ Slovenian & sl & 3 \\ Vietnamese & vi & 2 \\ Icelandic & ls & 2 \\ Hungarian & hu & 2 \\ Polish & pl & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Number of question per language included in L6 Yahoo! Answers.

 

are very nice." you should check both the paraphrase and the punctuation boxes.); b) At the end of the Google sheet there is a column named 'Other' you are welcome to write down in plain text any additional type that you observed and it is not covered by the existing classes. The provided classes are: capitalization, punctuation, paraphrase, delete fillers, completion, add context, contractions, spelling, normalization, slang/idioms, politeness, split sentences, and relativizers.

## Appendix I Quantitative analysis of xformal

Following, we include more details on the qualitative analysis procedures.

CapitalizationA rewrite performs a capitalization edit if it contains tokens that appear in capital letters in the informal text but in lowercase letters in the formal text.

PunctuationA rewrite contains a punctuation edits if any of the punctuation of the informal-formal texts differs.

SpellingWe identify spelling errors based on the character-based Levenshtein distance between informal-formal tokens.

NormalizationWe identify normalization edits based on a hand-crafted list of abbreviations for each language.

Split sentencesWe split sentences using the nltk toolkit.

ParaphraseA formal rewrite is considered to contain a paraphrase edit if the token level Levenshtein distance between the informal-formal text is greater than 3 tokens.

LowercaseA rewrite performs a capitalization edit if it contains tokens that appear in lower case in the informal text but in capital letters in the formal text.

RepetitionWe identify repetition tokens using regular expressions (a token that appears more than 3 times in a row is considered a repetition.)

## Appendix J xformal: Data Quality

We ask a **native speaker** to judge the quality of a sample of \(200\) rewrites for each language via choosing one of the following three options: "excellent", "acceptable", and "poor". The details of this analysis are included in A.H (**Task 1**).Results indicate that on average less than \(10\%\) of the rewrites were identified as of poor quality across languages while more than \(40\%\) as of excellent quality. The small number of rewrites identified as "poor" consists mostly of edits where humans add context not appearing in the original sentence. We choose not to exclude any of the rewrites from the dataset as we provide multiple reformulations of each informal instance.

## Appendix K OpenSubtitles data

### Qualification tests

Table 13 presents questions and answers used in qc2 of our annotation protocol. Turkers have to score \(80\) and above to participate in the task. To compute an average score for each test, we assume that an answer is incorrect if it deviates more than \(1\) point from the gold-standard scores given in the second column of Table 13.

### Trade-off plots

Trade-off plotsFigure 4 presents trade-off plots between multi-bleu vs. formality score and fluency, across different reruns as proposed by Tikhonov and Yamshchikov (2018). First, we observe that models exhibit small variations in terms of formality and fluency scores across different reruns, and larger variations across bleu for most cases. Notably, the single seed backtranslate systems are the most consistent across reruns for all metrics and languages. Furthermore, in almost all cases, models trained on 2M data, perform better that the naive copy baseline, across metrics. However, single seed models fail to consistently outperform the rule-based baselines in almost all cases, with the exception of backtranslate for br-pt and fr which report an improvement of about \(1\) bleu score.

## Appendix N Compute time \(\&\) Infrastructure

All experiments for benchmarking both nmt-based and unsupervised approaches use Amazon ec2 P3 instances: [https://aws.amazon.com/](https://aws.amazon.com/)

\begin{table}
\begin{tabular}{l c r r r} OpeSubtiles & Preprocessing & br-pt & it & fr \\  & ✗ & \(61.3\text{M}\) & \(35.2\text{M}\) & \(41.8\text{M}\) \\ All & ✓ & \(29.8\text{M}\) & \(16.8\text{M}\) & \(19.0\text{M}\) \\ Informal & ✓ & \(25.7\text{M}\) & \(14.7\text{M}\) & \(16.3\text{M}\) \\ \end{tabular}
\end{table}
Table 12: OpenSubtitles statistics in three languages.

 

ec2/instance-types/p3/, on Tesla V\(100\) gpus. Concretely, nmt-based experiments are run on \(4\) gpus (\(\sim 1\)-2 hours), while unsupervised models are run on single gpus, with average run time spanning from \(\sim 5\) hours (for out-of-domain data--\(100\)K), to \(\sim 72\) hours (for in domain data--\(2\)M).

Figure 4: BLEU vs. STYLE SCORE vs. PERPLEXITY trade-off plots across \(4\) reruns.

\begin{table}
\begin{tabular}{l r} \hline
**French** & \\ \hline Heureux veux qui ont la chance de pouvoir faire la sieste devant Derrick ! & –2 \\ je kiffe trop ce film il déchire carément!! & –3 \\ Moi je les données à des vieilles personnes qui ne comprenent pas le fonctionenement d’un lecteur DVD! & –1 \\ Le producture de ce moreau, c’ est Scott Torch. & 2 \\ Il si agit blu du groupe America et le titre “A horse wino no name”. & 2 \\ La France ne gangera jamais elle continuée avce des chansonsu qui ont l’air de dater des années 70. & 0 \\ Le but n’est pas de se faire du profitessus! & -1 \\ Il existe surtout sur les chaines du cable et du satellite, TL9, TPS foot, par exemple. & 2 \\ \hline
**ITALIAN** & \\ \hline TI do questo situ qui puoi trovare qualnquee cosa su Tru Calling buona lettura! & –1 \\ Parla proprio di amorpi i charot e ma matai?!!!!!!!!!! & –2 \\ ma parla cm magni!!!!!!!!!!!! & –3 \\ io ho un programma si kiama evil lyrics. & –1 \\ Se non ci si da una wegliate ”dove andremo a finire?" & –1 \\ Alla fine del Festival ci saranno due vincitori, une per categoria. & 2 \\ é quello che stó passando ora. & –1 \\ II montaggio visnità quinidi ridotto al minimo. & 2 \\ \hline
**brazilian-portuguese** & \\ \hline Me parece que você usa BASTANTE essa palavra e seas derivados, ne??!! & –2 \\ Mas verm ca, vo