SlowGuess/NAACL_2021_OCR huggingface dataset:
Olá, Bonjour, Salve!  
XFORMAL: A Benchmark for Multilingual Formality Style Transfer
Eleftheria Briakou*  
University of Maryland  
ebriakou@cs.umd.edu
Di Lu
Dataminr, Inc.
dlu@dataminr.com
Ke Zhang  
Dataminr, Inc.  
kzhang@dataminr.com
Joel Tetreault  
Dataminr, Inc.  
etreault@dataminr.com
We take the first step towards multilingual style transfer by creating and releasing XFORMAL, a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on XFORMAL suggest that state-of-the-art style transfer approaches perform close to simple baselines, indicating that style transfer is even more challenging when moving multilingual.
Style Transfer (ST) is the task of automatically transforming text in one style into another (for example, making an impolite request more polite). Most work in this growing field has focused primarily on style transfer within English, while covering different languages has received disproportional interest. Concretely, out of 35 ST papers we reviewed, all of them report results for ST within English text, while there is just a single work covering each of the following languages: Chinese, Russian, Latvian, Estonian, and French (Shang et al., 2019; Tikhonov et al., 2019; Korotkova et al., 2019; Niu et al., 2018). Notably, even though some efforts have been made towards multilingual ST, researchers are limited to providing system outputs as a means of evaluation, and progress is hampered by the scarcity of resources for most languages. At the same time, ST lies at the core of human communication: when humans produce language, they condition their choice of grammatical and lexical transformations to a target audience and a specific situation. Among the many possible stylistic variations, Heylighen et al. (1999) argue that "a dimension similar to formality appears as the most important and universal feature distinguishing styles, registers or genres in different languages". Consider the informal excerpts and their formal reformulations in French (FR) and Brazilian Portuguese
BRAZILIAN-PORTUGUESE  
saiam disso, fora de vontade!!  
get out of it, willpower!!  
Abandonem esta situacao, tenham fora de vontade.  
Abandon this situation, have willpower!  
FRENCH  
Il avait les yeux braqués ailleurs.  
He had his eyes fixed elsewhere.  
Il ne prétait pas attention à la situation.  
He was not paying attention to the situation.
ITALIAN  
in bocca al lupo!  
good luck!  
Ti rivolgo un sincero augurio!  
I send you a sincere wish!
Table 1: Informal-Formal pairs in XFORMAL.
(BR-PT) in Table 1. Both informal-formal pairs share the same content. However, the informal language conveys more information than is contained in the literal meaning of the words (Hovy, 1987). These examples relate to the notion of deep formality (Heylighen et al., 1999), where the ultimate goal is that of adding the context needed to disambiguate an expression. On the other hand, variations in formality might just reflect different situational and personal factors, as shown in the Italian (IT) example.
This work takes the first step towards a more language-inclusive direction for the field of ST by building the first corpus of style transfer for non-English languages. In particular, we make the following contributions: 1. Building upon prior work on Formality Style Transfer (FoST) (Rao and Tetreault, 2018), we contribute an evaluation dataset, XFORMAL that consists of multiple formal rewrites of informal sentences in three Romance languages: Brazilian Portuguese (BR-PT), French (FR), and Italian (IT); 2. Without assuming access to any gold-standard training data for the languages at hand, we benchmark a myriad
*Work done as a Research Intern at Dataminr, Inc.  
1Code and data: https://github.com/Elbria/xformal-FoST
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3199-3216 June 6-11, 2021. ©2021 Association for Computational Linguistics
of leading ST baselines through automatic and human evaluation methods. Our results show that FoST in non-English languages is particularly challenging as complex neural models perform on par with a simple rule-based system consisting of handcrafted transformations. We make XFORMAL, our annotations protocols, and analysis code publicly available and hope that this study facilitates and encourages more research towards Multilingual ST.
Controlling style aspects in generation tasks is studied in monolingual settings with an English-centric focus (intra-language) and cross-lingual settings together with Machine Translation (MT) (interlanguage). Our work rests in intra-language ST with a multilingual focus, in contrast to prior work.
ST datasets that consist of parallel pairs in different styles include: GYAFC for formality (Rao and Tetreault, 2018), Yelp (Shen et al., 2017) and Amazon Product Reviews for sentiment (He and McAuley, 2016), political slant and gender controlled datasets (Prabhumoye et al., 2018), Expert Style Transfer (Cao et al., 2020), PASTEL for imitating personal (Kang et al., 2019), SIMILE for simile generation (Chakrabarty et al., 2020), and others.
Intra-language ST was first cast as generation task by Xu et al. (2012) and is addressed through methods that use either parallel data or unpaired corpora of different styles. Parallel corpora designed for the task at hand are used to train traditional encoder-decoder architectures (Rao and Tetreault, 2018), learn mappings between latent representation of different styles (Shang et al., 2019), or fine-tune pre-trained models (Wang et al., 2019). Other approaches use parallel data from similar tasks to facilitate transfer in the target style via domain adaptation (Li et al., 2019), multi-task learning (Niu et al., 2018; Niu and Carpuat, 2020), and zero-shot transfer (Korotkova et al., 2019) or create pseudo-parallel data via data augmentation techniques (Zhang et al., 2020; Krishna et al., 2020). Approaches that rely on non-parallel data include disentanglement methods based on the idea of learning style-agnostic latent representations (e.g., Shen et al. (2017); Hu et al. (2017)). However, they are recently criticized for resulting in poor content preservation (Xu et al., 2018; Jin et al., 2019; Luo et al., 2019; Subramanian et al., 2018) and alternatively, translation-based models are proposed
that use reconstruction and back-translation losses (e.g., Logeswaran et al. (2018); Prabhumoye et al. (2018)). Another line of work, focuses on manipulation methods that remove the style-specific attribute of text (e.g., Li et al. (2018); Xu et al. (2018)), while recent approaches use reinforcement learning (e.g., Wu et al. (2019); Gong et al. (2019), probabilistic formulations (He et al., 2020), and masked language models (Malmi et al., 2020).
Inter-language ST is introduced by Mirkin and Meunier (2015) who proposed personalized MT for EN-French and EN-German. Subsequent MT works control for politeness (Sennrich et al., 2016a), voice (Yamagishi et al., 2016), personality traits (Rabinovich et al., 2017), user-provided terminology (Hasler et al., 2018), gender (Vanmassenhove et al., 2018), formality (Niu et al., 2017; Feely et al., 2019), morphological variations (Moryossef et al., 2019), complexity (Agrawal and Carpuat, 2019) and reading level (Marchisio et al., 2019).
3 XFORMAL Collection
We describe the process of collecting formal rewrites using data statements protocols (Bender and Friedman, 2018; Gebru et al., 2018).
Curation rational To collect XFORMAL, we firstly curate informal excerpts in multiple languages. To this end, we follow the procedures described in Rao and Tetreault (2018) (henceforth RT18) who create a corpus of informal-formal sentence-pairs in English (EN) entitled Grammarly's Yahoo Answers Formality Corpus (GYAFC).
Concretely, we use the L6 Yahoo! Answers corpus that consists of questions and answers posted to the Yahoo! Answers platform. The corpus contains a large number of informal text and allows control for different languages and different domains. Similar to the collection of GYAFC, we extract all answers from the Family & Relationships (F&R) topic that correspond to the three languages of interest: Familia e Relacionamente (BR-PT), Relazioni e famiglia (IT), and Amour et relations (FR) (Step 1). We follow the same pre-processing steps as described in RT18 for consistency (Step 2). We filter out answers that: a) consist of questions; b) include URLs; c) have fewer
$^{2}$ https://webscope.sandbox.yahoo.com/catalog.php?datatype  $= 1$  &did  $= 11$
3More details are included under A.F.
Corpus	BR-PT	FR	IT
L6 Yahoo!	230,302	225,476	101,894
+ Step 1	37,356	34,849	13,443
+ Step 2	14,448	14,856	4,095
+ Step 3	8,617	11,118	2,864
than five or more than 25 tokens; or d) constitute duplicates.4 We automatically extract informal candidate sentences, as described in  $\S 5.3$  (Step 3). Finally, we randomly sample 1,000 sentences from the pool of informal candidates for each language. Table 2 presents statistics of the curation steps.
Procedures We use the Amazon Mechanical Turk (MTurk) platform to collect formal rewrites for our informal sentences. For each language, we split the annotation into 20 batches of 50 Human Intelligence Tasks (HITs). In each HIT, Turkers are given an informal excerpt and asked to generate its formal rewrite in the same language without changing its meaning. We collect 4 rewrites per excerpt and release detailed instructions under A.G.
Annotation Workflow & Quality Control Our annotation protocol consists of multiple Quality Control (QC) steps to ensure the recruitment of high-quality annotators. As a first step, we use location restrictions (QC1) to limit the pool of workers to countries where native speakers are most likely to be found. Next, we run several small pilot studies (of 10 HITs) to recruit potential workers. To participate in the pilot study, Turkers have to pass a qualification test (QC2) consisting of multiple-choice questions that test workers' understanding of formality (see A.L). The pilot study results are reviewed by a native speaker (QC3) of each language to exclude workers who performed consistently poorly. We find that the two main reasons for poor quality are: a) rewrites of minimum-level edits, or b) rewrites that change the input's meaning. Table 3 presents the number of workers at each QC step. Only workers passing all quality control steps (last row of Table 3) contribute to the final task. Finally, we post-process the collected rewrites by a) removing instances consisting of normalization-based edits only and b) correcting minor spelling errors using an off-the-shelf tool. $^5$
Table 2: Number of sentences in filtered versions of the L6 Yahoo! Corpus across curation steps and languages.
Step	Description	BR-PT	FR	IT
(QC1)	Location restriction	151	78	59
(QC2)	Qualification test	54	40	33
(QC3)	Rewrites review	9	16	11
Table 3: Number of Turkers after each QC step.
Turkers' demographics We recruit Turkers from Brazil, France/Canada, and Italy for BR-PT, FR, and IT, respectively. Beyond their country of residence, no further information is available.
Compensation We compensate at a rate of  $0.10 per HIT with additional one-time bonuses that bumps them up to a target rate of over$ 10/hour.
After this entire process, we have constructed a high-quality corpus of formality rewrites of 1,000 sentences for three languages. In the next section, we provide statistics and an analysis of XFORMAL.
4 XFORMAL Statistics & Analysis
Types of formality edits Following Pavlick and Tetreault (2016), we analyze the most frequent edit operations Turkers perform when formalizing the informal sentences. We conduct both an automatic analysis (details in A.I) of the whole set of rewrites, and a human analysis (details in A.H) of a random sample of 200 rewrites per language (we recruited a native speaker for each language). Table 4 presents both analyses' results, where we also include the corresponding statistics for the English language (GYAFC). In general, we observe similar trends across languages: humans make edits covering both the "noisy-text" sense of formality (e.g., fixing punctuation, spelling errors, capitalization) and the more situational sense (paraphrase-based edits). Although cross-language trends are similar, we also observe differences: deleting fillers and word completion seems to be more prominent in the English rewrites than in other languages; normalizing abbreviations is a considerably frequent edit type for Brazilian Portuguese; paraphrasing is more frequent in the three non-English languages.
Surface differences of informal-formal pairs We quantify surface-level differences between the informal sentences and formal rewrites via computing their character-level Levenshtein distance (Figure 1) and their pairwise Lexical Difference (LeD) based on the percentages of tokens that are not found in both sentences (Table 5). Both analyses show that Italian rewrites have the most edits com
4We tokenize with nltk: https://www.nltk.org/ api/nltkTokenizer.html
<sup>5</sup>https://languagetool.org/
(a) BR-PT
(b) FR
(c) IT
(d) EN
Figure 1: Distribution of character-based Levenshtein distance between informal sentences and formal rewrites.
(a) BR-PT
Figure 2: Number of informal sentences and formal rewrites binned according to formality score.
EDIT TYPES	BR-PT	FR	IT	EN
CAPITALIZATION	0.24	0.33	0.32	0.43
0.43	0.56	0.60	0.46
PUNCTUATION	0.64	0.87	0.79	0.74
0.76	0.64	0.66	0.40
SPELLING	0.36	0.38	0.32	0.29
0.09	0.12	0.10	0.14
NORMALIZATION	0.17	0.01	0.04	0.07
0.27	0.02	0.06	0.10
SPLIT SENTENCES	0.07	0.06	0.11	0.08
0.07	0.07	0.07	0.04
PARAPHRASE	0.68	0.75	0.86	0.68
0.74	0.75	0.58	0.47
UNCHANGED	0.00	0.00	0.00	0.01
LOWERCASE	0.13	0.06	0.07	0.13
REPETITION	0.01	0.01	0.02	0.09
DELETE FILLERS	0.02	0.05	0.11	0.26
COMPLETION	0.07	0.04	0.04	0.15
ADD CONTEXT	0.02	0.00	0.07
CONTRACTIONS	0.02	0.01	0.01
SLANG/IDIOMS	0.19	0.10	0.15
POLITENESS	0.82	0.00	0.10
RELATIVIZERS	0.01	0.00	0.00
pared to their corresponding informal sentences. French and Brazilian Portuguese follow, with English rewrites being closer to the informal inputs.
Diversity of formal rewrites Are Turkers making similar choices when formalizing text? Since
Table 4: Percentage categories of frequent edits calculated automatically (not highlighted) and manually (highlighted). Categories are not mutually exclusive.
METRIC	BR-PT	FR	IT	EN
LeD	0.44	0.39	0.54	0.36
self-BLEU	0.43	0.35	0.21	0.51
Table 5: Surface differences between informal-formal pairs (LeD) and within formal rewrites (self-BLEU).
a large number of reformulations consist of paraphrase-based edits (more than  $50\%$ ), we want to quantify the extent to which the formal rewrites of each sentence are diverse, in terms of their lexical choices. To that end, we quantify diversity via measuring self-BLEU (Zhu et al., 2018): considering one set of formal sentences as the hypothesis set and the others as references, we compute BLEU for each formal set and define the average BLEU score as a measure of the dataset's diversity. Higher scores imply less diversity of the set. Results (last row of Table 4) show that XFORMAL consists of more diverse rewrites compared to GYAFC.
Formality shift of rewrites We analyze the formality distribution of the original informal sentences with their formal rewrites in GYAFC and XFORMAL, as predicted by formality mBERT models (§5.3). The distributions of formal rewrites are skewed towards positive values (Figure 2).
5 Multilingual FoST Experiments
We benchmark eight ST models on XFORMAL to serve as baseline scores for future research. We describe the models (§5.1), the experimental setting (§5.2), the human and automatic evaluation methods (§5.3 and §5.4), and results (§5.5).
Simple baselines We define three baselines: 1. COPY Motivated by Pang and Gimpel (2019) who notice that untransferred sentences with no alterations have the highest BLEU score by a large margin for ST tasks, we use this simple baseline as a lower bound; 2. RULE-BASED Based on the quantitative analysis of §4 and similarly to RT18, we develop a rule-based approach that performs a set of predefined edits-based operations defined by handcrafted rules. Example transformations include fix casing, remove repeated punctuation, handcraft a list of contraction expansions—a detailed description is found at A.C; 3. ROUND-TRIP MT Inspired by Zhang et al. (2020) who identify useful training pairs from the paraphrases generated by round-trip translations of millions of sentences, we devise a simpler baseline that starts from a text in language  $x$ , pivots to EN and then backtranslates to  $x$ , using the AWS translation service. $^6$
NMT-based models with synthetic parallel data We follow the TRANSLATE TRAIN (Conneau et al., 2018; Artetxe et al., 2020) approach to collect data in multilingual settings: we obtain pseudo-parallel corpora in each language via machine translating an EN resource of informal-formal pairs ( $\S 5.2$ ).<sup>7</sup> Then, starting with TRANSLATE TRAIN we benchmark the following NMT-based models: 1. TRANSLATE TRAIN TAG extends a leading EN FoST approach (Niu et al., 2018) and trains a unified model that handles either formality direction via attaching a source tag that denotes the desired target formality; 2. MULTI-TASK TAG-STYLE Niu et al. (2018) augments the previous approach with bilingual data that is automatically identified as formal ( $\S 5.3$ ). The models are then trained in a multi-task fashion; 3. BACKTRANSLATE augments the TRANSLATE TRAIN data with back-translated sentences of automatically detected informal text (Sennrich et al., 2016b), using 1. as the base model. We exclude backtranslated pairs consisting of copies. The output of the RULE-BASED system is given as input to each model at inference time. For all three models, we run each system with 4 random seeds, and combine them in a linear ensemble for decoding.
Unsupervised approaches We benchmark two unsupervised methods that are used for EN ST: 1. UNPSUPERVISED NEURAL MACHINE TRANS
LATION (UNMT) (Subramanian et al., 2018) defines a pseudo-supervised setting and combines denoising auto-encoding and back-translation losses; 2. DEEP LATENT SEQUENCE MODEL (DLSM) (He et al., 2020) defines a probabilistic generative story that treats two unpaired corpora of separate styles as a partially observed parallel corpus and learns a mapping between them, using variational inference.
Training data For TRANSLATE TRAIN TAG we use GYAFC, a large set of 110K EN informal-formal parallel sentence-pairs obtained through crowdsourcing. Additionally, we augment the translated resource with OpenSubtitles (Lison and Tiedemann, 2016) bilingual data used for training MT models. Given that bilingual sentence-pairs can be noisy, we perform a filtering step to extract noisy bittexts using the Bicleaner toolkit (Sánchez-Cartagena et al.). Furthermore, we apply the same filtering steps as in §3 (Curation rational). Finally, each of the remaining sentences is assigned a formality score (§5.3), resulting in two pools of informal and formal text. Training instances are then randomly sampled from those pools: formal parallel pairs are used for MULTI-TASK TAG-STYLE; informal target side sentences are backtranslated for BACKTRANSLATE; both informal and formal target-side texts are independently sampled from the two pools for training unsupervised models. Finally, for unsupervised FoST in FR, we additionally experiment with in-domain data from the L26 French Yahoo! Answer Corpus that consists of 1.7M FR questions. Table 6 includes statistics on training sizes.
METHODS	GYAFC	OpenSubs.	L6 Yahoo!
TRANSL. TRAIN TAG	110K	-	-
MULTI-TASK TAG-STYLE	110K	2M	-
BACKTRANSLATE	110K	2M	-
UNMT/DLSM	-	110K	-
UNMT/DLSM (IN-DOMAIN)	-	-	2M
Table 6: Number of training instances for each model.
Data are available at: http://opus.nlpl.eu/.
<sup>9</sup>We use the publicly available pretrained Bicleaner models: https://github.com/bitextor/bicleaner, and discard sentences with a score lower than 0.5.
10https://webscope.sandbox.yahoo.com/ catalog.php?datatype  $\equiv$  1&did  $= 74$
11 Split into 6.2M/6.6M formal/informal sentences.
$^{12}$ Bilingual data statistics are in A.K.
$^{6}$ https://aws.amazon.com/translate/
Details on the AWS performance are found in A.B.
Preprocessing We preprocess data consistently across languages using MOSES (Koehn et al., 2007). Our pipeline consists of three steps: a) normalization; b) tokenization; c) true-casing. For NMT-based approaches, we also learn joint source-target BPE with 32K operations (Sennrich et al., 2016b).
Model Implementations For NMT-based and unsupervised models we use the open-sourced implementations of Niu et al. (2018) and He et al. (2020), respectively. $^{13,14}$  We include more details on model architectures in A.D.
Recent work on ST evaluation highlights the lack of standard evaluation practices (Yamshchikov et al., 2020; Pang, 2019; Pang and Gimpel, 2019; Mir et al., 2019). We follow the most frequent evaluation metrics used in EN tasks and measure the quality of the system's outputs with respect to four dimensions, while we leave an extensive evaluation of automatic metrics for future work.
Meaning Preservation We compute self-BLEU (Papineni et al., 2002) which compares system outputs with the informal sentences.
Formality We average the style transfer score of transferred sentences computed by a formality regression model. We fine-tune mBERT (Devlin et al., 2019) pre-trained language models on the machine-translated answers genre from Pavlick and Tetreault (2016) that consists of about 4K human-annotated sentences rated on a 7-point formality scale. To acquire an annotated corpus in the languages of interest, we follow the TRANSLATE TRAIN transfer approach: we propagate the original EN training data's human ratings to their corresponding translations, assuming that translation preserves formality.[15] To evaluate the multilingual formality regression models' performance, we crowdsourced human judgments of 5 Turkers for 200 sentences per language. We report Spearman correlations of 64 (BR-PT), 70 (IT), 71 (FR), and 81 (EN).
Fluency We compute the logarithm of each sentence's probability—computed by a 5-gram Kneser-Ney language model (Kneser and Ney, 1995)—and normalize it by the sequence length. We train each
language model on 2M random sample of the non-English side of OpenSubtitles formal data.
Overall We compute multi-BLEU (Post, 2018) via comparing with multiple formal rewrites on XFORMAL. Freitag et al. (2020) shows that correlation with human judgments improves when considering multiple references for MT evaluation.
5.4 Human evaluation
Given that automatic evaluation of ST lacks standard evaluation practices—even in cases when EN is considered—we turn to human evaluation to reliably assess our baselines following the protocols of RT18. We sample a subset of 100 sentences from XFORMAL per language, evaluate outputs of 5 systems, and collect 5 judgments per instance. We open the task to all workers passing QC2 in Table 3. We include inter-annotator agreement results in A.E.
Formality We collect formality ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a 7-point discrete scale of  $-3$  to 3, following Lahiri (2015) (Very informal  $\rightarrow$  Informal  $\rightarrow$  Somewhat Informal  $\rightarrow$  Neutral  $\rightarrow$  Somewhat Formal  $\rightarrow$  Formal  $\rightarrow$  Very Formal).
Fluency We collect fluency ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a discrete scale of 1 to 5, following Heilman et al. (2014) (Other  $\rightarrow$  Incomprehensible  $\rightarrow$  Somewhat Comprehensible  $\rightarrow$  Comprehensible  $\rightarrow$  Perfect).
Meaning Preservation We adopt the annotation scheme of Semantic Textual Similarity (Agirre et al., 2016): given the informal reference and formal human rewrite or the formal system outputs, Turkers rate the two sentences' similarity on a 1 to 6 scale (Completely dissimilar  $\rightarrow$  Not equivalent but on same topic  $\rightarrow$  Not equivalent but share some details  $\rightarrow$  Roughly equivalent  $\rightarrow$  Mostly equivalent  $\rightarrow$  Completely equivalent).