SlowGuess/NAACL_2022_OCR huggingface dataset:
Maria Nădejde, Anna Currey, Benjamin Hsu, Xing Niu, Marcello Federico, Georgiana Dinu
mnnadejd@amazon.com
The machine translation (MT) task is typically formulated as that of returning a single translation for an input segment. However, in many cases, multiple different translations are valid and the appropriate translation may depend on the intended target audience, characteristics of the speaker, or even the relationship between speakers. Specific problems arise when dealing with honorifics, particularly translating from English into languages with formality markers. For example, the sentence 'Are you sure?' can be translated in German as 'Sind Sie sich sicher?' (formal register) or 'Bist du dir sicher?' (informal). Using wrong or inconsistent tone may be perceived as inappropriate or jarring for users of certain cultures and demographics.
This work addresses the problem of learning to control target language attributes, in this case formality, from a small amount of labeled contrastive data. We introduce an annotated dataset (CoCoA-MT) and an associated evaluation metric for training and evaluating formality-controlled MT models for six diverse target languages. We show that we can train formality-controlled models by fine-tuning on labeled contrastive data, achieving high accuracy (82% in-domain and 73% out-of-domain) while maintaining overall quality.
The quality of neural machine translation (NMT) models has been improving over the years and is approaching that of human translation (Hassan et al., 2018). With fewer glaring accuracy or fluency errors, it is important to address other aspects of translation quality, such as tone and style, in order to generate context-appropriate translations and improve the end-user experience with MT systems. In particular, for spoken language and certain text domains (customer service, business, gaming chat), problems arise when translating from English into languages that have multiple formality
Source	Could you provide your first name please?
Informal	Könntest duitte deinen Vornamen angegeben?
Formal	Könnten Sieitte ihren Vornamen angegeben?
Source	OK, then please follow me to your table.
Informal	ははテaproルまて私て上来。
Formal	ははテaproルまて私て上来てんだoes。
Respectful	ははテaproルまて私て上来てんだoes。
Table 1: Contrastive translations for EN-DE and EN-JA with different formality. Phrases in bold were annotated by professional translators as marking formality.
levels expressed through honorifics or grammatical register. Taking the example from Table 1, the phrase 'Could you?' can have two equally correct German translations: 'Könnten Sie?' for the formal register and 'Könntest du?' for informal. This problem has been addressed previously with custom models trained on data with consistent formality (Viswanathan et al., 2019), or through side constraints to control politeness or formality (Sennrich et al., 2016a; Niu et al., 2018; Feely et al., 2019; Schioppa et al., 2021). Most prior research has been tailored to individual languages and has labeled large amounts of data using word lists or morphological analysers.
In this work we look at formality across multiple languages and frame formality control as a transfer learning problem, by leveraging a generic NMT system and a small amount of manually labeled data to obtain MT systems that are controllable for formality. Our main contributions are threefold. First, we release a novel multilingual and multi-domain benchmark for Contrastive Controlled MT (CoCoA-MT) consisting of contrastive translations with phrase-level annotations of formality and grammatical gender in six diverse language pairs: English (EN)  $\rightarrow$  French (FR), German (DE), Hindi (HI), Italian (IT), Japanese (JA), and Spanish (ES). Second, to accompany the CoCoA-MT dataset, we introduce a reference-based automatic metric with high precision at distinguishing formal from informal system hypotheses. Third, we propose training
Findings of the Association for Computational Linguistics: NAACL 2022, pages 616 - 632
formality-controlled models using transfer learning on contrastive labeled data. Our method is effective across six language pairs and robust across several datasets. We show that transfer learning using CoCoA-MT is complementary to automatically labeled data, while cost-effective compared to non-contrastive curated data.
We release the CoCoA-MT dataset, together with Sockeye  $3^{1}$  baseline models and evaluation scripts. These resources were also available to participants of the IWSLT 2022 (Anastasopoulos et al., 2022) shared task on Formality Control for Spoken Language Translation.
2 CoCoA-MT Dataset
We first introduce CoCoA-MT, our Contrastive Controlled MT by AWS AI dataset, which enables evaluation and training of formality-controlled models.
2.1 Source Data
The EN source data comes from three domains/ modalities: Topical-Chat $^4$  (Gopalakrishnan et al., 2019), as well as new Telephony and Call Center data. $^5$  Topical-Chat consists of text-based conversations about various topics, such as fashion, books, sports, and music. The Telephony domain contains transcribed spoken general conversations, unrestricted for topic. The Call Center data is also transcribed spoken data, where the conversations come from simulated customer support scenarios.
We use these three datasets to extract subsets containing utterances that are relevant to the formality control task. The subsets are designed to ensure coverage of diverse phenomena related to formality (honorifics or grammatical register) in the target languages. Specifically, we first selected segments (without the conversational context) having between 7 and 40 words and containing second-person pronouns (relevant for all target languages) and first-person pronouns (relevant for honorifics in JA). Through regular expressions, we ensured that the selected data contained the relevant pronouns in various positions (subject, object, object
of preposition). Second, we created a list of common EN verbs and used them in data selection in order to ensure lexical diversity of verbs and verb forms. Third, the automatically selected segments were further filtered or corrected by native English-speaking annotators who were asked to remove stock phrases (e.g. thank you), ensure that at least one addressee or speaker is referenced, and clean disfluencies from the speech data.
The selected source segments were then further filtered after the translation and phrase-level annotation steps described in the next section.
2.2 Translations and Annotations
For each source segment, we collected one reference translation for each level of formality (formal and informal). For JA, where more than two formality levels are possible, informal was mapped to kudaketa, formal to teineigo, and respectful to sonkeigo and/or kenjougo. We discarded segments if translators did not provide a translation for each formality level, because we considered these segments not relevant for the formality control task. Table 1 provides examples for EN→DE and EN→JA. Annotators also provided phrase-level annotations of formality markers in the target segments in order to facilitate evaluation and analysis (shown in bold in Table 1).
Reference translations were created by professional translators who were native speakers of the specified language and geographic variant.7 Formal translations were created from scratch as the canonical form, and informal translations were post-edited from the formal translations to ensure that there were no spurious differences between formal and informal references. Translators were instructed to generate natural translations that preserve the meaning and tone of the original sentence while addressing formality with minimal required changes. Such changes included swapping pronouns, editing verb forms, and additional lexical changes to obtain natural-sounding translations. We report dataset statistics in the next section and the full instructions given to translators in Appendix D.
$^{1}$ https://github.com/awslabs/socketye  
 $^{2}$ The full data, including train/test splits, will be released at https://github.com/amazon-research/contrastive-controlled-mt/ under a CDLA-Sharing-1.0 license.  
 $^{3}$ https://iwslt.org/2022/formality  
 $^{4}$ http://github.com/alexa/Topical-Chat/  
 $^{5}$ The Telephony and Call Center data is part of a larger conversational dataset that is currently a work in progress.
$^{6}$ In this work we only use the informal and formal contrastive translations. For Japanese, we release, when applicable, additional translations for the respectful formality level.  $^{7}$ For French and Spanish, we release variants from France and Spain, respectively. We will release additional references for Canadian French and Mexican Spanish in the near future.
2.3 Dataset Statistics
For each language pair, we release test data for all three domains (Topical-Chat, Telephony, and Call Center), and training data for Topical-Chat and Telephony. All segments in the test data have distinct formal/informal references, while the training data contains some segments with identical references for both formality levels.
Table 2 reports the number of training and test segments for each language pair, as well as the overlap (measured as BLEU) between informal and formal references in the test set. Note that EN-JA has more training data because we include both first-person and second-person formality segments. The similarity between formal and informal translations is lowest for EN-JA and highest for EN-HI, confirming that Hindi and Japanese are the two extremes with respect to the degree of formality marking among these six languages.
Target	#train	#test	overlap
DE	400	600	75.1
ES	400	600	79.0
FR	400	600	76.7
HI	400	600	81.1
IT	400	600	78.8
JA	1,000	600	74.6
In Table 3 we report corpus level statistics on the variety of phenomena represented in formal training set, including the number of unique and total phrases and tokens labeled for formality in the reference translation. Additionally, we report on the fraction of tokens labelled for formality that are either verbs or pronomials. To compute the part-of-speech for each token, for Hindi we utilized stanza (Qi et al., 2020). For the other target languages, we utilized  $\text{spaCy}^8$  and the respective large language models. For Japanese there was a significant number of tokens that were nouns or adjectives (7%) which was not true for the other target languages (on average 2%).
3 Formality Evaluation
In this section, we present a manual analysis of formality expressed in the outputs of two generic commercial systems for inputs sampled from CoCoA-MT. Next, we propose and evaluate a reference
Table 2: Number of segments in the training and test data, and overlap between the references in the test set as measured by BLEU (informal vs. formal).
Phrases	Tokens
Target	#unique	#total	#unique	#total	%VB	%PR
DE	183	754	123	1,103	35.4	64.6
ES	219	625	217	758	48.2	42.9
FR	149	624	118	921	35.5	60.5
HI	33	627	34	628	18.9	80.6
IT	179	615	167	747	43.4	52.5
JA	915	2,473	619	6,778	82.6	0.0
Table 3: Formal training set statistics for the phrases and tokens labelled for formality and the fraction of tokens that are verbs or auxiliary verbs ("VB") or pronomials ("PR"). Note that for Japanese, roughly  $7\%$  of tokens were either nouns or adjectives.
Lang.	Sys.	F	I	N	O	IAA
EN-DE	A	45.7	46.0	3.0	5.4	0.93
B	49.8	39.8	3.7	6.7
EN-ES	A	26.8	67.4	1.5	4.2	0.91
B	28.0	66.1	1.2	4.7
EN-FR	A	68.6	24.6	0.5	6.4	0.94
B	72.7	18.6	0.5	8.2
EN-HI	A	81.7	3.2	1.7	13.5	0.96
B	87.7	5.2	1.5	5.7
EN-IT	A	3.7	74.9	14.4	7.0	0.92
B	1.3	93.3	2.7	2.7
EN-JA	A	29.0	42.2	2.0	24.2	0.82
B	73.8	1.7	2.0	20.0
Table 4: Percentage of system outputs labeled by professional translators according to the formality level: formal (F), informal (I), neutral (N), other (O).
based automatic metric which we will later use to evaluate formality-controlled models.
Manual Analysis of Commercial Systems General-purpose commercial MT systems are trained on web-scale parallel and monolingual data with different formality levels. To understand how these systems behave with respect to formality, we analyzed two commercial MT systems on 300 random samples from CoCoA-MT. For each target language and each system, two professional translators were asked to label the translations according to the formality markers present in the output: formal, informal, neutral, other. The label "neutral" was used for output that can be considered both formal or informal (impersonal-passive or plural forms), while "other" was used to label inconsistent formality or incorrectly omitted formality markers<sup>9</sup>.
Table 4 reports the distribution of labels for the two systems and the inter-annotator agreement measured by Krippendorff's alpha (Hayes and Krippendorff, 2007). Agreement is high at 0.91 on average across languages. The distribution of formality in the outputs varies widely across languages for both
<sup>8</sup>http://spacy.io
<sup>9</sup>We give examples in the appendix in Table 14.
Formal	Informal
LP	P	R	P	R
EN-DE	0.96	0.86	0.98	0.68
EN-ES	0.90	0.60	0.97	0.59
EN-FR	0.98	0.78	0.94	0.66
EN-HI	0.92	0.73	0.87	0.54
EN-IT	0.80	0.53	0.98	0.67
EN-JA	0.71	0.43	0.69	0.54
average	0.88	0.66	0.91	0.61
systems. Both systems exhibit cases of inconsistent formality, with over  $20\%$  of segments labeled as "other" for Japanese. Overall, systems A and B are surprisingly similar in their behaviour, with significant differences in only two languages: system B is more formal than system A for Japanese ( $73.8\%$  vs  $29.0\%$ ); system A outputs more neutral forms than system B for Italian ( $14.4\%$  vs  $2.7\%$ ).
Automatic Evaluation To evaluate formality-controlled models, we propose a reference-based corpus-level automatic accuracy metric. Given a system hypothesis, we automatically label it as formal or informal: formal if the hypothesis contains: a) any of the formality-marking phrases annotated in the formal reference and b) none of the phrases annotated in the informal reference. We reverse the conditions to assign an informal label. Note that some hypotheses may not fall into either category.
Following segment-level assignments, we compute a corpus-level Matched-Accuracy (M-Acc) metric as the percentage of outputs that match the desired formality level, out of all the instances classified automatically as either formal or informal (hence matched). We use the notation M-Acc  $(\mathrm{F}) / (\mathrm{I})$  to denote this score when the desired formality level is formal/informal, respectively. We could not reliably classify neutral and other examples automatically and as such we did not include these labels when computing accuracy. Algorithm 1 formally describes the implementation of the reference-based automatic Matched-Accuracy metric.
To validate the M-Acc metric, we compare the predictions for formal and informal against the true labels given to outputs of system A and system B (described above). We report the breakdown of precision and recall for the two labels for each language in Table 5. The reference-based segment-level classification algorithm achieves a macro-average of 0.90 precision and 0.64 recall across
Table 5: Precision and recall of automatic segment-level classification of system outputs as formal or informal.
Algorithm 1: Algorithm for computing the formal, informal Matched-Accuracy.
Input :System hypotheses and annotated (formal, informal) references
Output :Matched formal accuracy
1 for hyp ∈ hypotheses, (formal_ref, informal_ref) ∈ references do
2 for marked Phrase in formal_ref do
3 if marked Phrase in hyp then
4 #match_formal += 1
5 for marked Phrase in informal_ref do
6 if marked Phrase in hyp then
7 #match_informal += 1
8 if #match_formal > 0 and #match_informal = 0 then
9 formality ← Formal
10 else if #match_informal > 0 and #match_formal = 0 then
11 formality ← Informal
12 if formality = Formal then
13 #formal ← #formal + 1
14 else if formality = Informal then
15 #informal ← #informal + 1
16 #matched ← #formal + #informal
17 formal_acc ← #formal/#matched
18 informal_acc ← #informal/#matched
19 return formal_acc, informal_acc
formal and informal, with the highest performance for DE (0.97 precision and 0.77 recall) and the lowest for JA (0.70 precision and 0.49 recall).
4 Transfer Learning for Formality Control
We approach formality-controlled NMT as a transfer learning problem, where we fine-tune a generic pre-trained MT model on labeled contrastive translation pairs from the CoCoA-MT dataset. For each source segment we create two labeled training data points: one for each contrastive reference translation (formal and informal). We use a special token with a randomly initialized embedding for the formality label which we attach to the beginning of the source segment.
To leverage the small amount of labeled data while maintaining the overall quality of the generic pre-trained MT model, we first up-sample the labeled data by concatenating multiple copies.10 Next, we augment the labeled data with an equal amount of unlabeled data sampled randomly from the generic training set. Finally, we fine-tune on the combined labeled and unlabelled data for one
<sup>10</sup>We study the effect of up-sampling the labeled data in Section 6.
Figure 1: Accuracy on the CoCoA-MT test sets and generic quality scores (BLEU) for EN-DE for an increasing amount of the labeled data (up-sampling up to  $5\mathrm{x}$ ).
epoch with a fixed learning rate following the approach proposed by Hasler et al. (2021) for domain adaptation. With this method we train models that can perform both tasks: generic translation and formality-controlled translation.
NMT models into DE, ES, FR and IT were trained on 20M pairs sampled from ParaCrawl v9 (Bañón et al., 2020), using WMT newstest for development. For evaluating generic quality, we used the WMT newstests as well as the MuST-C data (Di Gangi et al., 2019). The EN-JA model was trained on all 10M pairs from JParaCrawl v2 (Morishita et al., 2020) using the IWSLT17 development set. For testing we used WMT newstest2020 and the IWSLT17 test set. The EN-HI model was trained on all 15M pairs from CCMatrix (Schwenk et al., 2021), using the WMT newsdev2014 for development and newstest2014 for testing.
NMT models were built using the Transformer-base architecture (Vaswani et al., 2017), but with 20 encoder layers and 2 decoder layers as recommended by Domhan et al. (2020) and SSRU decoder layers for faster decoding (Kim et al., 2019).
We report the complete lists of pre-processing and training arguments Appendix C.
This section evaluates formality-controlled models trained using CoCoA-MT as described in Section 4. Section 6.1 evaluates performance on CoCoA-MT test sets, investigating in-domain versus out-of-domain performance as well as the effect of up-sampling the labeled data in training. Section 6.2 compares the use of the contrastive CoCoA-
MT data with other sources of labeled data. In Section 6.3 we perform additional evaluations on existing (non-contrastive) test sets for which a single formality level is naturally appropriate: forum discussions (informal) and customer support conversations (formal). This is a common scenario, requiring consistent translations that are appropriate for the domain and target audience.
6.1 CoCoA-MT Performance
To maximize the effectiveness of transfer learning with the small amount of curated labeled data, we first experiment with up-sampling the contrastive labeled data for EN-DE. Figure 1 shows accuracy on the CoCoA-MT test sets for different upsampling factors. We report both formal and informal M-Acc values, obtained by setting the desired formality level to formal/informal and evaluating against formal/informal references respectively. As previously described, the training data covers the Telephony and Topical Chat domains, but not the Call Center domain. For this reason, Telephony and Topical Chat results show in-domain performance while Call Center, out-of-domain (distinction also used in Table 6). BLEU scores are reported as a measure of generic quality: in this setting translations are generated without any formality control.
Results show that by increasing the up-sampling factor (up to  $5\mathrm{x}$ ), accuracy improves up to  $80\%$  on the combined test sets, while generic quality is fairly stable (small degradation of up to  $-0.6$  BLEU). To avoid over-fitting on the labeled data, we fix the up-sampling factor to five for all language pairs throughout the rest of the paper.[12] When comparing the learning curves for the three
12This corresponds to 4,000 total labeled sentence pairs for EN-DE,ES,FR,HI,IT and 10,000 for EN-JA. The up-sampling factor can be tuned further for each language to achieve the optimal trade-off between accuracy and generic quality (we report additional results for EN-JA in Appendix B).
<sup>11</sup>We used newtest 2020 for DE, 2014 for ES, 2015 for FR, and 2009 for IT.
M-ACC - In-domain	M-ACC - Out-of-domain	BLEU
Lang.	Labeled data	F	I	Avg.	F	I	Avg.	WMT	TED
EN-DE	none	41.0	59.0	-	82.5	17.5	-	42.1	32.7
CoCoA-MT	89.2	82.2	85.7	97.8	45.0	71.4	41.4	32.1
EN-ES	none	15.9	84.1	-	51.5	48.5	-	35.1	36.7
CoCoA-MT	61.8	80.4	71.1	89.1	47.8	68.4	35.0	36.9
EN-FR	none	89.9	10.1	-	100.0	0.0	-	38.2	43.0
CoCoA-MT	76.4	61.9	69.1	98.3	13.4	55.8	39.4	45.4
EN-IT	none	3.6	96.4	-	6.4	93.6	-	31.5	31.4
CoCoA-MT	98.5	98.2	98.4	98.5	96.5	97.5	31.7	32.0
EN-HI	none	98.1	1.9	-	100.0	0.0	-	10.0	-
CoCoA-MT	93.7	70.1	81.9	96.3	36.7	66.5	9.9	-
EN-JA	none	64.5	35.5	-	65.0	35.0	-	21.7	14.3
CoCoA-MT	84.8	84.4	84.6	68.8	83.2	76.0	22.2	14.3
Average	CoCoA-MT	84.1	79.5	81.8	91.4	53.8	72.6	-	-
domains, we find that Telephony and Topical Chat show similar trends, with high accuracy for both formal and informal, while on Call Center, the out-of-domain setting, the gap between formal and informal accuracy remains large (ca. 50 points).
Table 6 reports results on all language pairs. On the in-domain test set, accuracy averaged across formal and informal ranges from  $69.1\%$  for EN-FR to  $98.4\%$  for EN-IT, with generally high accuracy of over  $70\%$  across languages for both formal and informal. On the out-of-domain set, accuracy across languages is high for formal  $(91.4\%)$  but low for informal  $(55.4\%)$ . Accuracy for informal is particularly low on this domain for target languages where the generic models have a strong bias toward formal: DE, ES, FR, and HI. For these languages, we find this setting adversarial for generating informal outputs as the test set is out-of-domain and at the same time the generic training data biases the models towards formal. We leave for future work exploration of whether increasing data size can overcome this bias, as seems to be the case for EN-JA where informal accuracy is  $92.9\%$ .
From these results we conclude that transfer learning with as little as 400 to 1,000 labeled contrastive examples is effective for formality control on in-domain data and can generalize to out-of-domain data, while generic quality is maintained.[13]
Finally, a manual investigation of the outputs
reveals that formality-controlled models appear to transfer knowledge from the generic training data to generalize to other aspects of formality, beyond grammatical register. We observe examples of changes in lexical choice, punctuation or syntactic structure, even when such variations are not present in the labeled data for that target language. Table 7 shows some anecdotal examples. We leave a full investigation of this aspect to future work.
Table 6: Accuracy of baseline and formality-controlled models on in-domain (Telephony, Topical Chat) and out-of-domain (Call Center) test splits. The TED test sets are MuST-C for EN-DE,ES,FR,IT and IWSLT for EN-JA. For controlled models, M-Acc (F)/(I) scores are computed using formal/informal translations respectively, resulting in performance upper bounds of  $100\%$ . Baseline un-controlled models generate a single translation, leading to M-Acc (F) and M-Acc (I) to sum up to  $100\%$ .
Src	I am doing well. Thanks so much for asking.
I	Mir Goes es gut. Danke so viel, dass du gefragt hast.
F	Mir Goes es gut. Vielen Dank dafür, dass Sie Fragen.
Src	I will need to know the availability of the day you want to check in.
I	Tendré que saber la disponibilitad del día en que querres hacer el check-in.
F	Tendré que saber la disponibilitad del día en que desea registrarse.