SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 7: Examples from formality-controlled models of induced formality features beyond grammatical register.
6.2 Effect of Labeled Data Variations
Next, we compare the effectiveness of manually labeled and curated data with that of rule-based automatically labeled data. We create a balanced sample of informal and formal sentence pairs by labeling the target side of the generic training data with methods introduced in prior work: word lists for ES, FR (Viswanathan et al., 2019), and JA (Feely et al., 2019) $^{14}$  and morpho-syntactic rules
14We used the informal and polite entries from Table 3 of their paper.
<sup>13</sup>We observe a side effect on EN-FR where generic quality improves by more than 2 BLEU points on MuST-C. We attribute this to an adaptation effect as both the CoCoA-MT training set and MuST-C test set cover spoken language, which is less represented in web crawled parallel data.
Labeled data conditions	M-ACC - All domains	BLEU
Lang.	Contrastive	Manual	#Src	F	I	Avg.	WMT	TED
EN-DE	yes	yes	400	92.3	68.6	80.4	41.4	32.1
no	yes	800	95.7	76.6	86.2	41.2	32.2
no	no	800	37.3	77.9	57.6	40.8	31.8
no	no	4,000	38.5	77.0	57.7	41.6	32.0
EN-ES	yes	yes	400	71.5	68.8	70.2	35.0	36.9
no	yes	800	65.6	74.6	70.1	35.0	36.8
no	no	800	43.4	78.7	61.1	34.9	36.3
no	no	4,000	46.1	74.3	60.2	34.7	36.2
EN-FR	yes	yes	400	84.3	44.2	64.2	39.4	45.4
no	yes	800	82.5	47.9	65.2	39.1	45.5
no	no	800	44.2	69.3	56.8	38.7	42.6
no	no	4,000	50.9	66.6	58.7	39.1	43.3
EN-JA	yes	yes	1,000	80.0	84.0	82.0	22.2	14.3
no	no	2,000	44.1	57.0	50.6	21.3	13.6
no	no	10,000	45.8	54.5	50.1	21.6	13.8
Table 8: Accuracy of formality-controlled models trained with different sources of labeled data. We consider the fallowing conditions: contrastive vs non-contrastive references and manually vs automatically labeled. The total number of training data points is fixed across all conditions, however the number of unique source segments varies. The TED test sets are MuST-C for EN-DE,ES,FR and IWSLT for EN-JA.
for DE (Sennrich et al., 2016a). While these methods are known to have limited coverage for languages with complex honorifics systems such as Japanese, or to introduce errors (see examples in Table 9), their advantage is that they can be used to label large amounts of data.[15]
Source what are your thoughts on the goates some of the players grow?
Target qué piensas de las barbas de chivo que se DEAan crecer algunos jugadores?
Table 9: Example of an informal sentence from CoCoA- MT classified as formal by the rule-based classifier.
We compare models trained on the rule-based labeled data against two models: one trained on the contrastive CoCoA-MT data and another trained on non-contrastive CoCoA-MT data, with twice as many source segments. For comparability, we keep the total number of data points constant across all conditions. However, as additional rule-based labeled data is easy to obtain and may improve results, we test two settings: 800 data points up-sampled  $5\mathrm{x}$  (same as the other models), as well as 4000 unique data points.
Results are shown in Table 8. Fine-tuning on noisy rule-based labeled data results in lower average accuracy across all language pairs and significantly worse performance on EN-DE and EN-JA. On FR, DE, and ES, results shift to better informal
accuracy with a trade-off in formal performance. For EN-JA the rule-based data is not effective for either formal or informal control. Increasing the diversity of the rule-based data by using more unique source segments does not lead to significant improvements. However, given the complementary performance observed, combining the two labeled datasets is a promising future work direction.
The non-contrastive use of CoCoA-MT leads to accuracy improvements of 5.8 points for EN-DE and 1 point for EN-FR. This suggests that improving coverage by sourcing and annotating additional training data is beneficial. However, contrastive data is more efficient to create, as swapping formality levels is done through post-editing.
6.3 Human Evaluation on Held-Out Domains
We conduct human evaluation of accuracy and generic quality of formality-controlled models on non-contrastive data from two held-out domains. The first domain comprises noisy comments on Reddit forums from the MTNT dataset (Michel and Neubig, 2018b) and the second domain comprises task-based (customer service) dialog turns from the Taskmaster dataset (Byrne et al., 2019; Farajian et al., 2020). For the human evaluation we select source segments that have at least one second person pronoun and set the formality level
17The dialog topics are: ordering pizza, creating auto repair appointments, setting up ride service, ordering movie tickets, ordering coffee drinks and making restaurant reservations. We use the first 35 dialogues included in the WMT 2020 Chat Translation shared task. https://github.com/Unbabel/BConTrasT
$^{15}$ CoCoA-MT could be used to train a formality classifier that can annotate more data. We leave this to future work. For EN-JA we did not have additional annotated data for the non-contrastive setting.
Informal	Formal
Accbl_Δ	Scorebl_%	Accbl_Δ	Scorebl_%
DE	79.9+30.2	4.3+0.0%	90.4+33.3	4.7+0.4%
ES	75.4+2.0	5.1+0.5%	70.1+49.5	5.6+2.2%
FR	38.0+31.2	4.3+4.0%	89.7-3.4	5.0+6.9%
IT	93.1+21.7	4.3+1.4%	92.6+91.9	4.9+2.6%
JA	80.5+67.5	3.9+3.0%	69.1-13.0	4.3+6.4%
Table 10: Human evaluation of formality-controlled models on held-out domains. Formality is set to Informal on MTNT and to Formal on Taskmaster. We report absolute accuracy difference  $(bl_{\Delta})$  and relative quality gain  $(bl_{\%})$  between the controlled and baseline models.
to informal for the MTNT data and formal for the Taskmaster data, matching the typical formality level used for each domain. Translators were instructed to rate the quality of translations on a scale of 1 (poor) to 6 (perfect) and to mark whether the translation matches the desired formality level. We did not include Hindi as we believed translators would have difficulties with this task given the low level of generic quality (10 BLEU on newstest).
In Table 10, we report the accuracy and quality scores $^{18}$  for the formality-controlled models as well as the improvements over the generic baseline models. Human evaluation results confirm that our formality-controlled models can generalize to unseen domains. Their accuracy is generally high (at or above  $70\%$ ) and better than the baseline across languages for both Formal and Informal (with the exception of Formal for French and Japanese). At the same time, generic quality is retained or even slightly improved in some cases (up to  $6.9\%$  for French and  $6.4\%$  for Japanese on Taskmaster) compared to the generic baseline.
7 Gender-Specific Translations
While creating the CoCoA-MT formality-controlled dataset, we observed that for target languages with grammatical gender (all except JA), some reference translations require gender to be expressed in the target even though it is ambiguous in the source.[19] Table 11 shows one such sentence from the EN-ES training set.
In fact, this is similar to formality: a grammatical distinction must be made in the target language, even though the source is under-specified with re
Source	Did you play with Legos growing up?
Feminine	¿DeLEEVEQUENAGUGABANpiezas de Lego?
Masculine	¿De PequeñONUGABANpiezas de Lego?
Table 11: Sentence from CoCoA-MT where grammatical gender is expressed in the target but ambiguous in the source. We show formal translations for illustration.
target	train	test
DE	1%	1%
ES	11%	12%
FR	9%	10%
HI	38%	54%
IT	5%	8%
Table 12: Percent of training and test segments that express gender distinctions in the reference.
spect to gender. Therefore, we create references with feminine and masculine grammatical gender using the same approach as for formality: translators post-edit segments altering only what is necessary to change the grammatical gender.[20] This results in up to four translations for each source: {feminine, masculine}  $\times$  {formal, informal}. Table 12 shows the percent of gendered references in the data for each target language.
Effect on Gender Translation Accuracy In Section 6, for segments with gendered translations, we selected a single gender (in that case, masculine) to use consistently in all training and evaluation data. Here, we perform an initial evaluation of the effect of gender-specific formality-controlled data on gender translation accuracy using WinoMT (Stanovsky et al., 2019) on EN-ES.[21] We compare the baseline (without formality control) to separate models trained using masculine (msc-trg; same as in Table 6) and feminine (fem-trg) target data. These results, along with formality and quality metrics, are shown in Table 13.
Using only masculine target sentences causes a drop in feminine F1, whereas feminine target segments improve feminine F1 without harming masculine F1. For easy comparison with Section 6, we report formality matched accuracy with respect to the masculine-reference test set, which explains the slight drop in formality accuracy for fem-trg. These results show that gender-specific translations are useful for maintaining gender translation accuracy when creating formality-controlled models.
18We average the scores of the two annotators and for all sentences.
19Here, we refer to grammatical gender of the language; we do not infer or ascribe gender to any speaker or utterance.
20We restrict this initial work to two genders because most of the languages examined contain two grammatical genders.
21We evaluate on EN-ES because it is the language pair with the most gendered references of those supported by WinoMT.
model	WinoMT	M-ACC - All domains	BLEU
Acc.	Fem. F1	Msc. F1	F	I	AVG	WMT	TED
base	59.5	51.2	68.0	29.0	71.0	-	35.1	36.7
msc-trg	57.7	47.9	66.9	71.5	68.8	70.2	35.0	36.9
fem-trg	60.0	53.1	68.0	70.2	65.1	67.7	35.0	36.7
Table 13: WinoMT, formality, and BLEU scores on English→Spanish models trained without formality control (base), and with grammatically masculine and feminine target data.
We release the gender-specific translations to open up opportunities to explore the best use of this data. The data could also be for gender control given user-specified preferences for gender in translation (similar to formality control explored here). We leave these possibilities for future work.
Controlling politeness for NMT was first tackled by Sennrich et al. (2016a) for EN-DE translation. They appended side constraints to the source text to indicate the preference of verbs or T-V pronoun choices (Brown and Gilman, 1960) in the output.[22] A similar approach was applied to control the presence of honorific verb forms in EN-JA MT by Feely et al. (2019). Viswanathan et al. (2019) controlled T-V pronoun choices of EN-ES/FR/Czech translations by adapting generic models with T-V distinct data. They collected politeness parallel data using heuristics. In a task of FR-EN formality-sensitive MT (Niu et al., 2017), translation and EN formality transfer were trained jointly in a multi-task setting (Niu et al., 2018; Niu and Carpuat, 2020). They assumed cross-lingual formality parallel data is not available and leveraged monolingual formality data instead (Rao and Tetreault, 2018).
Prior work has also investigated control for attributes besides formality: speaker role and gender (Mima et al., 1997; Rabinovich et al., 2017; Elaraby et al., 2018; Bentivogli et al., 2020), voice (Yamagishi et al., 2016), length/verbosity (Takeno et al., 2017; Lakew et al., 2021), readability/complexity (Stymne et al., 2013; Marchisio et al., 2019; Agrawal and Carpuat, 2019), monotonicity (Schioppa et al., 2021), translator traits (Wang et al., 2021) and a writer's proficiency level and native language (Nadejde and Tetreault, 2019). Controlling multiple attributes with a single NMT system was investigated by Michel and Neubig (2018a); Schioppa et al. (2021). Annotation toolkits or parallel corpora annotated with some of these
attributes has also been released, including gender and age (Rabinovich et al., 2017; Vanmassenhove et al., 2018; Bentivogli et al., 2020), complexity (Agrawal and Carpuat, 2019), and speaker traits (Michel and Neubig, 2018a).
This work addresses the problem of controlling MT output when translating into languages that make formality distinctions through honorifics or grammatical register. To train and evaluate formality-controlled MT models, we introduce CoCoA-MT—a novel multilingual and multidomain benchmark—and a reference-based automatic metric. Our experiments show that formality-controlled MT models can be trained effectively with transfer learning on labeled contrastive translation pairs from CoCoA-MT, achieving high targeted accuracy and retaining generic translation quality. We release the CoCoA-MT dataset to enable future work on controlling multiple features (formality and grammatical gender) simultaneously.
We would like to thank Tanya Badeka and Jen Wang for their valuable help in creating the CoCoAMT dataset; Weston Feely for helping with the Japanese guidelines; Natawut Monaikul and Tamer Alkhouli for their comments on previous versions of the paper; and the anonymous reviewers for their suggestions.
10 Ethical Considerations
As part of this paper, we created and are releasing formality-controlled contrastive parallel data from English into French, German, Hindi, Italian, Japanese, and Spanish. The translations and annotations were created by professional translators who were recruited by a language service provider and were compensated according to industry standards. The translations are based on existing English corpora which are not user-generated. Before creating
22https://en.wikipedia.org/wiki/T-V_. distinction
the translations, we obtained approval for our use case from the creators of the existing artifacts.
As part of our formality-controlled dataset, we noticed that translations often required the gender of the speaker or the addressee to be specified, even when the English source was gender-neutral. As a result, for each such case, we include grammatically feminine and grammatically masculine reference translations. We hope that this will open up opportunities for future work in avoiding gender bias when controlling for politeness, and even in improving translations by customizing to the user's desired gender,[23] in a similar way to how we customize for the desired formality in this paper. In creating gender-specific reference translations, we limit the differences to words that are grammatically gendered in the target languages, rather than stereotypical or other differences. It is important to note that while this paper addresses grammatical gender in translation, it does not use human subjects, infer or predict gender, or otherwise use gender as a variable.
We would like to emphasize that the work on gender in this paper is very much a work in progress. We provide this dataset as an initial contribution; we will continue to improve on this work and this data, and we hope other groups also use and expand on it. Most notably, so far we have only produced translations for two genders. In the future, we plan on expanding the references translations to more genders, in consultation with native speakers of the target languages and other stakeholders. We also would like to analyze gender bias in formality-controlled models, as well as create models that can control for multiple features (e.g., formality and grammatical gender) simultaneously.
Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1549-1564, Hong Kong, China. Association for Computational Linguistics.
Antonios Anastasopoulos, Luisa Bentivogli, Marcely Z. Boito, Ondrej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Marcello Federico, Christian Federmann, Hongyu Gong,
Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, David Javorsky, Vera Kloudova, Surafel M. Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, Juan Pino, Elizabeth Salesky, Jiatong Shi, Sebastian Stuker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alex Waibel, Changhan Wang, and Shinji Watanabe. 2022. FINDINGS OF THE IWSLT 2022 EVALUATION CAMPAIGN. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), Dublin, Ireland. Association for Computational Linguistics.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrias, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555-4567, Online. Association for Computational Linguistics.
Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, and Marco Turchi. 2020. Gender in danger? evaluating speech translation technology on the MuST-SHE corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923-6933, Online. Association for Computational Linguistics.
Roger Brown and Albert Gilman. 1960. The pronouns of power and solidarity. In T. A. Sebeok, editor, Style in Language, pages 253-276. MIT Press, Cambridge, Mass.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4516-4525, Hong Kong, China. Association for Computational Linguistics.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a multilingual speech translation corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017, Minneapolis, Minnesota. Association for Computational Linguistics.
Tobias Domhan, Michael Denkowski, David Vilar, Xing Niu, Felix Hieber, and Kenneth Heafield. 2020. The Sockeye 2 neural machine translation toolkit at
23Note that we do not recommend inferring gender from the user, but customizing according to user-specified gender.
AMTA 2020. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 110-115, Virtual. Association for Machine Translation in the Americas.
Mostafa Elaraby, Ahmed Y. Tawfik, Mahmoud Khaled, Hany Hassan, and Aly Osama. 2018. Gender aware spoken language translation applied to English-Arabic. In 2nd International Conference on Natural Language and Speech Processing, ICNLSP 2018, Algiers, Algeria, April 25-26, 2018, pages 1-6. IEEE.
M. Amin Farajian, Antonio V. Lopes, Andre F. T. Martins, Sameen Maruf, and Gholamreza Haffari. 2020. Findings of the WMT 2020 shared task on chat translation. In Proceedings of the Fifth Conference on Machine Translation, pages 65-75, Online. Association for Computational Linguistics.
Weston Feely, Eva Hasler, and Adrià de Gispert. 2019. Controlling Japanese honorifics in English-to-Japanese neural machine translation. In Proceedings of the 6th Workshop on Asian Translation, pages 45-53, Hong Kong, China. Association for Computational Linguistics.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019. Topical-Chat: Towards knowledge-grounded open-domain conversations. In Proc. Interspeech 2019, pages 1891-1895.
Eva Hasler, Tobias Domhan, Jonay Trenous, Ke Tran, Bill Byrne, and Felix Hieber. 2021. Improving the quality trade-off for neural machine translation multi-domain adaptation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8470-8477, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving human parity on automatic Chinese to English news translation. CoRR, abs/1803.05567.
Andrew Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1:77-89.
Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, and Nikolay Bogoychev. 2019. From research to production and back: Ludriciously fast neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 280-288, Hong Kong. Association for Computational Linguistics.
Surafel Melaku Lakew, Marcello Federico, Yue Wang, Cuong Hoang, Yogesh Virkar, Roberto Barra-Chicote, and Robert Enyedi. 2021. Machine translation morbidity control for automatic dubbing. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7538-7542. IEEE.
Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level of machine translation output. In Proceedings of Machine Translation Summit XVII: Research Track, pages 193-203, Dublin, Ireland. European Association for Machine Translation.
Paul Michel and Graham Neubig. 2018a. Extreme adaptation for personalized neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 312-318, Melbourne, Australia. Association for Computational Linguistics.
Paul Michel and Graham Neubig. 2018b. MTNT: A testbed for machine translation of noisy text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 543-553, Brussels, Belgium. Association for Computational Linguistics.
Hideki Mima, Osamu Furuse, and Hitoshi Iida. 1997. Improving performance of transfer-driven machine translation with extra-linguistic information from context, situation and environment. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, IJCAI 97, Nagoya, Japan, August 23-29, 1997, 2 Volumes, pages 983-989. Morgan Kaufmann.
Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2020. JParaCrawl: A large scale web-based English-Japanese parallel corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3603-3609, Marseille, France. European Language Resources Association.
Maria Nadejde and Joel Tetreault. 2019. Personalizing grammatical error correction: Adaptation to proficiency level and L1. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 27-33, Hong Kong, China. Association for Computational Linguistics.
Xing Niu and Marine Carpuat. 2020. Controlling neural machine translation formality with synthetic supervision. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8568-8575. AAAI Press.
Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output.