SlowGuess/NAACL_2022_OCR huggingface dataset:
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2814-2819, Copenhagen, Denmark. Association for Computational Linguistics.
Xing Niu, Sudha Rao, and Marine Carpuat. 2018. Multi-task neural models for translating between styles within and across languages. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1008-1021, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages.
Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074-1084, Valencia, Spain. Association for Computational Linguistics.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
Andrea Schioppa, David Vilar, Artem Sokolov, and Katja Filippova. 2021. Controlling machine translation for multiple attributes with additive interventions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6676-6696, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6490-6500, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 35-40, San Diego, California. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Sara Stymne, Jörg Tiedemann, Christian Hardmeier, and Joakim Nivre. 2013. Statistical machine translation with readability constraints. In Proceedings of the 19th Nordic Conference of Computational Linguistics, NODALIDA 2013, May 22-24, 2013, Oslo University, Norway, volume 85 of Linköping Electronic Conference Proceedings, pages 375-386. Linköping University Electronic Press.
Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. 2017. Controlling target features in neural machine translation via prefix constraints. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages 55-63, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003-3008, Brussels, Belgium. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998-6008.
Aditi Viswanathan, Varden Wang, and Antonina Kononova. 2019. Controlling formality and style of machine translation output using AutoML. In SIM-Big, volume 1070 of Communications in Computer and Information Science, pages 306-313. Springer.
Yue Wang, Cuong Hoang, and Marcello Federico. 2021. Towards modeling the style of translators in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1193-1199, Online. Association for Computational Linguistics.
Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. 2016. Controlling the voice of a sentence in Japanese-to-English neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 203–210, Osaka, Japan. The COLING 2016 Organizing Committee.
A Formality Evaluation
Manual Analysis We give examples of system outputs labeled as Neutral or Other in Table 14.
We report additional results for increasing the upsampling factor (up to 8x) for EN-JA in Figure 2. On this larger labeled dataset, a higher up-sampling factor can improve accuracy up to  $94\%$  across domains, significantly increasing the out-of-domain (Call Center) accuracy while generic quality remains stable. The up-sampling factor can be tuned further for each language to achieve the optimal trade-off between accuracy and generic quality.
EN	Wow, that's awesome! Who is your favorite Baseball team? I like my Az team lol
JA	ウ王者荣耀！おelnの好きな野球チムは誰ですか？おelnのAZチムは好感eel笑
Label	OTHER: "す北京冬奥" (informal) - "てす" and "你好てす" (formal).
EN	You know what I'm saying. You want them to teach you something new.
DE	Du weiß, was ich meine. Sie möchten, dass sie Ihnen etwas Neues beibringen.
Label	OTHER: "Duweisst"(informal) - "Siemöhlen" and "Ihnen" (formal)
EN	So I will need an early check-in and if you have a airport shuttle, that will be great.
IT	Quindi avrò bisogno di un check-in anticipato e se si dispone di una navetta aeroportuale, sarà fantasticico.
Label	NEUTRAL: "si dispone" (impersonal)
Table 14: System outputs labeled as "Other" or "Neutral".
Figure 2: Accuracy on the CoCoA-MT test sets and generic quality scores (BLEU) for EN-JA for an increasing amount of the labeled data (through up-sampling up to  $8\mathrm{x}$ ). The generic baseline scores correspond to 0 on the x-axis. Each source sentence in the CoCoA-MT dataset corresponds to two data points - one for each formality level. For computing the BLEU scores we translate the generic tes tset(IWSLT) without controlling formality.
All training and development data was tokenized using the Sacremoses tokenizer.[24] Words were segmented using BPE (Sennrich et al., 2016b) with 32K operations. Source and target subwords shared the same vocabulary. Training segments longer than 95 tokens were removed.
The source embeddings, target embeddings, and the output layer's weight matrix are tied (Press and Wolf, 2017). Training is done on 8 GPUs with Sockeye 2's large batch training. It has an effective batch size of 327,680 tokens, a learning rate of 0.00113 with 2000 warmup steps and a reduce rate of 0.9, a checkpoint interval of 125 steps, and learning rate reduction after 8 checkpoints without improvement. After an extended plateau of 60 checkpoints, the 8 checkpoints with the lowest validation perplexity are averaged to produce the final model parameters.
Fine-tuning is done on 4 GPUs with an effective batch size of 8,192 tokens, a learning rate of 0.0002, and only one epoch, as per Hasler et al. (2021).
Parameters for standard training:
Parameters for fine-tuning:
D Instructions for Creating Formality-Specific References
In this section, we reproduce the instructions given to the translators for DE, ES, FR, HI, and IT. Instructions for JA are similar but include some language-specific notes. We make minor edits for anonymity purposes. For brevity, we also remove example translations show to the translators.
Overview This project is to create a test set that content consists of short conversations or utterances taken from conversations. Many segments are taken out of context, but all of them are utterances said during a conversation. Sometimes you will understand the relationship between the speakers from the context, and sometimes you will not.
With your translations, we are creating a very specific test set. We will use it to test the capability of an MT engine to produce a translation with the required formality of both speakers. In other words, imagine if we could ask an MT engine: now translate this conversation as if it is between two speakers, where the relationship between them is formal. To test how well it can do that, we will be using your translations (the golden set).
You will receive a source file that will consist of utterances that were initially part of a conversation; some segments will appear with the surrounding context utterances, and some will be taken out of
the conversation. Each segment might consist of several sentences.
Terminology Formality marker: a (form of the) word(s) that indicates the tone of that utterance or relationship between speaker and addressee. Even if you take this word(s) out of context, by looking at it you will clearly know the tone/formality level of the conversation in which this word is used. For example,
- English: "you" (2nd person pronoun) has no formality marker in English (meaning, by looking at the word you cannot tell if the tone of addressing them is formal or informal).
- German: has formality markers in the 2nd person pronoun and corresponding verb forms - "du" (informal) versus "Sie" (formal). This means, just by looking at the pronoun "du" or the verb next to it, I know the tone is informal. So, I will mark "du bist" in DE with Formality tags.
- Spanish: has formality markers for the second person pronouns and their verb conjugations - "tu" (informal) and "usted" (formal). Since Spanish is a pro-drop language, verb conjugations may be the only indicator of this information.
Italian: similarly to Spanish, Italian has formality markers for the second person pronouns and their verb conjugations - "tu" (informal) and "lei" (formal). Since Italian is a pro-drop language, verb conjugations may be the only indicator of this information.
- French: similarly to Spanish and Italian, French has formality markers for the second person pronouns and their verb conjugations - "tu" (informal) and "vous" (formal). French, however, is NOT a pro-drop language.
- Hindi: There are Formality markers for 2nd person in Hindi (meaning, I can address someone respectfully or in a casual way by changing the pronoun). In this case, I will mark the pronoun in Hindi with Formality tags.
For Japanese, translators were additionally provided with examples of formality levels (Table 15) and formality markers (Table 16).
Tags We are interested in finding marker words for formality.
Formality tag: [F]X[/F] NOTE: in Spanish, German, French, Italian and Hindi, Formality is not expressed in the 1st person.
Task There will be two iterations of translating the same source file. Each iteration will be for translating segments into a certain formality. Iteration One will be translated and tagged by Translator 1. Iteration Two will be translated and tagged by Translator 2 (using the Translation Memory from Iteration One).
1. Iteration One - Step One. Will be done by Translator 1. Translate the segment into the suggested formality.
2. Iteration One - Step Two. As you are translating it, think of which words are getting translated with formality markers into your language (if any). Tag them in the translation.
(a) (!) You need to first get the translation, then tag the target words that change because of formality - in this order!
(b) It should be the words that have NO formality markers in English - but WILL have formality markers when translated into the target language. These are ambiguous source words that only acquire markers in the target language.
(c) Do not confuse the TONE/STYLE of the utterance overall with the presence of the formality marker words. "Yo, dang, it's Sunday already!" sounds informal overall, but there are no marker words for 1st or 2nd person in Es, De or Hi, so we should NOT tag anything for formality.
(d) If there are no markers in the translation, do not add any tags, just translate it.
(e) NOTE! If there are ways to translate a sentence with or without markers (for instance using a passive voice), please try to create a natural translation. Do NOT force usage of markers if it creates unnatural translations.
(f) ! TIP: to determine if a word is ambiguous or a marker, take it out of context and see if you can still determine formality.
Casual speech (常語jougo)	Polite speech (敬語：keigo)
Jougo / Kudaketa (Informal)	Teineigo (Formal)	Kenjougo (Humble)	Sonkeigo (Honorific)
Subject for verb	I and you/others	I and you/others	I	You
eat	食ごる	食ごます	頂く	召上かる
come	来る	来ます	参上する	いらっしやる
Table 15: Formality levels in Japanese and examples of changes in inflection or lexical choice for a main verb.
Jougo / Kudaketa (Informal)	Teineigo (Formal)	Kenjougo (Humble)	Sonkeigo (Honorific)
1st PersonI'll wait here.	ごこて[F]待て[/F].koko de [F]matsu[/F]	ごこて[F]待まます[/F].koko de [F]machimasu[/F]	N/AN/A	ごこて[F]お待ちります[/F].koko de [F]o-machi shimasu[/F]
2nd PersonWhat did you buy?	何を[F]買った[/F]?nani o [F]katta[/F]?	何を[F]買んだた[/F]か?nani o [F]kaimashita[/F] ka?	何を[F]お買のはうしおた[/F]か?nani o [F]o-kai ni narimashita[/F] ka?	N/AN/A
3rd PersonThe dog chased the cat.	犬は猫を[F]追った[/F].inu wa neko o [F]otta[/F]	犬は猫を[F]追いまuting[/F].inu wa neko o [F]oimashita[/F]	N/AN/A	N/AN/A
Table 16: Examples of labeled formality markers in Japanese.
3. Iteration 2 (if applicable). This will done by Translator 2.
(a) You will only need to translate and annotate the segments that were tagged with at least one marker during Iteration One. Treat the rest of them as context, where applicable.
(b) One iteration equals one formality level: you will be translating a whole file into one iteration at a time to minimize possible confusion.
(c) Leveraging the Translation Memory from Iteration one, translate the source text into the suggested Formality.
- For Spanish and German, please try to only change the markers in the translation to the requested Formality combination ( we expect mostly pronouns and verb inflections) and preserve the rest of the translation, if possible. If you disagree with the provided translation or tagging, please raise this to your project manager!
- For Hindi, you may have to introduce some additional changes to a sentence when changing its formality levels (choice of words, etc.). Please do that as needed in order to provide a natural translation, but try to be faithful to the source as much as possible.