Controlling Styles in Neural Machine Translation with Activation Prompt
Year: 2023

Abstract:
Controlling styles in neural machine translation (NMT) has attracted wide attention, as it is crucial for enhancing user experience. Earlier studies on this topic typically concentrate on regulating the level of formality and achieve some progress in this area. However, they still encounter two major challenges. The first is the difficulty in style evaluation. The style comprises various aspects such as lexis, syntax, and others that provide abundant information. Nevertheless, only formality has been thoroughly investigated. The second challenge involves excessive dependence on incremental adjustments, particularly when new styles are necessary. To address both challenges, this paper presents a new benchmark and approach. A multiway stylized machine translation (MSMT) benchmark is introduced, incorporating diverse categories of styles across four linguistic domains. Then, we propose a method named style activation prompt (StyleAP) by retrieving prompts from stylized monolingual corpus, which does not require extra fine-tuning. Experiments show that StyleAP could effectively control the style of translation and achieve remarkable performance.

heading: 1 Introduction
text: Natural language texts can be written in various styles while preserving the content, such as politeness, formal, classical, and many others (Hovy, 1987; Jing et al., 2020; Fu et al., 2018). Styles are crucial for communication since every sentence should fit a specific scenario and the appropriate style makes it more user-centric. A speaker needs to switch the styles of words to adapt to different conditions. Using the inappropriate style can be impolite or ridiculous in some societies and result in serious cultural conflicts (Nida and Taber, 2021).
∗*Work done while Y. Wang was an intern at ByteDance.
As a cross-lingual generation problem, machine translation performance heavily relies on the appropriate style. Therefore, many commercial translation systems provide multiple style choices, such as Portuguese (European vs. Brazilian) and English (American vs. British) in DeepL1, Korean (Honorific vs. Non-honorific) in Papago2, Chinese (Modern vs. Classical) in Volctrans3.
Recently, controlling style in machine translation has also drawn much attention in the academic community (Yamagishi et al., 2016; Michel and Neubig, 2018; Feely et al., 2019). Formally, stylized machine translation refers to translating the source sentence into different styles with certain attributes while the translation quality remains satisfactory, as the cases showed in Figure 1. Many previous studies have explored the task and gained promising results (Sennrich et al., 2016; Rabinovich et al., 2017; Wang et al., 2021). However, challenge still remains in two aspects.
The first challenge is about the benchmark. The style of natural languages consists of many aspects like word preference and grammar structure. However, the well-studied benchmark datasets mainly focus on the formality and politeness of European languages. Due to this limitation, previous work re-
1https://www.deepl.com 2https://papago.naver.com 3https://translate.volcengine.com
2606
stricts styles to a relative narrow scene. In addition, most of the test sets of the previous work have only one reference rather than multiple stylized references, which hinders the automatic evaluation for different styles. As such, a benchmark involving more diverse styles, multiple stylized references and beyond European languages is greatly needed.
The second challenge is about the iterative training framework. Most related work heavily relies on fine-tuning with new stylized data (Sennrich et al., 2016; Wang et al., 2021). Basically, they collect stylized bilingual texts and append tags before the sentence, then conduct fine-tuning to adapt the model to the given style (Sennrich et al., 2016). However, parallel data in specific styles is pretty sparse and costly to gather. Furthermore, in this way, we have to re-tune the model every time we want to add new styles, which is inconvenient.
Correspondingly, this paper contributes in terms of both benchmark and approach:
For the benchmark, we re-visit this task and push the boundary of styles to a wider range of language phenomena. We propose a dataset MSMT, including four directions with diverse language styles. We collect related public corpus as training sets and provide newly labeled sentences as test sets. Each source sentence has two references in different styles, which is convenient for automatic evaluation. By broadening the category and providing standard datasets, we hope to effectively push the development of this field.
For the approach, we propose style activation prompt (StyleAP), a method to avoid re-tuning time after time. The main idea is to extract one sentence of the target style as a prompt to guide the main sentence translation style. The intuition is straightforward. We assume that once the model has been trained on all kinds of data with various styles, it has the potential to generate any style as far as correctly activated. We can activate the ability by language model since it tends to maintain the sequence consistency (Sun et al., 2022b). And the prompt can be easily retrieved in a specific stylized monolingual corpus. In a word, we can obtain a “plug-and-play” model for any new generation style with mere stylized monolingual data instead of iterative fine-tuning. The experiments show that our approach achieves explicit style transformation while well maintaining the text semantics.

heading: 2 Related Work

heading: 2.1 Style Transfer for Machine Translation
text: Existing studies on style transfer mainly focus upon formality (Feely et al., 2019; Wu et al., 2021). They can be roughly divided into two groups: supervised methods and unsupervised methods. Sennrich et al. (2016) propose side constraints to control politeness and shows that substantial improvements can be made by limiting translation to the required level of politeness. Niu et al. (2017) propose a FormalitySensitive Machine Translation (FSMT) scenario where lexical formality models are used to control the formality level of NMT product. Since the parallel sentences are of unknown formality level, some work focus on the unsupervised way. Niu and Carpuat (2020) introduce Online Style Inference (OSI) to generate labels via a pre-trained FSMT model. Feely et al. (2019) use heuristics to identify honorific verb forms to classify the unlabeled parallel sentences into three groups of different level formality. Wang et al. (2021) propose to use source token, embedding and output bias to control different styles and achieve a remarkable performance. Wu et al. (2020b) propose a machine translation formality corpus. Diverse translation is also related to this work (Sun et al., 2020; Wu et al., 2020a).

heading: 2.2 Adaptive via In-Context Learning
text: Recent work shows that prompting the large language models (LMs) like GPT-3 (Brown et al., 2020) with a few examples in the context can further leverage the inductive bias of LMs to solve different NLP tasks (Wang et al., 2022). This part of work shows the adaptive ability of LMs learned from analogy. Our work is inspired by it, but we work under the iterative training situation where the supervised data is pretty sparse.
As prompts play a vital role in generic in-context learning, recent work propose different prompting strategies. Ben-David et al. (2021); Sun et al. (2022a) select the representative keywords of the field for domain adaptation. Zhu et al. (2022) capture keywords of images as prompts for multimodal translation. Hambardzumyan et al. (2021) put special tokens into the input and use continuous embeddings as prompts and Li and Liang (2021) directly optimize prompts in the continuous space. Besides, there is a research direction focusing on retrieval. These methods use two main representations for generating demonstrations. As for the sparse representations, they focus on a rule-
based score such as Okapi BM25 (Robertson and Zaragoza, 2009) for retrieval. Wang et al. (2022) use this method to improve model performance on four NLP tasks. The dense representations are generated by the pre-trained autoencoder model and have higher recall performance on most NLP tasks such as machine translation (Cai et al., 2021). For the sake of accuracy and storage, we use dense representations for retrieval in this paper.

heading: 3 Task Definition & MSMT: A Multiway Stylized Translation Benchmark
text: Stylized machine translation refers to the translations with certain language characteristics or styles on the basis of ensuring the quality of translation. Based upon the definition, we construct a stylized machine translation benchmark including four language directions. In each language direction, we give the illustration of various styles and provide corresponding training and test sets.
Different from traditional stylized machine translation studies, each group of our test sets is consist of one single source and multiple references in parallel. For example, for English-to-Chinese direction, for each English source sentence, we have two parallel Chinese references: classical style and modern style. In this way, we can automatically evaluate the style transformation by measuring the similarity between the stylized hypothesis and stylized references.
All the data has been publicly released and the detailed number is in Table 1. In this section, we will introduce our benchmark construction.

heading: 3.1 English-to-Chinese Translation
text: There are two common styles for Chinese: Classical and Modern. Classical Chinese originated from thousands of years ago and was used in ancient China. Modern Chinese is the normal Chinese that is commonly used currently.
The former is adopted on especially solemn and elegant occasions while the latter is used in daily life. They vary in many aspects like lexis and syntax so can be regarded as two different styles. In this direction, we aim at translating texts from English to Chinese in both styles. Specific data usage is as follows:
• Basic Parallel Data: Cleaned WMT2021 corpus plus the back translation of the subset of an open source corpus containing classical Chinese and
modern Chinese 4.
• Stylized Monolingual Data: The open source corpus containing classical Chinese and modern Chinese and the Chinese part of WMT2021.
• Development Set: Newstest2019.
• Test Set: English-Classical-Modern triplet parallel data annotated by language experts.

heading: 3.2 Chinese-to-English Translation
text: There are two common styles for English: Early Modern and Modern. Early Modern English in this paper refers to English used in the Renaissance such as Shakespearean plays. Modern English is the normal English used currently.
The former one is mostly seen in Shakespearean play scripts like Hamlet while the latter one is used in the daily life. They vary in many aspects like grammatical constructions such as two second person forms, thou and you. Therefore, they can be regarded as two styles. In this direction, we aim at translating texts from Chinese to English in both styles. Specific data usage is as follows:
• Basic Parallel Data: Cleaned WMT2021 corpus plus the back translation of a crawled corpus: The Complete Works of William Shakespeare 5.
• Stylized Monolingual Data: An open source dataset6 containing early modern and modern English and the English part of WMT2021.
• Development Set: Newstest2019.
• Test Set: Chinese-Early-Modern triplet parallel data annotated by language experts.

heading: 3.3 English-to-Korean Translation
text: There are seven verb paradigms or levels of verbs in Korean, each with its own unique set of verb endings used to denote the formality of a situation. We simplify the classification and roughly divide them into two groups: Honorific and Non-honorific
The former is used to indicate the hierarchical relationship with the addressee such as from the young to the old, from the junior to the senior. The latter is used in daily conversations between friends. They vary in some lexical rules so can be regarded
4https://github.com/NiuTrans/ Classical-Modern
5http://shakespeare.mit.edu/ 6https://github.com/harsh19/ Shakespearizing-Modern-English
as two styles. In this direction, we aim at translating texts from English to Korean in both styles. Specific data usage is as follows:
• Basic Parallel Data: IWSLT2017 7 plus the back translation of an open source dataset 8 containing honorific and non-honorific.
• Stylized Monolingual Data: The open source dataset and the crawled corpus from a public translation tool9.
• Development Set: IWSLT17.
• Test Set: English-Honorific-Non-honorific triplet parallel data annotated by language experts language experts.

heading: 3.4 English-to-Portuguese Translation
text: There are two common styles for Portuguese: European and Brazilian. European Portuguese is mostly used in Portugal. Brazilian Portuguese is mostly used in Brazil.
They vary in some detailed aspects like pronunciation, grammar and spelling, so can be regarded as two different styles. In this direction, we aim at translating texts from English to Portuguese in both styles. Specific data usage is as follows:
• Basic Parallel Data: IWSLT2017.
• Stylized Monolingual Data: European & Brazilian part of the parallel data.
• Development Set: IWSLT17.
• Test Set: English-European-Brazilian triplet parallel data annotated by language experts.

heading: 3.5 Evaluation
text: Previous style evaluation relies on human resources, which is costly and slow. Since our test sets are all multiway, we can evaluate our stylized hypothesis with the corresponding reference to take both quality and style into consideration at a small
7https://wit3.fbk.eu/ 8https://github.com/ezez-refer/
Korean-Honorific-Translation 9https://papago.naver.com/
cost. Moreover, human evaluation is inevitably subjective while our test sets can guarantee the comparison stability.

heading: 4 Style Activation Prompt
text: Prior work usually uses the fixed tag to control the generation with expected attributes (Sennrich et al., 2016). However, tag-based methods rely on a large amount of labeled parallel data, requiring relabeling and retraining of models when new styles need to be generated.
We go back to a standard NMT model. During the generation process, the NMT model tries to maximize the probability of the generation sentence. When predicting the i-th token, the model searches from the vocabulary to maximize:
P (yi|x, yj<i) where yj<i means the past words, indicating that the previous inference results can affect the subsequent generation. Therefore, our intuition is to control the generation style by taking advantage of stylized language model. We suggest that once the basic model has been trained on the data in various kinds of styles, we can activate the ability with the contextual influence.
Specifically, we retrieve an instance as a prompt from the stylized corpus and use it to instruct the NMT model to generate the sentence with the same attributes. To adapt to the prompt training, we extract every sentence in the basic parallel data and retrieve one similar sentence as the prompt. The whole framework is in Figure 2. We introduce the details of our proposed method as follows.

heading: 4.1 Prompt Retrieval
text: The prompt retrieval procedure aims at finding the proper sentence prompt. First, we construct a candidate datastore D that contains many (r, y) keyvalue pairs, where r is the representation of y. In this paper, we use a multilingual pre-trained language model XLM-R (Conneau et al., 2019) to obtain the sentence representation. By calculating
the similarity between the query representation h and keys, we can extract the needed sentence y:
y = argmin r∈D Distance(h, r)
where the search tool is Faiss(Johnson et al., 2021), a library for efficient similarity search and clustering of dense vectors.

heading: 4.2 Training
text: In the training stage, the goal is to retrieve a similar prompt with the current sentence to make the model adapt to the inference pattern. Specifically, we iterate each target-side sentence in the basic parallel data as a query and retrieve the most similar sentence as its prompt.
After obtaining the prompt, we concatenate the prompt and the query sentence by a special token as:
prompt, [s], src → prompt, [s], trg
We train the model with this kind of data and normal data together to learn the prompt-based generation as well as basic translation.

heading: 4.3 Inference
text: In the inference stage, we first translate the source sentence roughly. Then the draft hypothesis is used as the query. The candidate datastore is constructed with the monolingual data in the given style. After retrieving the prompt, we append it to the beginning of the source sentence with the special token. After the second inference, the hypothesis can be obtained by splitting the token.

heading: 4.4 Advantages
text: We conclude the advantages of StyleAP as follows:
• StyleAP does not need any architecture modification and is easy to deploy.
• StyleAP does not need to assign various tags to all kinds of styles.
• Extra tuning is no longer needed when it comes to a new style. We only need to retrieve the prompt from the new monolingual stylized corpus and then generate the given style.

heading: 5 Experiments
text: In this section, we will introduce the details of our experiments.

heading: 5.1 Setup
text: We first compare our method with other baseline models on four tasks. Then, we design a manual evaluation to assess whether our method maintains translation quality and achieves diversity. All experiments are implemented in the following settings.

heading: 5.1.1 Data & Preprocessing
text: In the previous section, we introduce our provided stylized NMT benchmark MSMT. Our designed experiments and analysis are based upon this benchmark. The statistics information of this benchmark is shown in the Table 1.
We use SentencePiece(Kudo and Richardson, 2018) to jointly learn an unsupervised tokenizer. We preprocess the training data and filter the parallel sentences with length greater than 256. We set
en→ zh zh→en en→ko en→pt
hyper-parameters min frequency 5 and max vocabulary 32k.

heading: 5.1.2 Implementation Details
text: Here, we introduce more details of our experiment settings. Our experiment is implemented on the open source Seq2Seq tool Neurst10 (Zhao et al., 2021). Our seq2seq model uses a transformer-base structure with 6 encoder layers and 6 decoder layers, attention with a layer size of 512, and word representations of size 512. We apply post-layer normalization (Ba et al., 2016), adding dropout to embeddings and attention layers with a dropout rate of 0.1. We tie the source and target embeddings. The main training parameters are as follows. We use Adam Optimizer with β1 = 0.9 and β2 = 0.98. We use label smoothed cross entropy as a criterion with a label smoothing rate of 0.1. We set batch size per GPU 4096 and batch by tokens. And, we use four A100 GPUs to train our model from scratch. We save checkpoints every 1000 steps and stop training when there is no improvement in the continuous 50 checkpoints.

heading: 5.1.3 Comparing Systems
text: We use sacreBLEU (Post, 2018) as our metrics and compare StyleAP with three common systems:
• Baseline: Transformer that is trained on the raw parallel data.
• Transfer: A two-phases processing: Translate first and conduct style transfer (Syed et al., 2020). We train the translation model with normal parallel data and train the transfer model with stylized data.
• Tag-tuning: A tag-based model which is generally used in other work (Sennrich et al., 2016). They add a special token as the tag at the start of the source text with the known style. In this way, the model can generate different styles with different tokens. This method needs explicit extra fine-tuning.
10https://github.com/bytedance/neurst

heading: 5.2 Results
text: As is shown in Table 2, we calculate the BLEU score on the test set to compare StyleAP with the mentioned baselines. The Transfer methods have many drawbacks. Not only does it need two-phases training, but also yields poor results. The attempt in English-to-Korean direction even fails. Tag-tuning gains some improvements and even achieves the best performance in some directions. But overall, StyleAP obtains the best results and outperform the other methods. At the same time, StyleAP needs no extra tag or extra tuning when it comes to new styles. After acquiring the ability to translate with style prompt, StyleAP can handle various styles.

heading: 5.3 Human Evaluation
text: We also design a human evaluation to manually check the style transfer ratio as well as the translation quality preservation during the transfer. The quality score ranges from 0 to 4. The style transfer ratio means the percentage of the hypothesis that meets the required style, ranging from 0 to 100. Refer to the appendix for the specific scoring criteria and rules.

heading: 5.3.1 Quality Preservation
text: The quality results are in Table 3. StyleAP achieves a comparable performance with the baseline model, which means little semantic loss within the stylized translation.

heading: 5.3.2 Style Transfer Ratio
text: As is shown in Figure 3, StyleAP significantly enhances the transfer ratio. The only unsatisfactory is the Chinese-to-English Translation in the Early
Modern style. The reason is that many sentences in the test set are very short like “What about that?” vs “What of that?”. The styles for this kind of extremely short sentences are not meaningful.

heading: 5.3.3 Conclusion
text: The quality scores are comparable with the baseline model, while the transferred ratios are much higher than the baseline model in the four tasks. That indicates that our method can effectively translate the source text into a sentence with specific attributes without quality loss.

heading: 6 Analysis

heading: 6.1 Retrieval Strategy Matters
text: There are many retrieval methods to select a similar sentence from stylized monolingual sentences. We conduct a detailed comparison in English-toChinese inference with the following strategies:
• Source: Directly use the source text representation generated from the pre-trained multilingual language model.
• Random: Randomly choose a prompt from candidates.
• Fixed: Use the same prompt for all samples.
The results are shown in Table 4. We can see that the our strategy performs the best and the other retrieval methods face different levels of the BLEU loss. The retrieval strategy plays an important role in the translation.

heading: 6.2 Even Unsupervised Prompts Works
text: In the training phase, we assume that the retrieval range lies within the specific styles. However, one condition that is more close to the real world is that we need to retrieve prompts from more general data, which may cause the style mismatch between the sentence and the prompt. Therefore, we also conduct a unsupervised prompt retrieval in training for English-to-Chinese direction.
As is shown in Table 5, the unsupervised version of StyleAP slightly drops in terms of BLEU but still outperforms Tag-tuning. It is worth mentioning that we have none of the style label of parallel data in this setting. General parallel data and monolingual stylized data is all you need. This again shows the universality and robustness of StyleAP.

heading: 6.3 Consistent Performance across Sizes
text: We are also interested in the situation of unbalanced data and even few stylized data. We implement a comparative experiment of stylized monolingual data on the en→zh task. We control the amount of labeled data at four levels: 1 million, 100 thousand, 10 thousand, and 1 thousand. For the tag-based method, we train the tag-based model from scratch at each level. For a fair comparison, we use the same stylized labeled data as the tag-based method but only as the target monolingual part for retrieval.
The results are shown in Figure 4 , where the horizontal axis represents the sample size, and the vertical axis represents the BLEU score. For the classical direction, our method performs better in all situations. Even when we only use 1,000 labeled stylized monolingual sentences, there is still an
improvement compared with the baseline model. On the contrary, the tag-based method performs poorly in few data and even has a lower BLEU score than the baseline model.
In conclusion, our method has an overall better performance than the tag-based method at different levels. Especially for extremely few samples, our method still gains significant improvements.

heading: 6.4 Attention Score Interprets the Effect
text: We are also interested in how the retrieved prompt affects the translation style. Here is an example of Chinese-to-English task in Figure 5. The model is translating a Chinese sentence meaning “Yeah, you have given me great comfort.” into English and the next token is “thou”, which means “you” in early modern English.
We show the average attention scores in the Transformer decoder, left for self-attention and right for cross-attention. For self-attention, except for some adjacent tokens, the model mainly pays attention to the token “thou” in the prompt which corresponds to the generating token. For
the cross-attention, the model concentrates on the corresponding Chinese token “Ni” (means “you” in Chinese) and the token “thou” in the prompt again.
This result suggests that our retrieval prompt could affect the generation process through the attention mechanism.

heading: 6.5 Case Study
text: Finally, one stylized Chinese-to-English translation example is listed in Table 6 to show the effectiveness of our method more intuitively. As in these examples, the Chinese word “Ni” (“you” in English) is translated into “you” in Baseline and the Tag-tuning method. However, under the guidance of the prompt which uses the early modern English word “thou”, StyleAP translates the word into “thou” correspondingly. Obviously, StyleAP can explicitly affect the translation style with prompts.

heading: 7 Conclusions
text: In cross-lingual generation fields, most studies focus on the translation quality but ignore the style issue, which happens to be important in the realworld communication. However, the previous studies face two major challenges including the bench-
mark as well as the approach. For those purposes, we re-visit this task and propose a standard stylized NMT benchmark MSMT with four well-defined tasks to push the boundary of this field. We also propose a new translation style controlling method with activation prompt. With stylized prompts that are retrieved from the stylized monolingual corpus, we successfully guide the translation generation style without iterative fine-tuning. Through automatic evaluation and human evaluation, our method achieves a remarkable improvement over baselines and other methods. A series of analysis also show the advantages of our method.
Limitation
One limitation of StyleAP is that one extra inference is needed for retrieval. It is mainly due to the monolingual retrieval accuracy is higher than that of crosslingual retrieval (refer to Section 6.1). In the future, we will try stronger multilingual model to mitigate this effect.

heading: Acknowledgments
text: We would like to thank the anonymous reviewers for their constructive comments. Weiguo Zheng is the corresponding author.

heading: A Appendix
text: In this section, we supplement human evaluation criterion in this paper and more stylized translation cases. Table 7 fully illustrates the score standard of our language experts. Table 8 shows more English examples of stylized translation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3,4,5
3 B1. Did you cite the creators of artifacts you used? 3,4,5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3,4,5

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3,4,5

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 3,4,5

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 3,4,5
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3,4,5
C 3 Did you run computational experiments? 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5,6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 5

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? A
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will try stronger multilingual model to mitigate this effect