Low-Resource Formality Controlled NMT Using Pre-trained LM
Year: 2023

Abstract:
This paper describes the UCSC’s submission to the shared task on formality control for spoken language translation at IWSLT 2023. For this task, we explored the use of “additive style intervention” using a pre-trained multilingual translation model, namely mBART. Compared to prior approaches where a single style-vector was added to all tokens in the encoder output, we explored an alternative approach in which we learn a unique style-vector for each input token. We believe this approach, which we call “style embedding intervention,” is better suited for formality control as it can potentially learn which specific input tokens to modify during decoding. While the proposed approach obtained similar performance to “additive style intervention” for the supervised English-toVietnamese task, it performed significantly better for English-to-Korean, in which it achieved an average matched accuracy of 90.6 compared to 85.2 for the baseline. When we constrained the model further to only perform style intervention on the <bos> (beginning of sentence) token, the average matched accuracy improved further to 92.0, indicating that the model could learn to control the formality of the translation output based solely on the embedding of the <bos> token.

heading: 1 Introduction
text: In the past decade, neural machine translation has made remarkable strides, achieving translation quality that is increasingly comparable to human-level performance across various languages. However, despite these advancements, the field of controllable machine translation remains relatively under-explored. One crucial aspect of translation variation is formality, which manifests through grammatical registers, adapting the language to suit specific target audiences. Unfortunately, current neural machine translation (NMT) systems lack the capability to comprehend and adhere to grammatical registers, specifically concerning formality.
Consequently, this limitation can result in inaccuracies in selecting the appropriate level of formality, potentially leading to translations that may be deemed inappropriate in specific contexts. Recognizing the significance of formality control, we aim to build a formality-controlled machine translation system to foster smooth and reliable conversations and enhance communication across languages and cultures, facilitating more nuanced and effective linguistic exchanges.
Formality-controlled Neural Machine Translation is the IWSLT 2023 task (Nădejde et al., 2022) under the Formality track. The goal of the task is to achieve formality controlled machine translation for the English-Vietnamese (En-Vi), EnglishKorean (En-Ko) in a supervised setting and EnglishPortuguese (En-Pt) and English-Russian (En-Ru) in a zero-shot setting as detailed in (Agarwal et al., 2023). We provide an example of formal and informal translations of an English sentence into Vietnamese in Figure 1. The formal and informal tokens are in bold.

heading: 2 Related Works
text: Machine translation (MT) research has primarily focused on preserving the meaning between languages. However, it is widely recognized that maintaining the intended level of formality in communication is a crucial aspect of the problem (Hovy, 1987) (Hovy, 1987). This field of research was named formality-sensitive machine translation (FSMT) (Niu et al., 2017), where the target formality level is considered in addition to the source segment in determining the translated text. Further, several studies have attempted to regulate formality in MT through side constraints to control politeness, or formality (Sennrich et al., 2016); (Feely et al., 2019); (Schioppa et al., 2021a). Other studies have tried to address this with custom models trained on data with consistent formality (Viswanathan et al., 2020). Most prior research
321
has been tailored to individual languages and has labeled large amounts of data using word lists or morphological analyzers.

heading: 3 Approach

heading: 3.1 Overview
text: The task of formality-controlled generation can be viewed as a seq2seq machine translation task. More formally, given an input sequence x, we design a model that does the following:
ŷ = argmax y∈Y
p(y|x, ls, lt, f ; θ) (1)
Where, x is the input sequence, ls is the source language, lt is the target language, f is the formality, ŷ is the formality controlled translation
We propose a single model that produces an output, given input x, and formality setting f. Despite being part of the unconstrained task, our proposed approach does not mine or develop any formality annotated data for training and just uses a pretrained checkpoint of mBART.

heading: 3.2 Design
text: We looked at previous works incorporating contrasting styles Rippeth et al., 2022, and Schioppa et al., 2021b as motivation for our approach. For controlling styles, the aforementioned works use an additive intervention approach. This approach entails adding a single style intervention vector V to the pre-trained encoder output Z. The same vector V is added to all the tokens of the encoder outputs, thereby changing the encoder outputs uniformly.
We modify the above approach to allow for more flexibility while learning. Instead of a single intervention vector V, we propose a unique vector Vi
for every token i in the input space. In short, we repurpose an Embedding layer as a style intervening layer between the encoder and the decoder. This design resulted from our original question: will allowing more flexibility in the encoder enable it to identify which tokens require stylization, thus making it more interpretable. The hypothesis that originated from this question was: by giving each token its own intervention vector Vi, the model will learn each intervention vector Vi differently based on whether the token at that time step has a contrasting translation that is dependent on the formality setting. In short, we let the model learn different Vi’s for each token. If true, this will provide some interpretability on which tokens the model recognizes as having a formality marker and translates them differently in formal and informal settings. This approach is visualized in Figure 2. Since our approach uses an embedding layer for style intervention, we call our approach ’style embedding intervention.’
We learn the style embedding layer only in the formal setting and use a zero vector in the informal setting. In other words, the style embedding intervention is performed only in the formal setting, and encoder outputs are not perturbed in the informal setting. We do not have separate Embedding layers to learn each formality style, simply because, it would be difficult to switch between layers during batched training. Looking at (Schioppa et al., 2021b), the combination of a style vector and a zero vector for contrasting styles was sufficient to learn the style.

heading: 4 Experimental Apparatus

heading: 4.1 Dataset
text: The IWSLT formality shared task provided a formality annotated dataset (Nadejde et al., 2022). This dataset comprises source segments paired with two contrastive reference translations, one for each
formality level (informal and formal) for two language pairs: EN-KO, VI in the supervised setting and two language pairs: EN-PT, RU in the zeroshot setting. The data statistics can be seen in Table 1. We use a random split of 0.2 to construct the validation dataset during model development.

heading: 4.2 Training Setup
text: For all our modeling experiments, we use mbartlarge-50-one-to-many-mmt, a fine-tuned checkpoint of mBART-large-50 (Liu et al., 2020). This model, introduced by (Tang et al., 2020), is a finetuned mBART model which can translate English to 49 languages, including the languages we are interested in: KO, VI, PT, and RU.
For our baseline, we perform zero-shot inference on the mBART model for the four language pairs. The results are shown in tables 3 - 6.
Based on the findings of (Nakkiran et al., 2019) and (Galke and Scherp, 2022) we fixed our loss function to be ‘cross entropy with logits‘ and optimizer to AdamW (Loshchilov and Hutter, 2017). We use the default learning rate of 10-3, standard weight decay of 10-2 and set β1, β2 and ϵ to 0.9, 0.998 and 10-8 respectively.
To effectively train the transformer-based mBART model, we used a learning rate scheduler - a linear schedule with a warm-up, as introduced by (Vaswani et al., 2017). This creates a schedule with a learning rate that decreases linearly from the initial learning rate to 0 after a warm-up period. The warm-up period is set to 10% of the total training steps, during which the learning rate increases linearly from 0 to the initial learning rate set in the optimizer. All the other hyper-parameters are left at their defaults.
We trained our models using one NVIDIA A100 GPU with 80GB memory. To fit our model in this GPU we used a batch size of 16 and a max sequence
length of 128. We trained for 15 epochs with an early stopping callback set at 3.
We have implemented all the models in PyTorch (Paszke et al., 2019) leveraging Huggingface (Wolf et al., 2019) transformers and evaluate libraries.

heading: 4.3 Evaluation
text: To assess the performance of the models, we use four metrics to evaluate the two main underlying tasks - translation quality and formality control.
For evaluating the translation quality, we use the following two metrics:
• Bilingual Understudy Evaluation (BLEU) score: BLEU score (Papineni et al., 2002) calculates the similarity between a machine translation output and a reference translation using n-gram precision. We use SacreBLEU 2.0 (Post, 2018) implementation for reporting our scores.
• Cross-lingual Optimized Metric for Evaluation of Translation (COMET) score: COMET score (Rei et al., 2020) calculates the similarity between a machine translation output and a reference translation using token or sentence embeddings. We use COMET wmt22-comet-da (Rei et al., 2022) model for reporting our scores.
For evaluating the formality control, we use the following two metrics:
• Matched-Accuracy (M-Acc): A referencebased corpus-level automatic metric that leverages phrase-level formality markers from the references to classify a system-generated translation as either formal or informal. This metric was provided by the IWSLT Formality shared task organizers.
• Reference-free Matched-Accuracy (RF-MAcc): A reference-free variant of M-Acc that uses a multilingual formality classifier, based on xlm-roberta-base, fine-tuned on humanwritten formal and informal text, to label a system-generated hypothesis as formal or informal. This metric was provided by the IWSLT Formality shared task organizers.
In addition to this, we evaluate our generic translation quality on FLORES-200 (Goyal et al., 2022) for all language pairs under supervised and zeroshot settings. We use the devtest set of FLORES200 and compute the BLEU and COMET scores.

heading: 5 Grounding results and observations
text: Along with the validation splits, we ground our approach by comparing our results with the 2022 formality track submission Rippeth et al., 2022. We compare our results on one language pair i.e. English-Spanish. The comparison is shown in Table 2.
As seen in Table 2, the BLEU scores between our approach - “style embedding intervention” - and the approach in Rippeth et al., 2022 - "additive style intervention" - are similar but our approach makes significant gains in Matched Accuracy, especially in the informal setting indicating improved formality control.

heading: 5.1 Style embedding layer analysis
text: In this section, we analyze the style embedding layer and compare the analysis with the original hypothesis - giving each token its own intervention vector Vi, the model will learn each vector differently based on whether the token at that time step has a contrasting translation that is dependent on the formality setting. Due to the unique nature of our training setup - learning zero vector in the informal setting - for our hypothesis testing, we compare the encoder vectors with and without the style embedding intervention. For this purpose, we use the dot product similarity. At each time step, we compute the dot product similarity between the encoder output before style intervention and the output after style intervention. This is equivalent to comparing the encoder outputs in the formal and
the informal setting. The similarity scores are visualized in Figure 3. For a closer look, Table 8 displays the similarity scores.
As seen from the token representation similarity scores, the model does not seem to learn new information in tokens that have a contrasting settingdependent translation - the tokens’ similarity scores are very near 1. Instead, it uses the 
’s representation to store the style ’signal’, by creating a style vector that makes the 
’s representation ∼11% different between formality settings.
Another interesting observation is the extremely slight dissimilarity produced at the beginning of the sentence or ’en_xx’ token. Did the model learn the same style information in ∼1% of information space in the ’en_xx’ token compared to the ∼11% of information space in the ’
’ token? To an-
swer this question, we added another modification to our approach - we masked out the intervention vectors for all tokens except the ’en_xx’ token.
For naming purposes, we call this approach ’bos style intervention’ respectively.

heading: 6 Official Results
text: Along with the approach from Rippeth et al., 2022 taken as a baseline and an adapted version of it, we submit the results of our approach and of the ’bos style intervention’ approach. We analyse the performance of our models under the supervised setting and the zero-shot setting. We also generate results on the FLORES-200 test split.

heading: 6.1 Supervised Setting
text: We trained our models multi-lingually on EN-VI and EN-KO for the supervised setting. In the for-
mal setting, we obtain a BLEU score of 44.6 for EN-VI and 23.3 for EN-KO on the official test split. In the informal setting, we obtain a BLEU score of 43.5 for EN-VI and 22.8 for EN-KO. Tables 3 and 5 have detailed results of all our models. Our primary model - ’bos style intervention’ - outperforms the UMD baseline significantly for both languages with around 20 BLEU increase and more than double the COMET score. This answers our hypothesis that the model can learn the formality style in the small ∼1% information space at the beginning of the sentence in ’en_xx’ token. Moreover, we obtain higher scores on the metrics M-Acc% & C-F% that compute the degree of formality/informality induced.
Qualitative analysis of the translations, especially for KO, revealed that code-switching was a major issue. For example, some translations have
entire phrases or latter parts of sentences in English as shown in Figure 4.

heading: 6.2 Zero-shot Setting
text: We evaluate the above multi-lingually trained model on RU and PT in a zero-shot setting. In the formal setting, we obtain a BLEU score of 26.6 for
EN-PT and 18.4 for EN-RU on the official test split. In the informal setting, we obtain a BLEU score of 28.4 for EN-PT and 14.9 for EN-RU. Tables 4 and 6 have detailed results of all our models. We observe that our model does not transfer the style knowledge very well. In both cases, the model is often biased toward formal translations. Moreover, our models have a slightly degraded performance in the translation quality than UMD baseline model. This cements our earlier observation that style knowledge transfer is incomplete. Qualitative analysis of the translations revealed that the zero-shot language translations also suffer from code-switching.

heading: 6.3 Testing on FLORES-200 dataset
text: In addition to evaluating formality, we assess the translation quality of our models by evaluating on the FLORES-200 test split. The results can be seen in Table 7.

heading: 7 Conclusion
text: In this paper, we presented and explored "style embedding intervention," a new approach for lowresource formality control in spoken language translation. By assigning unique style vectors to each input token, the proposed approach shows promising results in understanding and controlling the nuances of formal and informal style translation. It outperforms previous "additive style intervention" methods, specifically for the English-
to-Korean translation task, resulting in an average matched accuracy improvement from 85.2 to 90.6. Further, on analysis of our "style embedding intervention" model, we find that most of the style information is learnt in the <bos> token. Constraining style addition to the <bos> token - "bos style intervention" - further improved our averaged matched accuracy from 90.6 to 92.
We also observed that in a zero-shot setting, the formality control doesn’t seem to transfer well, and the model leans towards biases learnt during pretraining rather than the transferred style interventions. This is more pronounced for En-Ru translations where the model is more biased towards the formal style, with a matched accuracy of 98.8, than the informal style, with a matched accuracy of 51.6.
Future works focused on alleviating the style biases of pre-trained models might be necessary to ensure style transfer works equally well in a zero-shot setting.
We hope our work on translation models with interpretable formality control can serve as a base for other future works on interpretable models, especially in low-resource settings.
Code used for our implementation can be accessed at https://github.com/Priyesh1202/ IWSTL-2023-Formality.

heading: 8 Acknowledgements
text: We thank Prof. Lane, Prof. Rao and Brendan King from UC Santa Cruz for their constant guidance and support.

heading: Future Work
text: Future works focused on alleviating the style biases of pre-trained models might be necessary to ensure style transfer works equally well in a zero-shot setting. We hope our work on translation models with interpretable formality control can serve as a base for other future works on interpretable models, especially in low-resource settings. Code used for our implementation can be accessed at https://github. com/Priyesh1202/ IWSTL-2023-Formality