Improving Formality-Sensitive Machine Translation using Data-Centric Approaches and Prompt Engineering
Year: 2023

Abstract:
In this paper, we present the KU x Upstage team’s submission for the Special Task on Formality Control on Spoken Language Translation, which involves translating English into four languages with diverse grammatical formality markers. Our methodology comprises two primary components: 1) a language-specific data-driven approach, and 2) the generation of synthetic data through the employment of large-scale language models and empirically-grounded prompt engineering. By adapting methodologies and models to accommodate the unique linguistic properties of each language, we observe a notable enhancement in performance relative to the baseline, substantiating the heightened efficacy of data-driven approaches. Moreover, our devised prompt engineering strategy yields superior synthetic translation instances.

heading: 1 Introduction
text: Neural machine translation (NMT) models have achieved remarkable progress in recent years, as evidenced by their high BLEU scores (Britz et al., 2017; Stahlberg, 2020). Nonetheless, these models generally rely on generic parallel corpora and assume a single target translation for a given source sentence, often overlooking the significance of style and pragmatic aspects in translation, such as formality or politeness (Li et al., 2022). To address this issue, formality-sensitive machine translation (FSMT) has emerged as a research area, aiming to control grammatical formality in translated text across languages (Niu et al., 2017).
The Special Task on Formality Control on Spoken Language Translation introduces a new benchmark with high-quality training datasets for diverse languages, encompassing both supervised and zero-shot language pairs. Despite these new datasets (Nădejde et al., 2022), controlling formality in MT remains a challenging problem due to the
absence of gold translations with alternate formality levels and the extensive variation in grammatical formality markers across languages.
In the 2023 shared task, an English source segment is paired with two references that are minimally contrastive in grammatical formality, representing both formal and informal levels as shown in Table 1. Training and test samples are provided in the domains of “telephony data” and “topical chat” (Gopalakrishnan et al., 2019) for two supervised language pairs, English-Korean (EN-KO) and English-Vietnamese (EN-VI), and two zero-shot language pairs, English-Portuguese (EN-PT) and English-Russian (EN-RU). Grammatical formality markers differ across these languages. Personal pronouns and verb agreement signal formality in many Indo-European languages (e.g., PT, RU), while in Korean, formality control is notably challenging due to the widespread use of morphological markers to convey polite, respectful, and humble speech, making it an intriguing test case for FSMT.
In this paper, we present our approach to FSMT, focusing on the supervised setting for the EnglishKorean (EN-KO) and English-Vietnamese (ENVI) language pairs and evaluating our methods on the zero-shot English-Portuguese (EN-PT) and English-Russian (EN-RU) pairs. Our method consists of two main strategies: 1) a language-specific data-driven approach, and 2) synthetic data gener-
420
ation using large-scale language models and empirical prompt engineering. We apply techniques and models tailored to the linguistic features of each language. For Korean, we utilize a morphemecentric subword tokenization method, while for Vietnamese, we employ a pre-trained EnViT5 model with high-quality Vietnamese parallel corpora. Additionally, we generate synthetic translation datasets for Portuguese and Russian using prompt engineering and refine these datasets using formality classifiers for fine-tuning our models. Furthermore, we founded significant performance improvements in EN-KO and EN-VI and conducted an ablation study to utilize high-quality synthetic examples.

heading: 2 Proposed Method

heading: 2.1 Task Definition
text: In this submission, we focus on the supervised and zero-shot settings on unconstrained formality control machine translation task. Formally, provided with a source segment X = {x1, x2, . . . , xm} and a formality level l ∈ {formal, informal}, the objective is to identify a model defined by parameters Θ that produces the most probable translation Y = {y1, y2, . . . , yn} in accordance with the formality level:
Y = arg max Yl P (X, l;Θ)
In simpler terms, the goal is to find the optimal model parameters Θ that produce the most likely translation Y , given the source segment X and the desired formality level l (either formal or informal). This is achieved by maximizing the probability P (X, l;Θ) of obtaining the translation Y at the specified formality level.

heading: 2.2 Language Specialized Data-Centric Approach
text: In this work, we employ a language specialized data-centric approach by integrating transfer learning techniques from Zoph et al. (2016) and language-specific subword methods, such as Unigram (Kudo, 2018) or byte-pair encoding (BPE) (Sennrich et al., 2015b). This combination effectively captures the unique morphological and syntactic structures of the target language, resulting in substantial improvements in translation performance, especially for low-resource languages (Zoph et al., 2016; Bojanowski et al., 2017;
Park et al., 2020, 2021). Finally, we fine-tuned the pre-trained model (PLM) on the supervised train set each language pair.
EN-KO We discuss our approach to improve the English-Korean (EN-KO) translation performance by pre-training a Transformer using a high-quality dataset and leveraging morpheme-aware subword tokenization to better capture the linguistic characteristics of the Korean language such as agglutinative nature and structure.
We adopted a data-centric approach by pretraining a Transformer for EN-KO translation. To do so, we used a high-quality dataset from the AI Hub (Park et al., 2022)1 data platform, which is operated by the Korean government. This comprehensive dataset includes various parallel corpora encompassing diverse domains such as technical and scientific fields, daily life and colloquial expressions, news articles, government and local government websites, publications, administrative regulations, Korean culture, and formal and informal language. By using a dataset specifically tailored for English-Korean translation, we aimed to capture finer nuances in both languages and enhance the translation quality by incorporating domainspecific knowledge and addressing the linguistic variations in different contexts.
Furthermore, we addressed the linguistic characteristics of the Korean language by applying a morpheme-aware subword tokenization method, which combines a segmentation strategy based on linguistic features with subwords. This approach has been shown to be effective in various Korean NLP tasks (Park et al., 2020). We utilized MeCabko 2, a widely-used morphological analyzer for the Korean language, for morpheme analysis. After obtaining the morphemes, we applied the Unigram subword tokenization method, which allowed our model to capture linguistic patterns specific to the Korean language, ultimately improving the overall translation performance.
EN-VI For the EN-VI language pair, we employed the EnViT5 (Ngo et al., 2022), a Text-toText Transfer Transformer (T5) model proposed by Raffel et al. (2020). We aimed to improve the fine-tuning translation performance of EN-VI in a low-resource setting by applying this data-centric approach to the multi-domain pre-trained EnViT5
1https://aihub.or.kr/ 2https://bitbucket.org/eunjeon/
mecab-ko-dic
model, which has been specifically designed for Vietnamese language tasks. Notably, EnViT5 models outperformed existing multilingual models such as mBART and M2M-100 while maintaining a significantly smaller parameter size, making them scalable and promising for both academic and industry applications (Ngo et al., 2022).
EnViT5 was pre-trained with the CC100 Dataset (Wenzek et al., 2020) which comprises monolingual data for over 100 languages. Subsequently, EnViT5 was fine-tuned on the MTet (Ngo et al., 2022) and PhoMT (Doan et al., 2021) datasets. MTet is a multi-domain EN-VI machine translation dataset encompassing a diverse range of domains, including educational videos, software user interfaces, COVID-related news articles, religious texts, subtitles, Wikipedia, and TED Talks (Reimers and Gurevych, 2020). Ultimately, when combined with PhoMT and IWSLT’15 (Cettolo et al., 2015), the final MTet dataset expands the training set size to 6 million examples, covering previously neglected areas such as law and biomedical data, which contains monolingual data for over 100 languages.

heading: 2.3 Synthetic Data Generation via Prompt Engineering
text: Leveraging synthetic examples in machine translation is crucial for improving translation quality, especially in low-resource settings (Edunov et al., 2018; Sennrich et al., 2015a). ChatGPT with GPT-4 engine (OpenAI, 2023), in particular, exhibits translation performance comparable to state-of-the-art WMT system and demonstrate good quality of generation conditioned translation generation in both few-shot and zero-shot settings (Hendy et al., 2023). To generate synthetic data, we employ ChatGPT to condition on formality and translate the IWSLT’22 Formality Track (Salesky et al., 2022) for all language pairs with English as the source language. Furthermore, we use a formality classifier (Rippeth et al., 2022) to filter synthetic examples, ensuring that both formal and informal examples are accurately translated for each language.
Supervised Setting We follow the prompt template depicted in Appendix A, which is based on the approach proposed by Hendy et al. (2023). To provide context for the model, we utilize n randomly selected shots from the English training set of other language pairs in the IWSLT 23 Formality Track (Agarwal et al., 2023). The few-shot exam-
ples are sourced from the target language’s training set and include both informal and formal levels. ChatGPT is then tasked with translating the input text into either an informal or formal target language, depending on the specified prompt. For the input text, we use English source sentences from the IWSLT 22 Formality Track’s other language pairs. After filtering the translated examples using a formality classifier, we fine-tuned the respective PLMs for EN-KO and EN-VI by incorporating synthetic examples into the training sets for each language pair. To verify the effectiveness of data augmentation through prompt engineering, we conduct experiments comparing the results with and without the augmented data.
Zero-shot Setting In the EN-PT and EN-RU zero-shot settings, we generate synthetic examples for fine-tuning using the IWSLT’22 train set. We translate the source into both formal and informal target language levels, employing suitable prompts and filtering with a formality classifier to ensure conditioned formality. The template, shown in Appendix A, is adapted from the OpenAI Playground’s default sentence-level translation task3. The model is instructed to translate English input into either informal or formal target language, guided by n random shots from the training set. Generated examples are then filtered using a formality classifier before fine-tuning the pre-trained multilingual translation model.
This zero-shot approach enables effective conditioned task performance with limited exposure to specific language pairs and formality levels. By generating synthetic translation data for fine-tuning, we capitalize on the model’s generalization ability across languages and formality levels, enhancing translation performance in zero-shot settings. This highlights the potential of synthetic data in extending pre-trained language models’ capabilities, even
3https://platform.openai.com/examples/ default-translate
with novel language pair and formality combinations.

heading: 3 Experiment Settings

heading: 3.1 Dataset Details
text: The IWSLT shared task provides Formality Dataset which contains English source segments, each accompanied by two contrasting reference translations representing informal and formal formality levels. This is available for two language pairs, EN-{KO, VI}, in the supervised setting and two additional language pairs, EN-{PT, RU}, in the zero-shot setting. The statistics for the train and test sets of the dataset are shown in Table 2
For training and testing purposes, we randomly sampled 50 pairs of examples across each domain from the train set of Formality Dataset, and set them aside as validation sets (TASK DEV) for each supervised language. The remaining samples were utilized for training (TASK TRAIN).
Additionally, we utilized external datasets in conjunction with the data provided in the shared task. For EN-KO, we employed a parallel corpus comprising Formal/Informal, Social Science, Technology Science, and News domains from AI Hub for the pretraining of the PLM. For EN-VI, we utilized EnViT5, which was fine-tuned using the MTet (Ngo et al., 2022) and PhoMT (Doan et al., 2021) datasets.
In our research, we leverage ChatGPT for the augmentation of the EN-KO and EN-VI and the generation of synthetic examples for fine-tuning on EN-PT and EN-RU. This was done by using the source data from all available English-other language pairs (EN-XX) in the IWSLT’22 Formality Track (Anastasopoulos et al., 2022). To secure the quality and uniqueness of our training set, we implemented a preprocessing step that excludes duplicate sentences. Furthermore, to determine the optimal hyperparameters, we conducted a case study utilizing TASK DEV (details can be found in Section 4.3). The hyperparameters that led to the highest Matched-Accuracy (M-Acc) were selected for use. For all language pairs, we utilized a temperature of 0.9; specifically, we implemented 4-shot learning for EN-KO and 2-shot learning for ENVI. For EN-PT and EN-RU, we proceeded with a zero-shot setting. More detailed information regarding the datasets and the preprocessing steps are presented in Table 3.
Language Size Source
EN-KO 6M AI Hub (Formal/Informal+ Tech/Sci + Social/Sci + News) EN-VI 6.2M MTet (Ngo et al., 2022)+ PhoMT (Doan et al., 2021) EN-{PT, RU} 1.6K EN source from IWSLT’22(Anastasopoulos et al., 2022)
Table 3: Additional external datasets used for the formality track in various language pairs.

heading: 3.2 Training Details
text: In the training details for the EN-KO language pair, we applied a morpheme-aware tokenization method to the translation dataset. To achieve this, we followed the training methods proposed by Park et al. (2020) and Gowda and May (2020), using MeCab-ko and Unigram to construct a vocabulary of 48K tokens. We then pre-trained the Transformer model (Vaswani et al., 2017). We used the fairseq library with 12 encoder and 12 decoder layers, each having 16 attention heads. Both encoder and decoder had an embedding dimension of 1024 and a feed-forward network (FFN) dimension of 4096. During pre-training, we trained for 20 epochs with a learning rate of 5e-4 and 4000 warmup updates. For fine-tuning, we trained for 200 epochs using a learning rate of 4e-5 and 100 warmup updates. We fine-tuned using the TASK TRAIN for all language pairs.
For EN-{VI, PT, RU} pairs, we fine-tuned using the huggingface library. For EN-VI, we used the VietAI/envit5-translation as the PLM. Fine-tuning was performed for 200 epochs with a learning rate of 4e-5, 200 warmup steps, and a batch size of 64. For EN-{PT,RU} pairs, we used facebook/mbart-large-50 and trained for 200 epochs with a learning rate of 3e-5, 100 warmup steps, and a batch size of 16. All models were trained using four RTX A6000 GPUs. Detailed hyperparameters and training information can be found in the Appendix B.

heading: 3.3 Evaluation Details
text: In our experimental setting, we used the official test set from Formality Dataset (IWSLT’23) to evaluate our translation model’s performance. The evaluation was conducted across two dimensions: overall translation quality and formality control. To assess the overall translation quality, we employed BLEU (Papineni et al., 2002) and COMET (Rei et al., 2020) (eamt22-cometinho-da) as au-
tomatic evaluation metrics. We use 13A tokenizer to report SACREBLEU (Post, 2018) scores for all languages.
For formality control, we utilized MatchedAccuracy (M-Acc), a reference-based corpuslevel metric that leverages phrase-level formality markers from the references to classify systemgenerated hypotheses as formal or informal. The corpus-level score is the percentage of system outputs that match the desired formality level.
Additionally, we used a reference-free variant of M-Acc (C-F) 4, which relies on a multilingual formality classifier to label system-generated hypotheses as formal or informal, with the corpuslevel score representing the percentage of system outputs matching the desired formality level.

heading: 3.4 Prompt Design
text: We conducted experiments using ChatGPT with GPT-4 engine with langchain5. For EN-KO and EN-VI language pairs, we used a supervised set-
4https://github.com/amazon-science/ contrastive-controlled-mt/tree/main/ IWSLT2023
5https://python.langchain.com/
ting, while for EN-PT and EN-RU pairs, we employed a zero-shot setting. In the supervised setting, we extracted arbitrary n-shot samples using the TASK TRAIN. We designed prompts by leveraging langchain’s prompt guide and prompt examples from Hendy et al. (2023). Detailed examples and explanations of the prompts can be found in Appendix A.

heading: 4 Result & Findings

heading: 4.1 Results for Supervised Setting
text: Table 4 presents our experimental results in the supervised setting. As demonstrated by our results, our model, trained with the high-quality humanannotated Formality Dataset, exhibited outstanding performance. In particular, with respect to the C-F metric, our model shows almost perfect formality control performance (100% accuracy) for most of the tasks, except for the EN-KO informal task. Additionally, our model shows superior performance for the conventional NMT metrics (i.e. BLEU, COMET), outperforming ChatGPT with a 21.50 BLEU score for the EN-KO informal task. The EN-VI pair also exhibits high NMT metric
scores, M-Acc, and C-F scores compared to the baseline. These results suggest that our languagespecific data-centric approach is effective.
Through our experiments, we observed a significant degradation in the quality for supervised settings EN-{KO, VI}. This phenomenon can be attributed to the limitations of synthetic data produced by ChatGPT. While the data generated through ChatGPT exhibits considerable quality, it was not up to par with the sentences derived from our data-centric approach. We found that the integration of ChatGPT-augmented data inadvertently introduced noise into the system, leading to a decrease in overall performance. Despite the exceptional capabilities of ChatGPT, it appears that in this context, the quality of data augmented by conventional NMT methods is still superior. This observation further emphasizes the critical role of data quality over quantity in supervised learning environments, and highlights the potential benefits of more sophisticated prompting techniques that consider formality control, such as stylistic or sentence endings, for improving overall performance.

heading: 4.2 Results for Zero-shot Setting
text: The experimental results for the zero-shot setting are shown in Table 5. As can be seen from the experimental results, our model significantly out-
performs the official baseline on all tasks except the EN-PT informal task. Notably, our model demonstrates consistently higher performance in terms of C-F metric compared to ChatGPT, achieving 100% M-ACC and C-F in the majority of tasks.
Exceptionally for EN-PT informal task, the performance of our model is markedly subpar, and ChatGPT even fails to exceed the official baseline. We find this result is highly noteworthy, as it suggest that ChatGPT may generate semantically accurate and plausible data, while the formality can hardly be controlled, especially for the EN-PT language pair. In our experiments, we utilized the same prompt for both EN-PT and EN-RU language pairs, differing only in language specification. The disparity in results between these two language pair suggests that specialized techniques for controlling formality are required for each language pair. This issue can be partially attributed to a data bias in ChatGPT, indicating a potential training data bias concerning formality.

heading: 4.3 Case Study
text: Impact of In-context Shots In this section, we examine the changes in performance based on the number of few-shot samples used for in-context learning, particularly when employing prompt engineering for translation. Previous research sug-
gests that increasing the number of shots beyond 10 does not significantly impact translation performance when using large language models (Zhang et al., 2023). However, we argue that applying the same perspective to formality control tasks proves challenging. This complexity arises as formality introduces a unique element required for these tasks. Additionally, previous research did not consider unintended consequences arising from this factor.
In pursuit of this, we conducted experiments where the number of shots was incrementally increased from 1 to 32, in powers of 2, using TASK DEV. The aim was to verify the differences in performance resulting from these changes. This process involved translating data via ChatGPT with an increasing number of shots and then evaluating the resulting translation data for its appropriateness. The experimental results are depicted in Figure 1. For this particular experiment, we selected one temperature (from the options of 0.2, 0.5, 0.7, 0.9) that demonstrated the highest performance and evaluated the changes in performance based on the number of shots.
As observed in our experimental results, increasing the number of shots for in-context learning led
to an improvement in the general translation performance metric, BLEU. However, the scores of MAcc and C-F, we found that the best performance was achieved with a smaller number of shots. This suggests that the nature of formality as a feature makes the “formality control” task distinct from conventional NMT, and it may be challenging to directly apply perspectives from conventional NMT to this task. We propose two hypotheses based on these results: (i) there exists a trade-off between translation performance and formality control as the number of shots increases, and (ii) increasing the number of shots while applying random sample selection may have caused confusion in performing formality control. We leave the analysis and validation of these hypotheses for future work.
Impact of Temperature Temperature is an important parameter to make ChatGPT generates varied responses to human queries (Peng et al., 2023). Basically, higher temperatures leads to the higher linguistic variety, while the lower one generates grammatically correct and deterministic text (Ippolito et al., 2019). Previous work suggested that for machine translation, a diverse generation may
impede its translation quality with a high degree of certainty(i.e. high temperature) (Peng et al., 2023). In this sense, we experiment with different temperature setting and find the optimal temperature for the formality control data augmentation. In our experiments, we select the most appropriate one among seven shot-candidates (1, 2, 4, 8, 16, 32) for each language pair.
Experimental results reveal that varying temperature can lead to significant performance fluctuations. It is particularly noteworthy that the performance disparity due to temperature changes is exceptionally high for the informal tasks. For formal tasks, the impact of temperature is relatively minor, with the variation in BLEU score is at most 0.95 (EN-RU). However, for informal tasks, the performance shift can reach up to 4.82 points (ENRU) as temperature changes. Additionally, we find that in informal task, the performance variation depending on the temperature shows distinct trend for each language pair. This is evident from the fact that a moderate temperature(0.7) yielded the highest BLEU performance in the EN-PT informal task, while a similarly moderate temperature(0.5) resulted in the lowest performance. Our findings suggest that handling ChatGPT in informal task necessitates more elaborate control compared to dealing with formal data.

heading: 5 Background
text: In this work, we focus on data-centric approaches to improve Neural Machine Translation (NMT) performance. Several studies have investigated different strategies to address the challenges of lowresource languages and enhance translation quality. Kudo (2018) proposed subword regularization to improve NMT models using multiple subword candidates, effectively increasing data diversity and robustness. Gu et al. (2018) introduced a universal NMT model for extremely low-resource languages, leveraging multilingual knowledge from high-resource languages to assist in translation. Zoph et al. (2016) explored transfer learning for low-resource NMT, utilizing pre-trained models on related high-resource languages to improve the performance on the target low-resource language. Additionally, Sennrich et al. (2015a) proposed a method of improving NMT models by generating synthetic parallel data through back-translation, which has proven successful in various translation tasks. These studies highlight the diverse data-
centric approaches in NMT, aiming to improve translation quality and overcome the limitations of low-resource languages.

heading: 6 Conclusion
text: In this paper, we presented the KU x UpStage team’s submission for four languages, employing two main strategies: 1) a language-specific data-driven approach, and 2) synthetic data generation using large-scale language models and empirical prompt engineering. While our data-driven approach excelled, particularly in EN-KO and ENVI, the quality of synthetic data generation was called into question. In light of this feedback, we propose to enhance the quality of synthetic data by integrating Quality Estimation (QE) techniques as an additional filter in the generation process. This step aims to further refine our synthetic examples, potentially improving the overall system performance. We also plan to explore the use of translation models with larger parameters and conduct a thorough analysis through more shot examples and linguistically-grounded data augmentation techniques. Finally, we aim to extend our understanding of factors influencing FSMT performance, such as the impact of formal register versus grammatical formality in training data and a detailed examination of zero-shot transfer.

heading: Acknowledgments
text: This work was generously supported by multiple grants. The Core Research Institute Basic Science Research Program, funded by the Ministry of Education through the National Research Foundation of Korea (NRF) (Grant NRF2021R1A6A1A03045425), provided valuable support. Additionally, this research received backing from the Information Technology Research Center (ITRC) support program (IITP-2023-2018-001405), which is supervised by the Institute for Information & Communications Technology Planning & Evaluation (IITP) and funded by the Ministry of Science and ICT (MSIT), Korea. Finally, the Korea government’s MSIT also funded a grant through the IITP (No. 2020-0-00368) dedicated to "A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques." We extend our gratitude for this comprehensive support.
Limitations
Due to the random sampling of shots, the results of the experiment may vary between repeated trials. However, we did not conduct repeated experiments under identical conditions, and thus we acknowledge the potential inconsistency of our experimental results.
Ethics Statement
This research study did not involve any human or animal subjects, and no personal data or sensitive information was used in this research. Therefore, no ethical issues were encountered in this study. The authors confirm that the research was conducted in accordance with the relevant ethical guidelines and principles.

heading: A Prompt Template
text: A.1 Superviesd Setting
You are a helpful assistant that translates English to: 1. Informal [target language] or 2. Formal [target language]
####
[shot 1 source]
[shot 2 source]
[shot n source]

heading: 1. Informal [target language]: [shot 1 reference]

heading: 2. Formal [target language]: [shot 1 reference]

heading: 1. Informal [target language]: [shot 2 reference]

heading: 2. Formal [target language]: [shot 2 reference]

heading: 1. Informal [target language]: [shot n reference]
text: 2. Formal [target language]: [shot n reference]
####
Translate this into only [1. Informal | 2. Formal] [target language]: [input]
Figure 3: Prompt template for supervised setting based on Hendy et al. (2023). We utilize n randomly selected shots from the English training set of other language pairs in the IWSLT 23 Formality Track as input for our model, with few-shot examples derived from the target language’s training set.
A.2 Zero-shot Setting
You are a helpful assistant that translates English to:

heading: 1. Informal [target language] or 2. Formal [target language]
text: [shot n source]
Translate this into only [1. Informal | 2. Formal] [target language]: [input]
Figure 4: Prompt template for zero-shot setting, following the recommended instruction and format for the default sentence-level translation task in OpenAI playground6. This consistency enables us to maximize the benefits of the instruction finetuning protocol. We use n random shots from the training set.

heading: B Experimental Setup
text: B.1 EN-KO In the experimental setup for the EN-KO language pair, we employed a Transformer architecture with shared decoder input-output embeddings. The model’s parameters included 1024-dimensional embeddings for both encoder and decoder, 16 attention heads for each, and 12 layers for both encoder and decoder. We used the Adam optimizer with beta values (0.9, 0.98) and a learning rate of 5e-4 scheduled by an inverse square root scheduler with a 4000-step warm-up. To prevent overfitting, we applied a dropout rate of 0.3 and weight decay of 0.0001. Our translation task utilized a label-smoothed cross-entropy criterion with a label smoothing factor of 0.1. The training process was performed with a maximum token limit of 4096 per batch and an update frequency of 4. Model performance was evaluated using BLEU scores with a beam size of 1 and detokenization using the Moses tokenizer. The training process was executed for a maximum of 20 epochs with a log interval of 200 and without epoch checkpoints, while sharing all embeddings.
Parameters for pre-training: fairseq-train \
--fp16 \ --fp16-init-scale 4096 \ --arch transformer --share-decoder-input-output-embed \ --encoder-embed-dim 1024 --decoder-embed-dim 1024 \ --encoder-attention-heads 16 --decoder-attention-heads 16 \ --encoder-ffn-embed-dim 4096 --decoder-ffn-embed-dim 4096 \ --encoder-normalize-before --decoder-normalize-before \ --encoder-layers 12 --decoder-layers 12 \ --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \ --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \ --dropout 0.3 --weight-decay 0.0001 \ --task translation \ --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \ --max-tokens 4096 \ --update-freq 4 \ --eval-bleu \ --eval-bleu-args '{"beam": 1, "max_len_a": 1.2, "max_len_b": 10}' \ --eval-bleu-detok moses \ --eval-bleu-remove-bpe \ --best-checkpoint-metric bleu --maximize-best-checkpoint-metric \ --log-interval 200 \ --max-epoch 20 \ --skip-invalid-size-inputs-valid-test \ --no-epoch-checkpoints \ --share-all-embeddings
Parameters for fine-tuning: fairseq-train \
--batch-size 32 \ --lr 4e-5 --warmup-updates 200 \ --max-epoch 200 \ --restore-file $MODELDIR/checkpoint_best.pt \ --reset-optimizer --reset-meters --reset-dataloader --reset-lr-scheduler
B.2 EN-VI We fine-tuned our model using the Hugging Face library and the code available at their repository7. The fine-tuning was performed with a learning rate of 4e-5, Adam optimizer with beta1 and beta2 values set to 0.9 and 0.98, respectively, and a weight decay of 0.0001. We also used mixed precision training (fp16) to accelerate the process. The learning rate scheduler was set to inverse square root with a warm-up of 200 steps. The training was conducted for 200 epochs with a maximum gradient norm of 0.0, label smoothing factor of 0.1, and a batch size of 64 for both training and evaluation. The model was saved and evaluated at the end of each epoch, and the logging was performed after each training step.
7https://github.com/huggingface/transformers/tree/main/examples/pytorch/ translation
Parameters for fine-tuning: python train_mt_trainer.py \
--fp16 \ --model_name_or_path VietAI/envit5-translation \ --do_train \ --do_eval \ --do_predict \ --source_lang en \ --target_lang vi \ --source_prefix "translate English to Vietnamese: " \ --learning_rate 4e-5 \ --adam_beta1 0.9 \ --adam_beta2 0.98 \ --max_grad_norm 0.0 \ --num_train_epochs 200 \ --lr_scheduler_type inverse_sqrt \ --warmup_steps 200 \ --weight_decay 0.0001 \ --label_smoothing_factor 0.1 \ --save_strategy epoch \ --logging_steps 1 \ --evaluation_strategy epoch \ --per_device_train_batch_size=64 \ --per_device_eval_batch_size=64
B.3 EN-{PT, RU} We utilized the same training code as for the EN-VI task and employed the facebook/mbart-large-50 model.
Parameters for fine-tuning: export langs=ar_AR,cs_CZ,de_DE,en_XX,es_XX,et_EE,fi_FI,fr_XX,gu_IN,hi_IN, it_IT,ja_XX,kk_KZ,ko_KR,lt_LT,lv_LV,my_MM,ne_NP,nl_XX,ro_RO,ru_RU,si_LK, tr_TR,vi_VN,zh_CN
python train_mt_trainer.py \ --fp16 \ --model_name_or_path facebook/mbart-large-50 \ --do_train \ --do_eval \ --do_predict \ --source_lang en_XX \ --target_lang pt_XX \ --learning_rate 3e-5 \ --adam_beta1 0.9 \ --adam_beta2 0.98 \ --max_grad_norm 0.0 \ --num_train_epochs 200 \ --lr_scheduler_type inverse_sqrt \ --warmup_steps 100 \ --weight_decay 0.0001 \ --label_smoothing_factor 0.1 \ --save_strategy epoch \ --logging_steps 1 \ --evaluation_strategy epoch \ --per_device_train_batch_size=16 \ --per_device_eval_batch_size=16

heading: Future Work
text: We leave the analysis and validation of these hypotheses for future work. Impact of Temperature Temperature is an important parameter to make ChatGPT generates varied responses to human queries (Peng et al. , 2023). Basically, higher temperatures leads to the higher linguistic variety, while the lower one generates grammatically correct and deterministic text (Ippolito et al. , 2019). Previous work suggested that for machine translation, a diverse generation may
impede its translation quality with a high degree of certainty(i. e. high temperature) (Peng et al. , 2023). In this sense, we experiment with different temperature setting and find the optimal temperature for the formality control data augmentation. In our experiments, we select the most appropriate one among seven shot-candidates (1, 2, 4, 8, 16, 32) for each language pair. Experimental results reveal that varying temperature can lead to significant performance fluctuations. It is particularly noteworthy that the performance disparity due to temperature changes is exceptionally high for the informal tasks. For formal tasks, the impact of temperature is relatively minor, with the variation in BLEU score is at most 0. 95 (EN-RU). However, for informal tasks, the performance shift can reach up to 4. 82 points (ENRU) as temperature changes. Additionally, we find that in informal task, the performance variation depending on the temperature shows distinct trend for each language pair. This is evident from the fact that a moderate temperature(0. 7) yielded the highest BLEU performance in the EN-PT informal task, while a similarly moderate temperature(0. 5) resulted in the lowest performance. Our findings suggest that handling ChatGPT in informal task necessitates more elaborate control compared to dealing with formal data