Venue: NAACL 2021

# Towards Modeling the Style of Translators

in Neural Machine Translation

Yue Wang\({}^{*}\) and Cuong Hoang &Marcello Federico

Amazon AI

hoacuong@amazon.com

###### Abstract

One key ingredient of neural machine translation is the use of large datasets from different domains and resources (e.g. Europarl, TED talks). These datasets contain documents translated by professional translators using different but consistent translation styles. Despite that, the model is usually trained in a way that neither explicitly captures the variety of translation styles present in the data nor translates new data in different and controllable styles. In this work, we investigate methods to augment the state-of-the-art Transformer model with translator information that is available in part of the training data. We show that our style-augmented translation models are able to capture the style variations of translators and to generate translations with different styles on new data. Indeed, the generated variations differ significantly, up to \(+4.5\) BLEU score difference. Despite that, human evaluation confirms that the translations are of the same quality.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1193–1199

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1193–1199

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1193–1199

## 1 Introduction

Translators often translate the original content with provided guidelines for styles.1 However, guidelines are supposed to be high level and not comprehensive. Personal stylistic choices are thus welcome as creative part of the translator's job, as long as their translation style consistency is ensured to the task. By contrast, although neural machine translation (NMT) models Cho et al. (2014); Sutskever et al. (2014) are trained from these human translations (e.g. Europarl, TED Talks), the models do not explicitly learn to capture the rich variety of translators' styles from the data. This limits their capability to creatively translate new data with different and consistent styles as translators do. We believe that modeling the style of translators is an important yet overlooked aspect in NMT. Our contribution, to the best of our knowledge, is to fill this gap for the first time.

In particular, our work investigates ways to integrate translator information into NMT, with an emphasis on mimicking the translator's style. Our study uses the TED talk dataset, with four language pairs with translator annotations. We present and compare a set of different methods of using a discrete translator token to model and control translator-related stylistic variations in translation. Note that using a discrete token is a common approach to model and control not only specific traits in translation such as verbosity, politeness and speaker-related variances Sennrich et al. (2016a); Michel and Neubig (2018)) but also other aspects in NMT such as language ids Johnson et al. (2017); Fan et al. (2020). However, our study is the first to use such a discrete token to model the style of translators. It also provides several insights regarding translation style modeling as follows.

First, we show that the state-of-the-art Transformer model implicitly learns the style of translators only to a limited extent. Moreover, methods that add translator information to the decoder surprisingly result in NMT that fully _ignores_ the additional knowledge. This is regardless of whether the token is added to the bottom (i.e. the embedding layer) or to the top (i.e. the softmax layer) of the decoder. Meanwhile, methods that add the information to the encoder seem to model the translator's style effectively.

Second, we show that our best style-augmented NMT method is able to control the generation of translation in a way that mimics the translator's style, e.g. lexical and grammatical preferences, verbosity. While output produced by the style-augmented NMT can vary significantly with the translator-token values, with BLEU score variations up to \(+4.5\), a human evaluation confirms that observed differences are all about style and not

translation quality. Finally, we show that the translator information has more impact on NMT than the speaker information, which was investigated by Michel and Neubig (2018).

## 2 Related Work

Style itself is a broad concept Kang and Hovy (2019). It includes both simple high-level stylistic aspects of language such as verbosity Marchisio et al. (2019); Agrawal and Carpuat (2019); Lakew et al. (2019), formality Niu et al. (2017); Xu et al. (2019), politeness Mirkin et al. (2015) and complex aspects such as demography Vanmassenhove et al. (2018); Moryossef et al. (2019); Hovy et al. (2020) and personal traits Mirkin and Meunier (2015); Rabinovich et al. (2017); Michel and Neubig (2018).

Our study focuses on capturing the personal style of translators. The closest work to our study is thus the work of Michel and Neubig (2018), where they study instead the effects of using the speaker information in NMT. In our results, we show that the translator information has indeed more impact to NMT than the speaker information.

Finally, another distantly related research line tries to improve the diversity in the top rank translations of an input Li et al. (2016); Shen et al. (2019); Agrawal and Carpuat (2020). In fact, adding the translator information to NMT also provides means to generate translations with significantly different stylistic variations.

## 3 NMT with Translator Information

NMT reads an input sequence \({\bf x}=x_{1},...,x_{n}\) in the source language with an encoder and then produces an output sequence \({\bf y}=y_{1},...,y_{m}\) in the target language. The generation process is performed in a token-by-token manner and its probability can be factored as \(\prod_{j=1}^{m}P(y_{j}\,|\,{\bf y}_{<j},{\bf x})\), where \({\bf y}_{<j}\) denotes the previous sub-sequence before \(j\)-th token. The prediction for each token over the vocabulary \(\mathcal{V}\) is based on a softmax function as follows:

\[P(y_{j}|{\bf y}_{<j},{\bf x})=\text{softmax}({\bf W}_{V}{\bf o}_{j}+{\bf b}_{ V}). \tag{1}\]

Here, \({\bf o}_{j}\in R^{d}\) is an output vector with size \(d\) (e.g. \(512\) or \(1024\)), encoding both the context from the encoder and the state of the decoder at time \(j\). Meanwhile, \({\bf W}_{V}\in R^{|\mathcal{V}|\times d}\) and \({\bf b}_{V}\in R^{|\mathcal{V}|}\) are a trainable projection matrix and bias vector.

We adjust NMT in different ways as below to let it mimic and control the translator's style.

**Source Token.** In our first approach, we insert the translator token \(T\) as the beginning of each input sentence. The translator token is thus assigned with an embedding vector like any other source token. Hence, the embedding sequence \(E_{enc}\) for the MT encoder becomes:

\[E_{enc}=[e(T),e(x_{1}),...,e(x_{n})], \tag{2}\]

where \(e(\cdot)\) is an embedding lookup function.

**Token Embedding.** We also consider adding the embedded translator token \(e(T)\) to every token embedding in the encoder and/or decoder as follows:

\[E_{enc}=[e(T)+e(x_{1}),...,e(T)+e(x_{n})], \tag{3}\] \[E_{dec}=[e(T)+e(y_{1}),...,e(T)+e(y_{m})]. \tag{4}\]

Our motivation is to reinforce the influence of the translator token in MT.

**Output Bias.** Following Michel and Neubig (2018), we add the translator token information to the output bias at the final layer of the decoder (Full-Bias variant). Specifically, the method directly modulates the word probability over vocabulary \(\mathcal{V}\) as follows:

\[P(y_{j}|{\bf y}_{<j},{\bf x},T)=\text{softmax}({\bf W}_{V}{\bf o}_{j}+{\bf b}_ {V}+{\bf b}_{T}). \tag{5}\]

Here, \({\bf b}_{T}\in R^{|\mathcal{V}|}\) is the translator-specific bias vector, which can be thought of as a translator-token embedding with dimension \(|\mathcal{V}|\) rather than \(d\). We also explore another variant, named Fact-Bias, as in Michel and Neubig (2018). This variant instead learns the translator bias through the factorization:

\[{\bf b}_{T}={\bf W}{\bf s}_{T}, \tag{6}\]

with parameters \({\bf W}\in R^{|\mathcal{V}|\times k}\) and \(s_{T}\in R^{k\times 1}\) where \(k<<|\mathcal{V}|\).

Note that while the above methods digest the translator token at an earlier stage, this one consumes translator signals in a late fusion manner.

## 4 Experiments

### Dataset and Models

We run experiments with the WIT\({}^{3}\) public dataset of TED talks Cettolo et al. (2012), with four language pairs: English-German (en-de), English-French (en-fr), English-Italian (en-it) and English-Spanish (en-es). The dataset contains both speaker and translator information for each talk and translation, thus allowing to measure the effects of translators and speakers 

We construct training, validation and test sets for each translation direction as follows. We first extract all talks that are translated by the \(10\) most popular translators (see Figure 1) and split them into parallel sentences. From the data of each translator, we then sample \(500\) sentences for testing, and, from the remaining data, 90% for training and 10% for validation. All training, testing, and validation sentence pairs are put together and annotated with training and speaker labels. Table 1 shows the data statistics for four language pairs.

For preprocessing, we employ Moses Koehn et al. (2007) tool2 for tokenization and apply subword-nmt3Sennrich et al. (2016b) to learn subword representations.

Footnote 2: [https://github.com/moses-smt/mosesdecoder](https://github.com/moses-smt/mosesdecoder)

Footnote 3: [https://github.com/rsennrich/subword-nmt](https://github.com/rsennrich/subword-nmt)

We choose Transformer Vaswani et al. (2017) as the baseline and employ Fairseq Ott et al. (2019) for our implementations. Our Transformer model is comprised of \(6\) layers of encoder-decoder network, where each layer contains \(16\) heads with a self-attention hidden state of size \(1024\) and a feed-forward hidden state of size \(4096\). We employ Adam optimizer Kingma and Ba (2015) to update model parameters. We warm up the model by linearly increasing the learning rate from \(1\times 10^{-7}\) to \(5\times 10^{-4}\) for \(4000\) updates and then decay it with an inverse square root of the rest training steps by a rate of \(1\times 10^{-4}\). We apply a Dropout of \(0.3\) for en-de and \(0.1\) for both en-fr and en-it.

For all MT systems, we load weights from pretrained models to set up a better model initialization. Specifically, we employ models pretrained on WMT data for en-de and en-fr Ott et al. (2018), and pretrain models for en-it and en-es using our large in-house out-of-domain data, as there are no previous pretrained models for these pairs. We fine-tune models on TED talk data for \(10\) epochs4 and select the best model based on the validation loss.

Footnote 4: We try finetuning with more epochs and observe no further improvements.

During inference, we employ beam search with a beam size of \(4\) and add a length penalty of \(0.4\).

We use the BLEU score Papineni et al. (2002) to evaluate translation accuracy.

### Results

#### 4.2.1 Adding Translator Token

We first compare methods to integrate the translator token into the Transformer. Notice that we report performance of the model in two settings: (i) when fed with the oracle translator label (as at training time) and (ii): when fed with randomly assigned labels. Intuitively, if a model really leverages the translator information, we expect to see a performance drop in the randomized setting. Results are shown in Table 2.

Our findings are as follows. First, it is surprisingly ineffective to add the translator token into the decoder, whether to the input Dec-Emb or to the softmax Full-Bias, Fact-Bias). In most cases, our randomization experiment shows that the model simply ignores the information.

Second, methods adding the token to the encoder Src-Tok, Enc-Emb) are significantly more effective. Translation accuracy is also consistently better (at most by \(0.4\) BLEU) than with the Transformer baseline, indicating the translator token is useful. For those models, randomizing translator labels results in visible drops in BLEU score (up to \(1.0\) BLEU), indicating that the translator information has an important effect to the model.

Figure 1: TED talks translated by top 10 translators.

 



#### 4.2.2 Style Imitation

Following the common practice in evaluating the style imitation (e.g. see Michel and Neubig, 2018; Hovy et al., 2020)), we train a classifier to predict the translator style of the output of various models. We employ a Logistic Regression classifier based on both uni-gram and bi-gram word features. The classifier, trained on NMT training data, is applied on the outputs of NMT models. Figure 2 shows the results of this experiment.

As can be seen, the standard Transformer learns the style of translators only to a limited extent. The style of translation outputs are less consistent with the original translator's style, i.e. accuracy is between \(20\%\) and \(35\%\)). Meanwhile, the classification accuracy is significantly higher (up to \(+12\%\) relative) under Src-Tok and Enc-Emb. This confirms that explicitly incorporating translator information at the sentence level allows for transferring some of her/his personal traits into the translations.

Meanwhile, we notice higher accuracy achieved with the reference translations (e.g. 42% in ENES), suggesting there is room for improvement.

#### 4.2.3 Stylistic Variations

We analyzed stylistic variations using different translator token labels. In particular, we evaluate model outputs on _en-fr_ after translating the entire test set with the same translator token labels. As in Table 3, translator-informed NMT can produce quite different outputs, resulting in BLEU score variations up to \(+4.5\), (i.e. between \(T7\) and \(T3\), \(T8\), \(T10\)). We also observe differences in BLEU (albeit smaller) when testing with the WMT 2014 test set. In particular, BLEU score variations are up to \(+0.84\) between \(T7\) and \(T5\). We also compute the symmetric-BLEU distances between any two of the translators using their predictions for both TED and WMT test set and visualize their heatmaps in Figure 3. We observe that a similar BLEU distance between various translators in both test sets. Besides, T7 has a farther distance with others but its gap is closer on WMT than TED. These findings verify the consistency of translator styles in data from different domains.

Then, we asked 3 professional translators to grade the quality of translation produced with the labels \(T7\) and \(T3\) on the TED talks. The evaluation is on a 1-6 scale (higher is better) on a random sample of 100 sentences. This resulted in average scores of \(4.867\) and \(4.860\) for T3 and T7, respectively. A similar human evaluation with \(T7\) and \(T5\) labels was also run on a random sample of 100 sentences of the WMT 2014 test set. It provided the same conclusion: average scores are very similar: \(4.99\) and \(5.0\) for T5 and T7 respectively. Both evaluations confirm that there is no difference in translation quality when using different token labels, i.e. the low BLEU score of T7 is only an effect due to stylistic differences.

Table 4 shows examples of translations generated with labels T3 and T7. As we can observe, the translations show different use of grammars, words and verbosity.5

Footnote 5: Note that one could argue that it is not just about style here but also translation fidelity. We thank a reviewer for pointing it out.

Figure 2: Translator classification accuracy. Enc-Emb yields the best result in most cases.





#### 4.2.4 Translator vs. Speaker Effects

Finally, we compared the effect of the translator token with that of the speaker token, which was proposed in Michel and Neubig (2018) to perform extreme personalization. Results on all four directions (see Table 5) show that the translator token has more impact. 6 Given that speaker and author style has received much more attention in the literature, we hope that this final result will spark more interests on the style of translators.

Footnote 6: One probable reason is that the speaker signal is more sparse than the translator signal, i.e. each speaker is represented by one TED talk, while translators by multiple talks.

## 5 Conclusion

We designed various ways of incorporating translator information into NMT, in order to model and control the generation of translation with different translator styles. We show that resulting style-augmented NMT produces significantly different stylistic variations, mimicking professional translators. Human evaluation confirms that the generated variations are all of same translation quality.

## Acknowledgements

We thank the anonymous reviewers for their constructive feedback.

\begin{table}
\begin{tabular}{l l



## References

* Agrawal and Carpuat (2019) Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1549-1564, Hong Kong, China. Association for Computational Linguistics.
* Agrawal and Carpuat (2020) Sweta Agrawal and Marine Carpuat. 2020. Generating diverse translations via weighted fine-tuning and hypotheses filtering for the Duolingo STAPLE task. In _Proceedings of the Fourth Workshop on Neural Generation and Translation_, pages 178-187, Online. Association for Computational Linguistics.
* Cettolo et al. (2012) Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit\({}^{3}\): Web inventory of transcribed and translated talks. In _Proceedings of the 16\({}^{th}\) Conference of the European Association for Machine Translation (EAMT)_, pages 261-268, Trento, Italy.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 1724-1734.
* Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edoudar Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multilingual machine translation. _arXiv preprint_.
* Hovy et al. (2020) Dirk Hovy, Federico Bianchi, and Tommaso Fornacini. 2020. "you sound just like your father" commercial machine translation systems include stylistic biases. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1686-1690, Online. Association for Computational Linguistics.
* Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. _Transactions of the Association for Computational Linguistics_, 5:339-351.
* Kang and Hovy (2019) Dongyeop Kang and Eduard Hovy. 2019. xS-LUE: A Benchmark and Analysis Platform for Cross-Style Language Understanding and Evaluation. _arXiv:1911.03663 [cs]_. ArXiv: 1911.03663.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrey Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In _ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic_. The Association for Computational Linguistics.
* Lakew et al. (2019) Surafel Melaku Lakew, Mattia Antonino Di Gangi, and Marcello Federico. 2019. Controlling the output length of neural machine translation. _CoRR_, abs/1910.10408.
* Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A simple, fast diverse decoding algorithm for neural generation. _CoRR_, abs/1611.08562.
* Marchisio et al. (2019) Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level of machine translation output. In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_, pages 193-203, Dublin, Ireland. European Association for Machine Translation.
* Michel and Neubig (2018) Paul Michel and Graham Neubig. 2018. Extreme adaptation for personalized neural machine translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 312-318, Melbourne, Australia. Association for Computational Linguistics.
* Mirkin and Meunier (2015) Shachar Mirkin and Jean-Luc Meunier. 2015. Personalized machine translation: Predicting translational preferences. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 2019-2025, Lisbon, Portugal. Association for Computational Linguistics.
* Mirkin et al. (2015) Shachar Mirkin, Scott Nowson, Caroline Brun, and Julien Perez. 2015. Motivating personality-aware machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1102-1108. The Association for Computational Linguistics.
* Moryossef et al. (2019) Amit Moryossef, Roee Aharoni, and Yoav Goldberg. 2019. Filling gender & number gaps in neural machine translation with black-box context injection. In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_, pages 49-54, Florence, Italy. Association for Computational Linguistics.
* Niu et al. (2017) Xing Niu, Marianna Martindale, and Marine Carpuat. 2017. A study of style in machine translation: Controlling the formality of machine translation output.



In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2814-2819, Copenhagen, Denmark. Association for Computational Linguistics.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In _Proceedings of the Third Conference on Machine Translation (WMT)_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_, pages 311-318. ACL.
* Rabinovich et al. (2017) Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 1074-1084, Valencia, Spain. Association for Computational Linguistics.
* Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Controlling politeness in neural machine translation via side constraints. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 35-40, San Diego, California. Association for Computational Linguistics.
* Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
* Shen et al. (2019) Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 5719-5728. PMLR.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_, pages 3104-3112.
* Vanmassenhove et al. (2018) Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3003-3008, Brussels, Belgium. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreti, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_, pages 5998-6008.
* Xu et al. (2019) Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations.