言語処理学会 第30回年次大会 発表論文集（2024年3月）

Non-literal Neural Machine Translation
by Exploiting Non-literal Bitext
Lianhao Yu1 Naoki Yoshinaga2 Masato Neishi1 Yuma Tsuta1
1 The University of Tokyo
2 Institute of Industrial Science, The University of Tokyo{yu-l, ynaga, neishi, tsuta}@tkl.iis.u-tokyo.ac.jp
Table 1

Abstract
To avoid unnatural-sounding translations produced byexisting neural machine translation (NMT) systems, we
propose training an NMT model for non-literal translations
by exploiting possibly non-literal translations in training
data. Speciﬁcally, we split the training bitext into two sets
in terms of non-literalness and applied domain adaptation
techniques to acquire an NMT model adapted to non-literaltranslations. Our best-performing model achieved a BLEU
score of 25.20 and a COMET score of 0.7932 in producing
non-literal translations.

1

Introduction

Neural machine translation (NMT) utilizes a single large
neural network to directly transform the source sentence
into the target sentence [1, 2, 3], which brought great
improvements in terms of the translation quality. NMT
systems have already been developed and deployed for various practical uses. For example, Google Translate is aservice that enables to translate sentences, documents, andwebsites among over 100 languages, and helps its users to
overcome the obstacle of unknown languages and access awide range of information and resources.
Although the translations produced by NMT systems
are mostly correct, they are sometimes unnatural for native speakers and thus not quite understandable due to theexisting diﬀerences between languages and cultures [4].
For comparison, human translations of the same source
sentences are sometimes non-literal but more natural fornative speakers, and thus more appropriate, as shown in
Table 1. Considering the relatively common occurrencesof non-literal translations, it is natural and promising to
exploit such translations in existing parallel corpora to improve the performances of NMT models in terms of ﬁdelity,
adequacy, and ﬂuency of translations [5, 6, 4].

Examples of English-Chinese non-literal translations.

EN: In his hand, it became buoyant, in sync with the
motion of the waves that he made with his arms.
ZH: 它在他的手中浮 动，随着他用手臂模 仿的波浪一
同 沉浮。
( It ﬂoated in his hand, rising and falling with the waves
he mimicked with his arm. )EN: Success is in the doing, and failures are celebrated andanalyzed.
ZH: 成功就在 过程中。我 们庆祝和分析失败。
( Success is in the process. We celebrate and analyze
failure. )In this paper, we propose to train an NMT model for nonliteral translations from existing bitext. We obtain training
data specialized in non-literal translations by ranking parallel sentences in the bitext in terms of non-literalness andsplitting them into two sets in which one set is likely to
be more non-literal than the other. Regarding the nonliteralness as a kind of domain, we then apply domain
adaptation techniques to train an NMT model specialized
for non-literal translations using the training data. Speciﬁcally, we utilize multi-domain learning [7] and curriculum
learning [8] for training. The former is to add domain signals for non-literal and literal sets of the training data andmix them to ﬁne-tune a pre-trained language model. The
latter is to ﬁne-tune a pre-trained model ﬁrst on the literal
set and next on the non-literal set.
We evaluate our method on English-to-Chinese translation using OpenSubtitles and TED2020 datasets. To evaluate our method, we build a specialized evaluation dataset
by manually extracting non-literal translations to ﬁnd that
our methods achieved even better results when we forced
the models to generate non-literal translations. Experimental results on this specialized dataset show that the models
trained by our methods perform better than the baseline ingenerating translations that are natural for native speakers.

― 648 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

2scores surrounded by ones with high alignment scores.

Related WorkSpeciﬁcally, given a translation pair ⟨𝑒, 𝑓 ⟩ and its previ-

Non-literal translations have been long studied by translation theorists and linguists [9, 6]. Several studies haveous and next translations ⟨𝑒 prev , 𝑓prev ⟩, and ⟨𝑒 next , 𝑓next ⟩, weused the value ofvecalign (𝑒, 𝑓 )vecalign (𝑒 prev , 𝑓prev ) + vecalign (𝑒 next , 𝑓next )been done with diﬀerent motivations to detect non-literaltranslations in existing corpora. Chen et al. [5] aims tomine lexical and phrasal non-literal translations for human translators’ reference and to inspire improvements infor ranking.
NMT-based score

As NMT models are likely to gen-machine translation. The authors designed an algorithm

erate literal translations, the outputs of an oﬀ-the-shelfbased on attention scores for searching possible non-literal

NMT model 1） can be used to ﬁnd possibly non-literaltranslations in a bilingual corpus of Chinese and English.

translations. We assume that non-literal translations are

Several studies have been done for automatically detecting

semantically similar but superﬁcially dissimilar to literal

non-literal translations in existing corpora [10, 6, 4] fortranslations. Therefore, we computed and combined dif-

non-literal translations can bring diﬃculties for automaticword alignment, and the training of NMT systems [11].

ferent evaluation metrics as scores to rank translation pairs.
Speciﬁcally, given a parallel sentence, a pair of source

For example, Zhai et al. [4] ﬁrst annotated parallel corpora
with categories of non-literal translations and then con-sentence 𝑒 and target sentence 𝑓 , and a translation obtained by the oﬀ-the-shelf NMT model, 𝑓ˆ, we computed

ducted experiments on the detection of phrasal non-literal

BERTScore [13] and chrF [14] and used the value oftranslations. These studies focus on detecting non-literal−translations, whereas our study focuses on generating nonliteral translations.

for ranking.

3

3.2

Proposed Methods

chrF(f, f̂)BERTScore(f, f̂)

Training NMT Models

In this section, we ﬁrst describe our methods of ﬁnding

We design two strategies to train a non-literal NMT

possibly non-literal translations from existing bitext. We

model using the obtained possibly non-literal and literalthen use these possibly non-literal translations as the target-translations. We adopt a pre-trained 𝑚𝑇5 [15, 16] as adomain training data to perform multi-domain learning [7]backbone of our NMT models.

or curriculum learning [8].

3.1Multi-domain Learning (MDL)

We regard non-literaland literal translations as data from two domains and perform multi-domain learning (MDL) using all the data.

Ranking Translation Pairs

To begin with, we propose unsupervised and semi-

Speciﬁcally, we adopt domain-token mixing [7], which

supervised methods of ranking translation pairs of existing

predicts a domain token (here, <NLT> and <LT> for non-parallel corpora. We use the resulting rankings to obtain

literal and literal translations), before decoding a target

possibly non-literal and literal translations while changing

sentence. This enables us to train the model to predict thetheir proportion as a hyperparameter.

type of translations (non-literal or literal) and generate a

Alignment-based score

Since non-literal transla-

tions can bring diﬃculties for automatic sentential align-translation conditioned on the prediction.
Curriculum Learning

Another idea is to regard non-ment [11, 4], we assume that parallel sentences with low

literal translations as translations that are more diﬃcult

alignment scores are more non-literal translations. Given ato produce than literal ones. Then, following the ideatranslation pair < 𝑒, 𝑓 >, we use Vecalign [12] to computethe alignment cost 𝑣𝑒𝑐𝑎𝑙𝑖𝑔𝑛(𝑒, 𝑓 ) to extract possibly non-of curriculum learning [8, 17, 18] to mimic the humaneducation process, we ﬁrst ﬁnetune the model to generate

literal translations with high cost values. However, such

literal translations using the possibly literal translations andtranslations can also be noises or low-quality translations.
To reduce such cases, we design an additional score tothen to generate non-literal translations using the possibly
non-literal translations.

give higher ranks to parallel sentences with low alignment1） https://huggingface.co/K024/mt5-zh-ja-en-trimmed― 649 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

4

Experiments
We apply our methods to English-to-Chinese translation

using OpenSubtitles and TED2020 datasets. We evaluate
the ranking methods for non-literal translations and the
proposed non-literal NMT models obtained by our ranking
and training methods.

4.1

Settings

Data

Figure 1

We train and evaluate our methods on the whole

Top-N Precision of non-literal translations on Open-

Subtitles.

dataset of TED2020 on the English-Chinese language pair.
We prepare the data by ﬁrst doing some simple noise ﬁltering and then splitting the dataset by the ratio of 90 : 5 : 5
for training, validation, and test, respectively. As a result,
we obtained about 246𝑘, 14𝑘, and 14𝑘 sentence pairs forthe three parts, respectively.
As for the training data, we utilize the ranking methods
introduced in § 3.1 to sort the translation pairs and take the
top 𝑋% of data as the non-literal part and the rest as theliteral part where 𝑋 is a hyperparameter.
As for the test data, we additionally manually annotatedpart of the translation pairs and picked out actual non-literal

Figure 2

Top-N precision of non-literal translations on

TED2020.

10% of entire data and manually annotate non-literal ones
among them. We then compute top-𝑁 precision 𝑃 𝑁 ofby the proposed methods. Currently, this specialized test

non-literal translations as below.
#non-literal at rank 𝑁
𝑃𝑁 =𝑁
As for non-literal NMT models, we adopt BLEU [19]set contains 222 non-literal translation pairs, which will be

and COMET-22-DA [20, 21] scores. We utilize Sacre-

further expanded in the future.

BLEU [22] to compute BLEU scores with reference trans-

ones to form a specialized test set to evaluate the ability togenerate non-literal translations of the NMT models trained

To evaluate ranking methods of translation pairs, due tolimited time and labor for manual annotation for evaluation,
we only took the ﬁrst 10% data of both datasets, which
results in about 1.2𝑀 sentence pairs of OpenSubtitles and
46𝑘 of TED2020.

lations and predicted translations as inputs. As for COMET
scores, we use “Unbabel/wmt22-comet-da”2）model with
inputs of source sentences, reference translations, and predicted translations.
Training details

We choose the mT5-small model as

We ﬁnetune 𝑚𝑇5 [15, 16] to obtain diﬀerent

the base model and implement diﬀerent models by the

NMT models for non-literal translations using combina-

Transformers library of Huggingface [23]. We set theModelstions of ranking and training methods described in § 3.1

training epoch to 8 and save the best model checkpoints

and § 3.2. For each ranking method, we take top-𝑋% of

for ﬁnal evaluation, which have the lowest training loss.

data as the non-literal part and the rest as the literal part

As for models trained by curriculum learning, we addi-

and use the data for diﬀerent training methods. For mixed

tionally choose the best model trained on the literal part ofﬁne-tuning, we set the value of 𝑋 to 1, 5, 10, 20, and 40

data before training on the non-literal part. As for otherto train diﬀerent models, while for curriculum learning we

hyperparameters and settings, we keep them by default.

only set the value to 30 at the moment. As for baselines,
we ﬁnetuned a pre-trained mT5 model on the training data
we prepared.
Evaluation metrics

4.2

Results and Analysis

Results of Ranking Translations

As for ranking translation pairs,

we take the top-200 pairs of ranked translations from the

Figure 1 and 2

show the results of evaluating ranking methods for non2） https://huggingface.co/Unbabel/wmt22-comet-da― 650 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 2

Evaluation results of NMT models on the specialized test set of non-literal translations. ”𝑋%” indicates that the
top-𝑋% of translation pairs are used as the non-literal translations
and the rest as the literal ones in training.
Model

BLEU

COMET

Baseline
MDL, 1% non-literal by alignment
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%
MDL, 1% non-literal by NMT
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%
Curriculum, 30% non-literal by alignment
Curriculum, 30% non-literal by NMT

24.99
25.10
24.68
24.90
25.07
24.90
24.48
24.45
25.15
24.77
24.60
24.24
24.10

0.78680.78990.78000.78750.79070.78290.77950.78510.78580.78870.78570.77580.7785

Table 3

Evaluation results of NMT models on the specialized
test set when forced to generate non-literal translations.
Model

BLEU

COMET

MDL, 1% non-literal by alignment
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%
MDL, 1% non-literal by NMT%
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%

24.90
24.59
24.82
25.20
24.86
24.41
24.52
25.25
24.76
24.16

0.78670.77990.7906
0.79320.78180.77880.78590.78850.78880.7868

Table 4Generated non-literal translations by the MDL, 20%
and the MDL, 10% model. The English sentences are the source
sentences while the Chinese sentences are the forcedly generated
non-literal translations.
From MDL, 20%, non-literal by alignment-based scores

all precision, which resulted in 6.5% and 10.5% (shown

EN: It’s very expensive to drive that much, and as we’veseen, the middle class is struggling to hold on.
ZH: 开车非常昂 贵, 正如我 们所看到的, 中 产阶级挣扎不堪。
( Driving is very expensive, and as we’ve seen, the middle
class is struggling unbearably. )as a horizontal dotted line in the ﬁgures) for OpenSubti-

From MDL, 10%, non-literal by NMT-based scores

tles and TED2020, respectively. We can see from the re-

EN: But the suggestion I want to put to you today is that there’s
something fundamentally wrong with this model.
ZH: 但是我今天想告 诉你们的是, 这个模型根本上是错误的。
( But what I want to tell you today is that this model isfundamentally wrong. )literal translations. As for the baseline, we randomly sample 200 sentence pairs, manually annotate non-literal ones,
and compute the ratio of non-literal translations as the over-sults that the proposed methods outperform the baseline on
both datasets, which proves our hypothesis that top-ranked
translation pairs are more likely to be non-literal.
Results of Non-literal NMT Models

As for training

non-literal translators, we get results by evaluating models
on the specialized non-literal test set and the overall test set.

5

Conclusions

The latter is shown in Table 2 while the former can be foundIn this paper, we propose to train an NMT model for pro-in the Appendix. In addition, we force models trained byducing non-literal translations by exploiting translations

multi-domain learning to generate non-literal translations
by using <NLT> at the start of decoding when evaluated onthat are possibly non-literal in existing corpora. We ﬁrst

the specialized test set. We report this part of the results inlation pairs that are possibly non-literal. We then adopt

Table 3. We can see from the results in Table 2 and 3 that
the MDL, 20% and MDL, 10% models perform bettertwo domain adaptation techniques. One is to add domain
signals to the non-literal and literal parts of the data andthan other models. In addition, when forced to generate

train the model by multi-domain learning. The other is to

non-literal translations, they perform even better, whichﬁrst train the model on the literal part of data and second

proves the eﬀectiveness of our methods for training NMT

on the non-literal part by curriculum learning. The resultsmodels for non-literal translations.

show that our methods eﬀectively improve the NMT model

Examples of Generated Translations

We present

rank translation pairs by diﬀerent scores and extract trans-

for non-literal translations.

some generated samples of non-literal translators as shown

As for future work, we plan to conduct further experi-in Table 4. We can see from the examples that the gen-ments with improved methods and on other parallel cor-erated translations have common features of non-literalpora, such as OpenSubtitles.

translations in that they are ﬂuent and natural but havesome semantic divergences with the source sentences.

― 651 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgement
This work was partially supported by the special fundof Institute of Industrial Science, The University of Tokyo,
by JSPS KAKENHI Grant Number JP21H03494.

References[1] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neuralmachine translation: Encoder-decoder approaches. arXiv
preprint arXiv:1409.1259, 2014.
[2] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence
to sequence learning with neural networks. Advancesin neural information processing systems, Vol. 27, ,2014.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align andtranslate. arXiv preprint arXiv:1409.0473, 2014.
[4] Yuming Zhai, Gabriel Illouz, and Anne Vilnat. Detecting
non-literal translations by ﬁne-tuning cross-lingual pretrained language models. In Proceedings of the 28th
International Conference on Computational Linguistics, pp. 5944–5956, 2020.

[5] Qi Chen, Olivia OY Kwong, and Jingbo Zhu. Detecting
free translation in parallel corpora from attention scores.
In Proceedings of the 32nd Paciﬁc Asia Conferenceon Language, Information and Computation, 2018.
[6] Yuming Zhai, Lufei Liu, Xinyi Zhong, Gbariel Illouz, andAnne Vilnat. Building an English-Chinese parallel corpus annotated with sub-sentential translation techniques.
In Proceedings of the Twelfth Language Resources
and Evaluation Conference, pp. 4024–4033, 2020.
[7] Denny Britz, Quoc Le, and Reid Pryzant. Eﬀective domain
mixing for neural machine translation. In Proceedings of
the Second Conference on Machine Translation, pp.
118–126, 2017.
[8] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, andJason Weston. Curriculum learning. In Proceedings ofthe 26th annual international conference on machinelearning, pp. 41–48, 2009.

[9] Lucı́a Molina and Amparo Hurtado Albir. Translation techniques revisited: A dynamic and functionalist approach.
Meta, Vol. 47, No. 4, pp. 498–512, 2002.
[10] Yuming Zhai, Pooyan Safari, Gabriel Illouz, Alexandre
Allauzen, and Anne Vilnat. Towards recognizing phrasetranslation processes: Experiments on english-french.
arXiv preprint arXiv:1904.12213, 2019.
[11] Dun Deng and Nianwen Xue. Translation divergences inchinese–english machine translation: An empirical investigation. Computational Linguistics, Vol. 43, No. 3, pp.
521–565, 2017.
[12] Brian Thompson and Philipp Koehn. Vecalign: Improved
sentence alignment in linear time and space. In Proceed-[13] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[14] Maja Popović. chrf: character n-gram f-score for automatic
mt evaluation. In Proceedings of the tenth workshop
on statistical machine translation, pp. 392–395, 2015.
[15] Colin Raﬀel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li,
and Peter J Liu. Exploring the limits of transfer learningwith a uniﬁed text-to-text transformer. The Journal of
Machine Learning Research, Vol. 21, No. 1, pp. 5485–
5551, 2020.
[16] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale,
Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and ColinRaﬀel. mt5: A massively multilingual pre-trained textto-text transformer. arXiv preprint arXiv:2010.11934,
2020.
[17] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on
curriculum learning. arXiv preprint arXiv:2010.13166,
2020.
[18] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu
Sebe. Curriculum learning: A survey. International Journal of Computer Vision, Vol. 130, No. 6, pp. 1526–1565,
2022.
[19] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing
Zhu. Bleu: a method for automatic evaluation of machinetranslation. In Proceedings of the 40th annual meetingof the Association for Computational Linguistics, pp.
311–318, 2002.
[20] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. Comet: A neural framework for mt evaluation.
arXiv preprint arXiv:2009.09025, 2020.
[21] Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula
Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie,
Luisa Coheur, and André FT Martins. Comet-22:
Unbabel-ist 2022 submission for the metrics shared task. In
Proceedings of the Seventh Conference on MachineTranslation (WMT), pp. 578–585, 2022.

[22] Matt Post. A call for clarity in reporting BLEU scores.
In Proceedings of the Third Conference on MachineTranslation: Research Papers, pp. 186–191, Belgium,
Brussels, October 2018. Association for Computational
Linguistics.
[23] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac,
Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language
processing. arXiv preprint arXiv:1910.03771, 2019.

ings of the 2019 conference on empirical methods
in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 1342–1348, 2019.

― 652 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 5

Evaluation results of NMT models on the overall test

set.
Model

BLEU

COMET

Baseline
MDL, 1% non-literal by alignment
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%
MDL, 1% non-literal by NMT
MDL, 5%
MDL, 10%
MDL, 20%
MDL, 40%
Curriculum, 30% non-literal by alignment
Curriculum, 30% non-literal by NMT

23.97
24.06
23.96
23.92
24.15
24.06
24.02
23.82
23.92
24.15
24.23
23.45
23.41

0.78090.78030.77990.7816
0.78120.78120.78130.78060.77990.7816
0.78230.77240.7742

A

Appendix

A.1

Results on Overall Test Set

The results of evaluating non-literal translators on theoverall test set are shown in Table 5. As for proposedmodels, the MDL, 40% model performs relatively the best,
which is diﬀerent from the results on the specialized test
set. This diﬀerence may indicate that the non-literal neuralmachine translation can be considered as an individual
and important direction besides other sub-tasks of neuralmachine translation.

― 653 ―

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).