Venue: NAACL 2021

# Machine Translated Text Detection Through Text Similarity

with Round-Trip Translation

Hoang-Quoc Nguyen-Son\({}^{1}\), Tran Phuong Thao\({}^{2}\), Seira Hidano\({}^{1}\), Ishita Gupta\({}^{3}\), and Shinsaku Kiyomoto\({}^{1}\)

###### Abstract

Translated texts have been used for malicious purposes, i.e., plagiarism or fake reviews. Existing detectors have been built around a specific translator (e.g., Google) but fail to detect a translated text from a strange translator. If we use the same translator, the translated text is similar to its round-trip translation, which is when text is translated into another language and translated back into the original language. However, a round-trip translated text is significantly different from the original text or a translated text using a strange translator. Hence, we propose a detector using text similarity with round-trip translation (TSRT). TSRT achieves \(86.9\%\) accuracy in detecting a translated text from a strange translator. It outperforms existing detectors (\(77.9\%\)) and human recognition (\(53.3\%\)).

\({}^{1}\)KDDI Research, Inc., Japan

\({}^{2}\)The University of Tokyo, Japan

\({}^{3}\)Indian Institute of Technology, India

\({}^{1}\){ho-nguyen,se-hidano,kiyomoto}@kddi-research.jp

\({}^{2}\)tpthao@yamagula.ic.i.u-tokyo.ac.jp

\({}^{3}\)ishita.gupta.ee317@ee.iitd.ac.in

## 1 Introduction

A reader may misunderstand the original meaning of a translated text1. For example, Facebook translated "_good morning_" into "_attack them_," leading to an arrest2. Adversaries can use a translator for malicious tasks such as _round-trip translation_ used in plagiarism Jones and Sheridan (2015) to avoid human recognition or in adversarial text Iyyer et al. (2018) to fool AI.

Footnote 1: When we mention a translated text, translation, translator, and Google, all are related to machine translation systems

Footnote 2: www.theguardian.com/technology/2017/oct/24facebook-palestite-isreal-ternalates-good-morning-attack-them-arrested-mergant-to-all

Existing work has investigated the detection of translated texts in various approaches. The parse tree approach Chae and Nenkova (2009); Li et al. (2015) exploits text structure. The \(N\)-gram approach Aharoni et al. (2014); Arase and Zhou (2013) estimates text fluency. The text complexity approach uses complex words Nguyen-Son and Echizen (2017) and phrases Nguyen-Son et al. (2017). The text coherence approach is based on matching similar words on a paragraph level Nguyen-Son et al. (2018, 2019b). A three-layer CNN Riley et al. (2020) is trained on either one-way or round-trip translated texts. Our previous work Nguyen-Son et al. (2019a) combined round-trip translation with \(BLEU\) scores. All these approaches fail to detect a text translated by another translator or from a different language.

MotivationThe first translation round induces a low similarity between the translated and original texts, whereas the extent of similarity increases in later rounds Vanmassenhove et al. (2019). Let us consider an example in Fig. 1. We randomly selected an English text \(t\) from an English-Russian pair3; the Russian text was translated into English by Google, called \(t^{(Go,RU\to EN)}\). We measured the similarity between a text and its round-trip translation using the minimum edit distance (\(MED\)Levenshtein (1966). The translated text \(t^{\prime}\) is the result of using the translator once, and the similarity between \(t^{\prime}\) and its round-trip translation \(t^{(Go,RU\to EN\to RU)}\) is high (\(MED=1\)). Otherwise, the similarity between the original text \(t\) with \(t^{(Go,RU\to EN\to RU)}\) is low (\(MED=5\)). Based on the difference in similarity, we can distinguish the original from the translated text.

Footnote 3: This pair belongs to a Commentary News corpus Barrault et al. (2019)

In reality, a translator's source language is often unknown. The similarity decreases when using another language. For example, the similarity between \(t^{\prime(Go,RU\to EN)}\) translated from Russian and its round-trip translation \(t^{\prime(Go,RU\to EN\to DE\to EN)}\) from German is low (\(MED=6\)). It is close to the simila

\(\{t,t^{(Go,EN\to DE\to EN)}\}~{}(MED=4)\). A change in a translator induces a similar phenomenon. We thus detected the translator and the language before detecting the translated text.

ContributionsWe propose a novel translation detector that utilizes text similarity with round-trip translation (named TSRT). This detector can be used as a warning to prevent the risk of translated texts in a certain region where people are familiar with few languages and translators. First, we create round-trip translations from multiple configuration translator and language tuples. Second, we use each tuple's round-trip translations to train individual subclassifiers. Then, we use the tuple with the highest similarity between a suspicious text and its round-trip translation to choose a suitable subclassifier. Finally, we use the subclassesifer to determine if the text is an original or translated text. Experiments demonstrate that TSRT efficiently detects different kinds of translated texts (round-trip and one-way) when the translation translator and language is changed.

## 2 Text Similarity with Round-Trip Translation

Training PhaseFirst, we collect original texts \(T_{i}\) and translated texts \(T^{\prime}_{i}\), which are translated with a configuration tuple \(\pi_{i}=\{\text{language $\lambda_{i}$},\) translator \(\tau_{i}\}\) (see Fig. 2). Second, we generate round-trip translations \(T^{\pi_{i}}_{i}\) and \(T^{\prime\pi_{i}}_{i}\) for \(T_{i}\) and \(T^{\prime}_{i}\), respectively. Finally, \(T_{i}\) and \(T^{\prime}_{i}\) are combined with \(T^{\pi_{i}}_{i}\) and \(T^{\prime\pi_{i}}_{i}\) to train a subclassifier \(\chi^{\pi_{i}}\) by fine-tuning the BERT model Devlin et al. (2019). We repeat the procedure with other subclassifiers. In Fig. 1, \(t\), \(t^{\prime}\), \(t^{(Go,RU)}\), and \(t^{\prime(Go,RU)}\) belong to \(T\), \(T^{\prime}\), \(T^{(Go,RU)}\), and \(T^{\prime(Go,RU)}\), respectively, with \(\pi=(Go,RU)\).

Testing PhaseFor a suspicious text \(s\), we aim to determine if \(s\) is an original or a translated text. First, we generate round-trip translated texts \(s^{\pi_{i}}\) with all configuration tuples in the training phase. Next, we calculate the similarity \(\sigma^{\pi_{i}}\) between \(t\) and all \(s^{\pi_{i}}\) using the minimum edit distance (\(MED\)). Finally, we process \(s\) with the subclassifier associated with the best similarity \(\sigma_{b}\) corresponding to the lowest \(MED\). In the case of \(t^{\prime}\) in Fig. 1, two round-trip translations \(t^{\prime(Go,RU)}\) and \(t^{\prime(Go,DE)}\) are generated with respect to \(\sigma^{(Go,RU)}=1\) and \(\sigma^{(Go,DE)}=6\). The subclassifier \(\chi^{(Go,RU)}\) associated with the lower \(MED\) is chosen for classifying \(t^{\prime}\).

## 3 Evaluation

### Unchanged Translator and Language

Round-trip translation detection: We collected \(11,748\) distinct movie reviews from the Sentiment Treebank Socher et al. (2013) (19.1 words/review). We chose \(9,000\)/\(1,000\) reviews for training/developing and used the remaining pairs for testing. This ratio is reused in further experiments. We used the original reviews to generate round-trip translations by using configuration tuples of two translators and three languages (Table 1). In addition to Google, we chose Fairseq4Ng et al. (2019), the winner in the WMT'19 shared task. We compare TSRT5 with

Figure 1: Round-trip translations from an original text \(t\) and a translated text \(t^{\prime}\). The superscripts indicate the translator—Google(Go)—and the language—Russian(RU) or German(DE)—that are used to generate the round-trip translations.

 

existing methods using the accuracy metric (accuracy and \(F\)-score are equivalent in this balanced corpus). BERT and TSRT have the same optimized hyperparameters6. The first four methods do not work well with this parallel corpus. The round-trip translation Nguyen-Son et al. (2019a) based on \(BLEU\) and BERT Devlin et al. (2019) improves by approximately \(10\%\). TSRT provides the highest performance, as it captures round-trip information using deep learning.

Footnote 6: We optimize hyperparameters with recommended values from BERT (maximum size of \(128\), batch size of \(32\), learning rate of \(2e\)-\(5\), and epoch of \(3\)). Since the development accuracy is equivalent to the test accuracy, we use the test accuracy for further experiments.

We analyzed the text lengths of the top three detectors on the whole (Go,RU) test set (Fig. 3). BERT surpasses round trips in only short length ranges, while TSRT outperforms the others in all ranges.

_Human recognition_: We selected \(100\) random reviews from the test set for human recognition7. We sent them to \(14\) raters (6 were native English speakers), who decided whether each review was an original or a translated text. The average accuracy was \(53.3\%\) (\(55.0\%\) for the native speakers and \(52.0\%\) for the nonnative speakers), which was close to random. The low Fleiss' \(\kappa=0.13\) implied slight agreement in the native speakers' ratings. For nonnative speakers, \(\kappa\) was even lower (\(\kappa=-0.07\)). This indicates that the translated texts were indistinguishable by humans.

Footnote 7: The survey is available at [https://forms.gle/L8EkZxKuEH9Co3UB7](https://forms.gle/L8EkZxKuEH9Co3UB7).

_One-way translation detection_: We collected samples from the Commentary News corpus Barrault et al. (2019). We randomly selected \(11,748\) pairs with \(21.9\) words on average per sentence (same as the movie reviews). We experimented with two languages Russian and German) and two translators Google and Fairseq) (see Fig. 4). Since one-way translation is more challenging to detect, the accuracy is decreased for all methods. In the top three detectors, while BERT and round-trip translation yield unstable results, TSRT remains consistent.

### Changed Translator and Language

_Comparison_: Humans are familiar with limited languages and translators. Normally, they use their mother tongue and English (international language) and translate by choosing a popular translator such as Google or an open-source translator such as Fairseq. Table 2 presents the translation detection with translator and language changes. While the existing methods are trained with (Go,DE) or (Fa,RU), TSRT is trained on (Go,DE)+(Go,RU) or (Fa,RU)+(Go,RU), respectively. We tested all of them in (Go,RU). Our results showed that the existing methods were significantly downgraded in terms of accuracy, but TSRT remained stable.

Figure 3: Top three round-trip translation detectors.

Figure 2: Text similarity with round-trip translation process (training phase: solid lines, testing phase: dashed lines).

 

_Ablation Studies_: We trained TSRT on various configuration tuples and tested it on (Go,RU) (Table 3). Training TSRT on the combination with the correct configuration tuple (Go,RU) boosts the performance.

_Configuration identification_: We identify the translator and language on round-trip translation detection while the one-way approach obtains similar results. For translator change (Table 4's second column), we used (Go,RU) and (Fa,RU). For the language change (the third column), we used (Go,RU) and (Go,DE). All were tested on (Go,RU). We used BERT as the identification baseline. We replaced \(MED\) with \(BLEU\) in TSRT. All the metric-based approaches outperformed the baseline. The translator detection outperformed language detection. While a specific translator often uses the same architecture for all languages, various translators have different architectures. Therefore, a translator change was more apparent than a language change. \(MED\) (designed for structure similarity) was better than \(BLEU\) (designed for corpus levels).

## 4 Conclusion

This paper proposed a one-way and round-trip translation detection mechanism using text similarity with round-trip translation (TSRT), which is robust to language and translator changes. First, we trained subclassifiers on specific lan

\begin{table}
\begin{tabular}{l c c c c c} \hline
**Method** & **(Go,RU)** & **(Fa,RU)** & **(Go,DE)** & **(Fa,DE)** & **(Go,JA)** \\ \hline Complexity [25] & \(52.7\) & \(54.9\) & \(52.2\) & \(51.5\) & \(53.6\) \\ Parse tree [10] & \(58.3\) & \(55.5\) & \(56.0\) & \(53.6\) & \(58.1\) \\ Coherence [25] & \(60.7\) & \(60.1\) & \(57.7\) & \(55.0\) & \(62.4\) \\ \(N\)-gram [1] & \(74.7\) & \(69.0\) & \(68.0\) & \(64.9\) & \(72.6\) \\ Round trip [25] & \(86.4\) & \(82.2\) & \(82.9\) & \(83.8\) & \(80.3\) \\ BERT [1] & \(85.2\) & \(80.4\) & \(77.7\) & \(72.9\) & \(86.8\) \\ \hline TSRT & \(\mathbf{90.2}\) & \(\mathbf{87.6}\) & \(\mathbf{85.5}\) & \(\mathbf{85.2}\) & \(\mathbf{89.8}\) \\ \hline \end{tabular}
\end{table}
Table 1: Round-trip translation detection with a combination of a translator—Google(Go) or Fairseq(Fa)—and a language—Russian(RU), German(DE), or Japanese(JA).

Figure 4: Detecting one-way translation with various translators and languages.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt}} \hline
**Training data** & **Acc(Red.)** \\ \hline Go,RU) & \(\mathbf{90.2(-00.0)}\) (Fa,RU) & \(70.2(-20.0)\) (Fa,RU) & \(70.2(-20.0)\) (Ga,DE) & \(73.4(-16.8)\) (Fa,DE) \\ \hline Go,RU)+(Fa,RU) & \(\mathbf{86.9(-03.3)}\) (Go,RU)+(Go,DE) & \(\mathbf{86.6(-03.6)}\) (Go,RU)+(Fa,RU)+(Go,DE)+(Fa,DE) 81.5(-08.7) \\ \hline \end{tabular}
\end{table}
Table 3: TSRT’s results with individuals and combinations of configuration tuples of translators and languages.

 

guages/translators using round-trip translation. Then, we identified the language and translator using the highest similarity between the suspicious and round-trip translation texts. Finally, we chose the corresponding subclassifier for translation detection. The evaluation results show that TSRT outperforms other methods, with an accuracy of up to \(90.2\%\). Moreover, TSRT could also identify the original translator and translation language with \(93.3\%\) and \(85.6\%\) of accuracy, respectively. In future work, we will exploit saturation after repeatedly using the same AI system to detect other artificial texts such as fake COVID-19 news.

## Acknowledgments

We would like to thank you very much for the anonymous reviewers to provide useful comments.

## References

* Aharoni et al. (2014) Roee Aharoni, Moshe Koppel, and Yoav Goldberg. 2014. Automatic detection of machine translated text and translation quality estimation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 289-295.
* Arase and Zhou (2013) Yuki Arase and Ming Zhou. 2013. Machine translation detection from monolingual web-text. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL)_, pages 1597-1607.
* Barrault et al. (2019) Loic Barrault, Ondrej Bojar, Marta R Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. 2019. Findings of the 2019 conference on machine translation. In _Proceedings of the 4th Conference on Machine Translation (WMT)_, pages 1-61.
* Chae and Nenkova (2009) Jieun Chae and Ani Nenkova. 2009. Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text. In _Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL)_, pages 139-147.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 4171-4186.
* Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_, pages 1875-1885.
* Jones and Sheridan (2015) Michael Jones and Lynnaire Sheridan. 2015. Back translation: an emerging sophisticated cyber strategy to subvert advances in 'digital age'plagiarism detection and prevention. _Assessment & Evaluation in Higher Education_, 40(5):712-724.
* Levenshtein (1966) Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In _Soviet physics doklady_, volume 10, pages 707-710.
* Li et al. (2015) Yitong Li, Rui Wang, and Hai Zhao. 2015. A machine learning method to distinguish machine translation from human translation. In _Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation (PACLCIc)_, pages 354-360.
* Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Asuli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. In _Proceedings of the 4th Conference on Machine Translation (WMT)_, pages 314-319.
* Nguyen-Son and Echizen (2017) Hoang-Quoc Nguyen-Son and Isao Echizen. 2017. Detecting computer-generated text using fluency and noise features. In _Proceedings of the International Conference of the Pacific Association for Computational Linguistics (PACLING)_, pages 288-300.
* Nguyen-Son et al. (2018) Hoang-Quoc Nguyen-Son, Huy H Nguyen, Ngoc-Dung Tieu, Junichi Yamagishi, and Isao Echizen. 2018. Identifying computer-translated paragraphs using coherence features. In _Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation (PACLIC)_.
* Nguyen-Son et al. (2019a) Hoang-Quoc Nguyen-Son, Thao Tran Phuong, Seira Hidano, and Shinsaku Kiyomoto. 2019a. Detecting machine-translated text using back translation. In _Proceedings of the 12th International Conference on Natural Language Generation (INLG)_, pages 189-197.
* Nguyen-Son et al. (2019b) Hoang-Quoc Nguyen-Son, Tran Phuong Thao, Seira Hidano, and Shinsaku Kiyomoto. 2019b. Detecting machine-translated paragraphs by matching similar words. In _ArXiv Preprint arXiv:1904.10641_.
* Nguyen-Son et al. (2017) Hoang-Quoc Nguyen-Son, Ngoc-Dung Tieu, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2017. Identifying computer-generated text using statistical

\begin{table}
\begin{tabular}{p{56.9pt} p{56.9pt} p{56.9pt}} \hline
**Method** & **Translator** & **Language** \\ \hline BERT & \(63.4\%\) & \(70.0\%\) \\ \hline \(BLEU\)-\(1\) & \(92.4\%\) & \(84.5\%\) \\ \(BLEU\)-\(2\) & \(92.3\%\) & \(84.6\%\) \\ \(BLEU\)-\(3\) & \(92.3\%\) & \(84.4\%\) \\ \(BLEU\)-\(4\) & \(92.2\%\) & \(85.0\%\) \\ \(MED\) & \(\mathbf{93.3}\%\) & \(\mathbf{85.6}\%\) \\ \hline \end{tabular}
\end{table}
Table 4: Translator and language identification.

 

analysis. In Proceedings of the 9th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1504-1511. Cited by: SS2.
* P. Riley, I. Caswell, M. Freitag, and D. Grangier (2020)Translationese as a language in multilingual" nmt. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7737-7746. External Links: Document, Link Cited by: SS2.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts (2013)Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642. Cited by: SS2.
* E. Vanmassenhove, D. Shterionov, and W. Way (2019)Local in translation: loss and decay of linguistic richness in machine translation. In Proceedings of the 17th Machine Translation Summit (MT Summit), pp. 222-232. Cited by: SS2.