Multilingual Machine Learning. Exploring BLEU Scores using Patent Data | by Lee Mackey | Towards Data Science

Does your machine learn in Chinese? I don’t speak Mandarin or Cantonese so Google Translate gets all the credit — good or bad — for translating the preceding sentence into “您的機器學習中文嗎?” But how might a researcher quickly evaluate the quality of machine translations? This question encapsulates the basic challenge that gives rise to the BLEU metric. BLEU, which stands for bilingual language understudy, is a default measure of machine translation quality and is also sometimes applied to cross-lingual natural language processing (NLP) tasks. The metric is well-established in the machine translation space but some analysts question the algorithm’s applicability to a wider set of tasks beyond the measure’s original objectives. This article takes an initial dive into the lessons, implementations, and limits of BLEU using examples drawn from multilingual patent documents.
Researchers at IBM developed the BLEU algorithm in 2002 as an efficient method to evaluate the quality of machine translation in reference to benchmark human translations. The original paper by the developers, Papineni, and colleagues, is a good place to start if you’re interested in the founding context and objectives of the algorithm. BLEU is an adjusted precision measure of the matching word sequences between a “candidate” machine translation and one or multiple “reference” human translations. BLEU counts “n-grams”, a term for word sequences of length n, in a machine translation that match the n-grams in a human translation, divided by the total count of n-grams in the machine translation. The measure is adjusted in that it clips the match count to the maximum number of n-gram occurrences in a human translation and also penalizes machine translations that diverge in word length from the reference translation.
The resulting BLEU score is a number between 0 and 1, in which 0 represents zero n-gram matches between candidate and reference texts and 1 might equal a machine translation that is exactly similar to one of the references. In practice, the measure counts matches across multiple word sequence lengths — 4-grams (four-word sequences), tri-grams (three-word sequences), bi-grams (two-word sequences), and uni-grams (one-word sequences) — via a geometric mean of the respective n-gram calculations. The algorithm was designed for comparisons at the level of a corpus of sentences, with n-gram matches calculated at the basic unit of a sentence and then combined into a corpus-level score. To clarify terminology, the use of the term “document” in the present article refers to a corpus of sentences. If you’re interested in additional resources to understand the algorithm, you might check out the video tutorial at deeplearning.ai that discusses the details of the algorithm or the written tutorial at machinelearningmastery.com that explores the NLTK implementation. To explore the tangible use cases of the metric, I next apply BLEU using translations of Chinese-language patents.
A growing share of patents in the machine learning space is written and filed in Chinese according to a recent report by WIPO (World Intellectual Property Organization), the global organization that governs patents. To explore the basics of BLEU in this multilingual space, you can first begin with a domestic Chinese patent of an NLP innovation that the e-commerce company Alibaba extended to global coverage. The title of the Chinese language patent is displayed below. For additional details surrounding the example patent,
original_title_cn = "机器处理及文本纠错方法和装置、计算设备以及存储介质"
WIPO’s data query tool provides both English and Chinese language versions that you can inspect in your browser.
Human specialists often produce translations of equivalent quality that nonetheless differ in word choice and structure. BLEU, therefore, accepts single or multiple human translations as the references for comparison. To explore the production of reference translations more directly, I obtain two Chinese-to-English translations of the Chinese language patent from two different human translators via Gengo, a web-based platform for human translators. Next, you can source “candidate” machine translations from two
reference_human1_summary = "The invention discloses a machine processing and text error correction method and device, a computing device, and a storage medium, specifically comprising corrected and rewritten text pairs of incorrect text and corresponding correct text. The corrected and rewritten text pairs serving as a training corpus to train the machine processing model, thereby preparing a machine processing model suitable for text error correction. Through extraction of corrected and rewritten text pairs from a log, the machine processing model can be trained and thus made fit for text correction by inputting the first text into the machine processing model to get the second text, that is the error correction result text. In addition, the language model or the common lexicon can be used to determine whether the first text needs to be corrected. The training corpus extracted from a log can be used to train the language model, or the common lexicon can be sorted by segmenting and counting text in the log. This is how to easily implement text error correction."reference_human2_summary = "This invention makes public a machine processing and text error correction method and hardware, computing equipment and storage medium, and specifically pairs error text with the corresponding corrected and modified correct text. It uses this text pair as training material for the machine processing model, and from there prepares the machine processing model that is applied to the text correction. It can train the machine processing model using a diary or daily journal and make it suitable for text correction. The first text version is inputted into the machine processing model to get the second text version, which is the corrected text. Additionally, it can also use a stored language model or common vocabulary bank to determine if the first text version needs correction. It can use the practice language material gathered from the diary or daily journal to train the language model, and it can also initialize the common vocabulary bank through the segmentation and analysis of the diary or daily journal text. Through all this, text correction is conveniently implemented."
separate machine learning algorithms, Google Translate and the World Intellectual Property Organization (WIPO), which are displayed below.
candidate_google_summary = "The invention discloses a machine processing and text error correction method and device, a computing device and a storage medium, and particularly comprises an error correction rewriting pair of an error text and a corresponding correct text, and an error correction rewriting pair as a training corpus, and a machine processing model. Training is performed, thereby preparing a machine processing model suitable for text correction. The machine processing model can be trained to mine the error correction by mining the error correction rewrite pair from the log. The first text is input into the machine processing model to obtain a second text, that is, an error correction result text. In addition, you can use the language model or common lexicon to determine whether the first text needs to be corrected. The language model can be trained using the training corpus extracted from the log, or the common lexicon can be organized by segmenting and counting the text in the log. Thereby, text correction is facilitated."candidate_wipo_summary = "The present invention discloses a machine processing and text correction method and device, computing equipment and a storage medium. Specifically comprising corrected and rewritten text pairs of incorrect text and corresponding correct text, the corrected and rewritten text pairs serving as a training corpus for training a machine processing model, and in this way developing a machine processing model for use in text correction. Through extraction of corrected and rewritten text pairs from a log, the machine processing model can be trained and thus made fit for text correction by inputting a first text into the machine processing model to obtain a second text i.e. a corrected text result. Moreover, a language model or a lexicon of commonly used words can be used to assess whether text needs correction. The training corpus extracted from the log can be used to train the language model and also, through text segmentation and statistical analysis of text in the log compile a lexicon of commonly used words. Thus, text correction can be made easier and more convenient."
There are multiple implementations and extensions of BLEU, such as the popular sacreBLEU package, among others. The present example begins by calculating scores using the bleu_score module in the Natural Language Toolkit (NLTK), which is based on the original implementation by Papineni et al. The corpus_bleu function accepts reference and candidate translations represented as individual word tokens that you can create through standard NLP pre-processing steps. For a document with multiple sentences, the original BLEU implementation computes the n-gram matches sentence by sentence, then sums the clipped n-gram counts for all the candidate sentences and, lastly, divides by the number of candidate n-grams in the document. As the NLTK documentation states: the BLEU metric “calculates the micro-average precision (i.e. summing the numerators and denominators for each hypothesis-reference(s) pairs before the division).” By default, corpus_bleu calculates the geometric mean of equally-weighted n-gram scores comprising 4-grams, tri-grams, bi-grams, and unigrams.
What are the results of the BLEU score calculations for the two machine translation examples? The results indicate that the Google and WIPO translations are of similar quality in relation to the reference translations from human translators on Gengo. The score of the first candidate translation by Google Translate is 0.53 and the score of WIPO is 0.54, as displayed below.
# calculate BLEU score of Google translationbleu_google = round(corpus_bleu(refs_list_6, can_google_summary[:]),                             2)# calculate BLEU score of WIPO translationbleu_wipo = round(corpus_bleu(refs_list_5, can_wipo_summary[:]), 2)print(f"Google Translate BLEU score: {bleu_google}")print(f"WIPO BLEU score: {bleu_wipo}")
With this tangible example of the application of BLEU in mind, it is helpful to establish the potential scope of the algorithm’s applicability to machine translation and natural language processing tasks.
There is a general agreement among researchers that BLEU is expedient for evaluations of machine translation systems. However, some researchers caution that BLEU may not be appropriate for certain aspects of machine translation or wider NLP tasks. A key critique is that BLEU scores may not necessarily correlate well with significant quality differences in human translations. As Callison-Burch and colleagues argue in a highly-cited paper: “there are instances when an improvement in BLEU is not sufficient to reflect a genuine improvement in translation quality, and in other circumstances that it is not necessary to improve BLEU to achieve a noticeable improvement in translation quality.” These authors also suggest that BLEU may not be appropriate for comparing machine translation systems that utilize different techniques, detecting aspects of translation that BLEU doesn’t model, or evaluating infrequent improvements within a test corpus.
Researchers that conduct a literature review of BLEU argue in a recent paper that BLEU–human correlations may vary with the details of the systems being evaluated, the particular corpus texts used, and the exact protocol utilized for human evaluations. The authors, Reiter and colleagues, suggest that BLEU should be evaluated by real-world human evaluations such as A/B tests to determine contexts in which BLEU reliably correlates with real-world effectiveness. Based on the results of the literature review, these authors caution that researchers should approach BLEU as a diagnostic for machine translation at the system level but not as an evaluation technique to measure the output of a system. With the literature suggesting a general recognition of both the utility and shortcomings of the measure, some researchers, such as the author of a recent paper, focus on increased clarity in the reporting of BLEU scores. This would include, for example, having researchers include preprocessing schemes and other standardized conventions for reporting the implementation and scoring details of BLEU scores. The points raised in these papers sound notes in caution in applying and interpreting BLEU scores in the patent space and across different use cases.
This article outlines objectives, examples, and limits of BLEU scores. To run the full Python code that accompanies this article in a Jupyter notebook, you can head to the Github repository for Multilingual Machine Learning. For more background on my interest in this space, check out my website at lee-mackey.com. As patent documents in the machine learning space continue to become increasingly multilingual, particularly from Chinese language patents, the points raised in this article hopefully help you to surface and contextualize considerations when applying this algorithm to your particular use cases. If you’re working across languages in your NLP workflows, understanding these details of BLEU will help to select if, when, and how to use this metric in your projects.