Google mT5 multilingual text-to-text transformer: A Brief Paper Analysis
By: Parth Chokhra
Tags: Programming, Technology, Data Science, Artificial Intelligence, Machine Learning

Unlike the C4 dataset, which was explicitly designed in such a way that any page whose probability is below 99% of being English would be discarded when detected by langdetect,mC4 uses cld3 to detect over 100 languages. ( すごい 🤩)

For anyone curious how this C4 and mC4 originally data is collected you can check out Common Crawl. It is an open-source text data corpus consisting of petabytes of data collected since 2008 scraped from the internet. It contains raw web page data, extracted metadata and text extractions.

Apart from this, some preprocessing steps were applied to mC4 dataset to maintain data quality while training.

A “line length filter” that requires pages to contain at least three lines of text with 200 or more characters.

cld3 was used to detect each page’s primary language was and discard any page whose confidence score less than 70%.

After these filters, the pages were grouped by languages and include in the corpus of all languages.

Pretrained Model: mT5

The model architecture of mT5 is quite similar to that of T5(In mT5 authors used GeGLU instead of RELU).

One of the major tasks in pre-training a multilingual model is; how do we sample data from each language(Since there is quite a disparity in the size of the dataset collected for each language). The authors used the term “zero-sum game” for this problem. Let's take a little detour to learn what is Zero-sum😅;(you can skip this part!)

Zero-sum is a situation in game theory in which one person’s gain is equivalent to another’s loss, so the net change in wealth or benefit is zero. A zero-sum game may have as few as two players or as many as millions of participants. In financial markets, options and futures are examples of zero-sum games, excluding transaction costs. For every person who gains on a contract, there is a counter-party who loses.

So coming back mT5, the main issue here is if they sampled a low resource language too often, the model would end up overfitting it; but if high-resource languages are not trained on enough the model will remain under fitted on them.

Therefore the authors to boost lower-resource languages use the following technique.

p(L) ∝ |L| ^(α),where

p(L) is the probability of sampling text from a given language during pre-training,

|L| is the number of examples in the language, and

α (typically with α < 1) is a hyperparameter that allows us to control how much to “boost” the probability of training on low-resource languages.

Finally, α = 0.3 is a reasonable compromise between high and low resource languages.

Other multilingual related models

Now that we have explored mT5 thoroughly let’s compare it with other existing massively multilingual pretrained language models.

Photo by Annie Spratt on Unsplash

mBERT is one such model. Similar to mT5, mBERT show strong resemblance to BERT.The primary difference in BERT and mBERT is the training dataset. Instead of training on English Wikipedia and the Toronto Books Corpus, mBERT is trained on up to 104 languages from Wikipedia.XLM is also based on BERT but includes improved methods for pre-training multilingual language models.

XLM-R was later released which is based on Roberta model. It is trained with a cross-lingual masked language modeling objective on data in 100 languages from Common Crawl.

mBART was the first sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in 25 languages using the BART objective. In terms of similarity, mBART is very similar to mT5.

Experimentations and Results

To validate the performance of mT5 it is been evaluated on XTREME, XNLI(consisting of task covering 14 languages), the XQuAD, MLQA and TyDi QA.

The largest model mT5-XXL reaches state-of-the-art on all of the task's authors considered. Below is the table showing the performance of mT5 and compared to other models on similar benchmarks.

So here I conclude my brief blog on mT5. Hope you find this an interesting read. If you have any suggestions on any research paper or Machine learning-based topic for you need a blog post for me, do comment.

References