Going Global —How to Multi-Task in Multiple Languages with the mT5 Transformer | by Thilina Rajapakse | Towards Data Science

The original T5 (Text-To-Text Transfer Transformer) model achieved state-of-the-art performance on a variety of NLP benchmarks by leveraging a unified text-to-text format and a gigantic training dataset (C4). With the unified text-to-text approach, all downstream tasks were reframed such that both the input and the output of the model are text sequences. At a whopping 750 GB, the C4 (Colossal Clean Crawled Corpus) dataset was orders of magnitude larger than most existing datasets. Released back in October 2019 by Google, T5 still sits pretty at the top of the SuperGLUE benchmark as a testament to its capabilities.
More information regarding the original T5 implementation and how to use it can be found in my article below.
towardsdatascience.com
As impressive as T5 was (and still is), it was trained entirely on English text and therefore, can only be used for English-language tasks. Sadly, this leaves out some 80% of the world population who don’t speak English.
The mT5 model is a multilingual variant of the original T5 model, aimed at remedying this problem. mT5 closely follows the architecture and the training procedure of T5 but is trained on mC4 (~26 Terabytes!), a multilingual variant of the C4 dataset. It retains all the advantages of T5, but it also supports a total of 101 different languages!
MT5 reframes any NLP task as a text-to-text task, which means that both the input and the output are text sequences. Just like with T5, the task to be performed can be specified by adding a prefix (a small text sequence) to the start of the sequence as shown below.
However, unlike T5, the mT5 model can also be trained on datasets in languages other than English.
Even more impressively, mT5 is also capable of cross-lingual, zero-shot transfer learning, i.e., the model is trained on an English dataset for a specific task, but it also learns to perform that task in other languages. This is quite useful when a certain dataset is only available in English, but you need it to work in another language (or multiple other languages)!
In this article, we will focus on the zero-shot transfer learning ability of mT5. We’ll also do a quick comparison of its performance against T5 on English data. We’ll be doing all of this with the Simple Transformers library.
In a previous article, I have trained a T5 model on the 3 tasks below:
Binary classification — Yelp Reviews DatasetMultilabel classification — Toxic Comments datasetSentence similarity (sentence pair regression) — STS-B dataset
Binary classification — Yelp Reviews Dataset
Multilabel classification — Toxic Comments dataset
Sentence similarity (sentence pair regression) — STS-B dataset
towardsdatascience.com
We’ll be using the same tasks and datasets to train our mT5 model so that we can easily make the comparison between mT5 and T5.
To test the zero-shot transfer learning, we’ll focus on the first task. We’ll translate each testing example of the Yelp dataset to five different languages and see how well the model performs on the translated data.
The five languages will be
DutchGermanFrenchSwedishSpanish
Dutch
German
French
Swedish
Spanish
We’ll be using MarianMT models to translate the text, which are also available through Simple Transformers!
Note: You can find all the code in this article in the examples/t5/mt5 directory (link) of the Simple Transformers Github repo.
Since we are going to be working with 3 datasets, we’ll put them in 3 separate subdirectories inside the data directory.
data/binary_classification
data/multilabel_classification
data/regression
Download the Yelp Reviews Dataset.Extract train.csv and test.csv to data/binary_classification.Download the Toxic Comments dataset.Extract the csv files to data/multilabel_classification.Download the STS-B dataset.Extract the csv files to data/regression.
Download the Yelp Reviews Dataset.
Extract train.csv and test.csv to data/binary_classification.
Download the Toxic Comments dataset.
Extract the csv files to data/multilabel_classification.
Download the STS-B dataset.
Extract the csv files to data/regression.
The task to be performed by an mT5 model is specified by the prefix prepended to the input. Because of this, the input data format for an mT5 model (or a T5 model) in Simple Transformers is a Pandas dataframe with the 3 columns — prefix, input_text, and target_text.
This makes it easy to combine all 3 of our datasets into a single dataframe, as we can simply assign a prefix value to each task and use that to differentiate between the 3 tasks.
In this case, the three prefixes are as follows:
binary classificationmultilabel classificationsimilarity
binary classification
multilabel classification
similarity
import pandas as pd
import json
from sklearn.model_selection import train_test_split

prefix = 'data/binary_classification/'

binary_train_df = pd.read_csv(prefix + 'train.csv', header=None)
binary_train_df.head()

binary_eval_df = pd.read_csv(prefix + 'test.csv', header=None)
binary_eval_df.head()

binary_train_df[0] = (binary_train_df[0] == 2).astype(int)
binary_eval_df[0] = (binary_eval_df[0] == 2).astype(int)

binary_train_df = pd.DataFrame({
    'prefix': ["binary classification" for i in range(len(binary_train_df))],
    'input_text': binary_train_df[1].str.replace('\n', ' '),
    'target_text': binary_train_df[0].astype(str),
})

print(binary_train_df.head())

binary_eval_df = pd.DataFrame({
    'prefix': ["binary classification" for i in range(len(binary_eval_df))],
    'input_text': binary_eval_df[1].str.replace('\n', ' '),
    'target_text': binary_eval_df[0].astype(str),
})


print(binary_eval_df.head())

prefix                                         input_text  \
0  binary classification  Unfortunately, the frustration of being Dr. Go...   
1  binary classification  Been going to Dr. Goldberg for over 10 years. ...   
2  binary classification  I don't know what Dr. Goldberg was like before...   
3  binary classification  I'm writing this review to give you a heads up...   
4  binary classification  All the food is great here. But the best thing...   

  target_text  
0           0  
1           1  
2           0  
3           0  
4           1  
                  prefix                                         input_text  \
0  binary classification  Contrary to other reviews, I have zero complai...   
1  binary classification  Last summer I had an appointment to get new ti...   
2  binary classification  Friendly staff, same starbucks fair you get an...   
3  binary classification  The food is good. Unfortunately the service is...   
4  binary classification  Even when we didn't have a car Filene's Baseme...   

  target_text  
0           1  
1           0  
2           1  
3           0  
4           1  

prefix = "data/multilabel_classification/"

multi_train_df = pd.read_csv(prefix + 'train.csv')
multi_train_df["comment_text"].str.replace('\n', ' ').str.replace('\t', ' ')

for col in multi_train_df.columns:
    if col not in ["id", "comment_text"]:
        multi_train_df[col] = multi_train_df[col].apply(lambda x: col if x else "")

multi_train_df["target_text"] = multi_train_df['toxic'].str.cat(multi_train_df[[col for col in multi_train_df.columns if col not in ["id", "comment_text", "toxic"]]], sep=',')
multi_train_df["target_text"] = multi_train_df["target_text"].apply(lambda x: ",".join(word for word in x.split(",") if word)).apply(lambda x: x if x else "clean")
multi_train_df["input_text"] = multi_train_df["comment_text"].str.replace('\n', ' ')
multi_train_df["prefix"] = "multilabel classification"
multi_train_df = multi_train_df[["prefix", "input_text", "target_text"]]

multi_train_df, multi_eval_df = train_test_split(multi_train_df, test_size=0.1)

multi_train_df.head()

prefix = 'data/regression/'

sts_train_df = pd.read_csv(prefix + 'train.tsv', sep='\t', error_bad_lines=False).dropna()
sts_eval_df = pd.read_csv(prefix + 'dev.tsv', sep='\t', error_bad_lines=False).dropna()

sts_train_df["sentence1"] = sts_train_df["sentence1"].str.replace('\n', ' ').str.replace('\t', ' ')
sts_train_df["sentence2"] = sts_train_df["sentence2"].str.replace('\n', ' ').str.replace('\t', ' ')
sts_eval_df["sentence1"] = sts_eval_df["sentence1"].str.replace('\n', ' ').str.replace('\t', ' ')
sts_eval_df["sentence2"] = sts_eval_df["sentence2"].str.replace('\n', ' ').str.replace('\t', ' ')

b'Skipping line 2509: expected 10 fields, saw 11\nSkipping line 2650: expected 10 fields, saw 11\nSkipping line 2727: expected 10 fields, saw 11\nSkipping line 3071: expected 10 fields, saw 11\nSkipping line 3393: expected 10 fields, saw 11\n'
b'Skipping line 1042: expected 10 fields, saw 11\nSkipping line 1066: expected 10 fields, saw 11\nSkipping line 1083: expected 10 fields, saw 11\nSkipping line 1137: expected 10 fields, saw 11\nSkipping line 1150: expected 10 fields, saw 11\n'

sts_train_df.drop(2001, inplace=True) # This line is badly formatted. Getting rid.

sts_train_df["input_text"] = sts_train_df.apply(lambda x: "sentence1: " + x["sentence1"] + " sentence2: " + x["sentence2"], axis=1)
sts_eval_df["input_text"] = sts_eval_df.apply(lambda x: "sentence1: " + x["sentence1"] + " sentence2: " + x["sentence2"], axis=1)

sts_train_df["target_text"] = sts_train_df["score"].apply(lambda x: round(x * 5) / 5).astype(str)
sts_eval_df["target_text"] = sts_eval_df["score"].apply(lambda x: round(x * 5) / 5).astype(str)

sts_train_df["prefix"] = "similarity"
sts_eval_df["prefix"] = "similarity"

sts_train_df = sts_train_df[["prefix", "input_text", "target_text"]]
sts_eval_df = sts_eval_df[["prefix", "input_text", "target_text"]]

train_df = pd.concat([binary_train_df, multi_train_df, sts_train_df]).astype(str)
eval_df = pd.concat([binary_eval_df, multi_eval_df, sts_eval_df]).astype(str)

train_df.to_csv("data/train.tsv", "\t")
eval_df.to_csv("data/eval.tsv", "\t")

 

The notebook above loads each of the datasets; preprocesses them for mT5 and finally combines them into a unified dataframe.
This gives us a dataframe with 3 unique prefixes, namely binary classification, multilabel classification, and similarity. Note that the prefixes themselves are fairly arbitrary, the important thing is to ensure that each task has its own unique prefix. The input to the model will take the following format:
<prefix>: <input_text>
The ": " is automatically added when training.
A few other things to note:
The output of the multilabel classification task is a comma-separated list of the predicted labels (toxic, severe_toxic, obscene, threat, insult, identity_hate). If no label is predicted, the output should be clean.
The input_text for the similarity task includes both sentences as shown in the following example;sentence1: A man plays the guitar. sentence2: The man sang and played his guitar.
The output of the similarity task is a number (as a string) between 0.0 and 5.0, going by increments of 0.2. (E.g. 0.0, 0.4, 3.0, 5.0).
Running the notebook should give you two files, train.tsv and eval.tsv, which we will use to train and test our model!
We will be using the Simple Transformers library (based on the Hugging Face Transformers) to train the mT5 model.
The instructions given below will install all the requirements.
Install Anaconda or Miniconda Package Manager from here.Create a new virtual environment and install packages.conda create -n simpletransformers pythonconda activate simpletransformersconda install pytorch>=1.6 cudatoolkit=10.2 -c pytorchInstall simpletransformers.pip install simpletransformers
Install Anaconda or Miniconda Package Manager from here.
Create a new virtual environment and install packages.conda create -n simpletransformers pythonconda activate simpletransformersconda install pytorch>=1.6 cudatoolkit=10.2 -c pytorch
Install simpletransformers.pip install simpletransformers
See installation docs
Training a model is quite easy with Simple Transformers. First, we start by importing the necessary libraries and setting up logging.
Next, we load the datasets that we generated earlier.
Note that we are casting all the data in the Dataframe as strings. This is because mT5 is a sequence-to-sequence model which expects all inputs and outputs to be text sequences. If we have numeric values (or any other non-string values), we’ll run into errors during training.
Next, we set up our pre-trained mT5 model.
Here, we are configuring our mT5 model through model_args and instantiating a model with the pre-trained mt5-base weights.
For more information about the different model_args and what they do, please refer to the Simple Transformers docs here and here.
A model with this configuration can be trained on a GPU with 24 GB of VRAM. If your GPU has less VRAM than that and you run into CUDA memory issues, you can try using smaller train_batch_size and eval_batch_size values. You can also try using the google/mt5-small model which requires less memory (replace “google/mt5-base” with "google/mt5-small" in line 18 above).
Now, we just have to train our model!
The evaluation line is optional as we’ll be doing some real testing in the next section. You can comment it out to save some time.
In our first test, we’ll follow the same evaluation procedure as in my original T5 experiment.
Specifically, we’ll use the following metrics.
Binary Classification: F1 score and Accuracy score
Multilabel Classification: F1 score (Hugging Face SQuAD metrics implementation) and Exact matches (Hugging Face SQuAD metrics implementation)
Similarity: Pearson correlation coefficient and Spearman correlation
First, we will define the functions to calculate the metrics given above.
Next, we read in the evaluation dataset and process it for testing.
Here,
to_predict: List of input sequences with the prefix and : prepended as expected by the mT5 model.
truth: List of true labels
tasks: List of tasks (binary classification, multilabel classification, or similarity) corresponding to each sequence in to_predict.
Then, we load our trained mT5 model and generate predictions for each input sequence.
As a sequence-to-sequence model, the decoding algorithm used and the hyperparameters (num_beams, do_sample, top_k, top_p) used to control it has a substantial impact on the quality of the predictions.
If you’d like to learn more about the decoding process, please refer to the decoding algorithms section in this article and this excellent notebook by Huggingface.
Now that we have our predictions (in df["predicted"]), it’s time to calculate the metrics.
Running this snippet of code yields the results shown below.
-----------------------------------Results: Scores for binary classification:F1 score: 0.8974482916528863Accuracy Score: 0.893921052631579Scores for multilabel classification:F1 score: 0.8971675648577516Exact matches: 0.8971675648577516Scores for similarity:Pearson Correlation: 0.012219660470091782Spearman Correlation: 0.00885628521396601
Note that there is a degree of randomness when using sampling decoding algorithms, so the scores will change when the test is repeated. However, the scores should be relatively close to these values.
Before we dig into these scores, let’s put them side-by-side (and round them off for clarity) with the T5 scores for comparison.
Binary Classification
The mT5 model does a pretty good job at the binary classification task, although it does fall short of the scores set by the T5 model.
Multilabel Classification
Similar story here with the multilabel classification task with the mT5 models doing well but not as well as the T5 model.
Typically, a monolingual model will outperform a comparable multilingual model, so, these results are as expected.
Similarity
Unlike the two previous tasks, the mT5 model fails miserably at sentence similarity.
While it’s difficult to pinpoint the exact cause without more experimentation, the two reasons given below are likely to be largely responsible!
One significant difference between T5 and mT5 is that the former undergoes supervised training as part of the pre-training process while the latter does not. That is, the pre-trained T5 model (before we fine-tune it) is already trained on multiple downstream tasks in addition to its primary unsupervised training objective.This means that the T5 model has the advantage of prior training on the similarity task (as well as other similar tasks).
The similarity task is significantly underrepresented compared to the other two tasks. While this factor does affect the T5 model during fine-tuning, it appears that the supervised training during pre-training is able to compensate for it. Without the benefit of multi-task training during the pre-training process, this becomes a much harder challenge for the mT5 model.
There might be other reasons (e.g. similarity being a sentence-pair task) and it’s likely that there are ways to get it to work for this task as well. Let me know if you find anything!
However, mT5 was never designed to beat T5 in English language tasks! So, let’s move on to the true test, cross-lingual zero-shot transfer.
In this test, we want to see if an mT5 model trained on an English task can perform the same task in other languages, without further training.
For this test, we’ll focus on the first task, i.e. binary classification. We won’t be testing on the other tasks as the second task involves a lot of toxic language (I don’t expect the translation models to perform well here). The third task is, unfortunately, disqualified by default as it didn’t do well even in English.
Before we can test the cross-lingual capabilities of mT5, we’ll first need to translate our evaluation data. For this, we’ll use the pre-trained MarianMT models originally trained by Jörg Tiedemann using the Marian C++ library.
First, we’ll set up our translation models in translation_models.py
Here, we are setting up functions to load four different translation models (english_to_romance_model will handle both French and Spanish).
Next, we’ll do use these models to translate our evaluation dataset. Note that, the translations may take some time depending on available computational resources. If you want to speed things up, you can test with one language only.
To keep the code clean, we’ll be doing this step in another file, translate_dataset.py.
You can drop the languages you don’t want from the languages list in line 51. You can also comment out the unused translation models defined in model_map to prevent them from being downloaded and loaded into memory.
Running the script above will generate the translated datasets and save them to the data directory.
The code to calculate the metrics can be easily adapted from our original evaluation code. The only differences are that we need to loop over the eval.csv files for each language and we’ll only be evaluating the binary classification task.
Let’s see how we did!
Very impressive, considering the mT5 model was only fine-tuned on an English dataset!
Based on these results, it seems like mT5 favours more common languages, although that is to be expected. However, it’s also possible that this difference is due to translation models for more common languages being relatively superior.
Unfortunately, it doesn’t perform well for all the supported languages. For example, I tried a few examples in my native language, Sinhalese, with fairly poor (although not completely useless) results.
If you want to try some quick predictions yourself, use the command simple-viewerin the terminal (in the directory which contains the outputs directory).
Overall, I’m quite impressed with the performance of the mT5 model, particularly in the context of cross-lingual, zero-shot transfer learning.
However, mT5 does seem to struggle with smaller datasets when compared to T5. Oversampling the similarity dataset to bring it in line with the others (or even training mT5 on this task alone) didn’t solve the issue, so it’s possible that the small size of the dataset isn’t the sole factor here. This calls for more investigation!
The ability to train a model on English data and have it automatically transfer that knowledge to a bunch of other languages can be incredibly useful, especially when you consider that most training datasets are only available in English.
With such versatility, I’m sure that people will come up with a ton of creative applications!
mT5: A massively multilingual pre-trained text-to-text transformer — https://arxiv.org/pdf/2010.11934.pdfExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — https://arxiv.org/abs/1910.10683Google AI Blog — https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.htmlHuggingface Transformers — https://huggingface.co/transformers/
mT5: A massively multilingual pre-trained text-to-text transformer — https://arxiv.org/pdf/2010.11934.pdf
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — https://arxiv.org/abs/1910.10683
Google AI Blog — https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html
Huggingface Transformers — https://huggingface.co/transformers/