MultiLingual Amazon Reviews Classification | by Priya Dwivedi | Towards Data Science

NLP is going to be the most transformational tech of this decade. And Transformer models are fuelling the advancement in NLP. In this blog, we look at an application of Transformers on multilingual data. Many businesses are now global in their reach and are collecting data in many languages. NLP is also building strong capabilities to deal with data in multiple languages. Transformer models can now be trained in many languages and seamlessly ingest text in them. This is amazing!
In this blog, we use a pretrained multilingual XLM Roberta model and fine-tune it on a downstream classification task. XLM-Roberta has been pretrained in over 100 different languages! The model is able to give a state-of-the-art performance on a dataset with product reviews in multiple languages. I think this opens new opportunities for businesses to integrate data from their operations in different countries and generate valuable insights.
The code for this model is available on my Colab Notebook here. I have also uploaded it to my GitHub here.
The code here is a general-purpose code to run a classification using HuggingFace and the Datasets library. It can be modified easily to many other data sets and problems involving classification.
For this blog, we use Amazon multilingual reviews corpus. This data set is made publicly available by Amazon and contains customer reviews across a variety of products in 6 different languages — English, Japanese, German, French, Spanish, and Chinese. The datasets library provides an easy API to download and access the data set.
!pip install datasetsdataset  = datasets.load_dataset('amazon_reviews_multi')
The Data set has1.2MM reviews in the training set and 30K in the validation set. Each review has a title, category, text body and star rating. The star rating ranges from 1–5 with equal distribution across the 5 classes.
An example of an English review with its rating is:
Review: 'I’ve had good luck with this product when I’ve bought it in stores, but this particular one I bought though Amazon was watered down! The texture was much more viscous than any of the tubes I bought in person and it lasted me about half the time. You’re better off finding it in a shop than wasting your money on watered down face wash.'Rating: 2
We load the XML-Roberta model and use it to tokenize the text. The max length for text is set at 512 tokens. To improve model accuracy, we concatenate review_body, review_title and review_category.
Since the actual data set is quite big, we train on 10% of the training sample. The shard functionality in the dataset library makes it easy to sample a fraction of the data for experimentation. The code to do this is below:
dataset = dataset.shuffle(seed=123)train_dataset = dataset["train"].shard(index=1, num_shards=10)val_dataset = dataset['validation'].shard(index=1, num_shards=5)
The tokenizer, encode_plus option is used to tokenize the text associated with the review body, review title and review category. The actual reviews have a star rating from 1 star to 5 stars. We convert this to target labels ranging from 0–4.
To simplify the process of training, we use the Trainer library from HuggingFace. We load the pretrained XLM-Roberta-Base model using AutoModel functionality.
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainerbatch_size = 8num_labels = 5model = AutoModelForSequenceClassification.from_pretrained('xlm-roberta-base', num_labels=num_labels)
To instantiate the Trainer, we define two things:
TrainingArguments — These include the location of our output directory, learning rate, batch size, num epochs etcCompute Metrics function for evaluation —We can define an evaluation metric that is run on val set during the training.
TrainingArguments — These include the location of our output directory, learning rate, batch size, num epochs etc
Compute Metrics function for evaluation —We can define an evaluation metric that is run on val set during the training.
The datasets package also has many custom metrics defined. I love this functionality in the datasets library as it makes it very easy to calculate metrics ranging for bleu, rouge or custom SQuAD metrics on your data set.
We first load the metric of our choice from the datasets library
from datasets import load_metricmetric = load_metric('accuracy')f1_metric = load_metric('f1')
The metric.compute() function can be used to get results.
import numpy as npfake_preds = np.random.randint(1, 6, size=(64,))fake_labels = np.random.randint(1, 6, size=(64,))metric.compute(predictions=fake_preds, references=fake_labels)f1_metric.compute(predictions=fake_preds, references=fake_labels, average='weighted')
To run it during training, we get predictions and labels from eval_pred and then run metric.compute(). That’s all!!
To start training, now we can pass all these arguments to the Trainer.
trainer = Trainer(model,args,train_dataset= encoded_train_dataset,eval_dataset=encoded_val_dataset,data_collator=SmartCollator(pad_token_id=tokenizer.pad_token_id),tokenizer=tokenizer,compute_metrics=compute_metrics)
The model takes about 2 hours 20 minutes to train on a P100 instance on Colab. This is for a 10% sample of the training data set.
The model gets to an accuracy of 59.9% which is slightly better than the accuracy of 59.2% reported in the paper that introduced the data set. Training on the full data set should further improve results.
I want to touch on a couple of other cool things that can be used to improve training speed/model results
The inspiration from this comes from the work here. During training time, we have to feed in samples of the same length. Since underlying text can be of different lengths, the typical approach is to pad everything to a max length. This is generally done at the full data set level where if the max length of the text is 512 tokens then everything is padded to that max length. However, this is inefficient. As an example, a sentence that is 20 tokens would also get padded to 512 tokens. Since attention is quadratic in computation, this would significantly slow down the training for such a sample.
An optimization that can be done here is to pad to the max length in the batch instead of the max length in the data set. The code in my notebook does that by defining a smart data collator. Inside the data collator, the main block that does this is:
max_size = max([len(ex['input_ids']) for ex in batch])for item in batch:batch_inputs += [pad_seq(item['input_ids'], max_size, self.pad_token_id)]
This optimization cuts down training time in half! So very useful for a small block of code.
Optuna is a hyperparameter searcher that can be easily integrated with the trainer class. It efficiently searches a large space of hyperparameters pruning unpromising trials early.
For our experiment, we define the training arguments and the Trainer as before and let Optune optimize across a selected number of trials.
trainer.hyperparameter_search(n_trials=15, direction=”maximize”)
This blog shows how Transformer models can be used to work on multilingual data. We used HuggingFace Library and its many features to prepare our data and train it with just a few lines of code. The trained model has state-of-the-art results. The code is written in a very modular format and can be used for any type of classification problem. We also discuss features for dynamic padding to significantly speed up training and look at hyperparameter optimization to further improve results.
I hope you give the code a try and train your own models. Please share your experience in the comments below.
At Deep Learning Analytics, we are extremely passionate about using Machine Learning to solve real-world problems. We have helped many businesses deploy innovative AI-based solutions. Contact us through our website here if you see an opportunity to collaborate.
XML-Roberta Paper
HuggingFace Classification Notebook