Sentiment Analysis with Deep Learning | by Edwin Tan | Towards Data Science

Sentiment analysis is a technique in natural language processing used to identify emotions associated with the text. Common use cases of sentiment analysis include monitoring customers’ feedbacks on social media, brand and campaign monitoring.
In this article, we examine how you can train your own sentiment analysis model on a custom dataset by leveraging on a pre-trained HuggingFace model. We will also examine how to efficiently perform single and batch prediction on the fine-tuned model in both CPU and GPU environments. If you are looking to for an out-of-the-box sentiment analysis model, check out my previous article on how to perform sentiment analysis in python with just 3 lines of code.
pip install transformerspip install fast_ml==3.68pip install datasets
import numpy as npimport pandas as pdfrom fast_ml.model_development import train_valid_test_splitfrom transformers import Trainer, TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForSequenceClassificationimport torchfrom torch import nnfrom torch.nn.functional import softmaxfrom sklearn.metrics import classification_reportfrom sklearn.preprocessing import LabelEncoderimport datasets
Enable GPU accelerator if it is available.
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")print (f'Device Availble: {DEVICE}')
We will be using an ecommerce dataset that contains text reviews and ratings for women’s clothes.
df = pd.read_csv('/kaggle/input/womens-ecommerce-clothing-reviews/Womens Clothing E-Commerce Reviews.csv')df.drop(columns = ['Unnamed: 0'], inplace = True)df.head()
We are only interested in the Review Text and Rating columns. The Review Text column serves as input variable to the model and the Rating column is our target variable it has values ranging from 1 (least favourable) to 5 (most favourable).
For clarity, let’s append “Star” or “Stars” behind each integer rating.
df_reviews = df.loc[:, ['Review Text', 'Rating']].dropna()df_reviews['Rating'] = df_reviews['Rating'].apply(lambda x: f'{x} Stars' if x != 1 else f'{x} Star')
This is how the data looks like now, where 1,2,3,4,5 stars are our class labels.
Let’s encode the ratings using Sklearn’s LabelEncoder.
le = LabelEncoder()df_reviews['Rating'] = le.fit_transform(df_reviews['Rating'])df_reviews.head()
Notice that the Rating column has been transformed from a text to an integer column.
The numbers in the Rating column ranges from 0 to 4. These are the class id for the class labels which will be used to train the model. Each of the class id corresponds to a rating.
print (le.classes_)>> ['1 Star' '2 Stars' '3 Stars' **'4 Stars'** '5 Stars']
The position index of the list is the class id (0 to 4) and the value at the position is the original rating. For example at position number 3, the class id is “3” and it corresponds to the class label of “4 stars”.
Let’s split the data into train, validation and test in the ratio of 80%, 10% and 10% respectively.
(train_texts, train_labels, val_texts, val_labels, test_texts, test_labels) = train_valid_test_split(df_reviews, target = 'Rating', train_size=0.8, valid_size=0.1, test_size=0.1)
Convert the review text from pandas series to list of sentences.
train_texts = train_texts['Review Text'].to_list()train_labels = train_labels.to_list()val_texts = val_texts['Review Text'].to_list()val_labels = val_labels.to_list()test_texts = test_texts['Review Text'].to_list()test_labels = test_labels.to_list()
Create a DataLoader class for processing and loading of the data during training and inference phase.
class DataLoader(torch.utils.data.Dataset):    def __init__(self, sentences=None, labels=None):        self.sentences = sentences        self.labels = labels        self.tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')                if bool(sentences):            self.encodings = self.tokenizer(self.sentences,                                            truncation = True,                                            padding = True)            def __getitem__(self, idx):        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}                if self.labels == None:            item['labels'] = None        else:            item['labels'] = torch.tensor(self.labels[idx])        return item    def __len__(self):        return len(self.sentences)            def encode(self, x):        return self.tokenizer(x, return_tensors = 'pt').to(DEVICE)
Let’s take a look at the DataLoader in action.
train_dataset = DataLoader(train_texts, train_labels)val_dataset = DataLoader(val_texts, val_labels)test_dataset = DataLoader(test_texts, test_labels)
The DataLoader initializes a pretrained tokenizer and encodes the input sentences. We can get a single record from the DataLoader by using the __getitem__ function. Below is the result after an input sentence is tokenized.
print (train_dataset.__getitem__(0))
The output data is a dictionary consisting of 3 keys-value pairs
input_ids: this contains a tensor of integers where each integer represents words from the original sentence. The tokenizer step has transformed the individuals words into tokens represented by the integers. The first token 101 is the start of sentence token and the102 token is the end of sentence token. Notice that there are many trailing zeros, this is due to padding that was applied to the sentences at the tokenizer step.
attention_mask: this is an array of binary values. Each position of the attention_mask corresponds to a token in the same position in the input_ids. 1 indicates that the token at the given position should be attended to and 0 indicates that the token at the given position is a padded value.
labels: this is the target label
We would like the model performance to be evaluated at intervals during the training phase. For that we require a metrics computation function that accepts a tuple of (prediction, label) as argument and returns a dictionary of metrics: {'metric1':value1,metric2:value2}.
f1 = datasets.load_metric('f1')accuracy = datasets.load_metric('accuracy')precision = datasets.load_metric('precision')recall = datasets.load_metric('recall')def compute_metrics(eval_pred):    metrics_dict = {}    predictions, labels = eval_pred    predictions = np.argmax(predictions, axis=1)        metrics_dict.update(f1.compute(predictions = predictions, references = labels, average = 'macro'))    metrics_dict.update(accuracy.compute(predictions = predictions, references = labels))    metrics_dict.update(precision.compute(predictions = predictions, references = labels, average = 'macro'))    metrics_dict.update(recall.compute(predictions = predictions, references = labels, average = 'macro'))    return metrics_dict
Next, we configure instantiate a distilbert-base-uncased model from pretrained checkpoint.
id2label = {idx:label for idx, label in enumerate(le.classes_)}label2id = {label:idx for idx, label in enumerate(le.classes_)}config = AutoConfig.from_pretrained('distilbert-base-uncased',                                    num_labels = 5,                                    id2label = id2label,                                    label2id = label2id)model = AutoModelForSequenceClassification.from_config(config)
num_labels: number of classes
id2label: dictionary that maps the class ids to class labels {0: '1 Star', 1: '2 Stars', 2: '3 Stars', 3: '4 Stars', 4: '5 Stars'}
label2id:mapping dictionary that maps the class labels to class ids {'1 Star': 0, '2 Stars': 1, '3 Stars': 2, '4 Stars': 3, '5 Stars': 4}
Let’s examine the model configuration. The id2label and label2id dictionaries has been incorporated into the configuration. We can retrieve these dictionaries from the model's configuration during inference to find out the corresponding class labels for the predicted class ids.
print (config)>> DistilBertConfig {  "activation": "gelu",  "architectures": [    "DistilBertForMaskedLM"  ],  "attention_dropout": 0.1,  "dim": 768,  "dropout": 0.1,  "hidden_dim": 3072,  "id2label": {    "0": "1 Star",    "1": "2 Stars",    "2": "3 Stars",    "3": "4 Stars",    "4": "5 Stars"  },  "initializer_range": 0.02,  "label2id": {    "1 Star": 0,    "2 Stars": 1,    "3 Stars": 2,    "4 Stars": 3,    "5 Stars": 4  },  "max_position_embeddings": 512,  "model_type": "distilbert",  "n_heads": 12,  "n_layers": 6,  "pad_token_id": 0,  "qa_dropout": 0.1,  "seq_classif_dropout": 0.2,  "sinusoidal_pos_embds": false,  "tie_weights_": true,  "transformers_version": "4.6.1",  "vocab_size": 30522}
We can also examine the model architecture using
print (model)
Set up the training arguments.
training_args = TrainingArguments(    output_dir='/kaggle/working/results',    num_train_epochs=10,    per_device_train_batch_size=64,    per_device_eval_batch_size=64,    warmup_steps=500,    weight_decay=0.05,    report_to='none',    evaluation_strategy='steps',    logging_dir='/kagge/working/logs',    logging_steps=50)
report_to enables logging of training artifacts and results to platforms such as mlflow, tensorboard, azure_ml etc
per_device_train_batch_size is the batch size per TPU/GPU/CPU during training. Lower this if you face out of memory issues on your device
per_device_eval_batch_size is the batch size per TPU/GPU/CPU during evaluation. Lower this if you face out of memory issues on your device
logging_stepdetermines how frequently are the metrics evaluation done during training
Instantiate the Trainer. Under the hood, the Trainer runs the training and evaluation loop based on the given training arguments, model, datasets and metrics.
trainer = Trainer(    model=model,    args=training_args,    train_dataset=train_dataset,    eval_dataset=val_dataset,    compute_metrics=compute_metrics)
Let’s start training!
trainer.train()
Evaluation is performed every 50 steps. We can change the interval of evaluation by changing the logging_steps argument in TrainingArguments. In addition to the default training and validation loss metrics, we also get additional metrics which we had defined in the compute_metric function earlier.
Let’s evaluate our training on the test set.
eval_results = trainer.predict(test_dataset)
The Trainer's predict function returns 3 items:
An array of raw prediction scores
An array of raw prediction scores
print (test_results.predictions)
2. The ground truth label ids
print (test_results.label_ids)>> [1 1 4 ... 4 3 1]
3. Metrics
print (test_results.metrics)>> {'test_loss': 0.9638910293579102,		'test_f1': 0.28503729426950286,		'test_accuracy': 0.5982339955849889,		'test_precision': 0.2740061405117546,		'test_recall': 0.30397183356136337,		'test_runtime': 5.7367,		'test_samples_per_second': 394.826,		'test_mem_cpu_alloc_delta': 0,		'test_mem_gpu_alloc_delta': 0,		'test_mem_cpu_peaked_delta': 0,		'test_mem_gpu_peaked_delta': 348141568}
The model prediction function outputs unnormalized probability scores. To find the class probabilities we take a softmax across the unnormalized scores. The class with the highest class probabilities is taken to be the predicted class. we can find this by taking the argmax of the class probabilities. The id2label attribute which we stored in the model's configuration earlier on can be used to map the class id (0-4) to the class labels (1 star, 2 stars..).
label2id_mapper = model.config.id2labelproba = softmax(torch.from_numpy(test_results.predictions))pred = [label2id_mapper[i] for i in torch.argmax(proba, dim = -1).numpy()]actual = [label2id_mapper[i] for i in test_results.label_ids]
We use Sklearn’s classification_reportto obtain the precision, recall, f1 and accuracy scores.
class_report = classification_report(actual, pred, output_dict = True)pd.DataFrame(class_report)
trainer.save_model('/kaggle/working/sentiment_model')
In this section, we look at how to load and perform predictions on the trained model. Let’s test out inference in a separate notebook.
import pandas as pdimport numpy as npfrom transformers import Trainer, TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForSequenceClassificationimport torchfrom torch import nnfrom torch.nn.functional import softmax
The inference can work in both GPU or CPU environment. Enable GPU in your environment if it is available.
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")print (f'Device Availble: {DEVICE}')
This is the same DataLoader as we have used in the training phase
class DataLoader(torch.utils.data.Dataset):    def __init__(self, sentences=None, labels=None):        self.sentences = sentences        self.labels = labels        self.tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')                if bool(sentences):            self.encodings = self.tokenizer(self.sentences,                                            truncation = True,                                            padding = True)            def __getitem__(self, idx):        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}                if self.labels == None:            item['labels'] = None        else:            item['labels'] = torch.tensor(self.labels[idx])        return item    def __len__(self):        return len(self.sentences)            def encode(self, x):        return self.tokenizer(x, return_tensors = 'pt').to(DEVICE)
The SentimentModel class helps to initialize the model and contains the predict_proba and batch_predict_proba methods for single and batch prediction respectively. The batch_predict_proba uses HuggingFace's Trainer to perform batch scoring.
class SentimentModel():        def __init__(self, model_path):                self.model = AutoModelForSequenceClassification.from_pretrained(model_path).to(DEVICE)        args =  TrainingArguments(output_dir='/kaggle/working/results', per_device_eval_batch_size=64)        self.batch_model = Trainer(model = self.model, args= args)        self.single_dataloader = DataLoader()            def batch_predict_proba(self, x):                predictions = self.batch_model.predict(DataLoader(x))        logits = torch.from_numpy(predictions.predictions)                if DEVICE == 'cpu':            proba = torch.nn.functional.softmax(logits, dim = 1).detach().numpy()        else:            proba = torch.nn.functional.softmax(logits, dim = 1).to('cpu').detach().numpy()        return proba                    def predict_proba(self, x):                x = self.single_dataloader.encode(x).to(DEVICE)        predictions = self.model(**x)        logits = predictions.logits                if DEVICE == 'cpu':            proba = torch.nn.functional.softmax(logits, dim = 1).detach().numpy()        else:            proba = torch.nn.functional.softmax(logits, dim = 1).to('cpu').detach().numpy()        return proba
Let’s load some sample data
df = pd.read_csv('/kaggle/input/womens-ecommerce-clothing-reviews/Womens Clothing E-Commerce Reviews.csv')df.drop(columns = ['Unnamed: 0'], inplace = True)df_reviews = df.loc[:, ['Review Text', 'Rating']].dropna()df_reviews['Rating'] = df_reviews['Rating'].apply(lambda x: f'{x} Stars' if x != 1 else f'{x} Star')df_reviews.head()
We will create two sets of data. One for batch scoring and the other for single scoring.
batch_sentences = df_reviews.sample(n = 10000, random_state = 1)['Review Text'].to_list()single_sentence = df_reviews.sample(n = 1, random_state = 1)['Review Text'].to_list()[0]
Instantiate the model
sentiment_model = SentimentModel('../input/fine-tune-huggingface-sentiment-analysis/sentiment_model')
Predict on a single sentence using the predict_proba method.
single_sentence_probas = sentiment_model.predict_proba(single_sentence)id2label = sentiment_model.model.config.id2labelpredicted_class_label = id2label[np.argmax(single_sentence_probas)]print (predicted_class_label)>> 5 Stars
Predict on a batch of sentences using the batch_predict_proba method.
batch_sentence_probas = sentiment_model.batch_predict_proba(batch_sentences)predicted_class_labels = [id2label[i] for i in np.argmax(batch_sentence_probas, axis = -1)]
Let’s compare the inference speed for between predict_proba and batch_predict_proba method
for 10k sample data in a CPU and GPU environment. We will iterate through 10k samples for predict_proba make a single prediction at a time while scoring all 10k without iteration using the batch_predict_proa method.
%%timefor sentence in batch_sentences:    single_sentence_probas = sentiment_model.predict_proba(sentence)%%timebatch_sentence_probas = sentiment_model.batch_predict_proba(batch_sentences)
GPU environment
Iterating through predict_proba takes ~2 minutes while batch_predict_proba takes ~30 seconds for 10k sample data. Batch prediction is almost 4 times faster than using single prediction in GPU environment.
CPU Environment
In CPU environment, predict_proba took ~14 minutes while batch_predict_proba took ~40 minutes, that is almost 3 times longer.
Therefore for large set of data, use batch_predict_proba if you have GPU. if you do not have access to a GPU, you are better off with iterating through the dataset using predict_proba.
In this article we examined:
How to train your own deep learning sentiment analysis model by leveraging on a pretrained HuggingFace model
How to create single and batch prediction methods for scoring
Inference speed for single and batch scoring in both CPU and GPU environments
The notebooks for this article can be found here:
Training
Inference
Join Medium to read more stories like this.