Sentiment classification in Python | by Zolzaya Luvsandorj | Towards Data Science

This post is the last of the three sequential posts on steps to build a sentiment classifier. Having done some exploratory text analysis and preprocessed the text, it’s time to classify reviews to sentiments. In this post, we will first look at 2 ways to get sentiments without building a model then build a custom model.
Before we dive in, let’s take a step back and look at the bigger picture really quickly. CRISP-DM methodology outlines the process flow for a successful data science project. In this post, we will do some of the tasks that a data scientist would go through during the modelling stage.
This post assumes that the reader (👀 yes, you!) has access to and is familiar with Python including installing packages, defining functions and other basic tasks. If you are new to Python, this is a good place to get started.
I have tested the scripts in Python 3.7.1 in Jupyter Notebook.
Let’s make sure you have the following libraries installed before we start:◼️ Data manipulation/analysis: numpy, pandas◼️ Data partitioning: sklearn◼️ Text preprocessing/analysis: nltk, textblob◼️ Visualisation: matplotlib, seaborn
Once you have nltk installed, please make sure you have downloaded ‘stopwords’ , ‘wordnet’ and ‘vader_lexicon’ from nltk with the script below:
import nltknltk.download('stopwords') nltk.download('wordnet')nltk.download('vader_lexicon')
If you have already downloaded, running this will notify you so.
Now, we are ready to import the packages:
# Set random seedseed = 123# Data manipulation/analysisimport numpy as npimport pandas as pd# Text preprocessing/analysisimport refrom nltk.corpus import stopwordsfrom nltk.stem import WordNetLemmatizerfrom nltk.tokenize import RegexpTokenizerfrom nltk.sentiment.vader import SentimentIntensityAnalyzerfrom textblob import TextBlobfrom scipy.sparse import hstack, csr_matrixfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.preprocessing import MinMaxScaler# Modellingfrom sklearn.model_selection import train_test_split, cross_validate, GridSearchCV, RandomizedSearchCVfrom sklearn.linear_model import LogisticRegression, SGDClassifierfrom sklearn.naive_bayes import MultinomialNBfrom sklearn.metrics import classification_report, confusion_matrixfrom sklearn.pipeline import Pipeline# Visualisationimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinesns.set(style="whitegrid", context='talk')
We will use IMDB movie reviews dataset. You can download the dataset here and save it in your working directory. Once saved, let’s import it to Python:
sample = pd.read_csv('IMDB Dataset.csv')print(f"{sample.shape[0]} rows and {sample.shape[1]} columns")sample.head()
Let’s look at the split between sentiments:
sample['sentiment'].value_counts()
Sentiment is evenly split in the sample data. Let’s encode the target into numeric values where positive is 1 and negative is 0:
# Encode to numericsample['target'] = np.where(sample['sentiment']=='positive', 1, 0)# Check valuessample.groupby(['sentiment', 'target']).count().unstack()
Let’s set aside 5000 cases for testing:
# Split data into train & testX_train, X_test, y_train, y_test = train_test_split(sample['review'], sample['sentiment'], test_size=5000, random_state=seed,                                                     stratify=sample['sentiment'])# Append sentiment back using indicestrain = pd.concat([X_train, y_train], axis=1)test = pd.concat([X_test, y_test], axis=1)# Check dimensionsprint(f"Train: {train.shape[0]} rows and {train.shape[1]} columns")print(f"{train['sentiment'].value_counts()}\n")print(f"Test: {test.shape[0]} rows and {test.shape[1]} columns")print(test['sentiment'].value_counts())
We will quickly inspect the head of the training dataset:
train.head()
Alright, let’s dive in! 🐳
In this section, I want to show you two very simple methods to get sentiments without building a custom model. We will extract polarity intensity scores with VADER and TextBlob.
‘VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.’
Let’s start with a simple example and see how we extract sentiment intensity scores using VADER sentiment analyser:
example = 'The movie was awesome.'sid = SentimentIntensityAnalyzer()sid.polarity_scores(example)
neg, neu, pos: These three scores sum up to 1. These scores show the proportion of text falling in the category.compound: This score ranges from -1 (the most negative) to 1 (the most positive.
Although not all reviews would be as simple as our example at hand, it’s good to see that scores for the example review looks mostly positive. Now, let’s add the intensity scores to the training data:
train[['neg', 'neu', 'pos', 'compound']] = train['review'].apply(sid.polarity_scores).apply(pd.Series)train.head()
All it takes to get sentiment scores is a single line of code once we initialise the analyser object. Shall we inspect the scores further? Let’s start with peaking at 5 records with the highest pos scores:
train.nlargest(5, ['pos'])
It’s great to see that all of them are indeed positive reviews. Let’s do the same for neg:
train.nlargest(5, ['neg'])
This looks good too. But we could be looking at the extreme ends of the data where the sentiment is more obvious. Let’s visualise the scores with histograms to understand better:
for var in ['pos', 'neg', 'neu', 'compound']:    plt.figure(figsize=(12,4))    sns.distplot(train.query("target==1")[var], bins=30, kde=False,                  color='green', label='Positive')    sns.distplot(train.query("target==0")[var], bins=30, kde=False,                  color='red', label='Negative')    plt.legend()    plt.title(f'Histogram of {var} by true sentiment');
From the histograms, it appears that pos, neg and potentially compound columns are useful in classifying positive and negative sentiments. We can quickly classify each review into either positive or negative classes using these scores. Let’s see how well it will do:
train['vader_polarity'] = np.where(train['pos']>train['neg'], 1, 0)target_names=['negative', 'positive']print(classification_report(train['target'],                             train['vader_polarity'],                             target_names=target_names))
With very little effort, we can get about 69% accuracy using VADER. The performance on positive and negative reviews look different though. We have higher recall and lower precision of positive reviews — meaning that we have more false positives (See what I did there? Do you see why I encoded positive reviews as 1? 🙊). Let’s see confusion matrix:
# Create function so that we could reuse laterdef plot_cm(y_test, y_pred, target_names=['negative', 'positive'],             figsize=(5,3)):    """Create a labelled confusion matrix plot."""    cm = confusion_matrix(y_test, y_pred)    fig, ax = plt.subplots(figsize=figsize)    sns.heatmap(cm, annot=True, fmt='g', cmap='BuGn', cbar=False,                 ax=ax)    ax.set_title('Confusion matrix')    ax.set_xlabel('Predicted')    ax.set_xticklabels(target_names)    ax.set_ylabel('Actual')    ax.set_yticklabels(target_names,                        fontdict={'verticalalignment': 'center'});# Plot confusion matrixplot_cm(train['target'], train['vader_polarity'])
As we can see, we have many true positives and false positives. In fact, about 67% of our predictions are positive. Let’s see whether performance improves if we use the compound score.
train['vader_compound'] = np.where(train['compound']>0, 1, 0)print(classification_report(train['target'],                             train['vader_compound'],                             target_names=target_names))
plot_cm(train['target'], train['vader_compound'])
Performance looks pretty similar. I used training dataset to assess because we are not training a model here. However, if you do the same on the test data, the results should be very similar.
🔗 More information on VADER and VADER in nltk.
Another way to get sentiment score is to leverage TextBlob library. Using sentiment property from the TextBlob object, we can also extract similar scores. Here’s how we can extract using our previous example:
TextBlob(example).sentiment
polarity: ranges from -1 (the most negative) to 1 (the most positive)subjectivity: ranges from 0 (very objective) to 1 (very subjective)
Our example was analysed to be a very subjective positive statement. It’s true, isn’t it? Of these two scores, polarity is more relevant for us. Let’s add the intensity scores to the training data and inspect 5 records with the highest polarity scores:
train[['polarity', 'subjectivity']] = train['review'].apply(lambda x:TextBlob(x).sentiment).to_list()columns = ['review', 'target', 'polarity', 'subjectivity']train[columns].nlargest(5, ['polarity'])
As you saw, adding sentiment intensity score with TextBlob is also quite simple. Let’s look at 5 records with the lowest polarity scores:
train[columns].nsmallest(5, ['polarity'])
Time to plot some histograms to understand the scores better:
for var in ['polarity', 'subjectivity']:    plt.figure(figsize=(12,4))    sns.distplot(train.query("target==1")[var], bins=30, kde=False,                  color='green', label='Positive')    sns.distplot(train.query("target==0")[var], bins=30, kde=False,                  color='red', label='Negative')    plt.legend()    plt.title(f'Histogram of {var} by true sentiment');
As expected, polarity score looks possibly useful in classifying positive and negative sentiments. Let’s classify using the polarity score and see performance:
train['blob_polarity'] = np.where(train['polarity']>0, 1, 0)print(classification_report(train['target'],                             train['blob_polarity'],                             target_names=target_names))
With very little effort, we can get about 69% accuracy using TextBlob. Again, we have many false positives, in fact, even more than before. Let’s look at confusion matrix:
plot_cm(train['target'], train['blob_polarity'])
This time, the number of false positives are higher than the number of true negatives. Predictions are skewed to positive sentiment as 76% of predictions are positive.
🔗 More information on TextBlob.
Let’s compare how similar the scores from VADER and TextBlob are:
pd.crosstab(train['vader_polarity'], train['blob_polarity'])
There is about 79% overlap in their classifications with majority in positive sentiments. Let’s visualise the polarity scores:
plt.figure(figsize=(12,12))sns.scatterplot(data=train, x='polarity', y='compound',                hue='target', palette=['red', 'green'],                 alpha=.3)plt.axhline(0, linestyle='--', color='k')plt.axvline(0, linestyle='--', color='k')plt.title('Scatterplot between polarity intensity scores');
This plot shows a little bit more information than the previous table. In the bottom left quadrant, we see mainly red circles since negative classifications in both methods were more precise. In the top right quadrant, there is a higher volume of circles that are mostly green but the color mix is not as pure as before. The remaining two quadrants show where the two scores disagree with each other. Overall, the colour is more mixed than the left half in the right half of the plot.
We get very similar overall accuracy of 69% from both; however, when we look at the predictions closely, the performance differs between the approaches.
Now you know how to get sentiment polarity scores with VADER or TextBlob. If you have unlabeled data, these two provide a great starting point to label your data automatically. It’s time to build a model! ✨
In this section, we will:
choose a suitable preprocessing approach and algorithmexplore whether adding VADER and TextBlob sentiment scores as features improves the predictive power of the modelbuild a pipeline and tune its hyperparameterstest the final pipeline on unseen data
choose a suitable preprocessing approach and algorithm
explore whether adding VADER and TextBlob sentiment scores as features improves the predictive power of the model
build a pipeline and tune its hyperparameters
test the final pipeline on unseen data
The sentiment classification is one application of supervised classification model .Therefore, the approach we are taking here can be generalised to any supervised classification tasks.
In my previous post, we have explored three different ways to preprocess text and shortlisted two of them: simpler approach and simple approach. Of these two, we will now test if there is any difference in model performance between the two options and choose one of them to use moving forward. To make things easier, we will create two functions (the idea of these functions is inspired from here):
# Define functionsdef create_baseline_models():    """Create list of baseline models."""    models = []    models.append(('log', LogisticRegression(random_state=seed,                                              max_iter=1000)))    models.append(('sgd', SGDClassifier(random_state=seed)))    models.append(('mnb', MultinomialNB()))    return modelsdef assess(X, y, models, cv=5, scoring=['roc_auc',                                         'accuracy',                                         'f1']):    """Provide summary of cross validation results for models."""    results = pd.DataFrame()    for name, model in models:        result = pd.DataFrame(cross_validate(model, X, y, cv=cv,                                              scoring=scoring))        mean = result.mean().rename('{}_mean'.format)        std = result.std().rename('{}_std'.format)        results[name] = pd.concat([mean, std], axis=0)    return results.sort_index()
I have picked three algorithms to try: Logistic Regression Classifier, Stochastic Gradient Descent Classifier and Multinomial Naive Bayes Classifier. Let’s initiate the models:
models = create_baseline_models()models
Now, let’s inspect model performance when using simpler approach:
# Preprocess the datavectoriser = TfidfVectorizer(token_pattern=r'[a-z]+',                              stop_words='english',                              min_df=30,                              max_df=.7)X_train_simpler = vectoriser.fit_transform(X_train)# Assess the modelassess(X_train_simpler, y_train, models)
Great to see we get much better performance: 86–89% accuracy with baseline models compared to using only the sentiment scores. Since the classes are pretty balanced, we will mainly focus on accuracy. But we will ensure to inspect the predictions closer later to evaluate the model. Performance metrics look pretty close between Logistic Regression and Stochastic Gradient Descent with the latter being faster in training (see fit_time). Naive Bayes is the fastest of the three in training but performs slightly worse than the other two. Now let’s assess simple approach:
# Define functiondef preprocess_text(text):    # 1. Tokenise to alphabetic tokens    tokeniser = RegexpTokenizer(r'[A-Za-z]+')    tokens = tokeniser.tokenize(text)        # 2. Lowercase and lemmatise     lemmatiser = WordNetLemmatizer()    tokens = [lemmatiser.lemmatize(t.lower(), pos='v')               for t in tokens]    return tokens# Preprocess the datavectoriser = TfidfVectorizer(analyzer=preprocess_text,                              min_df=30,                              max_df=.7)X_train_simple = vectoriser.fit_transform(X_train)# Assess modelsassess(X_train_simple, y_train, models)
The performance looks similar to before. Therefore, we will favour the simpler approach and use it moving forward. Of the three algorithms, we will choose Stochastic Gradient Descent because it balances both speed and predictive power the most.
In this section, we will explore whether adding VADER and TextBlob sentiment scores as features improves the predictive power of the model. Let’s quickly check if there are any highly correlated features:
plt.figure(figsize = (14,5))columns = ['target', 'neg', 'neu', 'pos', 'compound', 'polarity',            'subjectivity']sns.heatmap(train[columns].corr(), annot=True, cmap='seismic_r');
The most correlated features are compound and neg. Let’s run a quick model to see which scores are more useful to use:
# Initialise a modelsgd = SGDClassifier(random_state=seed)# Initialise a scalerscaler = MinMaxScaler()# Assess the model using scoresscores = train[['neg', 'neu', 'pos', 'compound', 'polarity',                 'subjectivity']]assess(scaler.fit_transform(scores), y_train, [('sgd', sgd)])
We get about 77% accuracy using the scores. Now let’s inspect coefficients:
# Fit to training datasgd.fit(scores, y_train)# Get coefficientscoefs = pd.DataFrame(data=sgd.coef_, columns=scores.columns).Tcoefs.rename(columns={0: 'coef'}, inplace=True)# Plotplt.figure(figsize=(10,5))sns.barplot(x=coefs.index, y='coef', data=coefs)plt.title('Coefficients');
Seems like we could only use neg, pos and polarity because they are the most dominant features among the scores. Let’s see if model results can be improved by adding these selected scores to the previously preprocessed data.
# Add features to sparse matrixselected_scores = train[['neg', 'pos', 'polarity']]X_train_extended = hstack([X_train_simpler, csr_matrix(scaler.fit_transform(selected_scores))])# Assessassess(X_train_extended, y_train, [('sgd', sgd)])
Since adding these scores didn’t improve the model, it is unnecessary to add them as features. That will keep our pipeline simple too!
It’s time to build a small pipeline that puts together the preprocessor and the model. We will fine tune its hyperparameters to see if we can improve the model. Firstly, let’s try to understand the impact of three hyperparameters: min_df, max_df for the vectoriser and loss for the model with random search:
# Create a pipelinepipe = Pipeline([('vectoriser', TfidfVectorizer(token_pattern=r'[a-z]+')),                 ('model', SGDClassifier(random_state=seed))])# Prepare a random searchparam_distributions = {'vectoriser__min_df': np.arange(10, 1000, 10),                       'vectoriser__max_df': np.linspace(.2, 1, 40),                       'model__loss': ['log', 'hinge']}r_search = RandomizedSearchCV(estimator=pipe, param_distributions=param_distributions,                               n_iter=30, cv=5, n_jobs=-1, random_state=seed)r_search.fit(X_train, y_train)# Save results to a dataframer_search_results = pd.DataFrame(r_search.cv_results_).sort_values(by='rank_test_score')
Here, we are trying 30 different random combinations of hyperparameter space specified. This will take a while to run. The output of the random search will be saved in a dataframe called r_search_results. Let’s create another dataframe containing a few columns that are more interesting to us:
columns = [col for col in r_search_results.columns            if re.search(r"split|param_", col)]r_summary = r_search_results[columns].copy()r_summary.columns = [re.sub(r'_test_score|param_', '', col)                      for col in r_summary.columns]columns = [col.split('__')[1] if '__' in col else col            for col in r_summary.columns ]r_summary.columns = columnsr_summary.head()
Let’s visualise the output to understand the impact of hyperparameters better:
# Create a long dataframer_summary_long = pd.melt(r_summary,                          id_vars=['min_df',                                   'max_df',                                   'loss'],                          value_vars=['split0',                                      'split1',                                      'split2',                                      'split3',                                      'split4'])# Plot hyperparameter 'loss'plt.figure(figsize=(8,4))plt.title('Performance by loss')sns.boxplot(x='value', y='loss', data=r_summary_long,             orient='h')plt.xlim(.8, .9);
It looks loss='hinge' results in slightly better performance. Let’s look at the numerical hyperparameters:
for param in ['min_df', 'max_df']:    plt.figure(figsize=(8,4))    sns.scatterplot(x=param, y="value", data=r_summary_long,                     x_jitter=True, alpha=0.5)    plt.ylim(.8, .9);
As there seems to be a negative relationship between min_df and accuracy, we will keep min_df under 200. There isn’t a clear trend in max_df probably because the performance was more impacted by min_df and loss. Although this is true for all three of them, it’s more obvious for max_df. Now we have some idea on how these hyperparameters impact the model, let’s define the pipeline more precisely (max_df=.6 and loss=’hinge') and try to further tune it with grid search:
# Create a pipelinepipe = Pipeline([('vectoriser', TfidfVectorizer(token_pattern=r'[a-z]+', max_df=.6)),                 ('model', SGDClassifier(random_state=seed, loss='hinge'))])# Prepare a grid searchparam_grid = {'vectoriser__min_df': [30, 90, 150],              'vectoriser__ngram_range': [(1,1), (1,2)],              'vectoriser__stop_words': [None, 'english'],              'model__fit_intercept': [True, False]}g_search = GridSearchCV(estimator=pipe, param_grid=param_grid, cv=5, n_jobs=-1)g_search.fit(X_train, y_train)# Save results to a dataframeg_search_results = pd.DataFrame(g_search.cv_results_).sort_values(by='rank_test_score')
Grid searching will also take a bit of time because we have 24 different combinations of hyperparameters to try. Like before, the output will be saved to a dataframe called g_search_results. Let’s extract more relevant columns to another dataframe:
columns = [col for col in g_search_results.columns            if re.search(r"split|param_", col)]g_summary = g_search_results[columns+['mean_test_score']].copy()g_summary.columns = [re.sub(r'_test_score|param_', '', col)                      for col in g_summary.columns]columns = [col.split('__')[1] if '__' in col else col            for col in g_summary.columns ]g_summary.columns = columnsg_summary.head()
With any of these combinations, we reach a cross validated accuracy of ~0.9. It’s nice to see a marginal increase.
# Create a long dataframeg_summary_long = pd.melt(g_summary,                          id_vars=['min_df',                                   'ngram_range',                                   'stop_words',                                   'fit_intercept'],                          value_vars=['split0',                                      'split1',                                      'split2',                                      'split3',                                      'split4'])g_summary_long.replace({None: 'None'}, inplace=True)# Plot performancefor param in ['ngram_range', 'stop_words', 'fit_intercept']:    plt.figure(figsize=(8,4))    plt.title(f'Performance by {param}')    sns.boxplot(x='value', y=param, data=g_summary_long, orient='h')    plt.xlim(.85, .95);
We can see that by changing to ngram_range=(1,2), model performs better. The same is true for stop_words=None. On the other hand, whether we fit intercept or not doesn’t have much impact, which means we can leave this hyperparameter to its default. I think this is good enough, we can now define the final pipeline.
Using the top combination from grid search, this is how our final pipeline looks like:
pipe = Pipeline([('vectoriser', TfidfVectorizer(token_pattern=r'[a-z]+', min_df=30, max_df=.6, ngram_range=(1,2))),                 ('model', SGDClassifier(random_state=seed, loss='hinge'))])pipe.fit(X_train, y_train)
Our pipeline is very small and simple. Let’s see its coefficients:
coefs = pd.DataFrame(pipe['model'].coef_,                      columns=pipe['vectoriser'].get_feature_names())coefs = coefs.T.rename(columns={0:'coef'}).sort_values('coef')coefs
Features with the highest or lowest coefficients look intuitive. But look at the number of features we have: 49,577! This is mainly due to having relaxed min_df, adding bigrams and not removing stop words. If we were keen to reduce the number of features, we could change these hyperparamaters in the pipeline. If we start cutting down features, we will notice a tradeoff between number of features and the model accuracy. What an optimal balance looks like depends on the context. Let’s evaluate the pipeline:
train_pred = pipe.predict(X_train)print(classification_report(train_pred,                             y_train,                             target_names=target_names))
test_pred = pipe.predict(X_test)print(classification_report(test_pred,                             y_test,                             target_names=target_names))
Accuracy on train and test set is about 0.94 and 0.92 respectively. Precision and recall by both sentiments look pretty similar. We have slightly more false negatives. Let’s plot confusion matrix:
plot_cm(test_pred, y_test, target_names=target_names)
Looks good. Yay 🎊, now we have a pipeline that classifies about 9 in 10 reviews into the correct sentiment. Let’s see how long it takes to make a single prediction. We will use Jupyter Notebook’s magic command %timeit:
for i in range(10):    lead = X_test.sample(1)    %timeit pipe.predict(lead)
Although %timeit runs multiple loops and gives us mean and standard deviation of run time, I notice that I get slightly different output every time. Hence, we are looking at 10 loops of %timeit to observe the range.
A single prediction takes about 1.5 to 4 milliseconds. This needs to be evaluated in the context of production environment for the use case.
Alright, that was it for this post. 💫
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. Hopefully, you have learned a few different practical ways to classify text into sentiments with or without building a custom model. Here are links to the other two posts of the series:◼️ Exploratory text analysis in Python◼️ Preprocessing text in Python
Here are links to the my other NLP-related posts:◼️ Simple wordcloud in Python(Below lists a series of posts on Introduction to NLP)◼️ Part 1: Preprocessing text in Python◼️ Part 2: Difference between lemmatisation and stemming◼️ Part 3: TF-IDF explained◼️ Part 4: Supervised text classification model in Python◼️ Part 5A: Unsupervised topic model in Python (sklearn)◼️ Part 5B: Unsupervised topic model in Python (gensim)
Bye for now 🏃💨