Sentiment Anaylsis with the flair NLP library
By: Riley Jones
Tags: Machine Learning, Flair, NLP, Sentiment Analysis, Data Science

If you’re relatively new to machine learning and natural language processing in Python or don’t want to dive right into PyTorch or TensforFlow for whatever reason, there are other lightweight libraries that make it easy to incorporate elements of NLP into your applications.

The Python package known as flair or flairNLP library is one such resource. Here I’ll show you how to do a basic sentiment anaylsis of Hacker News comments using it.

from flair.models import TextClassifier

from flair.data import Sentence

from segtok.segmenter import split_single

import pandas as pd

import re





classifier = TextClassifier.load('en-sentiment')

First we load the English corpus, then we define some functions to process our text.

def clean(raw):

""" Remove hyperlinks and markup """

result = re.sub("<[a][^>]*>(.+?)</[a]>", 'Link.', raw)

result = re.sub('>', "", result)

result = re.sub(''', "'", result)

result = re.sub('"', '"', result)

result = re.sub('/', ' ', result)

result = re.sub('<p>', ' ', result)

result = re.sub('</i>', '', result)

result = re.sub('>', '', result)

result = re.sub('<i>', ' ', result)

result = re.sub("

", '', result)

return result

Clean and split our text into sentences using a splitting tool…

def make_sentences(text):

""" Break apart text into a list of sentences """

sentences = [sent for sent in split_single(text)]

return sentences

Next we turn out text into a flair Sentence object and call the classifer on it and retrieve the labels.

def predict(sentence):

""" Predict the sentiment of a sentence """

if sentence == "":

return 0

text = Sentence(sentence)

# stacked_embeddings.embed(text)

classifier.predict(text)

value = text.labels[0].to_dict()['value']

if value == 'POSITIVE':

result = text.to_dict()['labels'][0]['confidence']

else:

result = -(text.to_dict()['labels'][0]['confidence'])

return round(result, 3)

Helper functions to get the scores and sum of the scores for a comment...

def get_scores(sentences):

""" Call predict on every sentence of a text """

results = []



for i in range(0, len(sentences)):

results.append(predict(sentences[i]))

return results



def get_sum(scores):



result = round(sum(scores), 3)

return result

Now that are functions are defined we can apply them to our data!

df = pd.read_json('small.json')

df = df.dropna()

df = df.reset_index(drop=True)

df.text = df.text.apply(clean)

df['sentences'] = df.text.apply(make_sentences)

df['scores'] = df['sentences'].apply(get_scores)

df['scores_sum'] = df.scores.apply(get_sum)

Initially this approach seemed to work reasonably well, although after I had completed this project I saw that others had used the TextBlob library for sentiment anaylsis and had generally gotten better results.

The flair sentiment classifer was originally trained on IMDB movie review data and not Hacker News comments. This difference may explain some of the limitations of this classifer but it also appears that the TextBlob libary captures other dimensions of “sentiment” that the flair NLP library just doesn’t have.

You can see the full code and more predictions in my repo below.

my repo:

https://github.com/best-salitest-hacker-news-trolls/machine-learning-engineers