Real-time Twitter Sentiment Analysis for Brand Improvement and Topic Tracking (Chapter 2/3)
By: Chulong Li
Tags: NLP, Sentiment Analysis, Twitter, In Depth Analysis, Data Science

Loading & Preparing Twitter data

First, we need to extract the Twitter data from data source. You can directly read from sample_data.csv for simplicity.

df = pd.read_csv("sample_data.csv")

Or in a more formal way, extract data from our MySQL database, which has been set up and filled with real-time Twitter data in the Chapter 1.

db_connection = mysql.connector.connect(

host="localhost",

user="root",

passwd="password",

database="TwitterDB",

charset = 'utf8'

)

time_now = datetime.datetime.utcnow()

time_10mins_before = datetime.timedelta(hours=0,minutes=10)) \

.strftime('%Y-%m-%d %H:%M:%S'

time_interval = time_now - time_10mins_before

And read data released in last 30 minutes and load it into Pandas DataFrame via SQL query.

query = "SELECT id_str, text, created_at, polarity, \

user_location FROM {} WHERE created_at >= '{}'" \

.format(settings.TABLE_NAME, time_interval)

df = pd.read_sql(query, con=db_connection)

Then convert DATETIME (MySQL data type) into Datetime (Pandas data type).

# UTC for date time at default

df['created_at'] = pd.to_datetime(df['created_at'])

Sentiment Analysis with TextBlob

The core of sentiment analysis is to use TextBlob in order to extract the polarity & subjectivity from tweet texts, which is actually done by the data preprocessing for better data storage in the previous chapter. Negative tweets represent as -1, positive tweets represent as +1, and neutral tweets represent as 0 .

from textblob import TextBlob

sentiment = TextBlob(tweet_text).sentiment

polarity = sentiment.polarity

subjectivity = sentiment.subjectivity

Convert the entire time serises into groups of 2 seconds, and count the number of sentiment for each kind of polarities (e.g. -1, 0, and 1) in each time-interval group.

Apply unstack-stack technology to make sure all categories in each group are displayed even if one of categories doesn't have any value. As we only display real-time tweets posted in last 30 minutes, groups of 2-second interval could best display on the screen in practice. After that, rename the columns to allow them self explanatory.

# Clean and transform data to enable time series

result = df.groupby( \

[pd.Grouper(key='created_at', freq='2s'), 'polarity'] \

).count().unstack(fill_value=0).stack().reset_index()



result = result.rename(columns= \

{ "id_str": "Num of '{}' mentions".format(TRACK_WORD), \

"created_at":"Time in UTC" })

Record the time series with 2-second interval for further index usage.

time_series = result["Time in UTC"][result['polarity']==0] \

.reset_index(drop=True)

Use simple Plotly Express to quick visualize the line chart. Note: Plotly Express is a terse, consistent, high-level wrapper of plotly.graph_objects for rapid data exploration and figure generation.

import plotly.express as px

fig = px.line(result, x='Time in UTC', \

y="Num of '{}' mentions".format(TRACK_WORD), \

color='polarity')

fig.show()

Line Chart for Sentiment Analysis

Topic Tracking with Natural Language Processing using RE & NLTK

In order to tracking the hottest words or the most frequency tokens in the tweets, we join all tweet texts, remove the urls, clean the ‘RT’ and ‘&’ (a.k.a. ‘&’ )symbols, and convert all characters into lowercases.

content = ' '.join(df["text"])

content = re.sub(r"http\S+", "", content)

content = content.replace('RT ', ' ').replace('&', 'and')

content = re.sub('[^A-Za-z0-9]+', ' ', content)

content = content.lower()

Use python script to download these 2 files from NLTK at the first time. Punkt Sentence Tokenizer is used to divide a text into a list of sentences by using an unsupervised algorithm.

import nltk

nltk.download('punkt')

nltk.download('stopwords')

Then tokenize the entire text from all tweets, use Stop Words to remove commonly used words, and extract 10 most common words in the Frequency Distribution of all words.

from nltk.probability import FreqDist

from nltk.tokenize import word_tokenize

from nltk.corpus import stopwords tokenized_word = word_tokenize(content)

stop_words=set(stopwords.words("english"))

filtered_sent=[]

for w in tokenized_word:

if w not in stop_words:

filtered_sent.append(w)

fdist = FreqDist(filtered_sent)

fd = pd.DataFrame(fdist.most_common(10), \

columns = ["Word","Frequency"]).drop([0]).reindex()

Use simple Plotly Express again to quick visualize the bar chart.

import plotly.express as px

fig = px.bar(fd, x="Word", y="Frequency")

fig.update_traces(marker_color='rgb(240,128,128)', \

marker_line_color='rgb(8,48,107)', \

marker_line_width=1.5, opacity=0.8)

fig.show()

Bar Chart for Topic Tracking

Geographic Segmentation Recognization with Text Processing

To explore users’ geographic distributions, we need to identify their locations through their user profile rather than the locations attached with tweets, since less than 1% people will attach their tweet locations. However, according to locations in users’ profile, they may include one or more of counties, cities, states, countries, or planets. Thus, filtering these data into US state-level locations is the core of geographic segmentation recognization.

Set up all state names and their abbreviations as constants for further abbreviation-name transformation.

STATES = ['Alabama', 'AL', 'Alaska', 'AK', \

...... # Text hided for readability \

'WV', 'Wisconsin', 'WI', 'Wyoming', 'WY'] STATE_DICT = dict(itertools.zip_longest(*[iter(STATES)] * 2, fillvalue=""))

INV_STATE_DICT = dict((v,k) for k,v in STATE_DICT.items())

Extract the state information from their locations by iterating the list of state names and the list of user locations.

is_in_US=[]

geo = df[['user_location']]

df = df.fillna(" ")

for x in df['user_location']:

check = False

for s in STATES:

if s in x:

is_in_US.append(STATE_DICT[s] if s in STATE_DICT else s)

check = True

break

if not check:

is_in_US.append(None)



geo_dist = pd.DataFrame(is_in_US, columns['State']) \

.dropna().reset_index()

Count the number of tweets posted in each state of US, and use logarithmic number to avoid the extreme values (e.g. 500+ in California and 3 in North Dakota) for better visualization.

geo_dist = geo_dist.groupby('State').count(). \

rename(columns={"index": "Number"}).sort_values( \

by=['Number'], ascending=False).reset_index()

geo_dist["Log Num"] = geo_dist["Number"] \

.apply(lambda x: math.log(x, 2))

Add explanatory text information for the hover text on later dashboard.

geo_dist['Full State Name'] = geo_dist['State'] \

.apply(lambda x: INV_STATE_DICT[x])

geo_dist['text'] = geo_dist['Full State Name'] + '<br>' + \

'Num: ' + geo_dist['Number'].astype(str)

This time, we use Plotly (not Plotly Express) to visualize the USA map.

Note: plotly.graph_objects is the core of Plotly Library containing more generic functions for more complex usage. locationmode is the set of locations match entries in “locations”. text is the hover text. marker_line_color is the color of line markers between states. Set geo_scope to limit map scope to USA.