Real-time Twitter Sentiment Analysis for Brand Improvement and Topic Tracking (Chapter 2/3) | by Chulong Li | Towards Data Science

The tutorial will teach you how to step-by-step apply 1) Nature Language Processing and Sentiment Analysis on Twitter data, and 2) utilize Plotly to build an interactive data visualization.
This a standalone component to perform sentiment analysis & topic tracking and build the analytical dashboard on Jupiter Notebook, although it’s the second part of my comprehensive real-time Twitter monitoring system tutorial. Streaming Twitter data collection is explained and implemented in previous chapter.
Chapter 1: Collecting Twitter Data using Streaming Twitter API with Tweepy, MySQL, & Python
Chapter 2 (You’re here!): Twitter Sentiment Analysis and Interactive Data Visualization using RE, TextBlob, NLTK, and Plotly
Chapter 3: Deploy a Real-time Twitter Analytical Web App on Heroku using Dash & Plotly in Python
Chapter 4 (Optional): Parallelize Streaming Twitter Sentiment Analysis using Scala, Kafka and Spark Streaming
The analytical dashboard (with full code) above is what we’ll do today, which lays the foundation of the Real-time Twitter Monitoring Web App below in next chapter, as Dash (python framework for analytical web apps) is written on top of Plotly. Check my GitHub Repo for more details.
Tech Stack of Chapter: RE, NLTK, TextBlob, Plotly, MySQL, Python (time, datetime, math, itertools)
RE: Regular expression operations to identify whether a pattern exists in a given sequence of characters or not
NLTK: Natural Language Toolkit, a leading platform for building Python programs to work with human language data
TextBlob: Natural Language Processing library for processing textual data, providing a simple API for diving into common NLP.
Plotly: An interactive, open-source, and browser-based graphing library for Python
First, we need to extract the Twitter data from data source. You can directly read from sample_data.csv for simplicity.
df = pd.read_csv("sample_data.csv")
Or in a more formal way, extract data from our MySQL database, which has been set up and filled with real-time Twitter data in the Chapter 1.
db_connection = mysql.connector.connect(    host="localhost",    user="root",    passwd="password",    database="TwitterDB",    charset = 'utf8' )time_now = datetime.datetime.utcnow()time_10mins_before = datetime.timedelta(hours=0,minutes=10)) \    .strftime('%Y-%m-%d %H:%M:%S'time_interval = time_now - time_10mins_before
And read data released in last 30 minutes and load it into Pandas DataFrame via SQL query.
query = "SELECT id_str, text, created_at, polarity,          \         user_location FROM {} WHERE created_at >= '{}'"     \        .format(settings.TABLE_NAME, time_interval)df = pd.read_sql(query, con=db_connection)
Then convert DATETIME (MySQL data type) into Datetime (Pandas data type).
# UTC for date time at defaultdf['created_at'] = pd.to_datetime(df['created_at'])
The core of sentiment analysis is to use TextBlob in order to extract the polarity & subjectivity from tweet texts, which is actually done by the data preprocessing for better data storage in the previous chapter. Negative tweets represent as -1, positive tweets represent as +1, and neutral tweets represent as 0 .
from textblob import TextBlobsentiment = TextBlob(tweet_text).sentimentpolarity = sentiment.polaritysubjectivity = sentiment.subjectivity
Convert the entire time serises into groups of 2 seconds, and count the number of sentiment for each kind of polarities (e.g. -1, 0, and 1) in each time-interval group.
Apply unstack-stack technology to make sure all categories in each group are displayed even if one of categories doesn't have any value. As we only display real-time tweets posted in last 30 minutes, groups of 2-second interval could best display on the screen in practice. After that, rename the columns to allow them self explanatory.
# Clean and transform data to enable time seriesresult = df.groupby(                                        \      [pd.Grouper(key='created_at', freq='2s'), 'polarity'] \    ).count().unstack(fill_value=0).stack().reset_index()result = result.rename(columns=                             \    { "id_str": "Num of '{}' mentions".format(TRACK_WORD),  \      "created_at":"Time in UTC" })
Record the time series with 2-second interval for further index usage.
time_series = result["Time in UTC"][result['polarity']==0]  \     .reset_index(drop=True)
Use simple Plotly Express to quick visualize the line chart. Note: Plotly Express is a terse, consistent, high-level wrapper of plotly.graph_objects for rapid data exploration and figure generation.
import plotly.express as pxfig = px.line(result, x='Time in UTC',                      \    y="Num of '{}' mentions".format(TRACK_WORD),            \    color='polarity')fig.show()
In order to tracking the hottest words or the most frequency tokens in the tweets, we join all tweet texts, remove the urls, clean the ‘RT’ and ‘&’ (a.k.a. ‘&amp;’ )symbols, and convert all characters into lowercases.
content = ' '.join(df["text"])content = re.sub(r"http\S+", "", content)content = content.replace('RT ', ' ').replace('&amp;', 'and')content = re.sub('[^A-Za-z0-9]+', ' ', content)content = content.lower()
Use python script to download these 2 files from NLTK at the first time. Punkt Sentence Tokenizer is used to divide a text into a list of sentences by using an unsupervised algorithm.
import nltknltk.download('punkt')nltk.download('stopwords')
Then tokenize the entire text from all tweets, use Stop Words to remove commonly used words, and extract 10 most common words in the Frequency Distribution of all words.
from nltk.probability import FreqDistfrom nltk.tokenize import word_tokenizefrom nltk.corpus import stopwordstokenized_word = word_tokenize(content)stop_words=set(stopwords.words("english"))filtered_sent=[]for w in tokenized_word:    if w not in stop_words:        filtered_sent.append(w)fdist = FreqDist(filtered_sent)fd = pd.DataFrame(fdist.most_common(10),                    \    columns = ["Word","Frequency"]).drop([0]).reindex()
Use simple Plotly Express again to quick visualize the bar chart.
import plotly.express as pxfig = px.bar(fd, x="Word", y="Frequency")fig.update_traces(marker_color='rgb(240,128,128)',          \    marker_line_color='rgb(8,48,107)',                      \    marker_line_width=1.5, opacity=0.8)fig.show()
To explore users’ geographic distributions, we need to identify their locations through their user profile rather than the locations attached with tweets, since less than 1% people will attach their tweet locations. However, according to locations in users’ profile, they may include one or more of counties, cities, states, countries, or planets. Thus, filtering these data into US state-level locations is the core of geographic segmentation recognization.
Set up all state names and their abbreviations as constants for further abbreviation-name transformation.
STATES = ['Alabama', 'AL', 'Alaska', 'AK',                    \     ...... # Text hided for readability                       \    'WV', 'Wisconsin', 'WI', 'Wyoming', 'WY']STATE_DICT = dict(itertools.zip_longest(*[iter(STATES)] * 2, fillvalue=""))INV_STATE_DICT = dict((v,k) for k,v in STATE_DICT.items())
Extract the state information from their locations by iterating the list of state names and the list of user locations.
is_in_US=[]geo = df[['user_location']]df = df.fillna(" ")for x in df['user_location']:    check = False    for s in STATES:        if s in x:            is_in_US.append(STATE_DICT[s] if s in STATE_DICT else s)            check = True            break    if not check:        is_in_US.append(None)geo_dist = pd.DataFrame(is_in_US, columns['State'])           \    .dropna().reset_index()
Count the number of tweets posted in each state of US, and use logarithmic number to avoid the extreme values (e.g. 500+ in California and 3 in North Dakota) for better visualization.
geo_dist = geo_dist.groupby('State').count().                 \     rename(columns={"index": "Number"}).sort_values(          \    by=['Number'], ascending=False).reset_index()geo_dist["Log Num"] = geo_dist["Number"]                      \    .apply(lambda x: math.log(x, 2))
Add explanatory text information for the hover text on later dashboard.
geo_dist['Full State Name'] = geo_dist['State']               \    .apply(lambda x: INV_STATE_DICT[x])geo_dist['text'] = geo_dist['Full State Name'] + '<br>' +     \    'Num: ' + geo_dist['Number'].astype(str)
This time, we use Plotly (not Plotly Express) to visualize the USA map.
Note: plotly.graph_objects is the core of Plotly Library containing more generic functions for more complex usage.locationmode is the set of locations match entries in “locations”. text is the hover text. marker_line_color is the color of line markers between states. Set geo_scope to limit map scope to USA.
import plotly.graph_objects as gofig = go.Figure(data=go.Choropleth(    locations=geo_dist['State'], # Spatial coordinates    z = geo_dist['Log Num'].astype(float), # Data to be color-coded    locationmode = 'USA-states',     colorscale = "Reds",    text=geo_dist['text'],    marker_line_color='white', # line markers between states    colorbar_title = "Numbers in Log2"))fig.update_layout(    geo_scope='usa', )fig.show()
Now we could integrate all data visualization parts into one dashboard using Plotly.subplots. Displaying multiple figures simultaneously can greatly improve the reading efficiency and enhance the comparability among multiple insights.
First, create a Plotly Figure containing 2×2 subplots with Line Chart in the left, Map in the top right, and Bar Chart in the bottom right.
from plotly.subplots import make_subplotsfig = make_subplots(        rows=2, cols=2,        column_widths=[1, 0.4],        row_heights=[0.6, 0.4],        specs=[[{"type": "scatter", "rowspan": 2},                 {"type": "choropleth"}],               [None, {"type": "bar"}]]        )
Add three Lines of negatives, neutrals, and positives in the first subplot using add_trace and go.Scatter . In addition, row and col represent the place of this subplot in the big figure.
fig.add_trace(go.Scatter(    x=time_series,    y=result["Num of '{}' mentions"                     \        .format(settings.TRACK_WORDS[0])]               \        [result['polarity']==0].reset_index(drop=True), \    name="Neural",    opacity=0.8), row=1, col=1)   fig.add_trace(go.Scatter(    x=time_series,    y=result["Num of '{}' mentions"                     \        .format(settings.TRACK_WORDS[0])]               \         [result['polarity']==-1].reset_index(drop=True),    name="Negative",    opacity=0.8), row=1, col=1)fig.add_trace(go.Scatter(    x=time_series,    y=result["Num of '{}' mentions"                     \        .format(settings.TRACK_WORDS[0])]               \        [result['polarity']==1].reset_index(drop=True), \    name="Positive",    opacity=0.8), row=1, col=1)
Add the Bar Chart for the topic frequency distribution using add_trace and go.Bar . You may use rgb(xx,xx,xx) or rgba(xx,xx,xx,x) to change the color of certain elements in the figure.
fig.add_trace(go.Bar(x=fd["Word"], y=fd["Frequency"],       \    name="Freq Dist"), row=2, col=2)fig.update_traces(marker_color='rgb(59, 89, 152)',          \    marker_line_color='rgb(8,48,107)',                      \    marker_line_width=0.5, opacity=0.7, row=2, col=2)
Then insert the Map in the top right, and set locations and numbers per location.
fig.add_trace(go.Choropleth(    locations=geo_dist['State'], # Spatial coordinates    z = geo_dist['Log Num'].astype(float), # Data to be color-coded    locationmode = 'USA-states',     colorscale = "Blues",    text=geo_dist['text'], # hover text    showscale=False,    geo = 'geo'    ), row=1, col=2)
Add title in the layout of our figure, reduce the geo-scope of our map, turn the template theme into the dark, and add annotations for the layout using go.layout.Annotation.
fig.update_layout(    title_text =                                            \      "Real-time tracking '{}' mentions on Twitter {} UTC"  \      .format(settings.TRACK_WORDS[0]                ,datetime.datetime.utcnow().strftime('%m-%d %H:%M') \      ),    geo = dict(        scope='usa',    ),    template="plotly_dark",    margin=dict(r=20, t=50, b=50, l=20),    annotations=[        go.layout.Annotation(            text="Source: Twitter",            showarrow=False,            xref="paper",            yref="paper",            x=0,            y=0)    ],    showlegend=False,    xaxis_rangeslider_visible=True)
Last, display all subplots in the single figure.
fig.show()
Author’s Words: A little bit complex in this chapter as it uses NLP and sentiment analysis approaches to transfer data points into insights. Further improvement on topic tracking can be implemented by using advanced NLP methods.
Next chapter will be the most exciting part to integrate and deploy all functions on the Heroku server using Dash. As always, I appreciate for any feedback! 😃
Update on 9/20: Chapter 3 has been published !