Real-time Twitter Sentiment Analysis for Brand Improvement and Topic Tracking (Chapter 1/3)
By: Chulong Li
Tags: Sentiment Analysis, Twitter, In Depth Analysis, Data Science, Live Streaming Data

Photo by Sebastian B on Unsplash

Streaming Twitter Data with Tweepy, MySQL, & Python

We’ll use a python library called Tweepy to listen to streaming Twitter data. Create a new file called Main.ipynb and let’s get started. Note: I use Jupyter Notebook for better dev exploration, and you can use .py instead. You could check this Streaming With Tweepy online doc from Tweepy if you’re interested. You could check my full-code version here.

First, we need to create a Tweepy’s StreamListener with on_status method for receiving tweets and on_error method for stopping grabbing data due to rate limits. Tweets are known as “status updates”. So the Status class in

Tweepy has properties describing the tweet.

# Part of MyStreamListener in Main.ipynb

# Streaming With Tweepy

# Override tweepy.StreamListener to add logic to on_status

class MyStreamListener(tweepy.StreamListener):



def on_status(self, status):

# Extract info from tweets

id_str = status.id_str

created_at = status.created_at

user_created_at = status.user.created_at

# ...... and more!

# I'll talk about it below! (Or check full-code link above) def on_error(self, status_code):

'''

Since Twitter API has rate limits,

stop srcraping data as it exceed to the thresold.

'''

if status_code == 420:

# return False to disconnect the stream

return False

Then, use Oauth in Tweepy for authentication to register our application with Twitter.

# Oauth part in Main.ipynb

# Import api/access_token keys from credentials.py

import credentials.py

auth = tweepy.OAuthHandler(credentials.API_KEY, \

credentials.API_SECRET_KEY)

auth.set_access_token(credentials.ACCESS_TOEKN, \

credentials.ACCESS_TOKEN_SECRET)

api = tweepy.API(auth)

Now, start StreamListener with Oauth. In filter method, set tweet language preference and tracking words which is a list of words (e.g. [‘Facebook’ ]). Currently, we use Twitter’s free Standard Stream. Check Twitter Streaming API Documentation to see different streams with different capabilities and limitations.

myStreamListener = MyStreamListener()

myStream = tweepy.Stream(auth = api.auth, listener = myStreamListener)

myStream.filter(languages=["en"], track = settings.TRACK_WORDS) # However, this part won't be reached as the stream listener won't stop automatically. Press STOP button to finish the process.

Then, create and connect with MySQL local database for further analysis and storage. You could download the latest version here. The version we use is MySQL Community Edition, a very popular open source database. Set up the MySQL database according to the built-in guide, with database information provided below.

mydb = mysql.connector.connect(

host="localhost",

user="root",

passwd="password",

database="TwitterDB",

charset = 'utf8'

)

Then we need to create a new table if the table doesn’t exit. Of course, it should not exist during the first time. So we create it with following automatic-checking code.

if mydb.is_connected():

mycursor = mydb.cursor()

mycursor.execute("""

SELECT COUNT(*)

FROM information_schema.tables

WHERE table_name = '{0}'

""".format(TABLE_NAME))

if mycursor.fetchone()[0] != 1:

mycursor.execute("CREATE TABLE {} ({})" \

.format(TABLE_NAME, TABLE_ATTRIBUTES))

mydb.commit()

mycursor.close()

Specify the details for receiving and preprocessing Twitter Data in on_status method. The attributes in status is the same in Tweet Object.

id_str is Tweets Id in String format

created_at is the time when the tweet was created

text is the content of the tweet

polarity & subjectivity is the sentiment value by using TextBlob NLP library to perform core sentiment analysis

Attributes like user_created_at, user_location, user_description, and user_followers_count are in User Object, which is part of Tweet Object. In addition, coordinates including longitude and latitude is another object called Geo Object in JSON format.

# Extract info from tweets

def on_status(self, status):

if status.retweeted:

# Avoid retweeted info, and only original tweets will

# be received

return True

# Extract attributes from each tweet

id_str = status.id_str

created_at = status.created_at

text = deEmojify(status.text) # Pre-processing the text

sentiment = TextBlob(text).sentiment

polarity = sentiment.polarity

subjectivity = sentiment.subjectivity



user_created_at = status.user.created_at

user_location = deEmojify(status.user.location)

user_description = deEmojify(status.user.description)

user_followers_count =status.user.followers_count

longitude = None

latitude = None

if status.coordinates:

longitude = status.coordinates['coordinates'][0]

latitude = status.coordinates['coordinates'][1]



retweet_count = status.retweet_count

favorite_count = status.favorite_count # Quick check contents in tweets

print(status.text)

print("Long: {}, Lati: {}".format(longitude, latitude))



# Store all data in MySQL

if mydb.is_connected():

mycursor = mydb.cursor()

sql = "INSERT INTO {} (id_str,created_at,text,polarity,\

subjectivity, user_created_at, user_location,\

user_description, user_followers_count, longitude,\

latitude, retweet_count, favorite_count) VALUES \

(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)" \

.format(TABLE_NAME)

val = (id_str, created_at, text, polarity, subjectivity,\

user_created_at, user_location, user_description,\

user_followers_count, longitude, latitude,\

retweet_count, favorite_count)

mycursor.execute(sql, val)

mydb.commit()

mycursor.close()

Last, in order to allow tweet texts to store in MySQL, we need to do a little bit processing to remove emoji characters through striping all non-ASCII characters, even though the charset of database is utf8.