Real-time Twitter Sentiment Analysis for Brand Improvement and Topic Tracking (Chapter 1/3) | by Chulong Li | Towards Data Science

This is a comprehensive step-by-step tutorial to teach you how to build an end-to-end real-time Twitter monitoring system to track customer behaviors towards brands by identifying sentiments fluctuation, analyzing trending topics and geographic segmentation, and detecting anomaly on scandals, with consideration of improving brands’ customer engagement and retention.
You could try this awesome web app and find the full code!
Tech Stack: Python, Tweepy, Numpy, Pandas, Seaborn, Plotly, Dash, Scala Kafka, Spark, MySQL, RE, TextBlob, NLTK, HTML, CSS, Flask, JavaScript, React.js and Git
Since it’s a comprehensive end-to-end Twitter monitoring system, I’ll break it down into 3 chapters to explain the details under the hood and intuition behind the action.
Chapter 1 (You’re here!): Collecting Twitter Data using Streaming Twitter API with Tweepy, MySQL, & Python
Chapter 2: Twitter Sentiment Analysis and Interactive Data Visualization using RE, TextBlob, NLTK, and Plotly
Chapter 3: Deploy a Real-time Twitter Analytical Web App on Heroku using Dash & Plotly in Python
Chapter 4 (Optional): Parallelize Streaming Twitter Sentiment Analysis using Scala, Kafka and Spark Streaming
Why this system? This real-time end-to-end Twitter monitoring system is designed for the enterprise to evaluate Twitter data to inform business decisions. As we know twitter is a great place for the real-time high-throughput data source with 6000 tweets per seconds on average, we could use it to uncover the breaking news stories, identify industry-wide trends, and take actions on time. In practice, keep tracking all relevant Twitter content about a brand in real-time, perform analysis as topics or issues emerge, and detect anomaly with alerting. By monitoring brand mentions on Twitter, brands could inform enagement and deliver better experiences for their customers across the world.
To get started with Twitter APIs, we need to apply for a developer account first. It’ll allow you yo access Twitter APIs and other tools.
Then we create an app to generate two API keys and two access tokens. That’s the most important part to call Twitter APIs. The whole process may take a few minutes but should be done easily.
Now click on Details button on the right of your app in APPs under your Twitter Name on the top right (That’s NOT Dashboard button). Then click on Keys and tokens in the subtitle. Create a file called credentials.py, and copy & paste your Consumer API keys and Access token & access token secret into that file.
#credentials.pyAPI_KEY = 'XXXXXXXXXX'API_SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXX'ACCESS_TOKEN = 'XXXXXXXXXXXXXXXXXXXXXXX'ACCESS_TOKEN_SECRET = 'XXXXXXXXXXXXXXXXXXX'
We’ll use a python library called Tweepy to listen to streaming Twitter data. Create a new file called Main.ipynb and let’s get started. Note: I use Jupyter Notebook for better dev exploration, and you can use .py instead. You could check this Streaming With Tweepy online doc from Tweepy if you’re interested. You could check my full-code version here.
First, we need to create a Tweepy’s StreamListener with on_status method for receiving tweets and on_error method for stopping grabbing data due to rate limits. Tweets are known as “status updates”. So the Status class in  Tweepy has properties describing the tweet.
# Part of MyStreamListener in Main.ipynb# Streaming With Tweepy # Override tweepy.StreamListener to add logic to on_statusclass MyStreamListener(tweepy.StreamListener):    def on_status(self, status):        # Extract info from tweets        id_str = status.id_str        created_at = status.created_at        user_created_at = status.user.created_at        # ...... and more!         # I'll talk about it below! (Or check full-code link above)    def on_error(self, status_code):        '''        Since Twitter API has rate limits,         stop srcraping data as it exceed to the thresold.        '''        if status_code == 420:            # return False to disconnect the stream            return False
Then, use Oauth in Tweepy for authentication to register our application with Twitter.
# Oauth part in Main.ipynb# Import api/access_token keys from credentials.pyimport credentials.pyauth  = tweepy.OAuthHandler(credentials.API_KEY, \                            credentials.API_SECRET_KEY)auth.set_access_token(credentials.ACCESS_TOEKN,  \                      credentials.ACCESS_TOKEN_SECRET)api = tweepy.API(auth)
Now, start StreamListener with Oauth. In filter method, set tweet language preference and tracking words which is a list of words (e.g. [‘Facebook’ ]). Currently, we use Twitter’s free Standard Stream. Check Twitter Streaming API Documentation to see different streams with different capabilities and limitations.
myStreamListener = MyStreamListener()myStream = tweepy.Stream(auth = api.auth, listener = myStreamListener)myStream.filter(languages=["en"], track = settings.TRACK_WORDS)# However, this part won't be reached as the stream listener won't stop automatically. Press STOP button to finish the process.
Then, create and connect with MySQL local database for further analysis and storage. You could download the latest version here. The version we use is MySQL Community Edition, a very popular open source database. Set up the MySQL database according to the built-in guide, with database information provided below.
mydb = mysql.connector.connect(    host="localhost",    user="root",    passwd="password",    database="TwitterDB",    charset = 'utf8')
Then we need to create a new table if the table doesn’t exit. Of course, it should not exist during the first time. So we create it with following automatic-checking code.
if mydb.is_connected():    mycursor = mydb.cursor()    mycursor.execute("""        SELECT COUNT(*)        FROM information_schema.tables        WHERE table_name = '{0}'        """.format(TABLE_NAME))    if mycursor.fetchone()[0] != 1:        mycursor.execute("CREATE TABLE {} ({})" \            .format(TABLE_NAME, TABLE_ATTRIBUTES))        mydb.commit()    mycursor.close()
Specify the details for receiving and preprocessing Twitter Data in on_status method. The attributes in status is the same in Tweet Object.
id_str is Tweets Id in String format
created_at is the time when the tweet was created
text is the content of the tweet
polarity & subjectivity is the sentiment value by using TextBlob NLP library to perform core sentiment analysis
Attributes like user_created_at, user_location, user_description, and user_followers_count are in User Object, which is part of Tweet Object. In addition, coordinates including longitude and latitude is another object called Geo Object in JSON format.
# Extract info from tweetsdef on_status(self, status):    if status.retweeted:        # Avoid retweeted info, and only original tweets will         # be received        return True    # Extract attributes from each tweet    id_str = status.id_str    created_at = status.created_at    text = deEmojify(status.text)    # Pre-processing the text      sentiment = TextBlob(text).sentiment    polarity = sentiment.polarity    subjectivity = sentiment.subjectivity            user_created_at = status.user.created_at    user_location = deEmojify(status.user.location)    user_description = deEmojify(status.user.description)    user_followers_count =status.user.followers_count    longitude = None    latitude = None    if status.coordinates:        longitude = status.coordinates['coordinates'][0]        latitude = status.coordinates['coordinates'][1]                 retweet_count = status.retweet_count     favorite_count = status.favorite_count     # Quick check contents in tweets     print(status.text)     print("Long: {}, Lati: {}".format(longitude, latitude))             # Store all data in MySQL     if mydb.is_connected():         mycursor = mydb.cursor()         sql = "INSERT INTO {} (id_str,created_at,text,polarity,\            subjectivity, user_created_at, user_location,\            user_description, user_followers_count, longitude,\            latitude, retweet_count, favorite_count) VALUES \            (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)" \             .format(TABLE_NAME)          val = (id_str, created_at, text, polarity, subjectivity,\             user_created_at, user_location, user_description,\              user_followers_count, longitude, latitude,\              retweet_count, favorite_count)          mycursor.execute(sql, val)          mydb.commit()          mycursor.close()
Last, in order to allow tweet texts to store in MySQL, we need to do a little bit processing to remove emoji characters through striping all non-ASCII characters, even though the charset of database is utf8.
def deEmojify(text):    if text:        return text.encode('ascii', 'ignore').decode('ascii')    else:        return None
It’s my first technical article, and I appreciate for any feedback!