Machine Learning Python Investing Social Media
By: Szymon Zaczek
Tags: Machine Learning, Social Media, Data Visualization, Data Science, NLP

Data science

Leveraging ML and social media data to improve investing strategies based on the current market sentiment

Creating pipeline for visualising results of the sentiment analysis of the latest Tweets

Image by Open Studio from Adobe Stock

Machine learning and investing are among the most commonly searched-for topics related to practical applications of computer science. Unfortunately, they’re also often abused by self-proclaimed data scientists who have only watched a few tutorials on YouTube and claim that by applying linear regression to previous values of a given stock you are very likely to predict its future price.

However, nothing could be further from the truth. Investing in any financial instrument is a very complex matter. It is extremely challenging to teach a computer how to mimic informed business decisions — for instance, what would be the best time to buy a particular stock. Such decisions should be made based on the extensive expertise and rigorous research — no one can make a better decision about what to do with your money than yourself. However, I do believe that machine learning can buy you one of the most precious commodities in today’s world: time.

If you are an active investor, you most likely regularly skim through news websites to find out whether some new governmental regulation or yet another global event will drag down the price of the stock you have just purchased. If that’s the case, buckle up, and let me show you how you can set up your own sentiment analysis pipeline using the latest Tweets about the financial instrument of your choice.

Sidenote

Please bear in mind, that this article does not aim to be a comprehensive Python tutorial — it showcases a specific Python application. If you are a complete beginner, I recommend checking out the following article about the best resources for learning Python:

Or if you are specifically interested in learning basics of Machine Learning with Python, feel free to check out the article written by my colleague:

Getting started

The first thing you need to do to get started is getting credentials to use the official Twitter API (Application Programming Interface). You can find an excellent tutorial on how to do this here: click

To go ahead, you will need:

an API key,

an API secret key,

an Access token,

an Access token secret.

There are several ways to use API credentials, with the easiest one being simply to include API keys and tokens directly in the program itself. However, this is not a good idea: if your program is published online for whatever reason, your credentials would become publicly available. Instead, I recommend using credentials as conda environment variables. To do so, you need to download Anaconda or Miniconda distributions with the help of the official tutorial: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html). Conda is primarily a package and an environment management system that features Python and its most popular libraries for data analysis by default. We will use them later on.

Once Conda is installed, you should create and activate a separate environment for your project. You can do so by typing the following:

conda create --name twitter_sentiment python=3.7

conda activate twitter_sentiment

If you use Linux distribution or Mac, type it in the terminal. If you use Windows, type it in the Anaconda Prompt. Next, we’ll proceed to setting environment variables.

Setting environment variables — Linux/Mac

Next, let’s put Twitter API keys as variables unique to your environment. In the terminal write:

cd $CONDA_PREFIX

Afterwards, you need to create a new `etc` directory, and in this directory you should have two more directories: `activate.d` and `deactivate.d`.

mkdir ./etc

mkdir ./etc/conda/activate.d

mkdir ./etc/conda/deactivate.d

This is the step where you include credentials in a file. Change the directory to ./etc/activate.d and then edit a file called env_vars.sh with your favorite text editor (I love vi for these purposes) with your Twitter API credentials:

cd ./etc/conda/activate.d

vim env_vars.sh

Content of env_vars.sh file should be as follow:

#!/bin/sh

export consumer_key='your_consumer_key'

export consumer_secret='your_consumer_secret'

export access_token_key='your_acess_token_key'

export access_token_secret='your_acess_token_secret'

Next, change the directory to ../deactivate.d and then create another env_cars.sh file:

cd ../deactivate.d

vim env_vars.sh

Populate env_vars.sh file with the following content:

#!/bin/sh

unset consumer_key

unset consumer_secret

unset access_token_key

unset access_token_secret

Your credentials are now all ready. Right now, you can temporarily deactivate your environment (this must be done for envrionmental variables to be set properly), switch your current working directory to wherever you want it to be (let’s assume that this is $HOME/twitter_sentiment ) and reactivate Conda environment so that you :

conda deactivate

mkdir $HOME/twitter_sentiment

cd $HOME/twitter_sentiment

conda activate twitter_sentiment

Setting environment variables — Windows

This process is very similar to the one for Linux or Mac but there are some tiny changes, due to the fact that Windows does not support Bash shell. Once your new environment has been activated, feel free to change directory to the main directory of the environment and create necessary directories along with env_vars.bat files — type following commands in Anaconda Prompt:

cd %CONDA_PREFIX%

mkdir .\etc\conda\activate.d

mkdir .\etc\conda\deactivate.d

type NUL > .\etc\conda\activate.d\env_vars.bat

type NUL > .\etc\conda\deactivate.d\env_vars.bat

Then, open file .\etc\conda\activate.d\env_vars.bat with any text editor (you can use notepad):

cd ./etc/conda/activate.d

notepad env_vars.bat

And insert your Twitter API credentials to env_vars.bat file:

set consumer_key='your_consumer_key'

set consumer_secret='your_consumer_secret'

set access_token_key='your_acess_token_key'

set access_token_secret='your_acess_token_secret'

Next, change the directory to ../deactivate.d and then edit env_vars.bat file that is located there:

cd ../deactivate.d

notepad env_vars.sh

with the following content:

set consumer_key=

set consumer_secret=

set access_token_key=

set access_token_secret=

Your credentials are now all ready. Right now, you should temporarily deactivate your environment (this must be done for envrionmental variables to be set properly), switch your current working directory to wherever you want it to be (let’s assume that this is %HOME%/twitter_sentiment ) and reactivate Conda environment so that you :

conda deactivate

mkdir %HOME%/twitter_sentiment

cd %HOME%/twitter_sentiment

conda activate twitter_sentiment

Installing NLTK and python-tweeter packages

Before writing some code, we need to install packages that will allow us to do the job we intend to do — that is communicating with Tweeter API and then applying sentiment analysis to the text from acquired Tweets. For the former I suggest using python-twitter module. For the latter, there are many excellent choices, and one of the most popular ones is VADER. It has been integrated into the most powerful Natural Language Processing engine that Python has to offer — NLTK (Natural Language Toolkit). Just type in following commands:

pip install python-twitter

pip install nltk

python -m nltk.downloader vader_lexicon

This will install python-twitter and NLTK packages as well as VADER extension to the NLTK library. Now, let’s proceed to the actual code itself.

Sidenote

If you do not use conda distribution of Python, you will also need to install matplotlib and seaborn packages — you can do so by typing in the console:

pip install matplotlib

pip install seaborn

Coding — communicating with Twitter API

For your convenience, the code that I will use here can be also found on my Github page: https://github.com/szymonzaczek/towards-data-science/

Let’s start with importing the necessary packages.

The `Twitter` package is the Twitter API Python module; `os` allows us to interact with the operating system; `datetime` is used for dealing with dates (we will need it because we will try to get the most recent Tweets); and the `re` package enables using regular expressions. We will use `pandas` for a convenient way to process data, and `matplotlib.pyplot` and `seaborn` to create very informative and elegant plots. `SentimentIntensityAnalyzer` will be our workhorse — this is the class that will be used to evaluate the sentiment of given Tweets.

The next step is to set up our API access. Since we use a conda environment, we can easily access our credentials by calling the `os.environ.get()` method, and use the results of those calls as an argument to the actual Twitter.API object. Bear in mind that we will be taking advantage of `tweet_mode=”extended”`, since we do not want to truncate Tweets and we want to analyze them whole.

Next, we need to choose the keywords that will be the basis of our search for relevant Tweets. They should be chosen wisely — for instance, if you were to choose `apple` as a keyword, most of the search results would probably refer to the tech company, but some of them could also relate to our beloved fruit. Ideally, those keywords should not have multiple meanings. We will search for Twitter posts that include all of the specified keywords.For the program to run properly, pick between one and three keywords. I chose `”crude”` and `”oil”` and grouped the keywords in a list.

Now, let’s establish what day it is today. Python has a very neat and concise way to do so: since we have already imported the `datetime` module, just call the `datetime.date.today()` method and assign its value to a variable:

It’s at this point that we need to include our code within the `while` loop. Why? I will get to that slightly later, but for now, we’ll just initiate the loop:

From now on, any code should be indented (indentation in Python is treated as either four spaces or a tab by default); that is until we leave the loop. Then, we proceed to construct a query for Twitter search. There are several ways to do so, and I chose to search Twitter with the `raw_query` method. This means that the API will be called directly with a query that mimics the query search used in the browser (for more info, please see https://python-twitter.readthedocs.io/en/latest/searching.html). This offers a neat way to handle time frames for searches. However, it requires you to very strictly adhere to formatting rules that you can nevertheless forget about once the query is complete. Since we want to use time frames for searching Twitter, we must format `date_for_query` according to the `raw_query` requirements (which is a pretty standard way to format dates, by the way):

And, then, the query is constructed using string concatenation as follows:

If you are a beginner Python user, this might look pretty terrifying, but keep in mind that it has everything you need and it will automatically adapt to your use case. The `query` variable will contain all of the earlier specified keywords (by using `”%20".join(keywords)` command), the right day (`date_formatted`) and some other stuff that is required from `raw_query` ()(such as `%20` and `%3A` tags, filtering out links, keeping only replies without citation of the original post, etc.). Afterwards, we need to call the actual search method while assigning its result to a variable:

Just like this, we already have Twitter search results. However, we still don’t know what exactly is in there. If an analysis of any social media content is supposed to guide investing strategies, we should make sure that this analysis is not made using excluded examples but rather a batch of posts. Let’s assume that for our purposes we will need at least 30 Tweets. This is why we encapsulated our code in a `while` loop: right now we can specify that if our search results in more than 30 Tweets, we will exit the loop:

Here comes the tricky part. What do we do if we do not have enough Tweets? We can, of course, expand our search to earlier dates. To do so, we just need to change the value of the `date_for_query` variable to the day prior to the current `date_for_query`. Since the code is now in a loop, this will be easy. But, before doing so, let’s check if the current date in the `date_for_query` variable is not older than seven days. This is important for two reasons. Firstly, Tweets older than that should not really influence the current sentiment regarding your targeted keywords. Secondly,Twitter’s basic API does not allow searching for Tweets that are older than seven days. Mathematical operations on `datetime` objects can be done using the `datetime.timedelta()` method. Also, we can raise ValueError with an elegant error message if our search is already at its limit. It’s not really Pythonic, but here we’d rather be overeager than leave users with a hardly informative message (due to the very long string, formatting here is not perfect).

If we are not at the limit yet, we can widen our search to the previous day. This can be done pretty easily:

And, just like this, we are done with direct interactions with Twitter API. The following code sums up the whole `while` loop:

Using this piece of code, we will either receive at least 30 `twitter.models.Status` objects (contained within the `search_results` list), or we will be prompted by an error message stating that there were not enough Tweets found for given keywords. In this case, you should modify your keywords. Either use fewer of them, or try more general ones, because each keyword from the `keywords` list must be found in each Tweet that will be the result of your search.

Data processing

The next step is to extract the text of individual Tweets from the list of `twitter.models.Status` objects that are contained in the `search_results` list. Remember that our goal is to perform sentiment analysis of tweets and to get an understanding of people’s current feelings about the keywords we chose. In my case, it’s going to be `crude` and `oil`. For this analysis to be as unbiased as possible, we should make sure that each Tweet was posted by a different user. So, we will start with initializing two lists:

Now, we can start populating these lists. For this purpose we will need a loop, in which we will iterate over our `search_results`. For each iteration, we will check if a given Tweet was posted by a unique user. If there were multiple Tweets posted by the same user, we will use only the first one we found. For this purpose, we might use the `not in` membership operator embedded in the `if/else` clause. If there are multiple posts in our search results by the same author, we will just go to the next post using the `continue` statement. If we have a post by an author who was not featured in our search previously, let’s extract the `full_text` attribute from the Tweet (which, of course, is the text of the Tweet):

Twitter users quite often overuse various hashtags and mention other people. I see hardly any reason to include those in the sentiment analysis, so let’s remove them. This can be done using regular expressions — a powerful tool for looking up strings within strings. Importantly, regular expressions are programming-language agnostic, and, as a result, they are a great, universal tool to have in your pocket. In Python, the `re` module is responsible for dealing with them and its `re.sub()` method that allows substituting a string with another string. In this case, we are looking for words that start with either `@` or `#`, and we simply remove those words by replacing them with an empty string.

Also, very short posts tend to pose a challenge in sentiment evaluations, so we might want to make sure that the posts that will be analyzed are at least 20 characters long. Also, if there were any hashtags or mentions at the beginning of the Tweet, there will be whitespace in front of such posts . Let’s get rid of that by using `lstrip()` method:

To sum up extracting the actual text from Tweets, this is the whole loop that populates the `tweets_text` list:

Sentiment analysis of the acquired Tweets

So, until now, we have searched Twitter for Tweets containing specified keywords and we extracted text from them. The sentiment analysis, which is the exciting bit, begins now. Truth to be told, this part will not require a lot of code since we only need to feed the text to the `SentimentIntensityAnalyzer` object and extract the desired output. In this case, it will be the `compound` score taken from the `polarity_scores` method that is made available by the `SentimentIntensityAnalyzer` class. The `compound` value describes the overall sentiment of the provided text. If the sentiment of the text is positive, its `compound` score is positive. If the sentiment is negative, its `compound` score is negative. When the text is rather neutral, the `compound` score is very close to 0. The value of the `compound` score is accessed as a key from a dictionary that is the result of running the `polarity_scores` method. What’s more, the `compound` score is a floating point value formatted with four decimal digits. Since such precision is not really necessary for our purposes, let’s reduce it to two decimal digits. Last but not least: let’s embed using `SentimentIntensityAnalyzer` class (so basically our Sentiment Analysis engine) in a function and let’s decorate it with type hinting, for both the input (`tweet_text: str`) and the output (` -> float`):

To sum up this part, the `vader_sentiment_score` function takes as an argument the text of a single tweet and it returns a float that describes the sentiment itself.

Now, we can finally proceed to analyze our Tweets. Let’s create another list in which we’ll store the results and perform some machine learning by iterating over Tweets and feeding them to the function that performs the sentiment analysis:

And voila! You have just performed a sentiment analysis of Tweets. It wasn’t so hard, was it? You have officially applied machine learning to real-world data and you have accessed it through the API of one of the most popular websites in the world. Quite impressive, I’d say. But beware, we are still nowhere near the end. Even though we have the data, we haven’t even seen the results of our efforts yet. You can obviously just print the numerical values if you want but this will not be very informative. So, let’s generate a neat, high-quality bar plot that will sum up all of the findings. For this purpose, let’s assign the floating point numbers obtained in the earlier step to their categories (negative, neutral and positive) using simple `if`/`elif` statements, along with creating a `sentiment_list` that will store the results of this operation:

Visualising the results

And now, let’s take a look at how to create a bar plot. There are a few ways of doing so but my preferred one is using the great and versatile `seaborn` module. For our purpose, we might just create two lists that will contain data for the x and y axes of the plot. Let’s call them conveniently `x_axis` and `y_axis`. The first one will contain only our categories, whereas the second one will contain the count of each category. This is how they can be created:

If you cannot wait to see the results, you can visualize the outcome of your own sentiment analysis just by calling `sns.barplot(x=x_axis, y=y_axis)`. However, there are some additional things that might make our plot look much more smoother. I very much enjoy visualizations that communicate the findings as conveniently as possible. This can be aided by an appropriate and thoughtful use of colors. For instance, let’s make the negative Tweets red, the neutral ones blue, and the positive ones green. We can do that by changing the color pallette of the seaborn module:

colors = [“firebrick”, “dodgerblue”, “limegreen”]

sns.set_palette(sns.color_palette(colors))

We didn’t use standard shades of those colors — I really prefer the firebrick variation of red over standard red, and so on.

We can also include a grid on the plot, so it will be easier to compare the heights of the bars:

And right now we can initialize the `figure` object that works like a canvas for the plot. Let’s tweak its size and its resolution:

It’s always a good idea to put labels to axes. So, let’s put the `”Amount”` label to y axis (in bold, large font), along with creating an ‘ax’ variable that points to an instance of axes from our plot:

And now, we have all of our basic components to create the plot itself:

Let’s not stop there, though. It would be pretty cool to have automatic generation of a plot title according to the keywords we used. We might also put a date right into the plot title. You will then have all of the details of your analysis right on the single plot. There is one small caveat though — at first, we tried to look for Tweets from the current day. If it failed, we tried to look for posts from previous days, up to seven days before. It would therefore be cool for our automatic generation to accommodate those details as well as efficiently deal with both of those examples. For this purpose, we might check if our `date_for_query` variable is the same as `datetime.date.today()`,and then use a customized plot name:

At this point we have a very neat bar plot with a fully customized title in accordance with our use case. But let’s be honest: we, people, are lazy. We hardly ever want to stretch our brains too much. So, if any of the bars in the plot have very similar heights, we might get confused as to whether we have more negative or positive Tweets. We might make it all clearer to understand by putting the actual amount of Tweets assigned to each category on the plot. For this purpose we can utilize the `ax.annotate()` method. To use it properly, we need to figure out the exact location where the annotations should be placed. This can be done by starting from iterating over patches that might be accessed from the `ax` object. Then, for each patch, get the bounding box coordinates for each bar. The best option will be to put the actual amounts just over the bars, so the annotation should happen just above the bar. We also want to make sure that in the plot there will be enough room to accommodate everything that we want to find there. So, we set the end of the y_axis to 110% value of the one automatically assigned by seaborn. We also want to make sure that the numbers are very much centered above the bars themselves — that is why we check if the `val` is bigger or lesser than 10. If it is bigger, for it to be perfectly centered, we add twice the offset (because we have two digits instead of one):

The very last thing to do with the plot is just saving it on the disk. Ideally, plots that you save would have a fully descriptive name that contains each detail of your analysis, similarly to what was done in the title of the plot. This can be done using the following code:

Here you can find the complete code for creating and saving the plot:

Now, have a look at my plot which illustrates my search for ”crude” and ”oil” from the 19th of June:

Plot created by author

Summary

And there you have it! It wasn’t a rough ride, was it? Even though the code that we have produced was not terribly complicated, we did use some cool stuff here. In fact, not only have you performed a sentiment analysis of the Tweets that you downloaded directly using the official Twitter API but you also prepared awesome visualizations of your work that are 100% customized to suit your use case. To put it simply: we have just developed a universal pipeline for performing sentiment analysis of the Tweets that contain keywords that you specify and created an automatic generation of awesome and informative plots. With this pipeline, all you need to do is to execute your code each day in the morning and right off the bat you will receive a brand new plot that will inform you about the current outlook among Twitter users with regards to the keywords you’re interested in. This can surely make your life as an investor more convenient by saving your time for much more pressing matters and it will definitely aid your decision-making process. Of course, if you want you can extend the code to your preferences. For instance, you could set up this script in cloud services, make it run each day at a fixed time and it may even automatically send you an email with the created plot. This is outside of the scope of this article though— perhaps I will revisit it in the future. Nonetheless, if you coded alongside me, you have just created a machine learning-based analysis pipeline of the latest Tweets regarding the topic of your choice. Isn’t that cool?

If you want to learn more about software development process directly from experienced programmers, feel free to check out STX Next’s blog: