Do stock traders’ attitude on social media affect stock market?
By: Francis Zhang
Tags: Assignment 1, Sentiment Analysis, Stock Traders, Stock Market, Social Media

It is very common to invest in stock on the basis of suggestions from professional stock traders online. Many of stock traders have several social media accounts to post their thoughts, conjecture and comments. As one of the biggest social media platforms, Twitter creates an online community for people to follow many specific topics. Some famous stock traders such as Peter Brandt and Steve Burns have got tens of thousands of followers and tweets posted. They pass along their attitude to their followers and others who may see their tweets. What powerful in this process is that every tweet can be retweeted, hence the attitude will be transmitted over and over. Twitter is literally a social network with weighted notes and edges. In this network, any posted tweets somehow have influence to others who received them.

In this article, I picked 100 influential stock traders on twitter whose tweets may have impact on their audiences. Basically, I categorized all the tweets into three attitudes — Positive, Negative and Neutral. Positive tweets may deliver confidence to investors, which results in buying stock, and vice versa. The objective of this research is to find out if there is an insightful relationship between stock traders’ attitude and stock market trend. S&P 500 index is used as the criterion to exam the trend.

Data Collection

Part 1. Twitter accounts collection

Basically, my list comes from Google. I googled key works such as “famous stock traders” and “stock traders on Twitter”. There are many lists online contain those famous stock traders Twitter information. You can also generate your list by searching other key words.

Part 2. Data download and merge

Twitter API provides developers access to most of Twitter’s functionality and database. Tweepy, which is an open source Python package, gives developers a very convenient way to access the Twitter API with Python. Tons of tweets and other data can be downloaded through Tweepy without dealing with low-level details. There are many online tutorials talking about how to get start with Tweepy, you can check them if you are new to Tweepy. I just list steps below for reference.

1. Go to Twitter developer site to apply for a developer account. 2. Fill the required information and create an application. 3. Create the Authentication Credentials, which include Consumer key, Consumer secret, Access token and Access secret. 4. Install Tweepy in your Python package. 5. Create a new tweepy.API object and set your authentication credentials.

Then, you are good to go. Twitter allows developers to extract maximum 3200 recent tweets per account. In this case, I extracted two levels of data — account level and tweet level. For account level, variables include:

screen_name # Twitter unique account id

num_total_tweets # number of total tweets posted by account

num_followers # number of total followers of account

num_friends # number of friends an account following

created_at # the date account created

if_verified # if a verified account

For tweets level, variables include:

screen_name # Twitter unique account id that posted this tweet

text # text content of this tweet

num_retweet # number of retweets of this tweet

num_favorite # number of favorite of this tweet

created_at # the date this tweet created

I merged these two dataframes on screen_name for future analysis. Because they are from different level, I broadcast accounts information to match the number of tweets observations:

import pandas as pd

merge = pd.merge(tweet_info, account_info.loc[:,['screen_name', 'num_total_tweets', 'num_followers', 'num_friends',

'created_at', 'if_verified']], how='left', on='screen_name')

Now I have a dataset with shape(311404, 10), which contains 10 variables and 311,404 observations.

Data Processing

The popularity of tweets is strikingly different. Some tweet received less than 10 favorites while others received tens of thousands. There are many ways to clean the data. Tweets received extremely high retweet and favorite rate may be retweeted tweets from other celebrities. We need to remove those outliers. In this case, I used z-score to detect outliers. Any z-score greater than 3 or less than -3 is considered to be an outlier.

sub = merge.loc[:,['num_retweet','num_favorite']]

z_scores = stats.zscore(sub)

abs_z_scores = np.abs(z_scores)

filtered_entries = (abs_z_scores < 3).all(axis=1)

new_sub = sub[filtered_entries]

Rescaling the dataset can offset the effect of large variance. MinMaxScaler rescales the dataset such that all feature values are in the range[0,1].

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler() merge[['num_total_tweets','num_followers','date_diff','num_friends','num_retweet','num_favorite']] = scaler.fit_transform(merge[['num_total_tweets','num_followers','date_diff','num_friends','num_retweet','num_favorite']])

One problem I encountered is the date format. The format of created data variables extracted from Tweepy is “%Y-%m-%d %h%m%s”, we need to get rid of the hours, minutes and seconds, because the analysis sets day as index.

merge['created_at_x'] = pd.to_datetime(merge['created_at_x']).dt.date

merge['created_at_y'] = pd.to_datetime(merge['created_at_y']).dt.date

Sentimental Analysis

In this section let’s talk about text processing. There are many ways to do text analysis. You can train your classifier to do the job, or use already existed Python library for processing textual data. TextBlob is a very powerful text processing library that is built upon NLTK and provides an easy use interface to the NLTK library. Before training the tweets, an necessary step must be taken. You must remove the links and special characters from tweets.

from textblob import TextBlob

import re def clean_tweet(tweet):

return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)", " ", tweet).split())



def analize_sentiment(tweet):

analysis = TextBlob(clean_tweet(tweet))

if analysis.sentiment.polarity > 0:

return 1

elif analysis.sentiment.polarity == 0:

return 0

else:

return -1 merge['senti'] = np.array([analize_sentiment(tweet) for tweet in merge['text']])

From the classifying result we could know that the percentage of positive tweets is 40.8%, the percentage of negative tweets is 15.5% and the percentage of neural tweets is 43.7%.

merge['senti'].value_counts()

0 136095

1 127099

-2 48210

Name: senti, dtype: int64

Model building

Because every tweet’s influential ability is different, a influential model should be built for calculating the affect score. For example, an account with more followers has influential score higher than an account with less followers; a tweet has larger number of retweet possibly has higher influential score. Based on this sense, I assigned different weights to related variables. The amount of weights decided manually by my intuitive knowledge of social media. A new variable is created as date_diff, which is the days difference between the date tweet created and the date account created. The concept of variable date_diff is that the account created 10 years ago has more influential ability than a new account that created 2 years ago.

w0 # weight of if_varified

w1 # weight of date_diff

w2 # weight of num_friends

w3 # weight of num_total_tweets

w4 # weight of num_followers

w5 # weight of num_favorite

w6 # weight of num_retweet w6 > w5 > w4 > w3 > w2 > w1 > w0

According to (Nasir, 2011), negative news travel faster than good news. In order to consider this affect into model, I set a weight for stock traders’ attitude — w=1 if tweet is positive else w=p (p>1)if tweet is negative. Hence, a manual weight is added to negative tweets.

merge['inf_score'] = (w0 * merge['if_verified'] + w1 * merge['date_diff'] + w2 * merge['num_friends'] + w3 * merge['num_total_tweets'] + w4 * merge['num_followers'] + w5 * merge['num_favorite'] + w6 * merge['num_retweet']) * w * merge['senti']

The influential scores calculated as above. Now we can group by day and calculate the total score for that day. For instance, there are many positive and negative tweets posted for a specific day, and the sum of total positive and negative tweets’ influential score is considered as the score for that day. The tweets have positive influence on stock market if the score is greater than 0, and vice versa. Now we can plot the scores and compare the trend with S&P 500 index.

conclusion

The graph below looks ugly, but it shows the trend of score between April and September. It shows that stock traders on Twitter had a positive influence started from April to middle of August, then the attitude dropped to negative after middle of August.

Below is the S&P 500 index, we can see that the trends roughly matched.

Limitation

Because for each twitter account only 3,200 tweets can be extracted, there are several possible bias of my dataset. For the very active twitter accounts, their frequency of posting tweets is much higher than other accounts. Therefore, I could only collect tweets in a short time. For example, if a twitter account posts 100 tweets per day, I could only extract data for 32 days at maximum. But for those who are not very active account, I could extract data for years. This problem leads to a unbalanced dataset, which means the older the time, the less the data points. Besides, the weights are set manually from my perspective. Different weights may change the result completely. Another research is needed for analyzing the weights applied on model. Furthermore, the pandemic is a strong cause to turn stock traders’ attitude to negative.

There are tons of reasons that may have affect on stock market. Stock traders’ attitude on social media is one of them that may give us a hint to this rapidly changing world.