Web Scrapping, Downloading Twitter Data and Performing Sentimental Analysis using Python
By: Anastasia Kiiru
Tags: Python Programming, Sentiment Analysis, Twitter, Web Scraping, Data Science

Web Scrapping

This website has a predetermined list of 100 influential individuals in Africa and this website has the list to key government officials in Africa. To scrap these two websites, we first import the necessary libraries:

from requests import get

from requests.exceptions import RequestException

from contextlib import closing

from bs4 import BeautifulSoup

import pandas as pd

import re

import os, sys import fire

Next, we define a function for getting website content by making a HTTP GET request :

def simple_get(url):

try:

with closing(get(url, stream=True)) as resp:

if is_good_response(resp):

return resp.content

else:

return None

except RequestException as e:

log_error('Error during requests to {0} : {1}'.format(url, str(e)))

return None

We’ll also define another function for downloading a page specified by a URL and return a list of strings, one per tag element:

def get_elements(url, tag='',search={}, fname=None):

if isinstance(url,str):

response = simple_get(url)

else:

#if already it is a loaded html page

response = url if response is not None:

html = BeautifulSoup(response, 'html.parser')



res = []

if tag:

for li in html.select(tag):

for name in li.text.split('

'):

if len(name) > 0:

res.append(name.strip())





if search:

soup = html





r = ''

if 'find' in search.keys():

print('finding',search['find'])

soup = soup.find(**search['find'])

r = soup if 'find_all' in search.keys():

print('finding all of',search['find_all'])

r = soup.find_all(**search['find_all'])



if r:

for x in list(r):

if len(x) > 0:

res.extend(x)



return res

The above function uses the BeautifulSoup library for raw HTML selection and content extraction. Selection is done based on either the tag element or using search together with find or find_all .

We can now call our defined functions and pass in the URL of our two websites.

The functions return all the content from the websites that satisfy our search criteria and we’ll, therefore, need to clean the data to get twitter usernames only:

Data Cleaning

The .split() and .strip() methods are used in the data cleaning process to get rid of any white-spaces or special characters. For instance, cleaning the res data will be:

res_cleaned = []

for element in res:

if re.findall("@", element):

res_cleaned.append(element)





df = pd.DataFrame(res_cleaned)

df1 = df[0].str.split('@', expand = True) influencers = df1[1].tolist() final = []

for influencer in influencers:

influencer = influencer.strip(')')

final.append(influencer)

And finally, we save our scrapped and cleaned data to a CSV file:

pd.DataFrame(final, columns = ['Twitter handles']).to_csv('influencers_scraped.csv', index = False, encoding = 'utf-8')

Downloading Twitter Data

Just like its necessary for any python program, we start by importing the needed libraries. For this extraction, we will use the Tweepy library to extract Twitter data:

import tweepy

from tweepy.streaming import StreamListener

from tweepy import OAuthHandler

from tweepy import Stream

from tweepy import Cursor

from tweepy import API

Next, we initialize variables to store user credentials for accessing Twitter API, authenticate the credentials, and create a connection to Twitter Streaming API:

consumer_key = 'YOUR TWITTER API KEY'

consumer_secret = 'YOUR TWITTER API SECRET KEY'

access_token = 'YOUR TWITTER ACCESS TOKEN'

access_token_secret = 'YOUR TWITTER ACCESS TOKEN SECRET' #authentication and connection

auth = OAuthHandler(consumer_key, consumer_secret)

auth.set_access_token(access_token, access_token_secret)



auth_api = API(auth)

After creating a successful connection to twitter, we proceed with extracting data for the users and storing the data in a CSV file for easier access during analysis.

The following function receives a list of users and a CSV file name as arguments, iterates through the list getting data for each user, and saves that data to a CSV file.

def get_user_info(list, csvfile):

users_info = []

for user in list:

print ('GETTING DATA FOR ' + user)

try:

item = auth_api.get_user(user)



users_info.append([item.name, item.description, item.screen_name, item.created_at, item.statuses_count, item.friends_count, item.followers_count])



except Exception:

pass



print('Done!!')



user_df = (pd.DataFrame(users_info, columns = ["User", "Description", "Handle", "Creation Date", "Tweets", "Following", "Followers"])).to_csv(csvfile, index = False, encoding = 'utf-8')

Hashtags and mentions in a user’s tweets also help to determine the level of influence of an individual. We, therefore, get the hashtags and mentions in each tweet of a user with the help of the Cursor object and again save the data in a CSV file using the following function:

def get_tweets(list,csvfile1, csvfile2, csvfile3):

hashtags = []

mentions = [] for user in list:

print ("GETTING DATA FOR "+ user)

try:

for status in Cursor(auth_api.user_timeline, id = user).items():

if hasattr(status, "entities"):

entities = status.entities

if "hashtags" in entities:

for ent in entities["hashtags"]:

if ent is not None:

if "text" in ent:

hashtag = ent["text"]

if hashtag is not None:

hashtags.append(hashtag)

if "user_mentions" in entities:

for ent in entities["user_mentions"]:

if ent is not None:

if "screen_name" in ent:

name = ent["screen_name"]

if name is not None:

mentions.append([user, name])

except Exception:

pass



print("Done!")



hashtags_df = (pd.DataFrame(hashtags, columns = ["hashtags"])).to_csv(csvfile1, index = False, encoding = "utf-8")

mentions_df = (pd.DataFrame(mentions, columns = ["mentions"])).to_csv(csvfile2, index = False, encoding = "utf-8")

Sentimental Analysis

Now that we have mined all the necessary data, we can proceed with analysis to determine the rank of the influencers and the most commonly used hashtags. The formulas used for ranking are based on the ‘Measuring User Influence in Twitter: The Million Follower Fallacy’ paper.

According to Gummadi, Benevenuto, Haddadi, and Cha (2010), the influence of a person can be measured in terms of:

indegree influence — the audience of an individual,

retweet influence — the ability of the individual to generate valuable content, and

mention influence — the ability of the individual to hold an engaging conversation.

The three metrics can be calculated as reach score, popularity score, and relevance score respectively.

reach_score = item.followers_count - item.friends_count

popularity_score = retweet_count + favorites_count

relevance_score = mentions_count

The most common used hashtags, on the other hand, are calculated using the Counter module:

unique_hashtags = []

for item, count in Counter(hashtags).most_common(5):

unique_hashtags.append([item, count])

The most common hashtags data can be used to understand the distribution of hashtags and how they are used by different influencers. The following code outputs a bar-plot with this information.

df = pd.DataFrame(pd.read_csv('hashtags_data.csv')) unique_hashtags = ['...',...,'...']



# selecting rows based on condition

hashtags_grouping_data = df[df['hashtags'].isin(unique_hashtags)] grouped = hashtags_grouping_data.groupby(['hashtags', 'user_type']).agg({'user_type': ['count']}) grouped.columns = ['count']

grouped = grouped.reset_index() bp = sns.barplot(x = 'hashtags', y = 'count', hue = 'user_type', data = grouped) #Position the legend out the graph

bp.legend(bbox_to_anchor=(1.02, 1),

loc=2,

borderaxespad=0.0);

bp.set(title='Bar plot for Influencers and Top Government Officials by hashtag', xlabel='Hashtags', ylabel='Count')

A sample bar-plot constructed by the above code looks like: