Web Scrapping, Downloading Twitter Data and Performing Sentimental Analysis using Python | by Anastasia Kiiru | Towards Data Science

Social media influencers play an important role in the current society of increasing human interconnection; and business organizations are realizing the need for influencer marketing campaigns (Newberry, 2019). A successful influencer marketing campaign requires a partnership with the right social media influencer. So how do we determine the right influencer?
For the data science community, only data has the answer. Let’s take a case study for determining the right influencers from Africa to partner with, for a digital marketing campaign. We will explore scrapping of data from relevant websites, data extraction from twitter for specific users, and perform sentimental analysis to determine the right influencers to partner with.
This website has a predetermined list of 100 influential individuals in Africa and this website has the list to key government officials in Africa. To scrap these two websites, we first import the necessary libraries:
from requests import getfrom requests.exceptions import RequestExceptionfrom contextlib import closingfrom bs4 import BeautifulSoupimport pandas as pdimport reimport os, sysimport fire
Next, we define a function for getting website content by making a HTTP GET request :
def simple_get(url):    try:        with closing(get(url, stream=True)) as resp:            if is_good_response(resp):                return resp.content            else:                return None    except RequestException as e:        log_error('Error during requests to {0} : {1}'.format(url, str(e)))        return None
We’ll also define another function for downloading a page specified by a URL and return a list of strings, one per tag element:
def get_elements(url, tag='',search={}, fname=None):    if isinstance(url,str):        response = simple_get(url)    else:        #if already it is a loaded html page        response = urlif response is not None:        html = BeautifulSoup(response, 'html.parser')                res = []        if tag:                for li in html.select(tag):                for name in li.text.split('\n'):                    if len(name) > 0:                        res.append(name.strip())                                               if search:            soup = html                                                r = ''            if 'find' in search.keys():                print('finding',search['find'])                soup = soup.find(**search['find'])                r = soup            if 'find_all' in search.keys():                print('finding all of',search['find_all'])                r = soup.find_all(**search['find_all'])               if r:                for x in list(r):                    if len(x) > 0:                        res.extend(x)                    return res
The above function uses the BeautifulSoup library for raw HTML selection and content extraction. Selection is done based on either the tag element or using search together with find or find_all.
We can now call our defined functions and pass in the URL of our two websites.
res = get_elements('https://africafreak.com/100-most-influential-twitter-users-in-africa', tag = 'h2')url= 'https://www.atlanticcouncil.org/blogs/africasource/african-leaders-respond-to-coronavirus-on-twitter/#east-africa'response = simple_get(url)res_gov = get_elements(response, search={'find_all':{'class_':'twitter-tweet'}})
The functions return all the content from the websites that satisfy our search criteria and we’ll, therefore, need to clean the data to get twitter usernames only:
The .split() and .strip() methods are used in the data cleaning process to get rid of any white-spaces or special characters. For instance, cleaning the res data will be:
res_cleaned = []for element in res:    if re.findall("@", element):        res_cleaned.append(element)                df = pd.DataFrame(res_cleaned)df1 = df[0].str.split('@', expand = True)influencers = df1[1].tolist()final = []for influencer in influencers:    influencer = influencer.strip(')')    final.append(influencer)
And finally, we save our scrapped and cleaned data to a CSV file:
pd.DataFrame(final, columns = ['Twitter handles']).to_csv('influencers_scraped.csv', index = False, encoding = 'utf-8')
Just like its necessary for any python program, we start by importing the needed libraries. For this extraction, we will use the Tweepy library to extract Twitter data:
import tweepyfrom tweepy.streaming import StreamListenerfrom tweepy import OAuthHandlerfrom tweepy import Streamfrom tweepy import Cursorfrom tweepy import API
Next, we initialize variables to store user credentials for accessing Twitter API, authenticate the credentials, and create a connection to Twitter Streaming API:
consumer_key = 'YOUR TWITTER API KEY'consumer_secret = 'YOUR TWITTER API SECRET KEY'access_token = 'YOUR TWITTER ACCESS TOKEN'access_token_secret = 'YOUR TWITTER ACCESS TOKEN SECRET'#authentication and connectionauth = OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_token_secret)            auth_api = API(auth)
After creating a successful connection to twitter, we proceed with extracting data for the users and storing the data in a CSV file for easier access during analysis.
The following function receives a list of users and a CSV file name as arguments, iterates through the list getting data for each user, and saves that data to a CSV file.
def get_user_info(list, csvfile):    users_info = []    for user in list:        print ('GETTING DATA FOR ' + user)        try:            item = auth_api.get_user(user)                    users_info.append([item.name, item.description, item.screen_name, item.created_at, item.statuses_count, item.friends_count, item.followers_count])                    except Exception:            pass            print('Done!!')        user_df = (pd.DataFrame(users_info, columns = ["User", "Description", "Handle", "Creation Date", "Tweets", "Following", "Followers"])).to_csv(csvfile, index = False, encoding = 'utf-8')
Hashtags and mentions in a user’s tweets also help to determine the level of influence of an individual. We, therefore, get the hashtags and mentions in each tweet of a user with the help of the Cursor object and again save the data in a CSV file using the following function:
def get_tweets(list,csvfile1, csvfile2, csvfile3):    hashtags = []    mentions = []for user in list:        print ("GETTING DATA FOR "+ user)        try:            for status in Cursor(auth_api.user_timeline, id = user).items():                if hasattr(status, "entities"):                    entities = status.entities                    if "hashtags" in entities:                        for ent in entities["hashtags"]:                            if ent is not None:                                if "text" in ent:                                    hashtag = ent["text"]                                    if hashtag is not None:                                        hashtags.append(hashtag)                    if "user_mentions" in entities:                        for ent in entities["user_mentions"]:                            if ent is not None:                                if "screen_name" in ent:                                    name = ent["screen_name"]                                    if name is not None:                                        mentions.append([user, name])        except Exception:            pass    print("Done!")        hashtags_df = (pd.DataFrame(hashtags, columns = ["hashtags"])).to_csv(csvfile1, index = False, encoding = "utf-8")    mentions_df = (pd.DataFrame(mentions, columns = ["mentions"])).to_csv(csvfile2, index = False, encoding = "utf-8")
Now that we have mined all the necessary data, we can proceed with analysis to determine the rank of the influencers and the most commonly used hashtags. The formulas used for ranking are based on the ‘Measuring User Influence in Twitter: The Million Follower Fallacy’ paper.
According to Gummadi, Benevenuto, Haddadi, and Cha (2010), the influence of a person can be measured in terms of:
indegree influence — the audience of an individual,
retweet influence — the ability of the individual to generate valuable content, and
mention influence — the ability of the individual to hold an engaging conversation.
The three metrics can be calculated as reach score, popularity score, and relevance score respectively.
reach_score = item.followers_count - item.friends_countpopularity_score = retweet_count + favorites_countrelevance_score = mentions_count
The most common used hashtags, on the other hand, are calculated using the Counter module:
unique_hashtags = []for item, count in Counter(hashtags).most_common(5):        unique_hashtags.append([item, count])
The most common hashtags data can be used to understand the distribution of hashtags and how they are used by different influencers. The following code outputs a bar-plot with this information.
df = pd.DataFrame(pd.read_csv('hashtags_data.csv'))unique_hashtags = ['...',...,'...']   # selecting rows based on condition hashtags_grouping_data = df[df['hashtags'].isin(unique_hashtags)]grouped = hashtags_grouping_data.groupby(['hashtags', 'user_type']).agg({'user_type': ['count']})grouped.columns = ['count']grouped = grouped.reset_index()bp = sns.barplot(x = 'hashtags', y = 'count', hue = 'user_type', data = grouped)#Position the legend out the graphbp.legend(bbox_to_anchor=(1.02, 1),          loc=2,           borderaxespad=0.0);bp.set(title='Bar plot for Influencers and Top Government Officials by hashtag', xlabel='Hashtags', ylabel='Count')
A sample bar-plot constructed by the above code looks like:
Influencer marketing has become the new marketing niche and business organizations are on the lookout for the right social media influencers to partner with for an effective marketing campaign. Using data science, we can help drive the marketing decision based on factual data. It is as systematic as scrapping data from relevant websites, downloading user data from a social media platform such as Twitter, and performing sentimental analysis on the data to come up with visuals that business organizations can better understand. The post outlines the main codes used in the process, and the full code can be found here.
[1]: C. Meeyoung, H. Hamed, B. Fabricio, and G. Krishna, Measuring user influence in Twitter: The million follower fallacy (2010), http://twitter.mpi-sws.org/icwsm2010_fallacy.pdf
[2]: N. Christina, Influencer marketing guide: How to work with social media influencers (2019), https://blog.hootsuite.com/influencer-marketing/