Vice Presidential and Presidential Debate Analysis using Data Science
By: Aadit Kapoor
Tags: Data Analysis, Machine Learning, Python, Data Science, Politics

Vice Presidential and Presidential Debate Analysis using Data Science

Debate Analysis using Data Science: Using YouTube Comments to find the true intent of voters

I ntroduction

I believe Data Science allows me to express my curiosity in ways I’d never imagine. The coolest thing in Data Science is that I see data not as numbers but as an opportunity (business problem), insights(predictive modeling, stats, and data wrangling), and improvement (metrics). With this thought in mind, I decided to analyze the YouTube Comments of VP and Presidential debates.

After getting mixed results from the news sources, I thought to analyze the Vice Presidential and Presidential debates using Data Science.

The idea is to use YouTube comments as a medium to get the sentiment regarding the debate and getting insights from the data. In this analysis, we plot common phrases, common words, we also analyze sentiment and in the end for all my data science practitioners I present them a full-fledged dataset containing YouTube Comments of VP and Presidential debates.

How and Why

Why: After getting mixed results from the news sources about the outcome of the debate, I decided to use data science to help me see the outcome of the result. With the elections around the corner, technology or to be precise analytics plays a key role in shaping our thoughts and supporting our hypothesis.

How: To Analyze YouTube Comments we use Python and various other NLP Libraries followed by some data visualization tools. We will use the wonders of the awesome data wrangling library known as Pandas and we hope to find some interesting insights.

Requirements

For this project we require:

Python 3.8

Pandas

Scikit-Learn

Numpy

Seaborn

NLTK

Wordcloud

TextBlob

Creation of the Dataset

The dataset contains YouTube comments on the most popular/watched VP and Presidential debates. We use the YouTube Data API to get all comments (Due to the size limitation, we only get 100 comments per video). The videos have been selected through careful examination by the author, to be precise we focused on the highest number of views and the highest number of YouTube comments.

def clean_text(string):

string = re.sub(r'[^\w\s]', '', string)

return ''.join(i for i in string if ord(i) < 128) def remove_stopwords(string):

stop_words = set(stopwords.words('english'))

word_tokens = word_tokenize(string)

filtered_sentence = [w for w in word_tokens if not w in stop_words]

filtered_sentence = []



for w in word_tokens:

if w not in stop_words:

filtered_sentence.append(w) return ' '.join(filtered_sentence)

These functions define the cleaning of the text and removing stopwords.

YouTube Comments Extraction

The following function defines the video_ids we scrap and some basic extraction code. We get our data in the form of JSON, then we perform preprocessing and ultimately combine all comments followed by the sentiment score (using TextBlob) into a DataFrame.

video_ids = {"vp_debate":['65twkjiwD90','t_G0ia3JOVs','xXE6I3gWiMc'], "prez_debate":[

'yW8nIA33-zY','wW1lY5jFNcQ','K8Z9Kqhrh5c']} video_id = "&videoId="

content = None def extract_comments(resp):

"""

Get comments from the resp (json) yt comment

"""

com = []

for i in resp['items']:

com.append(i['snippet']['topLevelComment']['snippet']['textOriginal'].lower())

return com def get_videos_comments():

"""

Extract video comments and store in the arrays

"""

vp_debate_data = []

prez_debate_data = []

for i,j in video_ids.items():

print("Getting for: ",i)

for id in j:

video_id = "&videoId=" + id

resp = requests.get(URL+video_id)



print("Length: ",len(resp.content))

content = resp.json()

comments = extract_comments(content)

if i == "vp_debate":

vp_debate_data.extend(comments)

else:

prez_debate_data.extend(comments)

return vp_debate_data, prez_debate_data

class Comment:

def __init__(self, text):

self.text = remove_stopwords(clean_text(text))

# set threshold as 0.5

self.sentiment = 0 if TextBlob(self.text).sentiment.subjectivity <= 0.5 else 1

def return_comment(self):

return self class DataGenerator:

def __init__(self, vp_debate_data, prez_debate_data):

self.vp_data = {"comments":[], "sentiment":[]}

self.prez_data = {"comments":[], "sentiment":[]}

for i in vp_debate_data:

c = Comment(i)

self.vp_data['comments'].append(c.text)

self.vp_data['sentiment'].append(c.sentiment)

for i in prez_debate_data:

c = Comment(i)

self.prez_data['comments'].append(c.text)

self.prez_data['sentiment'].append(c.sentiment)

self.df_vp = pd.DataFrame(self.vp_data)

self.df_prez = pd.DataFrame(self.prez_data)

def return_df(self):

print("Loaded dataframe.")

return self.df_vp, self.df_prez



# return corpus given the debate key

# to be used for getting the commonkeywords and plotting

def get_corpus(self, key="vp_debate"):

corpus = []

if key == "vp_debate":

corpus = [i for i in self.vp_data['comments']]

else:

corpus = [i for i in self.prez_data['comments']]



return corpus

A preview of our data

Analysis

Upon examining the data, we first lowercase the words, then remove all punctuations, and then we remove all the stopwords. We use the popular TextBlob (https://textblob.readthedocs.io/en/dev/) to perform quick sentiment analysis.

A bar plot representing the sentiment analysis of the comments.

This data does not provide much information but if we try categorizing or grouping by specific words then we can get a better understanding.

Most Common Words

We find the most common words in each of the debates (i.e df_vp and df_prez). We use the popular package known as NLTK to get the most common words (after preprocessing)

def most_common(corpus):

fd = nltk.FreqDist(corpus)

print(fd)

fd_t10=fd.most_common(10)

counter = dict(fd_t10)

# pd.Dataframe from dict to show bar plot

df = pd.DataFrame.from_dict(counter, orient='index')

names, values = zip(*fd_t10)

# plot method 1

fd.plot(10)

# plot method two

print("Bar Plot")

df.plot(kind='bar')

Frequency distribution of words in the VP debate video

Frequency distribution of words in the Presidential debate video

As you can see in the Presidential debate video the most common words are ‘trump’ and ‘biden’.

WordCloud

Word clouds or tag clouds are graphical representations of word frequency that give greater prominence to words that appear more frequently in a source text. The larger the word in the visual the more common the word was in the document(s).

def plot_wordcloud(kind="vp_debate"):

words = get_word_list(kind)

wordcloud = WordCloud(width = 800, height = 800,

background_color ='white',

min_font_size = 10).generate(' '.join(words))

plt.imshow(wordcloud)

plt.axis("off")

plt.tight_layout(pad = 0)

plt.show()

A WordCloud representation of vp_debate and prez_debate

Find Keywords: A unique way to search specific words.

This feature finds the specified keyword (most prob top 10 words). It acts on each sentence.



"""

Find the specified keyword (most prob top 10 words)

Acts on each sentence

Add to the corresponding dataframe

Returns: DataFrame





Used inconjunction with get_most_common_words()



"""

for i in df.comments:

if word in i.split():

c = Counter(i.split())

if c[word] >= 2:

print(f'Word is {word} that occured {c[word]} times where sentence is {i}

')

(keyword="biden", debates=['vp_debate','prez_debate'])

def run(keyword, debates):

df = None

if debates == "vp_debate":

df = df_vp

else:

df = df_prez

find_keywords(keyword,df) def find_keywords(word, df):"""Find the specified keyword (most prob top 10 words)Acts on each sentenceAdd to the corresponding dataframeReturns: DataFrameUsed inconjunction with get_most_common_words()"""for i in df.comments:if word in i.split():c = Counter(i.split())if c[word] >= 2:print(f'Word is {word} that occured {c[word]} times where sentence is {i}

') @interact (keyword="biden", debates=['vp_debate','prez_debate'])def run(keyword, debates):df = Noneif debates == "vp_debate":df = df_vpelse:df = df_prezfind_keywords(keyword,df)

A glimpse of the output. (Demo available)

Conclusion

We see that by using Data Science we were able to gather interesting insights about the data. We can certainly improve our work by using a better sentiment classifier and testing for the different hypotheses. We can also use n-grams and compare their frequency.

Try here: https://gesis.mybinder.org/binder/v2/gh/aaditkapoor/DebateAnalysis/mainDataset here: http://www.kaggle.com/dataset/43532333d82042a1287e00672b86a2c76e76ffbc4d85569715309714635172b0