Jordan Peterson NLP Word Frequency Analysis
By: Michelangiolo Mazzeschi
Tags: Naturallanguageprocessing, Psychology, Data Science, Spacy, Jordan Peterson

To perform this experiment, I will need the textual content of his entire course. Because I would need to use the Youtube API to access the subtitles (it would take an article per se to explain how to connect to Youtube using python), I will use third-party software for this purpose: JDownloader. If you wish to replicate the experiment on a similar set of videos, know that you can find other free and open-source software to download Youtube subtitles in the .srt format.

Screenshot of JDownloader

I will select subtitles only and will download everything under the same folder:

list of .srt to download

Create Functions

To perform the experiment, I will need to preprocess the textual data:

Clean the data from punctuation, symbols, and stop words

Lemmatize every word: convert them to the original root

!pip install spacy

!pip install pysrt #per sottotitoli #find top words

def top_frequent(text, num_words):

#frequency of most common words

import spacy

from collections import Counter nlp = spacy.load("en")

text = text #lemmatization

doc = nlp(text)

token_list = list()

for token in doc:

#print(token, token.lemma_)

token_list.append(token.lemma_)

token_list lemmatized = ''

for _ in token_list:

lemmatized = lemmatized + ' ' + _

lemmatized #remove stopwords and punctuations

doc = nlp(lemmatized)

words = [token.text for token in doc if token.is_stop != True and token.is_punct != True]

word_freq = Counter(words)

common_words = word_freq.most_common(num_words)

return common_words

Using the function top_frequent(text, num_words), I can input any string of text (even an entire book, if I want to), and I will get the count of lemmatized words ranked in ascending order. With the argument num_words, I can specify the number of words I want to return (returning the entire list of 2000 words, for example, would not be much of use).

Analysis of the Entire Course

One interesting experiment is performing the analysis on the entire course and then comparing the results with individual lessons separated.

Why should I perform the experiment on the whole text and different partitions? Most importantly, why do I have enough confidence to claim that the most frequent lemmatized words used by Peterson may not be the same across all the lessons? By analyzing the whole course, I do not have an idea of the word distribution because it represents all the individual courses chunked together. By getting data from all individual lessons, and by analyzing each one of them separately, I can build a word frequency distribution that can tell me if the results are uniform or not.

# parse sub

#total words of JP in the course

import pysrt total_string = ''

for text_n in range(1, 10):

subs = pysrt.open('/content/drive/My Drive/Colab Notebooks/Projects/20200729_NLP_frequency/JP_'+str(text_n)+'.srt')

for sub in subs:

#print(sub.text)

total_string += sub.text

total_string len(total_string)

Output

I asked the software to chunk all subtitles together and to rank the 1000 most used words by Peterson:

common_words = top_frequent(total_string, 1000)

common_words

[('-PRON-', 13496),

('know', 1133),

('like', 924),

('thing', 592),

('think', 495),

('people', 397),

('right', 361),

('way', 294),

('world', 272),

('happen', 235),

('mean', 225),

('good', 223),

('look', 219),

('time', 192),

('want', 185),

('maybe', 185),

('tell', 183),

('idea', 181),

('sort', 168),

('little', 156),

('come', 145),

('story', 143),

('actually', 142),

('kind', 134),

('problem', 133),

...

Analysis of individual lessons

With this partition of code, I will build a distribution of values and see how if it follows a uniform distribution. If that is the case, I might claim that INDEPENDENTLY FROM THE CONTENT OF A SPEECH, Jordan Peterson will maintain a consistent frequency of words.

def find_common_factors(X):

#flat everything in one list

one_row = list()

for k in X.values:

for n in k:

one_row.append(n)

one_row = pd.DataFrame(one_row)

one_row #labeled list

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

le.fit(one_row[0])

one_row_labeled = le.transform(one_row[0])

one_row_labeled #rebuild the original dataset and convert it to DataFrame

import numpy

X_labeled = numpy.array_split(one_row_labeled, (X.shape[0]))

X_labeled = pd.DataFrame(X_labeled)

X_labeled m = [[0 for x in range(max(one_row_labeled)+1)] for x in range(len(X_labeled))] #turn each corresponding label to 1

for row in range(len(X_labeled.values)):

for num in range(len(X_labeled.values[1])):

m[row][X_labeled.values[row][num]] = 1

m = pd.DataFrame(m)

m original_shape = m.shape

original_shape #convert column num in corresponding text

m.loc[original_shape[0]] = [0 for x in range(0, original_shape[1])]

m for _ in range(original_shape[0]):

m.loc[original_shape[0]] += m.loc[_]

m #rename columns with label values

m.columns = a = le.inverse_transform(m.columns)

m #isolate count of columns

m.loc[m.shape[0]] = m.columns #we add last col with names

f = m.transpose().drop([x for x in range(m.shape[0]-2)], axis=1).transpose() #we only save the last 2 cols

f = f.reset_index()

f.pop('index') #sorting values

f = f.transpose().reset_index().drop(['index'], axis=1)

return f.sort_values(0, axis=0, ascending=False, inplace=False, kind='quicksort', na_position='last') #parse individual lesson

#total words of JP in the course

import pysrt

import pandas as pd

Now that the algorithm is ready, I can access all the subtitles and collect the results of an experiment into a pandas DataFrame. REMEMBER to change the path folder. I will be using my Google Drive to store my code. You will have stored your data in a different location.

total_subs = list()

for text_n in range(1, 15):

subs = pysrt.open('/content/drive/My Drive/Colab Notebooks/Projects/20200729_NLP_frequency/JP_'+str(text_n)+'.srt')

top_words = top_frequent(subs.text, 12) #n_words qui, metto 12 perchè le prime 2 si eliminano

df = pd.DataFrame([top_words[x][0] for x in range(10)])

total_subs.append(df) X = pd.concat(total_subs, axis=1)

X = X.transpose()

X

X[[x for x in range(2, 12)]]

Output for each individual course:

I am taking out the first 2 columns because they count the -PRON- and \ an in the text, probably a spacy bug.

Conclusion of the experiment

The results are incredible! The distribution of the used words appears to be s uniform. The choice of our vocabulary remains consistent in the future.

find_common_factors(X).reset_index().drop(['index'], axis=1).iloc[2:20]

By counting the words in the dataset, this is the result: