How I used machine learning to classify emails and turn them into insights (part 1). | by Anthdm | Towards Data Science

Today I wondered what would happen if I grabbed a bunch of unlabeled emails, put them all together in one black box and let a machine figure out what to do with them. Any idea what will happen? I didn’t.
The first thing I did was look for a dataset that contained a good variety of emails. After looking into several datasets, I came up with the Enron corpus. This dataset has over 500,000 emails generated by employees of the Enron Corporation, plenty enough if you ask me.
As the programming language, I used Python along with its great libraries: scikit-learn, pandas, numpy and matplotlib.
For clustering the unlabeled emails I used unsupervised machine learning. What, how? Yes, unsupervised, because I have training data with only inputs, also known as features and contains no outcomes. In supervised machine learning we work with inputs and their known outcomes. In this case I wanted to classify emails based on their message body, definitely an unsupervised machine learning task.
Instead of loading in all +500k emails, I chunked the dataset into a couple of files with each 10k emails. Trust me, you don’t want to load the full Enron dataset in memory and make complex computations with it.
import pandas as pdemails = pd.read_csv('split_emails_1.csv')print emails.shape # (10000, 3)
I now had 10k emails in the dataset separated into 3 columns (index, message_id and the raw message). Before working with this data I parsed the raw message into key-value pairs.
This is an example of a raw email message.
Message-ID: ❤0965995.1075863688265.JavaMail.evans@thyme>Date: Thu, 31 Aug 2000 04:17:00 -0700 (PDT)From: phillip.allen@enron.comTo: greg.piper@enron.comSubject: Re: HelloMime-Version: 1.0Content-Type: text/plain; charset=us-asciiContent-Transfer-Encoding: 7bitX-From: Phillip K AllenX-To: Greg PiperX-cc:X-bcc:X-Folder: \Phillip_Allen_Dec2000\Notes Folders\’sent mailX-Origin: Allen-PX-FileName: pallen.nsf
Greg,
How about either next Tuesday or Thursday?
Phillip
To work with only the sender, receiver and email body data, I made a function that extracts these data into key-value pairs.
def parse_raw_message(raw_message):    lines = raw_message.split('\n')    email = {}    message = ''    keys_to_extract = ['from', 'to']    for line in lines:        if ':' not in line:            message += line.strip()            email['body'] = message        else:            pairs = line.split(':')            key = pairs[0].lower()            val = pairs[1].strip()            if key in keys_to_extract:                email[key] = val    return emaildef parse_into_emails(messages):    emails = [parse_raw_message(message) for message in messages]    return {        'body': map_to_list(emails, 'body'),         'to': map_to_list(emails, 'to'),         'from_': map_to_list(emails, 'from')    }
After running this function, I created a new dataframe that looks like this:
email_df = pd.DataFrame(parse_into_emails(emails.message))index   body           from_             to0       After some...  phillip.allen@..  tim.belden@..
To be 100% sure there are no empty columns:
mail_df.drop(email_df.query(    "body == '' | to == '' | from_ == ''").index, inplace=True)
Which is short for term frequency–inverse document frequency and is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. I need to feed the machine something it can understand, machines are bad with text, but they shine with numbers. Which is why I converted the email bodies into a document-term matrix:
vect = TfidfVectorizer(stop_words='english', max_df=0.50, min_df=2)X = vect.fit_transform(email_df.body)
I made a quick plot to visualize this matrix. To do this I first needed to make a 2d representation of the DTM (document-term matrix).
X_dense = X.todense()coords = PCA(n_components=2).fit_transform(X_dense)plt.scatter(coords[:, 0], coords[:, 1], c='m')plt.show()
That being done, I wanted to find out what the top keywords were in those emails. I made this function doing exactly that:
def top_tfidf_feats(row, features, top_n=20):    topn_ids = np.argsort(row)[::-1][:top_n]    top_feats = [(features[i], row[i]) for i in topn_ids]    df = pd.DataFrame(top_feats, columns=['features', 'score'])    return dfdef top_feats_in_doc(X, features, row_id, top_n=25):    row = np.squeeze(X[row_id].toarray())    return top_tfidf_feats(row, features, top_n)
After running this function on a document, it came up with the following result.
features = vect.get_feature_names()print top_feats_in_doc(X, features, 1, 10) features     score0     meetings  0.3831281         trip  0.3243512          ski  0.2804513     business  0.2762054        takes  0.2041265          try  0.1612256    presenter  0.1584557    stimulate  0.1558788        quiet  0.1480519       speaks  0.14805110  productive  0.14507611      honest  0.14022512      flying  0.13918213     desired  0.13388514        boat  0.13036615        golf  0.12631816   traveling  0.12530217         jet  0.12481318  suggestion  0.12433619     holding  0.12089620    opinions  0.11604521     prepare  0.11268022     suggest  0.11143423       round  0.10873624      formal  0.106745
All making sense if you look into the corresponding email.
Traveling to have a business meeting takes the fun out of the trip. Especially if you have to prepare a presentation. I would suggest holding the business plan meetings here then take a trip without any formal business meetings. I would even try and get some honest opinions on whether a trip is even desired or necessary.As far as the business meetings, I think it would be more productive to try and stimulate discussions across the different groups about what is working and what is not. Too often the presenter speaks and the others are quiet just waiting for their turn. The meetings might be better if held in a round table discussion format.My suggestion for where to go is Austin. Play golf and rent a ski boat and jet ski’s. Flying somewhere takes too much time.
The next step was writing a function to get the top terms out of all the emails.
def top_mean_feats(X, features, grp_ids=None, min_tfidf=0.1, top_n=25):    if grp_ids:        D = X[grp_ids].toarray()    else:        D = X.toarray()D[D < min_tfidf] = 0    tfidf_means = np.mean(D, axis=0)    return top_tfidf_feats(tfidf_means, features, top_n)
Returning the top terms out of all the emails.
print top_mean_feats(X, features, top_n=10) features     score0     enron  0.0440361       com  0.0332292       ect  0.0270583       hou  0.0173504   message  0.0167225  original  0.0148246   phillip  0.0121187     image  0.0098948       gas  0.0090229      john  0.008551
What I got so far is interesting, but I wanted to see more and find out what else the machine was able to learn from this set of data.
KMeans is a popular clustering algorithm used in machine learning, where K stands for the number of clusters. I created a KMeans classifier with 3 clusters and 100 iterations.
n_clusters = 3clf = KMeans(n_clusters=n_clusters, max_iter=100, init='k-means++', n_init=1)labels = clf.fit_predict(X)
After training the classifier it came up with the following 3 clusters.
Because I now knew which emails the machine assigned to each cluster, I was able to write a function that extracts the top terms per cluster.
def top_feats_per_cluster(X, y, features, min_tfidf=0.1, top_n=25):    dfs = []    labels = np.unique(y)    for label in labels:        ids = np.where(y==label)         feats_df = top_mean_feats(X, features, ids,    min_tfidf=min_tfidf, top_n=top_n)        feats_df.label = label        dfs.append(feats_df)    return dfs
Instead of printing out the terms, I found a great example on how to plot this graph with matlibplot. So I copied the function, made some adjustments and came up with this plot:
I immediately noticed cluster 1, had weird terms like ‘hou’ and ‘ect’. To get more insights about why terms like ‘hou’ and ‘ect’ are so popular, I basically needed to get more insight in the whole dataset, implying a different approach..
To know how I came up with that different approach and how I found new and interesting insights will be available for reading in part 2.
Code available on Github