Document Classification Part 1: Intuition & How Do We Work With Documents?
By: Chan Woo Kim
Tags: Machine Learning, Python, Spam, Text, Classification

In this article I will give an introduction to conducting machine learning on documents to classify them into categories. This is the part 1 of a series outlined below:

Part 1: Intuition & How Do We Work With Documents?

Part 2: Part 2: Text Processing (N-Gram Model & TF-IDF Model)

Part 3: Detection Algorithm (Support Vector Machines & Gradient Descent)

Part 4: Variations of This Approach (Malware Detection Using Document Classification)

Let us first define what a document classification task looks like. Given a new document that the program has not encountered before, we want to train a model that will output the correct category that the document belongs to. These categories are pre-defined and the training process is based on the same set of categories.

So what data are we working with in classifying text? There are more advanced approaches using semantics and heuristics, but it is all fundamentally based on analysis we do on the words (depends on how you define this, though) that the documents are comprised of.

Throughout this series, I will use the example of classifying emails as spam or not spam. When classifying emails, we really are just looking out for certain words. If the word “royal”, “Nigerian”, “Prince”, “millions”, “dollars”, “deal”, “sex”, and more keeps appearing throughout, the given email will likely be spam.

So we want to find common occurring words in both category that is generally representative of that piece of text. But this process can not be as simple as regarding an email as spam immediately as it sees the word “prince”:

def isItSpam(document):

if document.contains(“prince”) OR document.contains(“sex”):

return True;

else:

return False;

It can not be this simple. Theres much more nuance than this to text.

What we want to do instead is to say that there is a greater chance in a body of text including certain words to be in a certain category. This is the case with most machine learning tasks. We do this by assigning a weight to each word. That is to say, each word influences the program to think that it is a spam email or not by different amount. This is the fundamental objective of a machine learning task classifying text.

But wait… how do you assign weights to words? It makes sense when you talk about it but it’s unintuitive in terms of implementation. See, it makes sense to think of weights as the following:

x = [1,6,8] //inputs

w = [0.2,0.7,0.6] //weights

The weighting process could be like: x*w = [0.2, 4.2, 4.8]

Then we can train a classifier that says that if the sum of all elements is greater than 0, it’s a certain category.

For instance:

Sum of x*w = 0.2+4.2+4.8 = 9.2, which is positive, and therefore a certain category.

But if we try this with text:

x = [“this”, ”is”, “text”]

W = [0.2, 0.7, 0.6]

x*W =

…

What is “this” multiplied by 0.2?

This was the source of confusion with document classification when I first started out.

Machine learning requires that one works with numerical data, but text is not numerical. We require a numerical representation to compute machine learning algorithms on.

Vectorizing Text

Vectorizing the text is what allows us to have numerical representation of text that we can work with. To vectorize text is to represent text as numerical vectors. Below is an example that represents this in a very intuitive manner:

Let’s say we have the following sentences as part of our document corpus:

Sentence 1 (S1): “vectorize this text to have numbers!”

Sentence 2 (S2): "what does it mean to vectorize?"

Sentence 3 (S3): "document classification is cool"

What we do then is create a vocabulary set:

V = ['classification', 'cool', 'document', 'does', 'have', 'is', 'it', 'mean', 'numbers', 'text', 'this', 'to', 'vectorize', 'what']

If you haven’t guessed already, the vocabulary set contains all words that occur throughout the corpus.

And now we can represent each sentence as a vector comprised of the number of times certain words from the set V occurs in each sentence, where each component corresponds to the index of that word in the vocabulary set:

S1 = [0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0]

S2 = [0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1]

S3 = [1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0]

Now we have a numerical representation of the sentences! We can now do all kinds of cool things with this representation, like training a machine learning model on it. This model is what is called a Vector Space Model.

Implications of Vectorized Text

An implication of this representation is that each word is a feature. This is something I had a hard time grasping at first (none of what I was reading really said this clearly). For those that are not familiar with what features are, here’s an example: say you want to classify if a certain day is from summer or winter. Putting the obvious metric aside (like what date it is), the characteristics that you would look out for would be things like precipitation, temperature, and more. These are what we call features — characteristics of each category that we want to look out for to determine the right category, and in the case of document classification, each word is treated as a characteristic of a sentence.

Math With Text (?!)

To give you a sense of how cool this model is, here’s a way to compute document similarity for those familiar with high school math in vectors:

The angle between two vectors is equal to the dot product of the two vectors divided by the magnitude of them (in the formulation above, i represents each component, each number in a vector). For those interested but not yet familiar with dot products, you can learn about it here.

This angle can be regarded as a testament to how similar two vectors are. For instance, in the image below, most would agree that the red vector is more similar to the green vector than to the purple vector. The reason this is true is because it has a lesser angle to the green vector than to the purple vector.

Taken from this post

Going back to our sentences that we were working with:

Sentence 1 (S1): “vectorize this text to have numbers!”

Sentence 2 (S2): "what does it mean to vectorize?"

Sentence 3 (S3): "document classification is cool"

I think most would agree that sentence 1 and 2 are more similar to one another than to sentence 3. Let me prove that mathematically with what I just presented above.

Given the vector representations provided above:

S1 = [0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0]

S2 = [0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1]

S3 = [1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0

Let’s compute the similarities:

Dot product of S1 and S2:

Magnitude of S1:

Magnitude of S2: 2.4494897428

So the angle between S1 and S2 is:

Now that is a large angle, but if you calculate the angle between S1 and S3 (I’ll spare the dot product and magnitude calculation for this):

It’s even greater than 70 degrees. We can agree that 90 degrees is greater than 70 degrees, and thus we prove that S1 is more similar to S2 than to S3.

I was pretty mind-blown when I first encountered this. I had learned cosine similarity before in my sophomore year in high school but I couldn’t believe that I could calculate angles between documents. That’s just insane.

Some document classification tasks do involve this idea. But we will explore a different algorithm in this series, called Support Vector Machines.