Engineering Intuitive Computer Vision | by Raveena Jayadev | Towards Data Science

Image classification is one of the hottest fields of machine learning, data science, and AI, and often used to benchmark certain types of AI algorithms — from logistic regression to deep neural networks.
But for now, I want to take your mind away from those hot techniques, and ask ourselves a question: if us humans saw an image of a handwritten character, or a dog or cat, how would our brains intuitively classify different types of images? Below is an example of digits in an image; “2”, “0”, “1” and “9”.
In the example above of digits (or numbers/numerals), how would our brains differentiate between, say, the 1 and 9 at the bottom? Well, intuitively our brains have a sort of “mental model” of what 1s look like, and a mental model of what 9s look like. For a new image of a digit, our brains can compare the digit based on each mental model and decide if it’s a 1, a 9, or an entirely different handwritten character. In fact, our mental model can do more than just classify: it can generate new instances of similar data. Show young children digits for the first time so their brains can form a mental model, and I’m pretty sure they can learn to draw new instances of, say a 1 or a 9, or the other digits in the photo above.
Now this type of decision-making, classification, is quite different from what you see in popular AI vision systems deployed around the world in cameras, hospitals, etc. — those rely on something called convolutional deep learning — check this visual demo out. While the convolutional-only layers of a convolutional neural network (CNN) might resemble specific parts of the vision system in our eyes according to this author, the actual decision making of a deep CNN network mainly comes from the thousands of tuned weights in the feedforward “final fully connected layer” of the network.
BTW, if you haven’t heard of neural networks or not familiar with how neural networks are used to classify images, I definitely recommend this video by Grant Sanderson on the 3Blue1Brown (3b1b) Youtube channel; his exposition and explanations are fantastic.
But the ML system we’ll be engineering to decide and classify digits will not be incorporating thousands of weights and parameters as is the case in artificial neural networks.
Before we go on, we need to clarify what a “mental model” of digits in the computer would look like.
What is a “Mental Model” in the Brain vs. Computer?
The short answer is that the actual mental models in the brain — well, we’re still figuring it out. There’s a lot we don’t know yet about the brain, and neuroscientists, cognitive scientists, and others have a lot to chime in.
However, there are theories growing about what mental models in the brain could resemble computationally. This article by computational neuroscientist Karl Friston talks more about the computational theory of how brains “call on an internal, mental generative model of the world” to infer and predict the causes behind the sensations signals it receives from the rest of the body, including the visual system. In this theory, the brain basically has to infer the causes of sensations based on a mental model it has about which probable causes lead to probable bodily perceptions. Friston also mentions that a wide range of real biological physiological and behavioral phenomena has been predicted by computational models like this. I’m leaving out a bunch of of details, so if you’re interested, read the article :)
In our machine learning case, we’re thankfully not having to deal with the whole host of body sensations — just 28x28 black and white images on a screen. But it’s still a challenging problem! — otherwise computer vision would have been “solved” already. So now the issue remains: how do we create, in a sense, a “mental generative model” the computer can hold as an object in its memory?
Generative Models as “Distributions”
Earlier I said how a “mental model”, or a generative model, allows you to basically create new instances of similar data. If I trained a generative model on, say, the digit 2, then the computer would be able to draw new instances of a 2. Having a model to create these instances is like a rough blueprint: it allows you to fill in the details and create different instances of houses, buildings, etc. This “rough blueprint” is called a distribution, because, intuitively, you basically are generating a collection of new data — and distribution is just another word for collection. The technical term for this is called a “probability distribution” (don’t worry, I won’t be revisiting the horrors of stats class!) because in your generated collection, some samples are more likely than others. For example if the computer were generating new instances of a “2”, then a sample of 2 with the “curly hook” missing at the top would be calculated as “less likely to resemble” a 2, than if it had a 2. Being less likely just means the model will generate that type of 2 less than the usual way a 2 looks.
Another way of looking at generative models in the language of machine learning is that they create models of “clusters” — they help cluster the data into specific regions. This would be called “unsupervised learning” — inferring structure within the data without any training labels to teach the machine.
Now that we know generative models are distributions, what we need to do first is see how to train a mental model over each of the 28x28 images of digit types (0,1,2,..9). In Python’s sci-kit learn library, there’s a function called density estimation model that will help us.
The Final Step in Classification: Inference
Now our computer calls on its own internal, generative “mental model” for each type of digit, and generate new types of images — and that’s all cool for sure! — but we still need to finish our first goal: classify and predict new images of digits it hasn’t seen before.
The key to classifying and predicting in our intuitive classifier is called inference. Our brains do it all the time, actually! Imagine the following question:
“You’re only gonna be eating greens for dinner today!” What does greens refer to?
Well, green is a color, and greens used in that plural way, it’s inferred that it refers to green objects — but what objects? our brains infer that the “objects” mean green vegetables like broccoli, spinach, kale, etc. because what other green things can be cooked and eaten? Our brains easily do inferences like this all the time, but getting a natural language processing (NLP) computer system to do inferences like this is very hard without billions of data points, parameters , and high energy usage— something human brains don’t really need. Luckily we’re not dealing with NLP inference for now, but don’t get me wrong — visual human-like inference is also a longstanding challenge in AI to overcome.
Anyway, back to vision. We want our computer to infer what the new digit will be, based on its “mental models” (distributions) for what digits look like. How would such an inference look like? Well, let’s say we were only dealing with two types of digits; a 2 and a 7. Suppose the computer has created distributions for what it thinks 28x28 images of 2 and 7 look like. When a new un-seen image comes in, how does it infer? The computer asks itself: under which mental model is the image more likely: my model of a 2, or a 7? If it’s more likely under a 2, then I’ll decide the image is a 2; otherwise I decide 7. In this way, the actual classification has turned into an if-else condition statement, a statement very pervasive in computer science languages from Python, to Java, to C. In an interesting spin, the decision is quite programmatic!
If you want to learn more about probability distributions or get familiarity, I recommend this video by Grant Sanderson from the “3Blue1Brown” Youtube channel, along with KhanAcademy videos on probability and statistics!
So...let’s begin the implementation! Feel free to take a pause before we dive right in!
Data Importing
The first order of business is to import the handwritten digit data from somewhere. For that, TensorFlow and Keras (the high-level language running on top of Tensorflow) have a beautifully organized collection of datasets, and Keras happens to have handwritten digits. The MNIST dataset can be found here (and its license is: CC BY-SA 3.0). To import the digit dataset, we just use the code:
from keras.datasets import mnist(x_train, y_train), (x_test, y_test) = mnist.load_data()
The MNIST dataset from Keras has 6,000 training samples per digit-class and we have 10 classes (so 60,000 training digits total), but we won’t be using that much data. Why? Well, for one, to show in principle we can create these generative models without as much data. But also, in the real world, 60,000 training data points might not exist — there might only be tens or a few hundred training data per class. So what we’re going to do is just select some random group of, say 6,000 digits from the training group. That gives us about 600 training digit samples per digit-class. That’s already a decrease of 90% of the original data! So if we can train generative “mental” models of each digit-class based on this smaller training set, we would be more data efficient.
We can randomly sample and create a new smaller training set with the following code:
np.random.seed(10)random_indices = np.random.choice(60000, size=6000, replace=False)random_x_train = x_train[random_indices, :]random_y_train = y_train[random_indices]
Feature Extraction
In order to create the models, we first have to do what’s called “feature engineering” (or “feature selection” if you’re working with data where the features are columns of a table). Why do we have to do this? Well, we have to realize something: these images live in a very, very high-dimensional space. Each image has 784 pixels, so as a vector, each image lives in the same 784-dimension space. We can’t even plot a graph in 4D space for that matter, so just give up on graphing in 784D space!
Working with data in high dimensions can be pretty difficult, and high-dimensional space does some weird, weird things — for example, spheres start to become “spiky” in 4D and higher: check out this video by Youtube channel Numberphile if you want to get a taste. You might have also heard of this phenomenon as the “curse of dimensionality” in machine learning.
If you have tabular data with features as columns, there are statistical ways to pre-select a smaller set of features based on statistical stuff like correlation and spread. If you don’t have tabular data, in general there are a few ways to do feature extraction. Luckily for us, there’s an amazing way to extract features — also called “dimension reduction” in the ML lingo — where we can save almost all the information about the data and go do all the way to 2D! It’s called UMAP — you can read the Python user manual here — and it was created in 2018 by mathematician Leland McInnes. UMAP stands for “Uniform Manifold Approximation and Projection” — if that’s a mouthful, don’t worry. (If you want an explanation, check this video out by Dr. Letitia) In the simplest way I can explain: the UMAP algorithm tries to connect the 784D image data by dots and lines depending on how clustered the data is (kinda like a social network!); then the whole graph is projected using topology, down into a lower-dimensional space. If you want to visualize the data points, you’ll have to project all the way down into 2D or 3D space, as I’ve done here below:
mapper = umap.UMAP(n_components=2, random_state=40).fit(random_x_train)umap_components = mapper.transform(random_x_train)
The distance you see up here between clusters reflects the fact that in some concrete way, certain digits like 0 and 7 are “far away” or “pretty separate” from 7 and 9. Of course you might say “Raveena! every digit is separate” and that’s true, but when handwriting digits, sometimes pen strokes can be confusing. And that’s the underlying structure UMAP is trying to compress into 2D so we can see it for ourselves!
(Disclaimer: Although the UMAP components show clear clustering in the scatterplot, the components axes themselves are not really human interpretable. However, the decision-making prediction process still is.)
Learning the “Mental” Generative Models
Now it’s time to learn the “mental models” — the generative probability models for each class of handwritten digit the computer will learn from data in order to make educated inferences at classification time. To create the actual model, we will use a technique in the sci-kit learn module called kernel density estimation (KDE). Imagine a data set of points, some of them close to each other, some more lonely and isolated. The estimation process is visualized below:
Once we learn the generative “mental” models for each digit, we can plot the models together and use 10 different colors to denote the digits 0–9 and their model shapes. Each “shapely blob” of color below represents the internal model the computer has learned from data for each digit, similar to how human brains have internal models for how digits should look like.
Now that our computer has generative models for each digit, it’s time to do...inference! What I want to emphasize in this project is two things: 1) we can program image classifier that infers and decides with internal “mental” models like a human brain can, and 2) this makes the AI system easy to troubleshoot. In an era where black-box deep AI systems are being deployed all over the world and often discriminate against marginalized people (article by Zachary Hay of Towards Data Science), it’s extremely hard to troubleshoot the discriminatory predictions those systems can make––and if you can’t troubleshoot, the system might as well be dangerous in my opinion.
Let me show you how we can check the system we’re creating in this article, though. Here’s the summarized code for how our system would infer and predict the content of an unseen test image:
def decision(sample, mental_models):... (summary)    if likelihood_ratio <= 3.0:        print("There's very little evidence that my first choice, is better than my second choice."       return "I refuse to decide!"   elif (likelihood_ratio > 4.0) and (likelihood_ratio <= 10.0):        print("There is a bit of evidence that my first choice is better than my second choice")       return "maybe digit {} over {}"   else: print('There is strong evidence that my first choice, is better than my second choice.'       return [confident machine choice]
(The machine prediction code looks just like an if-else statement (as said above), and the machine can predict with different levels of confidence, which is pretty intuitive!)
Now suppose we gave this AI system an image of a 3, and instead it predicted a 5. How could we troubleshoot the model to see how it made a mistake? Well, follow me here:
The machine decided from the if-else statement, that under its generative “mental” models, a 5 was more likely from the image data than a 3.
If-else decisions are pure logic, so that means the issue is not with the decision, but either the statistical mental models or the feature extraction. We can ask the machine to display the mental models and feature extraction, both as I did above. From there we can analyze the shortcomings of the models, and see how to adjust them.
In these 2 steps, we already identified where the problems might lie in this AI system if it made a mistake, By contrast, many of the deep learning systems connected to surveillance technologies around the world cannot even be troubleshooted in this way, and that makes for potentially dangerous AI.
I hope you, the reader, have come away from this article with a hope that 1) we can create AI systems that can potentially make decisions intuitively like humans can, and 2) we can potentially troubleshoot the AI model if it makes mistakes, and backtrack the error all the way into the internal parts of the system and understand what to fix and what to scrap.
Now admittedly, you might be telling me: “Raveena, the “UMAP” feature extraction doesn’t seem intuitive!” Yes, I know. The features are unfortunately not human-interpretable. However, there is a way to modify this system so that it can be interpretable, albeit in a different way. If you’re interested to know, this article in 2015 by MIT Prof. Joshua Tenenbaum talks about making a AI handwritten character classifier system using internal “mental models” that are quite similar to our probabilistic generative models. In fact, it’s so intuitive and data efficient, it only learns from 30 data points and imitates human handwritten symbols eerily well. In my opinion, it’s one of the most revolutionary creations for the future of AI.
By the way, here’s a link to my code if you want to see the full implementation details: my Github is public and free to access :)
References
Thomas Parr, Karl Friston, The Anatomy of Inference: Generative Models and Brain Structure (2018), Frontiers in Computational Neuroscience, 2018.S.R. Kheradpisheh et al., Deep Networks Can Resemble Human Feed-forward Vision in Invariant-Object Recognition (2016), Nature.Joshua Tenenbaum, Brendan Lake, Human-Level Concept Learning through Probabilistic Program Induction (2015), Science.
Thomas Parr, Karl Friston, The Anatomy of Inference: Generative Models and Brain Structure (2018), Frontiers in Computational Neuroscience, 2018.
S.R. Kheradpisheh et al., Deep Networks Can Resemble Human Feed-forward Vision in Invariant-Object Recognition (2016), Nature.
Joshua Tenenbaum, Brendan Lake, Human-Level Concept Learning through Probabilistic Program Induction (2015), Science.