Animations of Neural Networks Transforming Data | by Angela Shi | Towards Data Science

When it comes to classification algorithms, the explanation can be intuitive about how they work:
Logistic Regression and SVM find a hyperplane to “cut” the space into two. (But the approach of these two algorithms is different, so the final hyperplane is different.)
In case the data are not linearly separable, with the kernel trick, SVM transforms the data into a space of higher dimension, then “cuts” it into two.
Decision trees group data into hyper-rectangles that will contain a majority of one class.
K Nearest-Neighbors analyses the neighbors of a new observation to predict the class of this one.
Now what about Neural Network?
Let’s take a simple example of dataset: two classes, two features x1 and x2, and the data are not linearly separable.
import numpy as npnp.random.seed(1)x1=np.concatenate((np.random.normal(0.5,0.1,100),np.random.normal(0.2,0.05,50),np.random.normal(0.8,0.05,50))).reshape(-1,1)x2=np.concatenate((np.random.normal(0.5,0.2,100),np.random.normal(0.5,0.2,50),np.random.normal(0.5,0.2,50))).reshape(-1,1)X=np.hstack((x1,x2))y=np.concatenate((np.repeat(0,100),np.repeat(1,100)))
The visualization is below and the data are also normalized.
from sklearn.preprocessing import StandardScalerscaler = StandardScaler()X=scaler.fit_transform(X)plt.scatter(X[:, 0], X[:, 1], c=y,s=20)plt.axis('equal')
For this dataset, the simplest suitable structure of the neural network is the one with one hidden layer of two neurons. If you don’t see why, just continue and you will understand intuitively.
We can also considerer that the activation function is the sigmoid function.
from sklearn.neural_network import MLPClassifierclf = MLPClassifier(solver=’lbfgs’,hidden_layer_sizes=(2,),activation=”logistic”,max_iter=1000)clf.fit(X, y)
The initial data are transformed by the hidden layer. Since the hidden layer has two neurons, we can visualize the outputs of the two neurons on a 2D plane.
So the initial 2D dataset is transformed into another 2D space.
We can calculate the values of the hidden layer A1 with the corresponding weights:
A1=1/(1+np.exp(-(X@clf.coefs_[0]+clf.intercepts_[0])))
Then we can visualize the layer A1:
fig, ax = plt.subplots(figsize=(5,5))ax.scatter(A1[:, 0], A1[:, 1], c=y,s=30)ax.set_xlim(-0.1, 1.1)ax.set_ylim(-0.1, 1.1)
Now, how are the initial data transformed into these values? We can animate this process:
in the beginning, we have the original dataset (which is not, remember, linearly separable)
in the end, we have will have the transformed data in the (0,1) square because the sigmoid function has an output between 0 and 1. And do you see the form of the data? Yes, that’s when the magic happens...
We can see that in the end, the dataset becomes linearly separable! For binary classification, the output neuron is a Logistic Regression. And it is easy to find a linear decision boundary. In this case, the hyperplane separator will be a straight line.
So the intuition behind a neural network is that the hidden layers transform the non-linearly separable initial data into a space where they are almost linearly separable.
Combining the hidden layer and the output layer, we have a surface, and we can visualize it below. It is a surface because the input has 2 variables, and the output is the probability, so a number that we can represent by the z-axis.
Now we can see that the two neurons “cut” the two frontiers between the two classes.
Now let consider the dataset below. One class is inside another with a circular form. Before using a Neural Network, which algorithms would solve this problem?
Quadratic Discriminant Analysis is a suitable candidate, because the two classes have different covariance matrices with the same centers. The decision boundary will an almost perfect circle to separate the yellow dots from the purple points.
SVM with an RBF (Radial Basis Function) kernel will also be perfect.
What about a Neural Network? Two neurons mean that you “cut” only twice, here we need at least three.
We can first see the final surface below and the decision boundary is actually a sort of triangle.
The initial 2D space is transformed into a 3D space, with 3 neurons. And we can animate this transformation.
In the beginning, the dataset is on a plane. We will consider that the value of the z-axis is zero.
In the end, the dataset is mapped into the (0,1) cube because the output of the sigmoid function is between 0 and 1.
Now can you see that the dataset is almost linearly separable in this cube? Let’s create another animation by rotating the cube. Now you see that with a plane, we can easily separate the yellow dots from the purple ones?
In order to understand how algorithms work, I like to create visualizations with simple data. If you also find them helpful, please comment.
The plots are first created with python, then I use gifmaker to animate them.