Neural Network: How Many Layers and Neurons Are Necessary | by Angela Shi | Towards Data Science

When creating Neural Networks, one has to ask: how many hidden layers, and how many neurons in each layer are necessary?
When it comes to complex real data, Andrew Ng suggests in his course on Coursera: Improving Deep Neural Networks, that it is a highly iterative process, and so we have to run many tests to find the optimal hyperparameters.
But, what is the intuition behind how the hidden layers and neurons impact the final model?
Let’s create a dataset using make_circles.
from sklearn import datasetsX,y=datasets.make_circles(n_samples=200,shuffle=True, noise=0.1,random_state=None, factor=0.1)
And you can visualize the dataset:
import matplotlib.pyplot as pltplt.scatter(X[:, 0], X[:, 1], c=y,s=20)plt.axis(‘equal’)
Each neuron creates a linear decision boundary. By looking at the dataset, the intuition is to create one layer of 3 neurons. And we should be able to create a perfect classifier. Now, what about only 2 neurons? It seems not enough, and what if we try to add more layers?
With these questions, let’s try to do some tests.
If we use 3 neurons with the following code:
from sklearn.neural_network import MLPClassifierclf =MLPClassifier(solver=’lbfgs’,hidden_layer_sizes(3,),activation=”tanh”,max_iter=1000)clf.fit(X, y)clf.score(X,y)
Then we are able to create 3 decision boundaries within the hidden layer. And we can notice that there are an infinite number of possibilities to create these 3 lines (so there are an infinite number of global minimum in the cost function of this neural network).
To create the visualization of the decision boundaries of the hidden layer, we can use the coefficients and intercepts of the hidden layer.
n_neurons=3xdecseq=np.repeat(np.linspace(-1,1,100),n_neurons).reshape(-1,n_neurons)ydecseq=-(xdecseq*clf.coefs_[0][0]+clf.intercepts_[0])/clf.coefs_[0][1]
Then we can plot these three lines with the original dataset:
fig, ax = plt.subplots(figsize=(8,8))ax.scatter(X[:, 0], X[:, 1], c=y,s=20)ax.plot(xdecseq[:, 0], ydecseq[:, 0])ax.plot(xdecseq[:, 1], ydecseq[:, 1])ax.plot(xdecseq[:, 2], ydecseq[:, 2])ax.set_xlim(-1.5, 1.5)ax.set_ylim(-1.5, 1.5)
If we use only 2 neurons, it is quite easy to see that we will not be able to perfectly classify all observations.
Using similar codes, we can plot the following graph with 2 decision boundaries from the hidden layer.
What if we create 2 neurons in many hidden layers? To better understand what happened in the first hidden layer with 2 neurons, I created this visualization.
Now it is quite clearly that no matter how many layers you add, you will never to able to classify the data in the red circle below.
You can test yourself: you can add more layers in the argument hidden_layer_sizes. For example (2,3,2) means 3 hidden layers with 2, 3, and 2 neurons for each.
from sklearn.neural_network import MLPClassifierclf = MLPClassifier(solver=’lbfgs’,hidden_layer_sizes=(2,3,2),activation=”tanh”,max_iter=1000)clf.fit(X, y)clf.score(X,y)
From this simple example, we can end with this intuition:
The number of neurons in the first hidden layer creates as many linear decision boundaries to classify the original data.
It is not helpful (in theory) to create a deeper neural network if the first layer doesn’t contain the necessary number of neurons.
If you want to see other animations to understand how neural networks work, you can also read this article.