The Simplest Neural Network: Understanding the non-linearity | by Tushar Seth | Towards Data Science

The first neural network you want to build using squaring of numbers. Yes it’s not XOR or MNIST
Every time you want to learn about NNs or data science or AI, you search through google, you go through Reddit, get some GitHub codes. There is MNIST dataset, GANs, convolution layers, everywhere. Everybody is talking about neural networks. You pick up your laptop, run the code, Voila! it works. But then scratching your head, you know this is maths but how did this work?
Suddenly you think: enough, I want to learn the basics. You go to google and search ‘basics of neural networks’ and result: lots of tutorials showing MNIST dataset, XOR gate. You implement, it works, still, the question pops up how the duck it’s working? You get to know about multilayer perceptron- the basics of NNs. Again working on gates doesn’t satisfy you as that doesn’t seem to be predicting something. It’s from the training dataset’s 1’s and 0’s. You want some real-world example, then you come across this article. You think again one more article on NNs. But in this article, no basic theory, but the applicability of NNs. I am not showing you the internal maths happening, but to get the feel of how neural networks behave the way you see them behaving. Hope this helps you understand the “basic” Neural networks working(the non-linearity).
Like outer space, the brain is one thing that is still inexplicable in many aspects. We know that there exist small alien-like structures (yes those are horrendous), neurons, which are responsible for the transfer of information and there are several parts of the brain like the cerebrum, cerebellum which has their different functions. Different Neurons get activated differently(in a non-linear fashion) and transfer the information through electrical impulses. But the exact function of how these organs work and how neurons combine and transfer and store information and make the brain, one of the most incredible thing out there and are still having the scope of huge research.
When you are starting out your journey as an AI enthusiast, you come across many ways to start your learning. Data science is more related to statistics and mathematics. But it has been observed that neural networks can increase the power of data science to a tremendous level as it learns the non-linear relationships between the data as well, which is difficult to observe through normal statistics.
There are many good posts out there explaining how neural networks work, backpropagation, how different nodes connect to each other using weights and so many hyperparameters control these. Hence, I am not giving those details in this post. This and this are very nice posts regarding neural networks and backpropagation of which, the second link I highly recommend, which helped me in understanding neural networks in deep as in the second link, there is a detailed explanation of how relus(activation function) help in the determination of non-linear intricacies in the network.
But since neural networks are all about mathematics, non-linearity, learning; I wanted to see how it is able to adapt to simple non-linear functions. Yes, square function
This is the jupyter notebook which I created for the analysis:
github.com
We will be mainly using TensorFlow’s Keras, and from Keras, Sequential and Dense. Tensorflow is a widely used framework for neural networks and deep learning. Keras used to be a wrapper to isolate Tensorflow/Pytorch so that the code you develop in Keras could work in any of the libraries. But in tensorflow2.0, Google realized that TF1.0 was way too complicated and they introduced Keras API in tensorflow2.0 as a part of it.
Sequential and Functional are two types of models present in Keras. Where Sequential allows connection of layers in sequence, ie. one after the another; the functional allows connection of any layer to any of the layers in the whole network. Since we will be using only one hidden layer with one input and one output layer.
Dense helps implement densely connected layers. Keras core offers a lot more implementations like convolution, LSTM etc. But this is not Keras post, so let’s focus on our neural network.
'''Training a neural network to predict the square of a number'''import numpy as npimport matplotlib.pyplot as ppfrom tensorflow import kerasfrom keras.models import Sequentialfrom keras.layers.core import Densefrom IPython.display import Audio, display
Let’s see what all hyperparameters we need to create and run this NN.
n_samples: how many samples are needed for 1 epoch.
epoch: when all the samples complete forward and backward propagation
batch_size: how many samples can be introduced in one time to the neural network
mid_range: the range within which the number of samples is taken.
num_neurons: the number of neural nodes to be kept in the hidden layer
eg. we want 100000 samples between 1 and 10. So, it will give us values like 1.02, 7.89
'''configurations for neural net'''# change these values to experimentn_samples = 100000 # number of samples between 0 and mid_rangeepochs = 400 # number of times for the fwd and backward propagationbatch_size = 1000 # number of samples sent through NN at oncemid_range = 10 # range within which data is requirednum_neurons = 10 # number of neurons in the hidden layer
get_data method takes n_samples = 100000 and mid_range=10 which means this will spit out random 1000000 between 1 and 10.
'''creates random samples of n_samples rows which are between 0 to 1.mid_range decides the range under which the samples will be created.'''def get_data(n_samples, mid_range):  X = np.random.random((n_samples,1))*mid_range-(mid_range/2)  # eg. if n_samples = 10000, mid_range = 10 then it will create samples b/w   # 0 and 5 positive and negative  y = X*X  #print(X)  return X,y
Here we create a neural network of one node as input, 10 nodes as a hidden layer (not any hard and fast rule. Check which works for you the best) and one output layer.
''' creating the neural net model of 1:20:1; relu activation, mse as loss and adam optimizer'''def get_model():  model = Sequential()  model.add(Dense(num_neurons, input_shape=(1,), activation='relu'))  model.add(Dense(1))  print(model.summary())  model.compile(loss='mse', optimizer='adam')  return model
As shown in the figure above, we train the neural network with all the 10,00,000 samples for a defined number of epochs. The network is provided with X as input and y as expected output which is x2.
''' train the model for specified number of epochs, batch_size'''def train_model(X, y, model, epochs, batch_size):  h = model.fit(X, y, validation_split=0.2,               epochs=epochs,               batch_size=batch_size,               verbose=1)  pp.figure(figsize=(15,2.5))  pp.plot(h.history['loss'])  return model
Here we predict the model for X_train which is training data and X_test is outside of the training data.
''' predict the model for X(training data) and or X2 which is outside the training dataset'''def predict_model():  pred = model.predict(X)  mid_range = 20  X2 = np.random.random((n_samples,1))*mid_range-(mid_range/2)  pred2 = model.predict(X2)    # uncomment below code to observe the expected and actual values  # in the form (inputData, expectedOutputData, predictedOutputData, difference)    #pred2 = model.predict([-6])  #for i,j in zip(X, pred):  #  print(i,i*i,j, j-(i*i))  pp.figure(figsize=(10,3))  pp.plot(X,pred, '.')  pp.xlabel('x')  pp.ylabel('prediction')  pp.title('Prediction within training sample space')    pp.figure(figsize=(10,3))  pp.plot(X2,pred2, '.')  pp.xlabel('x')  pp.ylabel('prediction')  pp.title('Prediction outside training sample space')
if __name__ == '__main__' :  X_train,y_train = get_data(n_samples, mid_range_train)  X_test,y_test = get_data(n_samples, mid_range_test)  model = get_model()  pp.figure(figsize=(10,3))  pp.plot(X_train, y_train,'.')  pp.title('Original Training Data')  model = train_model(X_train, y_train, model, epochs, batch_size)  predict_model(X_train, X_test)  # below is audio which will fireup(literally :D) when the training and   #prediction completes  display(Audio(url='https://sound.peal.io/ps/audios/000/000/537/ \                original/woo_vu_luvub_dub_dub.wav', autoplay=True))
The main reason is that neural networks are good at interpolating rather than extrapolating ie. if the data lies within the sample space, (may not be the training data but within the range of training data) eg. 1–10 in this case.
If I try to predict 112, 122 etc. it might predict it wrong. The reason is that beyond the sample space, the Neural networks leave the non-linearity and become linear as you can see from the above plot.
Change the mid_range to 100 and see how the performance degrades. The reason is that now the sample space is too huge to imagine between 1 and 2, there are infinitely large numbers. So, 1–100 becomes too huge for a sample space. You can train for this as well. But you might need 100,000,00 samples and a large number of epochs. And maybe more. For this, you need powerful computing environments and much more time.Change the num_neurons and see how it performs. You can see as you decrease num_neurons, it tends to become more and more linear and starts losing its non-linearity. The reason being that non-linearity comes with reinforcement of a number of linearities (product of many linear functions).Change the batch size to observe how lower and higher batch size performs. Too low the batch size is, too much time it takes to train, moreover it gets saturated after some time and doesn’t train well. Too high the batch size, the more epochs it needs to converge. So, best is to keep an optimum value(the sweet spot)
Change the mid_range to 100 and see how the performance degrades. The reason is that now the sample space is too huge to imagine between 1 and 2, there are infinitely large numbers. So, 1–100 becomes too huge for a sample space. You can train for this as well. But you might need 100,000,00 samples and a large number of epochs. And maybe more. For this, you need powerful computing environments and much more time.
Change the num_neurons and see how it performs. You can see as you decrease num_neurons, it tends to become more and more linear and starts losing its non-linearity. The reason being that non-linearity comes with reinforcement of a number of linearities (product of many linear functions).
Change the batch size to observe how lower and higher batch size performs. Too low the batch size is, too much time it takes to train, moreover it gets saturated after some time and doesn’t train well. Too high the batch size, the more epochs it needs to converge. So, best is to keep an optimum value(the sweet spot)
Note: This code I have shown you is based on tensorflow==1.X and fails if instead of using Keras, I use tensorflow==2.0 and use tf.keras. I have raised the issue for this on GitHub. I will update here when it gets resolved. In Jupyter notebook, it works since it still uses the 1.X version of TensorFlow.
In the next post, I will show you how you can deploy this neural network training and prediction to a Kubernetes cluster over Google Kubernetes Engine by creating two microservices and calling them using Flask rest and gRPC. Don’t miss it as it’s got all in one post. Till then, keep on playing with the neurons(not with the actual ones :P). Connect with me on LinkedIn. Signing off!!.