Only Numpy: Implementing Different combination of L1 /L2 norm/regularization to Deep Neural Network (regression) with interactive code | by Jae Duk Seo | Towards Data Science

I was always interested in different kind of cost function, and regularization techniques, so today, I will implement different combination of Loss function with regularization to see which performs the best. We’ll also take a look at absolute sum of each model’s weight to see how small the weights became.
L1-norm loss function and L2-norm loss function
I think the above explanation is the most simple yet effective explanation of both cost functions. So I won’t add anything more, know lets take a look at regularizes.
1. Making a Simple Neural Network
2. Keras Cheat Sheet: Neural Networks in Python
3. A noob’s guide to implementing RNN-LSTM using Tensorflow
L1 regularization and L2 regularization
Again the red box from top to bottom represent L1 regularization and L2 regularization. Crispy clear nothing for me to add. However since I have to drive derivative (back propagation) I will touch on something.
Derivative of Absolute Function
As seen above, derivative of absolute function have three different cases, when X > 1, X < 1 and X = 0.
Option 1 → When X > 1, derivative = 1Option 2 → When X = 0, derivative = undefinedOption 3 → When X < 1, derivative = -1
Since we can’t just let the gradient to be ‘undefined’ I BREAK THIS RULE.
Above, is the function that I will use to calculate the derivative of value X. And as seen above, I don’t have the second option, we merged that into first option.
Initialize Hyper parameters and Weights
Above is creating training data and declaring some noise as well as learning rate and the alpha value (these are for regularization).
Initialize weights and copy them to each cases.
Network Architecture + Forward Feed
There is nothing special about the network arch, simply put.
Denoted by the red marker below each layer.Layer 1 → 100 NeuronsLayer 2 → 104 NeuronsLayer 3 → 200 NeuronsLayer 4 → 1 Neuron
And the weights have appropriate dimension to perform transformation between the layers. However, there are two boxes that I wish to touch upon.
Red Box → Identity Activation Function
def IDEN(x):    return xdef d_IDEN(x):    return 1
If implemented in python it would look something like above, very simple linear function. We need this since we are going to perform regression on continuous values. If you want to know why we need activation functions please read my other blog post “Only Numpy: Why we need Activation Function (Non-Linearity), in Deep Neural Network — With Interactive Code”
Blue Box → Arctan Sigmoid Function
Think of this function as just as tanh() function but with a wider range. I personally believe that we don’t have to stick to logistic sigmoid or tanh. I think there is an whole wide variety of activation functions we need to explore. Anyway if you want the visualized archtan() please see the below image.
Cost Function and Regularization Cases
As seen above, there are in total of 6 (ignore 7) cases that we can work with.
Case 1 → L1 norm lossCase 2 → L2 norm loss Case 3 → L1 norm loss + L1 regularization Case 4 → L2 norm loss + L2 regularizationCase 5 → L1 norm loss + L2 regularizationCase 6 → L2 norm loss + L1 regularization
And we will see how each case function differ from one another!
Back Propagation (on case 1, 3, and 4)
Since, every other cases can be derived from those 3 cases, I won’t do every back propagation process. However, there is one thing that I want you to know. The parts written in red marker are the places where we BREAK THE RULE of taking derivative of absolute function! (Note this!!)
Regular Results
As expected the network with regularization were most robust to noises. However the model with pure L1 norm function was the least to change, but there is a catch! If you see where the green star is located, we can see that the red regression line’s accuracy falls dramatically.
Also, one thing to note is where the blue star lies, most of the model fails to predict the right value of Y at the beginning of X, this was very interesting to me. Know, lets take a look at the absolute sum of the weights.
Overall it becomes very clear that the models with the regularization have much smaller weights. Among them L1 cost function with L2 regularization had the smallest weight values.
Thinking outside the convention: Results
Where and how did I get the above result? Simple, rather than following the strict derivative of absolute function, I loosen up the derivative a bit. See below for the exact difference.
As seen above, rather than following the strict rule of derivation, I just adjusted the cost function to be (Layer_4_act — Y)/m.
I think when it comes to deep learning, sometimes creativity gives better results, I am not sure but Dr. Hinton did something with randomly decreasing weights in back propagation and still achieving good results. Anyway, lets take a look at the summed absolute value of the weights.
Interactive Code — Regular Results
Please follow this link to see the result and the code.
Interactive Code- Thinking outside the convention: Results
Please follow this link to see the result and the code.
I saw from other article that there are in total of 7 regularization techniques, so I am just scratching the surface. Can’t wait to know more.
If any errors are found, please email me at jae.duk.seo@gmail.com.
Meanwhile follow me on my twitter here, and visit my website, or my Youtube channel for more content. I also did deriving back propagation on simple RNN here if you are interested.
References
Seo, J. D. (2018, January 16). Only Numpy: Why we need Activation Function (Non-Linearity), in Deep Neural Network — With... Retrieved January 20, 2018, from https://medium.com/@SeoJaeDuk/only-numpy-why-we-need-activation-function-non-linearity-in-deep-neural-network-with-529e928820bcInverse Tangent. (n.d.). Retrieved January 20, 2018, from http://mathworld.wolfram.com/InverseTangent.htmlSigmoid function. (2018, January 18). Retrieved January 20, 2018, from https://en.wikipedia.org/wiki/Sigmoid_function(n.d.). Retrieved January 20, 2018, from http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/Finding the Derivative of |x| using the Limit Definition. (n.d.). Retrieved January 20, 2018, from https://math.stackexchange.com/questions/83861/finding-the-derivative-of-x-using-the-limit-definitionRobust linear model estimation using RANSAC¶. (n.d.). Retrieved January 20, 2018, from http://scikit-learn.org/stable/auto_examples/linear_model/plot_ransac.html#sphx-glr-auto-examples-linear-model-plot-ransac-pyJain, A., Shaikh, F., Choudhary, A., Singh, G., & Kaur, P. (2016, February 07). A Complete Tutorial on Ridge and Lasso Regression in Python. Retrieved January 20, 2018, from https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/Jerkic, M. (2017, October 30). Build A Multi Layer Neural Network With L2 Regularization Using Tensorflow. Retrieved January 20, 2018, from https://markojerkic.com/build-a-multi-layer-neural-network-with-l2-regularization-with-tensorflow/
Seo, J. D. (2018, January 16). Only Numpy: Why we need Activation Function (Non-Linearity), in Deep Neural Network — With... Retrieved January 20, 2018, from https://medium.com/@SeoJaeDuk/only-numpy-why-we-need-activation-function-non-linearity-in-deep-neural-network-with-529e928820bc
Inverse Tangent. (n.d.). Retrieved January 20, 2018, from http://mathworld.wolfram.com/InverseTangent.html
Sigmoid function. (2018, January 18). Retrieved January 20, 2018, from https://en.wikipedia.org/wiki/Sigmoid_function
(n.d.). Retrieved January 20, 2018, from http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/
Finding the Derivative of |x| using the Limit Definition. (n.d.). Retrieved January 20, 2018, from https://math.stackexchange.com/questions/83861/finding-the-derivative-of-x-using-the-limit-definition
Robust linear model estimation using RANSAC¶. (n.d.). Retrieved January 20, 2018, from http://scikit-learn.org/stable/auto_examples/linear_model/plot_ransac.html#sphx-glr-auto-examples-linear-model-plot-ransac-py
Jain, A., Shaikh, F., Choudhary, A., Singh, G., & Kaur, P. (2016, February 07). A Complete Tutorial on Ridge and Lasso Regression in Python. Retrieved January 20, 2018, from https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/
Jerkic, M. (2017, October 30). Build A Multi Layer Neural Network With L2 Regularization Using Tensorflow. Retrieved January 20, 2018, from https://markojerkic.com/build-a-multi-layer-neural-network-with-l2-regularization-with-tensorflow/