fabiochiu/medium-articles huggingface dataset:
Gradient Descent Algorithm
Tags: Machine Learning, Artificial Intelligence, Gradient Descent, Regression

Gradient Descent is an optimization algorithm that is applied in the negative part of the gradient, which is repetitive and used to minimize some functions.

Even in machine learning, we use Gradient Descent to improve the parameters of our model. The parameters mentioned here express the coefficients in Linear Regression as well as the weights in Neural Networks.

Learning Rate

We can call each of the steps above the learning rate. We progress by increasing our learning rate with each step and we can cover more space with each step. However, since the slope changes constantly, we must also consider the possibility of exceeding the lowest point. In this case, it may cause Overfitting problem.

Overfitting :Overfitting is an issue within machine learning and statistics where a model learns the patterns of a training dataset too well. This happens when a model learns the detail and noise in the training data to the extent that it negatively affects the performance of the model on new data.

Since recalculation is very frequent with a very low learning rate, this algorithm can also proceed safely in the negative gradient direction. A low learning rate is more reliable but it is time consuming to calculate.

Cost Function

It is a function that tests the accuracy of the Machine Learning model designed for the data we have. This function is used to quantify the error between predicted values (y-predict) and expected values (y) and presents it as a single real number. The cost function has its own curve and its own gradients. The slope of this curve tells how to update our parameters to make the model more accurate.

Thank you for reading .

References

https://www.researchgate.net/post/What_is_the_meaning_of_the_word_optimization_in_research
Gradient, Gradient Descent, Stochastic Gradient Descent
Tags: Gradient, Gradient Descent, Stochastic Gradient, Sgd, Machine Learning

Source from Wikipedia

While reading the chapter 4 of Deep Learning for Coders with fastai & PyTorch, by Jeremy Howard and Sylvain Gugger in 2020, I spotted SGD (Stochastic Gradient Descent) in the machine learning training process as an optimiser, and I wondered what gradient is, what gradient descent is, and finally, what exactly stochastic gradient descent is. Since I haven’t done many studies for a while, I thought that it could be a good idea to hit the books, dive into the world wide web and learn about them.

First things first, gradient. In layman’s terms or the definition of the Cambridge dictionary, it is defined as the steep level of a slope, as the steep side of a hill or mountain. Likewise, in mathematical terms, the gradient is how steep of a function is. It could be used to measure the steepness of a curve line from top to bottom is in a two-dimensional graph, or how great the change of a vector in a vector function. During the training process of neural networks, we need to find the right learning rate to save time and get an accurate result, meaning that we need to find the steepest gradient.

So, here comes the gradient descent. This is a way, or accurately, an algorithm to update the parameters of a neural network model. Imagine that the goal is to walk downhill blind-folded from the peak of a mountain to the bottom as quickly as possible, we only can take steps to achieve it under the condition that our bodies sense the steepness of the slope. So, the ideal scenario is that the steeper the surface, the bigger step we should take; likewise, the closer to the bottom, the gentler the gradient, the smaller step should be taken, so we won’t pass through the destination. In the training process of a model, we try to save time and get a relatively accurate result. Hence, the parameters start as random values, and in every turn, we make a comparison between the output data and the expected using a loss function. If the difference is too far-off, the parameters will be adjusted a bit more; otherwise, make a slight adjustment.

In reality, training a model requires data, and the more data to be trained, the longer the training time. Therefore, SGD is introduced to tackle this problem. Although it acts like gradient descent, instead of using the entire data set, SGD takes a subset of the data, randomly selected, to make the training process less expensive.

Simply put, SGD is a method to calculate the gradient iteratively with a smaller size of randomly picked data from a data set.
Gradient Descent in Machine Learning
By: Seth Larweh Kodjiku
Tags: Machine Learning, Slope, Gradient Descent

The objective of Gradient Descent

Gradient descent is an exceptionally well known and common algorithm utilized in different Machine Learning algorithms, above all forms the premise of Neural Networks. In this article, try to clarify it in detail, yet in straightforward terms.

Gradient, in plain terms, implies slant or inclination of a surface. So gradient descent in a real sense implies descending a slope with the aim of reaching the absolute bottom on that surface. Let us envisioned a 2D graph, such as a parabola in the figure below.

From the above graph, The goal of gradient descent algorithm is to discover “x” with the end goal that “y” is minimum. “y” here is known as the objective function that the gradient descent algorithm works on, to move to the absolute bottom point.

The Algorithm-Gradient Descent

I utilize a linear regression problem to clarify the gradient descent algorithm. The aim of regression is to limit the sum of squared residuals. We realize that a function arrives at its base value when the slant is equivalent to 0. By utilizing this procedure, we tackled the linear regression problem and learned the weight vector. A similar issue can be tackled by gradient descent procedure.

Gradient descent is an iterative algorithm, that begins from an irregular point on a function and goes down its slant in steps until it arrives at the lowest bottom of that function.”

This algorithm is useful in situations where the optimal points can’t be found by likening the slope of the function to 0. On account of linear regression, you can intellectually plan the sum of squared residuals as the function “y” and the weight vector as “x”.

Moving down the steps

This is the core of the algorithm. The overall thought is, to begin with an irregular point and figure out how to update this point with each iteration with the end goal that we descend the slope.

The means of the calculation are

Find the slope of the objective function concerning each feature, that is, calculate the gradient of the function. Pick an arbitrary initial value for the parameters. Update the slope or gradient function by connecting the parameter values. calculate the progression sizes for each component as step size = gradient * learning rate. Ascertain the new boundaries as new params = old params — step size Repeat steps 3 to 5 until the gradient is almost 0.

The “learning rate” referenced above is an adaptable parameter which intensely impacts the convergence of the algorithm. Bigger learning rates make the algorithm make tremendous strides down the slope and it may hop over the base point in this way missing it. Thus, it is in every case great to adhere to a low learning rate, for example, 0.01. It can likewise be numerically indicated that gradient descent algorithm makes bigger strides down the slant if the beginning stage is high above and makes baby strides as it arrives at nearer to the objective to be mindful so as not to miss it and furthermore be quick enough.
SGD in Machine Learning
By: Frederick Nii Ofei Bruce
Tags: Machine Learning, Sdg, Gradient Descent

Now, the theory gets deeper and I am a bit appreciative to look at the behind the scenes of the codes and the models in terms of the algorithms and mathematics.

It is week 5 and there is a Recap of Lesson 2 in the videos to a whole new session of ‘What’s next?’, resizing Images with DataBlock, Data Augmentation and item_tfms vs batch_tfms, Training your model, and using it to clean your data. With cleaning my data, I had quite a few sessions crashing before, but then it gets better.

Gradient Descent is a popular optimization technique in Machine Learning and Deep Learning, and it can be used with most, if not all, of the learning algorithms. A gradient is the slope of a function. It measures the degree of change of a variable in response to the changes of another variable.

Mathematically, Gradient Descent is a convex function whose output is the partial derivative of a set of parameters of its inputs. The greater the gradient, the steeper the slope.

Typically, there are three types of Gradient Descent:

Batch Gradient Descent Stochastic Gradient Descent Mini-batch Gradient Descent

Previously, I have applied these in varied contexts of a slope, and even in basic traditional machine learning applications but it gets way cooler in fast AI.

And from building the model to deployment, I think I am getting super excited to building very practical solutions in fast AI.

Once again to the mentors at unpackAI, kudos for all the push and consistent help.
It’s All About How You Fall
By: Akshat Katiyar
Tags: Machine Learning, Explanation, Gradient Descent

In life, you have to fall a million times to get it right once(Unless you are Chuck Norris). So today I will take you on a Journey on something that is a very important concept in Machine Learning and unravel a little about it. Sorry if I got your hopes high but this is not a motivational post about life.

Today, we will talk about Gradient Descent.

What is Gradient Descent

Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. For Example

I will explain more about it later for now just think of it as you need that black ball to fall to the lowest value possible. Now why you need to do it is a different question in itself. Say you need to find the black ball with the cheapest cost as its month-end and you are broke. So you find the lowest point in the curve to minimize the cost function. Similarly in Machine learning, you need to reduce the cost function.

Gradient Descent in Real life

My intention here is to give you an overview of what this algorithm does and to visualize it so I won't be stressing too much about the math behind it.

If you are interested in knowing the math behind Gradient Descent.

Check this article by Parul Pandey (You should definitely follow her)

How to reach the lowest Point

So now you know that you need to reach the lowest minima point but how to reach it is still a mystery.

Okay, Close your eyes and put yourself in Gradient descent Boots. You are stuck on the top of a mountain. Your task is to come down. How will you approach this problem ?.

— Look around and find a slope and keep repeating it until you are at the bottom. This is exactly what Gradient Descent does.

Some things come naturally to us humans but for Machine Learning Algos to understand them requires a huge effort. In this case, we know how much to fall down at a time without any explicit calculation, it's all intuition. But for the Algo to understand this we need something called the learning rate alpha. Alpha decides how much it falls from a given point

If alpha is too low — it falls a little by little but it will converge at some point.

If alpha is too high — it might skip the lowest point and never converge

Also once it reaches the lowest point it will stay there for all the next iterations.

Common Problems with this approach

Converging to a local minimum can be quite slow

2. If there are multiple local minima, then there is no guarantee that the procedure will find the global minimum.

For the Local Minima issue, there are solutions to fix it according to the type of problem.

Conclusion

It is one of the most commonly used optimization algorithms in ML. You should definitely check more about it.

Adios
Supervised Learning
By: Mehdi Razpoush Nazari
Tags: Supervised Learning, Gradient Descent, Machine Learning, Andrew Ng, Stanford University

This article is about machine learning and it is a free download of second session of machine learning course in Stanford University by Prof. Andrew Ng. It is about linear regression, gradient descent, and normal equations.

In supervised learning we tell the closest answer based on some examples (training set) and try it to others. He introduced Alvin which was an autonomous driving system using neural network to drive. It was trained first by capturing video and saving driver actions and then it started to steer independently. It increased its driving accuracy by collecting more data from more videos. It was an example of regression problem.

Supervised learning can be shown as below:

Where x represents input variables (features) and y is used to show output variable (target) and h is hypothesis.

In training phase we try to minimize squared differences between the predictions of algorithm and actual targets. One of algorithms to achieve that target is gradient descent. Gradient descent looks like the process of little steps to come down from a hill. If we use multiple training sets then we call it batch gradient descent and in case of very large number of training sets it called stochastic gradient descent or incremental gradient descent which performs calculations until convergence.

There is also another algorithm which uses matrix operations without need for iterative operations.
反向傳播算法（ Backpropagation Algorithm ）
By: Ken Huang
Tags: Backpropagation, Deep Learning, Gradient Descent, Machine Learning, Artificial Neural Network

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Directional Derivative and its relation with Gradient Vector
By: Manish Arya
Tags: Gradient, Gradient Descent

Consider any point P in the input plane (XY plane) whose coordinates are (x, y). Let’s move a very small displacement, known as elementary displacement. So, the co-ordinates of Q are (x+dx, y+dy). R is a point parallel to Y-axis. Co-ordinates of R, therefore, becomes (x, y+dy). Clearly, position vectors of P and Q are given as:

So, from the triangle law of vector addition,

Now output z corresponding to point P is the vertical height as shown. The vertical height corresponding to point Q is z+dz, as the magnitude of the elementary displacement is very small. Therefore, when we move an elementary displacement from P to Q, there will be an elementary change dz in the height z.

Well studied so far. Now let’s define the term directional derivative. As the name depicts, derivative refers to the slope(gradient), and “directional” says that we are moving in a particular direction (from P to Q here).

So directional derivative, from the diagram, is defined as

Also, the gradient vector, which we often encounter in Deep Learning to update the trainable parameters, is defined as:

The latter representation, matrix representation of a vector, is often encountered in Deep Learning.

Here, dz/dx refers to the partial derivative of z with respect to (w.r.t) x. It means an elementary change in z while moving an elementary displacement dx parallel to X-axis. Similarly, dz/dy refers to the partial derivative of z w.r.t y.

Now it’s time to establish the relation, which is our final objective, between directional derivative and gradient vector.

It can be intuitively observed from the diagram that elementary change dz while moving from P to Q is equal to the sum of the elementary change in z while first moving from P to R( i.e., parallel to Y-axis) and then moving from R to Q (i.e., parallel to X-axis).

Note that dz/dx is calculated at point R whose coordinates are (x, y+dy). Interestingly, it can also approximately be the same at point P as the separation between the points P and R is very, very small.

Equation (1) can be written beautifully as the dot product of two vectors (from the definition of the dot product of two vectors). So equation (1) becomes

Dividing both sides by the magnitude of vector PQ, we get

The left-hand side of this equation is nothing but a directional derivative. Also, we know that a vector divided by its magnitude gives a unit vector that specifies its direction. So, the above equation becomes

Here n-hat is a unit vector in the direction of elementary displacement.

This is the sought relation between directional derivative and gradient vector. Now that we have an awesome relation, we will have fun understanding the Gradient Descent, which forms the heart of Deep Learning! I have explained this concept in my Youtube video: https://youtu.be/n7_gFWyI-yQ
梯度下降學習法（Gradient Descent Learning Rule）
By: Ken Huang
Tags: Gradient Descent, 神經網路, Machine Learning, Neural Networks, Artificial Neural Network

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Tuning Your Keras SGD Neural Network Optimizer
By: Kyle Kieser
Tags: Deep Learning, Gradient Descent, Keras, Stochastic Gradient, Neural Networks

Momentum Optimization

Vanilla Gradient Descent vs Gradient Descent with Momentum, Source

Momentum optimization is an improvement on regular gradient descent. The idea behind it is essentially the idea of a ball rolling down a hill. It starts slow but picks up speed over time. The gradient for momentum optimization is used for acceleration rather than speed like in regular Gradient Descent. This simple idea is very effective. Think of what happens when regular gradient descent gets closer to a minimum. The gradient is small, so steps become really small and can take a while to converge. However, with momentum optimization, convergence will happen quicker because its steps utilize the gradients before it rather than just the current one.

Gradient Descent goes down the steep slope quite fast, but then it takes a very long time to go down the valley. In contrast, momentum optimization will roll down the valley faster and faster until it reaches the bottom (the optimum).*

Implementing momentum optimization is Keras is quite simple. You use the SGD optimizer and change a few parameters, as shown below.

optimizer = keras.optimizers.SGD(lr=0.001, momentum=0.9)

The momentum hyperparameter is essentially an induced friction(0 = high friction and 1 = no friction). This “friction” keeps the momentum from growing too large. 0.9 is a typical value.
Top 4 commonly used Machine Learning Algorithms
By: Dhiraj K
Tags: Logistic Regression, Decision Tree, Linear Regression, Gradient Descent, Machine Learning

Let's explore the basic Machine Learning algorithms

If you are a machine learning engineer or data scientist, you can use these techniques to create functional Machine Learning projects easily. These four basic machine learning algorithms are the base of artificial intelligence and Machine Learning.

Here is a detailed video about the Top 10 Machine Learning Algorithms For Data Scientists

1. Gradient Descent

Gradient Descent is an optimization algorithm. Optimizers help to converge a machine learning model during training by reducing the loss. Loss is nothing but the difference between the actual value and predicted value.

A model converges means, there is very little possibility of a further significant decrease in loss, if we do more training iteration. In other words, a model converged, means there is very little possibility of a significant increase in performance, by doing more training iteration.

You can see the implementation in Gradient Descent from Scratch in Python.

2. Linear Regression

Linear regression is one of the most fundamental and basic, machine learning algorithms. It is used to show, the relationship between, two or more variables.

The column values, that are used to predict the output, are called features, or independent variables. And the column value, that is predicted, is called the dependent variable, or label.

You can see the implementation in Linear Regression from Scratch in Python.

3. Logistic Regression

Logistic regression is one of the most fundamental and basic, machine learning algorithms. Logistic regression is a machine learning algorithm, used for classification problems.

It uses, the log odds ratio, and maximum likelihood method, to fit the model.

The column values, that are used to predict the output, are called features, or independent variables. And the column value, that is predicted, is called the dependent variable, or label.

You can see the implementation in Logistic regression from Scratch in Python.

4. Decision Tree

The decision tree algorithm, is a machine learning technique, for making predictions. As its name suggests, it behaves like a tree structure. The decision tree is built by, repeatedly splitting, training data, into smaller and smaller samples.

It works on, the principle of conditions. We check conditions, at a node, and split the data, as per the result, of the conditional statement.

Decision Tree algorithm belongs to, the family of, supervised machine learning algorithms. It can be used to, build classification, as well as regression models.

You can see the implementation in the Decision tree from Scratch in Python.

End Notes:

In this article, we discussed the Top 4 commonly used Machine Learning Algorithms and I hope you found this article useful.

You may like to explore more Machine Learning algorithms in the below video.

Happy Learning !!
Adaptive Learning Rate: AdaGrad and RMSprop
By: Rauf Bhat
Tags: Gradient Descent, Adaptive Learning, Learning Rate, Adagrad, Rmsprop

In my earlier post Gradient Descent with Momentum, we saw how learning rate(η) affects the convergence. Setting the learning rate too high can cause oscillations around minima and setting it too low, slows the convergence. Learning Rate(η) in Gradient Descent and its variations like Momentum is a hyper-parameter which needs to be tuned manually for all the features.

Image by author

When we use the above equation for updating weights in a neural net

Learning rate is the same for all the features Learning rate is the same at all the places in the cost space

Impact of constant learning rate on the convergence

Suppose, we are trying to predict the success/rating of a movie. Let's assume there are thousands of features and one of them being “is_director_nolan”. The feature “is_director_nolan” in our input space will be mostly 0 as Nolan has directed very few movies but his presence significantly impacts the success/rating of a movie. Essentially, this feature will be sparse but because of high information content, we cannot ignore it.

During the forward pass in a neural net, if the input(x)at iteration(t) is 0 then the output becomes the activation(φ) of bias(b) using the below equation.

Image by author

Hence, the local gradient calculated during back prop w.r.t this constant bias will be 1 and the weight update will be very small for this feature (look at the first equation)

For a sparse input feature like “is_director_nolan”, a large weight update will happen only when the input changes from 0 to 1 or vice versa. Whereas a dense feature will receive more updates. Therefore, using a constant and same learning rate for all the features is not a good idea.

Solution

From above we can deduce that the learning rate for a feature should be decayed such that it is inversely proportional to the weight update frequency of that feature. The new learning rate should be low for frequent features whereas it should be high for sparse features.

Image by author

A proxy for the update frequency of the weights is the sum of gradients obtained over the past updates. Instead of taking the absolute value of gradients, we should take the squared gradients so that the sign(direction) of a gradient doesn’t matter. We are only concerned with the gradient frequency. This can be captured using the below equation.

Image by author

Dense features will have a higher value for this proxy while for sparse features this will be very low.

Assuming 𝓥(0) = 0

Image by author

The new learning rate becomes

Image by author

Note: we are taking the square root of 𝓥(t) in order to preserve the dimension of our weight update equation

The new weight update equation becomes

Image by author

What happens if 𝓥(t) is zero(in case of a sparse feature)? In order to avoid the denominator from becoming zero, we should add a small value epsilon ε( usually 1e -07) in the denominator.

Image by author

The above equation is the weight update equation for AdaGrad( Adaptive Gradient).
Simple Native Implementation of Linear Regression with Gradient Descent
By: Bansi Maddali
Tags: Python, Linear Regression, Gradient Descent, Native Code, Machine Learning

In this blog, I will try to explain how to write native code for Linear Regression in Python.

1.We will start of by writing equation function , given a single value of x , it’s weight and the intercept. Y = B0 + B1 * X

#Linear Equation

def calculatey(x,B0,B1):

y = B0 + B1 *x

return y

2.Caluculate the error/delta for each instance

#Delta/error for Gradient Descent

def caluculateError(y,y_pred):

error = y_pred-y

return error

3.Add Gradient Descent Step

#Gradient Descent Step

def updateWeights(B0,B1,alpha,error,x):

B0 = B0 - (alpha * error)

B1 = B1 - (alpha* error * x)

return B0,B1

4.Root-Mean-Squared-Error

#Root Mean Square Error

def rmse(Y_pred,Y):

squarred_errors = [(y_pred-y)* (y_pred-y) for y_pred,y in zip(Y_pred,Y)]

return np.sum(squarred_errors)

5.Linear Regression-Putting all those together

#Linear Regression

#Intialize the weights to Zeros

def applyLinearRegression(X,Y,epochs,alpha):

B0 = 0.0

B1 = 0.0

Y_pred = []

B0_List = []

B1_List = []

Error_List = []

for epoch in range(epochs):

for x,y in zip(X,Y):

B0_List.append(B0)

B1_List.append(B1)

y_pred = calculatey(x,B0,B1)

error = caluculateError(y,y_pred)

Error_List.append(error)

B0,B1 = updateWeights(B0,B1,alpha,error,x)

print(f"Final B0 = {B0} B1 = {B1}")

Y_pred = [calculatey(x,B0,B1) for x in X]

print("Root Mean Squared Error = ",rmse(Y_pred,Y))



#return (B0_List,B1_List,Error_List)

6.Test the implementation

import numpy as np

import math

#Generate Data to test

x = np.random.randint(0, 10, 10)

y = 3 + 4 *x

7.Making the Call to our funtion

applyLinearRegression(x,y,epochs=100,alpha=0.01)

So, Gradient Descent almost matches to exact weights of the original equation. Further, we can plot B0,B1 with Error and Epochs to analyze how gradient descent is progressing.

Entire code in Github.
Logistic regression-Gradient descent
Tags: Gradient Descent, Beginner, Linear Regression, Logistic Regression

Till now the articles I have read about logistic regression and gradient descent are either too complex for beginners or they do not clear everything about this topic, this article contains everything to know about logistic regression

Algorithm

Classification using logistic regression

Why we need this in the first place

it is a part of a supervised learning algorithm and is a classification algorithm.

Before going to logistic regression let us understand the following aspects of Linear regression:-

defining hypothesis function

Loss function

Gradient of loss function

updating theta using Gradient descent

If we have m number of samples i.e. x and y values and hi represents the hypothesis vector for x then the loss function is given as:

loss function of linear regression

this loss function is convex i.e it converges and that can be shown mathematically.

if we take n =1 i.e. number of features =1, then the loss function looks like shown in the figure. theta value can be updated to get the minimum value of loss function using its gradient, if for a particular value of theta gradient is negative then theta shifts rightwards until it becomes zero.

gradient(linear regression)

for a particular value of theta gradient is calculated as shown:

In the final step just update theta as theta = theta-learning_rate*grad

Linear regression is used to predict continuous values but when we have to classify given data points into two classes say class0 and class1 then linear regression is not used
Logistic Regression With Gradient Descent in Excel
By: Angela Shi
Tags: Excel, Logistic Regression, Data Science, Gradient Descent, Machine Learning

Logistic Regression With Gradient Descent in Excel

While using Excel/Google Sheet for solving an actual problem with machine learning algorithms can be a bad idea, implementing the algorithm from scratch with simple formulas and a simple dataset is very helpful to understand how the algorithm works.

In a series of articles, I will present how to implement different algorithms from scratch in Excel.

Linear Regression With Gradient Descent in Excel

Neural Network With Back Propagation in Excel

In this article, I will share how I implemented a simple Logistic Regression with Gradient Descent.

You can use this link: Simple logistic regression with gradient descent, to get the Excel/Google Sheet file.

Now let’s get our hands dirty!

Photo by Karen Maes on Unsplash

Using a simple dataset

First I use a very simple dataset with one feature, you can see the graph below showing the target variable y and the feature variable x.

Creating the model

The Logistic Regression uses the following model:

We can see the final model in the “res” sheet in the Excel file, with a and b.

Now, how a and b were determined? Let see in the next sheet “mlh” for maximum likelihood.

The cost function of the model

First we can consider the likelihood of the model, and we have to maximize:

Then we take the log of the likelihood. To have the loss function, we have to take the opposite of the log likelihood.

We also name this loss function logistic loss, cross-entropy or log-loss.

You can modify the parameters a and b to see that the likelihood of the model will vary.

Gradient descent

Now let’s find the optimal parameters a and b.

For this, we have to find the derivatives of the loss function with respect to a and b.

In the sheet “graddesc”, you will find all the Excel formulas to implement the gradient descent. Enjoy!
Intuitive understanding of Gradient Descent
Tags: Data Science, Calculus, Gradient Descent, Machine Learning, Algorithms

As we are moving ahead with rapid pace with AI technological software developments , we ignore the basic math implementation in future and aim solving the big problems. For the future generations who wants to understand what is happening under the hood, I am leaving the traces of gradient descent from a mathematical perspective.

Code Here:

Read more here:
Gradient Descent Nedir?
By: Mehmet Fatih Akca
Tags: Optimizasyon, Gradient Descent, Makine Öğrenimi, Machine Learning, Makine Öğrenmesi

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Descent method — Steepest descent and conjugate gradient
By: Sophia Yang
Tags: Math, Linear Algebra, Gradient Descent

Descent method — Steepest descent and conjugate gradient

Let’s start with this equation and we want to solve for x:

The solution x the minimize the function below when A is symmetric positive definite (otherwise, x could be the maximum). It is because the gradient of f(x), ∇f(x) = Ax- b. And when Ax=b, ∇f(x)=0 and thus x is the minimum of the function.

In this article, I am going to show you two ways to find the solution x — method of Steepest Descent and method of Conjugate Gradient.

Method of Steepest Descent

The main idea of the descent method is that we start with a starting point of x, try to find the next point that’s closer to the solution, iterate over the process until we find the final solution.

For example, at step k, we are at the point 𝑥(𝑘). How do we decide where to go next? Which direction should we go? How much should we go?

Let’s assume the direction we decide to go is p(k) and how far we are going down the direction is 𝛼. Then the next data point can be written as:

For each step, the steepest descent method wants to go down towards the steepest slope, where it is the most effective (r here means residual):

Once we decided the direction to go, we want to make sure we minimize the function in this direction:

Now we can get calculate the x and the residual for the next data point:

This is basically the math behind the steepest descent method. Normally we would give a stopping criterion to the residual, and we iterate the process until we reach the stopping point.

For Python implementation, check out:

Conjugate gradient method

The steepest descent method is great that we minimize the function in the direction of each step. But it doesn’t guarantee that the direction we are going to minimize the function from all the previous directions. Here we introduce a very important term A conjugate directions. Directions p are A conjugate directions if they have the following property (note A is symmetric positive definite):

Only when the current direction p is A conjugate to all the previous directions, will the next data point minimize the function in the span of all previous directions.

The issue is how do we calculate the search direction p when p has to be A conjugate?

Remember that the steepest descent chose the steepest slope, which is also the residual (r) at each step. And we know that this is a good choice. How about we find an A-conjugate direction that’s the closest to the direction of the steepest descent, i.e., we minimize the 2-norm of the vector (r-p).

Through calculations, we know that the current direction is a combination of the current residual and the last direction.

Because of the property of A-conjugate directions:

We can then calculate 𝛾_𝑘:

In summary, the conjugate gradient method is as follows:

Again, for Python implementation, check out:

Now you know how to solve the system of linear equations using the steepest descent and the conjugate gradient method! Enjoy!

Reference:
Linear Regression With Gradient Descent in Excel
By: Angela Shi
Tags: Machine Learning, Data Science, Excel, Gradient Descent

Linear Regression With Gradient Descent in Excel

While using Excel/Google Sheet for solving an actual problem with machine learning algorithms can be a bad idea, implementing the algorithm from scratch with simple formulas and a simple dataset is very helpful to understand how the algorithm works. After doing it for almost all the common algorithms including the Neural Network, it helps me a lot.

In this article, I will share how I implemented a simple Linear Regression with Gradient Descent. You can use this link Simple linear regression with gradient descent to get the Excel/Google Sheet file.

Now let’s get our hands dirty!

Photo by Karen Maes on Unsplash

Using a simple dataset

First I use a very simple dataset with one feature, you can see the graph below showing the target variable y and the feature variable x.

Creating the linear model

In Google Sheet or Excel, you can add a trendline. So you get the result of Linear Regression.

But if you want to use the model to do predictions, then it is necessary to implement the model and in this case, the model is quite simple: for each new observation x, we can just create a formula: y=a*x + b. Where a and b are the parameters of the model.

The cost function of the model

How can we obtain the parameters a and b? Well, the optimal values for a and b are those minimizing the cost function, which is the Squared Error of the model. So for each data point, we can calculate the Squared Error.

Squared Error = (prediction-real value)²=(a*x+b-real value)²

In order to find the minimum of the cost function, we use the gradient descent algorithm.
Why convexity is the key to optimization
By: Nvs Yashwanth
Tags: Data Science, Gradient Descent, Convexity, Machine Learning, Optimization

Machine Learning

Why convexity is the key to optimization

A Convex function. Source Wikipedia.

The most interesting thing you would first come across when starting out with machine learning is the optimization algorithm and to be specific, it is the gradient descent, which is a first-order iterative optimization algorithm used to minimize the cost function.

The intuition behind gradient descent is converging to a solution, which could be a local minimum in the neighborhood or in best-case, the global minimum.

Everything seems fine and okay until you start questioning yourself about the convergence problem. Having a good understanding of convexity helps you prove the intuition behind the idea of gradient descent. So let us discuss the same.
GRADIENT DESCENT : Basics And Application
By: Society Of Ai
Tags: Cost Function, Gradient Descent, Application, Machine Learning, Algorithms

Gradient Descent is one of the most widely used optimization algorithms to train machine learning models. It is extensively employed in machine learning as well as deep learning.

APPLICATION: To estimate the values of function parameter so that the value of cost function is minimized.

Let’s study more about the cost function:

We place on the x-axis, on the y-axis, and on the z-axis. The graph below depicts the following:

Gradient Descent Algorithm

Our aim, here, is to somehow reach the minima of the cost function, i.e. the pits of the graph where the black marked lines end.

How do we achieve this?

We achieve this by taking the derivative of the cost function. The slope of the tangent represents the derivative at that point. We take steps down the graph in the path of steepest descent. Eventually, we reach the minima of the graph.

The learning rate parameter (α)determines the size of each step we take along the direction of steepest descent.

In the above example, the distance between each ‘star’ is a representation of the step taken, determined by “α”. Greater the learning rate (α) larger is the step taken.

NOTE: Depending on where we start, we could end up at two different minima (local optima) as shown in the above figure.

Gradient Descent Algorithm

[ NOTE: We know that a = b is a truth assertion to check if a is equal to b, however, a:=b is an assignment operation where we the value of b is assigned to a. ]

Here we have to simultaneously update θ0 & θ1 in the following manner:

Now, let us mathematically analyse the simultaneous update of function parameters (θ0 & θ1).

Let us consider one parameter (θ1) for the time-being. We will plot the graph cost function J(θ1) against the parameter taken into consideration (θ1).

We have to repeat this formula until the cost function reaches its minimum value.

In the above graph, the value of the slope is negative until (θ1) reaches its minimum value. On moving further towards right the value of the slope increases as the value of (θ1) increases.

On the other hand, the value of the learning rate parameter (α) should be kept such that the gradient descent algorithm converges in a reasonable amount of time.

How does gradient descent converge when the learning rate parameter is fixed?

The convergence should happen in such a way that ( d[J(θ1) ])/(d(θ1)) approaches zero at the bottom of the cost function.

At the minima we get:

θ1:=θ1+α* 0

The gradient descent can converge to local minimum even with a fixed value of the learning rate parameter (α) because the steps taken by gradient descent spontaneously decrease as we approach the local minimum.

If you liked the story and want to appreciate us you can clap as much as you can. Appreciate our work by your constructive comment and also you can connect to us on….

Youtube: https://www.youtube.com/channel/SocietyOFAI

LinkedIn : https://www.linkedin.com/company/society-of-ai

Facebook: https://www.facebook.com/societyofai/

Website : https://www.societyofai.in/
Mathematical justification of Stochastic Gradient Descent
By: Ravikiran Gunale
Tags: Deep Learning, Gradient Descent, Stochastic Gradient, Machine Learning, Statistics

Minimising cost function is the holy grail of machine learning. Gradient descent is our master key to all cost minimisation problems. But gradient descent is slow. It uses entire training dataset for one iteration. This problem is exaggerated when working with millions of rows dataset. Stochastic gradient descent (SGD) comes to our rescue.

Nearly all of deep learning is powered by SGD algorithm.

SGD approximates gradient by using one example. This approximation is fairly accurate and works. But how can we approximate calculation involving million of rows using only one row?

Let us see general equation of gradient

where L stands for loss function, J is the cost function and m is number of training examples.

1) Key insight here is that gradient is the expectation. The expectation can be approximated by few examples. This insight shows that our approximation is accurate and guarantees convergence of SGD algorithm.

2) Standard error of the mean calculated from m examples is inversely proportional to square root of number of examples (m)

Consider two scenarios, we are estimating mean using 10 examples and 100,00 examples. Later examples needs 100 times more computation but reduces error by only 10 times.

We get diminishing gains by using more examples to approximate mean.

3) Small batches can offer a regularizing effect by adding noise to the gradient.

4) In practise there is redundancy in training examples. Large number of examples all making very similar contributions to the gradient. SGD can eliminate this redundancy.

Mini batch gradient descent: Similar to SGD but instead of 1 example, we use more (generally few hundreds) examples. All of the above arguments are valid for mini batch gradient descent
Difference Between Stochastic, Mini-Batch and Batch Gradient descent
By: Danny Wilde
Tags: Gentle Introduction, Gradient Descent, Deep Learning, Machine Learning, Neural Networks

Gradient descent is how a neural network tweaks its weights. It can be visualised as a person moving down a hill. This hill is also a graph of weights vs the loss. The ultimate goal is to get as far down the hill as possible in order to minimise the loss for a given set of inputs.

The loss is measure of how bad your neural network is performing. This is why you want to minimise it. So if a neural network was given a picture of a cat and it classifies it as a dog then the loss is 1, the highest it can be. If the neural network correctly predicts the cat then the loss is 0.

The weights axis corresponds to a single weight in the neural network. So for instance if your neural network was just two neurons it would be the line connecting them together. Of course in practise your neural network contains millions of weights but they are ignored in order to visualise the concept better.

The inputs are the data instances going in to your neural network. There is an option of when to calculate the loss when feeding the data into the neural network. It can be done once per data instance (stochastic) or once per mini-batch or once per batch.

The loss function looks different for these different options.

Performing gradient descent per batch basis does not yield the lowest possible minimum. It super imposes a lot of the more nuanced losses which may have corresponded to important features in the dateset.

Whereas stochastic gradient descent is too jagged and has many local minimums where one can get stuck. If you imagine the person going down the hill he would find it very difficult to not get stuck in one of the many dips. It is so jagged because it is based on just a single data instance.

The mini-batch gradient descent is optimum because the loss function gets very low without too much extra jaggedness. Our imaginary friend would be able to reach the bottom of this hill.
Linear Regression Part II — using Gradient Descent Algorithm
By: Nandini Sekar
Tags: Linear Regression, Gradient Descent

Gradient Descent approach

Introduction:

In this article we will learn about Linear Regression using an algorithm called Gradient Descent[GD]. Please check the below link to get an overview of Linear Regression using SSE → https://medium.com/@nandinisekar27/linear-regression-part-i-cc1a19a3591e

Now lets get into our topic, the word Gradient means “slope” and Descent means an “act of moving downwards”. GD performs multiple iterations to find the point at which is slope is 0 or minimal. Hence it can be framed as a “first order iterative optimization technique to find the minimum of a function”.

Predicted Y value

Where Y hat[predicted Y] is replaced in the function above

Thereby if we substitute different values of “m” and “c” each time we will arrive at different set of functions, all those when plotted results in a “convex” shaped graph and the lowest point of that curve is what we need to find.

Flow of Gradient Descent Convex Graph

The small blue circles in the above figure represents the learning rate or the speed at which the point moves from one position to another to find the global minima. There may be more than one minima in a graph, all the minima’s are referred as “Local Minima” the point where the slope is 0 or close to 0 is referred as “Global Minima”

Steps Involved:

Calculate y actual and y predicted values. Calculate their difference and square them up.(Refer little above for the function) Substitute values for slope-“m” and intercept-“c” in the function, for every value in x the Mean of Squares is calculated:

4. Initially we can take m and c values as 0. Learning Rate-0.001(blue circles in figure above) this is handled depending on how the value of m /slope is varying at each point.

5. Learning rate should be minimum for good accuracy.

6. We need to substitute the values of “m” and “c” in the partial derivative of the function. Once with respect to “m” and next with respect to “c” the partial derivatives are found, for the above function.

7. Updating the values of m and c for each iteration, this process is continued until minimum value of the function(0 slope value) is found, so that the error is 0% and accuracy is 100%

Conclusion:

With this we have come to an end of this article. We learnt about how “gradient descent” helps in finding the global minimum point by taking iterative approach in linear regression model. GD algorithm is not only for linear regression, used in various other algorithms as well.

Disadvantages of Linear Regression:

If the data we are going to operate on is non linear, then it will be inappropriate to implement linear regression algorithm.

2. This is prone to Bias trade off and Variance problem.

Please wait for my next post which is on Error metrics, Bias and Variance problems!

Happy Learning! :)
Derivative the Gradient Descent of Linear and Logistic Regression
By: Abu Nayem
Tags: Cost Function, Logistic Regression, Derivatives, Linear Regression, Gradient Descent

Linear Regression and Logistic Regression are most useful models in Machine Learning field. We need to fit these models to predicting, forecasting, classification or reducing error. This why it is very essential to calculate gradient descent, cost function. I am don’t writing this blog to elaborate on the Linear and Logistic Regression. I will explain the derivate of the cost function for gradient descent. Let’s start,

We know,

Linear equation:

Let’s derivate gradient descent of the logistic regression.

Finally, we got our simplified formula to doing derivation.

Linear equation: