“Is it BAD?” When the Gradient Descent algorithm meets high dimensional data
Tags: Gradient Descent

Photo credit to Pinterest.com

Intro:

From this blog, I hope to deliver a brief and clear explanation of the gradient descent algorithm. More than that, I also want to invite readers to share your opinion on the question of whether the gradient descent algorithm would work well in high dimensional machine learning analysis.

The term “gradient descent” can be familiar to us. We might have seen it frequently appeared at the titles of some scholarly articles, combined with several unfathomably mathematical words, that you have never crossed in your lives. Nevertheless, it is actually a very popular method, as well as a practical one. Now, before jumping into showing its precise definition, an intriguing analogy can be given to better illustrate the intuition of this algorithm.

Imagine we are hiking in beautiful mountains. Someday, we are dedicating to find a bottom lake (a local minimum) from halfway up a small hill (a starting-point). But our sight is blocked by the heavy fog inside the forests of the mountain. The only helpful tool is a walking stick, which can tell us what direction is the best for us to go from our current position (gradient). And since we are very careful about time consumption and worry about circling around, we take the downhill path step by step (learning rate).

The analogy roughly depicts how the gradient descent algorithm came up. Since when solving real-world problems, people always want the “things to be better”. Namely, we might strive to either pu rsue more profits or fewer errors. Those purposes, similar to finding the bottom lake, can all be considered as the process of optimization. In the machine learning field, the focus is moving towards a loss function, which helps evaluate how well specific algorithm models the training data. Gradient descent, a first-order iterative optimization algorithm, then shed many lights on finding the more satisfying machine learning models for us.

Gradient Descent:

To give a formal understanding of gradient descent, we will break down the explanation into four parts in this part.

Gradient

What is the gradient?

In vector analysis, we define the gradient of a scalar-valued differentiable function f of several variables is the vector field whose value at a point p whose components are the partial derivatives of f at p. A simpler perception of the gradient can be the synonym of the slope.

Author Self-made

Gradient Descent

Move to the definition of gradient descent, we can summarize it as an approach that by iteratively updating its parameters in the negative direction of the gradient of our objective function to find its minimum value. we illustrate the concept in the following graph.

Author Self-made

How we implement it?

first, we initialize a random starting point for the algorithm.

second, we fix a value for the step size parameter, also named the learning rate, which plays an important role in finding the global minima during the iteration process.

Then, we just need to calculate the gradient value. We will use the initial value to subtract the product of gradient and learning rate repetitively until the optimized point is reached.

Finally, we can stop our algorithm by setting a threshold of the norm of the new point’s gradient value.

Example

Nothing can be more comparable than a concrete example.

Suppose we want to minimize an objective loss function f(x,y):

And we set our initialized point p as (0, 0). Thus, we have f(0,0) equals to 1²+7²=50. Now we calculate the gradient at the point p, which gives us:

So, the gradient of at p =(-2, -14). Given a reasonable learn rate as 0.1, we have the new point as (0,0) - 0.1(-2, -14)=(0.2, 1.4). At the time, we recalculate the objective loss function, and we get f(0.2, 1.4)=32. Clearly, we saw a great decrease from the starting point with this step. For the following process, we will repeat the above procedures, and in the end we would find that as the point moves closer to (1,7), the optimal solution is achieved.

Note that the gradient value becomes smaller during the iteration process. This makes sense to us because when we reached the global minimum of the loss function, the gradient of that point should equal zero, inferring there is no need to go any step further. That is also the time we stop the gradient descent algorithm.

As previously mentioned, the learning rate is a key parameter here. If we set the value of it to 1 in the above example, we might never reach the optimized point, the gradient descent algorithm will keep oscillating and never converge to the minimum value cause the stepwise is too big.

Usual Types

Gradient descent can be somewhat understood as a concept, for it guides people to find the optimal solutions. However, the paths can be different.

“All roads lead to Rome, we said.”

Based on this context, gradient descent algorithms can be categorized into three different primary classes. To make it more interesting for you to read, we show a scatch graph of the three methods and you may guess each method based on the name, explanation, and your intuition.

. Batch Gradient Descent

It is also known as the Vanilla gradient descent, which is a very common methodology. It takes all training data points into consideration for moving a single step in its algorithm. The core idea is to calculate the mean of all gradients at each move. According to this nature, the average cost versus epoch can decrease very smoothly. Meanwhile, the method is undoubtedly computational.

. Stochastic Gradient Descent

SGD is standing the opposite of the Batch method, for it takes only one example data point for consideration. We usually fit the point into a neural network, and then calculate the gradient. The new gradient will be used for tuning the weights parameters. The process will produce again and again until we approximate the optimal point. Based on the process, the cost function will decrease with various fluctuations and it will never reach the minima because of them. However, as we now take just one point for updating the algorithm, SGD can be a much efficient way when we dealing with high dimensional datasets.

. Mini Batch Gradient Descent

Mini Batch seems to take both the advantages of Batch and SGD method. It needs us to define a fixed number of data points and the number is much less than the total size of the training dataset. We called it a mini-batch. Like SGD, we will calculate the average gradient of those points, and utilize the value to tune weights. This method, unlike SGD that takes only one point for consideration, allows us to use vectorized implementation, which helps us to get faster computations.

Author Self-made

The answer is: (1) The blue arrow line is the Batch Gradient Descent; (2) The orange arrow line is the Mini Batch Gradient Descent; (3) The black arrow line is the Stochastic Gradient Descent.