Beautiful Trap of Neural Networks
By: Ilya Kvyatkovskiy
Tags: Neural Networks, Optimization, Keras, Machine Learning, Gradient Descent

Nowadays, Neural Networks (NN) have become a ‘fashionable’ attribute of every modern trend discussion. Most individuals with some degree of technical and even non-technical background can speculate about them with varying levels of confidence, oftentimes making a skeptical impression on an informed listener. If someone attempts to talk about the connection between these mysterious technologies and the human brain’s anatomical structure — this is a grave folly and not the best way to gain credibility as an experienced data scientist.

Simply put Neural Networks are a family of machine learning algorithms able to recognize patterns. While this definition might seem fairly vague, and not very telling, it is still probably the most accurate common language description. Indeed, the algorithms are quite similar in their structure as they all connect input with output through some rigorous and highly intense calculations and are capable of taking on practically any type of raw data with minimal pre-processing. And the outputs easily satisfy every engineer’s demand — whether a regression or a classification problem is at hand. Using these algorithms we can approximate any smooth (and even not-too-smooth) function with a required degree of precision. Sounds like a dream, what could go wrong?!

But, as is always the case, this dream comes at a steep price, which comes in the form of quite a high level of computational complexity of even the simplest NN models. The number of the model’s parameters follows a complexity increase in a multiplicative pattern, meaning these models are very prone to overfitting and require regularization. Model’s hyper-parameter tuning becomes an intuitive hide-and-seek game with very lengthy iterations.

Given these limitations, what is the cornerstone of NN’s success? It is their high performance level! In many situations they outperform other models and their results are unparalleled. Neural Networks simply keep adjusting their parameters until their error is further irreducible. This is called parameter optimization, and there’s a number of algorithms based on the gradient descent approach which I’ve included a non-exhaustive below:

Adam — adaptive moment estimation

— adaptive moment estimation AdaGrad — adaptive gradient

— adaptive gradient AdaDelta — AdaGrad extension

— AdaGrad extension Nesterov’s algorithm

algorithm RMSProp — Root-Mean-Square propagation

— Root-Mean-Square propagation SGD — Stochastic gradient descent

— Stochastic gradient descent LBFGS — Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm

— Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm Line — Bresenham’s line algorithm

— Bresenham’s line algorithm Conjugate gradient

Hessian Free (aka truncated Newton) algorithm

Sounds quite straight-forward so far, doesn’t it? A variety of optimization options with a common denominator in the gradient descent approach.

But wait a second, what can possibly be undesirable in such an idillic situation? It is the gradient descent approach which rings the bell for anyone with some background in calculus and applied math. The basic algorithm itself is as old as the hills, but has a very unpleasant caveat. It can fool you, to be blunt. As basic calculus tells us, we could be descending to a local minimum instead of the desired global one which is understandably a problem. There are several types of local minima — saddle, plateau, flat areas as well as other irregularities such as cliffs and exploding gradients. But even quite a simple polynomial

graphed around the origin can look like this beauty!