Non-Convex Optimization in Deep Learning
By: Er Raqabi El Mehdi
Tags: Deep Learning, Optimal Performance, Optimization, Convex Optimization, Approximation

Humans have been enjoying convex optimization (CO) for many years compared to the few contexts where they had to deal with non-convex optimization (NCO). However, over the last decade, non-convex optimization became more crucial and important than before. In fact, with the emergence of deep learning (DL), researchers needed to deal with non-convex optimization more and more given the benefits hidden behind its complexity.

In this article, I will first present both convex and non-convex optimization before getting into some interesting insights related to non-convex optimization. The objective is highlighting both benefits and challenges and see whether there are trade offs that the researcher can benefit from. For simplicity, we consider minimization problems.

Convex Optimization

CO is a subfield of mathematical optimization that deals with minimizing specific convex function over convex sets. It is interesting since in many cases, convergence time is polynomial. Linear programming (LP) and least square problem (LSP) are special cases of CO. A convex problem has the form presented below. There is, in general, no unique approach to tackle convex optimization problems. Many effective methods are available in the literature such as the interior point method. If we can formulate a problem as a convex optimization problem, then we can solve it efficiently, just as we can solve a LSP efficiently. Still, recognizing a convex function can be complicated and obviously recognizing convex optimization problems, or those that can be transformed to convex optimization problems can be more challenging challenging.

Non-Convex Optimization

A NCO is any problem where the objective or any of the constraints are non-convex. Even simple looking problems with as few as ten variables can be extremely challenging, while problems with a few hundreds of variables can be intractable. Here, generally, we face the obligation to compromise, i.e. give up seeking the optimal solution, which minimizes the objective over all feasible points. This compromise opens a door for local optimization where intensive research has been conducted and many developments were achieved. As a result, local optimization methods are widely used in applications where there is value in finding a good point, if not the very best. In an engineering design application, for example, local optimization can be used to improve the performance of a design originally obtained by manual, or other, design methods.

What makes non-convex optimization hard?

Such optimization problems may have multiple feasible and very flat regions, a widely varying curvature, several saddle points, and multiple local minima within each region. It can then take time exponential in the number of variables and constraints to determine that a non-convex problem is infeasible, that the objective function is unbounded, or that an optimal solution is the global optimum across all feasible regions. The optimization may also require an initial guess, which is critical and can greatly affect the objective value of the local solution obtained. Furthermore, little information is provided about how far from the global optimum the local solution is. Beyond this, local optimization methods are often sensitive to algorithm parameter values, which may need to be adjusted for a particular problem, or family of problems. To sum it up, roughly speaking, local optimization methods, used in non-convex optimization, are more art than technology. In contrast, except some cases on the boundary of what is currently possible, there is little art involved in solving a least-squares problem or a linear program. Consequently, there cannot be a general algorithm to solve efficiently NCO problems in all cases and scenarios. NCP is then at least NP-hard. Weak theoretical guarantees add to the downsides of this type of optimization. However, it has the upside of an endless set of problems that we can try to solve better while learning more theoretical insights and improve the practical implementations.

Why is it needed then?

NCO existed since the early days of operations research. Still, the topic gained more interest and focus with the emergence of DL in the recent years. In fact, neural networks (NN)are universal function approximators. With enough neurons, they have the ability to approximate any function well. This the beauty of such networks. Among the functions to be approximated, non-convex functions are gaining more focus. To approximate them, convex functions cannot be good enough. Hence, the importance of using NCO. The freedom to express the learning problem as a non-convex optimization problem gives immense modeling power to the algorithm designer. Since then, the issue of convex optimization became very important to the ML community and consequently some scientific meetings have been devoted solely to this issue, such as the NIPS 2015 Workshop on Non-convex Optimization for Machine Learning: Theory and Practice. Despite the guarantees achieved in individual instances, it is still complex to unify a framework of what makes NCO easy to control. On the practical side, conversations between theorists and practitioners can support the identification of what kind of conditions are rational for specific applications, and thus lead to the design of practically motivated algorithms for non-convex optimization with rigorous guarantees.

Non-convex Optimization Convergence

For NCO, many CO techniques can be used such as stochastic gradient descent (SGD), mini-batching, stochastic variance-reduced gradient (SVRG), and momentum. There are also specialized methods for solving non-convex problems known in operations research such as alternating minimization methods, branch-and-bound methods. But, these methods are not, generally, very popular for ML problems. In the same context, there are varieties of theoretical convergence results including convergence to a stationary point, convergence to a local minimum, local convergence to the global minimum, and global convergence to the global minimum. As an example, let us consider the widely used SGD. Non-convex SGD converges with a slow theoretical rate, but not necessarily to a local minimum, which certainly means it doesn’t necessarily reach the global optimum. These theoretical insights can be strengthened through the incorporation of stronger conditions and assumptions. With the previous ones, we can prove its convergence to a local minimum with (or without) an explicit rate of convergence. Sometimes, if we start close enough to the global optimum, we can achieve a local convergence to the global optimum. However, it is very expensive in terms of time and applied to specific cases. Beyond that, the global convergence to the global minimum happens when we can converge wherever we initialize. For deep neural networks, it known empirically that it doesn’t happen.

Practical Insights

Because of this, we almost always can’t prove convergence or anything like that when we run back-propagation SGD on a deep net. In DL as NCO, many things could go wrong. First, convergence to a bad local minimum may happen. I such a case, we can re-optimize the system with different initialization and/or add extra noise to gradient updates. Second, we may face convergence to a saddle point that can be tackled by finding the hessian and computing a descent direction. Third, getting stuck in a region of low gradient magnitude that can be solved using batchnorm or designing networks efficiently using a rectified linear unit (ReLU) activation function for instance. Fourth, we may take huge steps and diverging because of the high curvature. In that case, we can use adaptive step size or more intuitively limit the size of the gradient step. Finally, if we end up having a wrong setting of hyperparameters, we can go for hyperparameter optimization methods.

Finally, it seems that the NCO in DL will remain an art unless someone finds a generic framework that can be followed in any problem faced. Until then, it is really fascinating to witness the high rate with which things are evolving between theory and practice and how both machine learning and operations research are taking more and more place in our daily life !

Have you ever dealt with non-convex optimization in deep learning or elsewhere? What are your insights on it? I am looking forward to hear your ideas :-).