Gradient-free Neural Network Training by Multi-convex Alternating Optimization

In recent years , stochastic gradient descent ( SGD ) and its variants have been the dominant optimization methods for training deep neural networks . However , SGD suffers from limitations such as the lack of theoretical guarantees , vanishing gradients , excessive sensitivity to input , and difficulties solving highly non-smooth constraints and functions . To overcome these drawbacks , alternating minimizationbased methods for deep neural network optimization have attracted fast-increasing attention recently . As an emerging and open domain , however , several new challenges need to be addressed , including 1 ) Convergence depending on the choice of hyperparameters , and 2 ) Lack of unified theoretical frameworks with general conditions . We , therefore , propose a novel Deep Learning Alternating Minimization ( DLAM ) algorithm to deal with these two challenges . Our innovative inequality-constrained formulation infinitely approximates the original problem with non-convex equality constraints , enabling our proof of global convergence of the DLAM algorithm under mild , practical conditions , regardless of the choice of hyperparameters and wide range of various activation functions . Experiments on benchmark datasets demonstrate the effectiveness of DLAM . 1 INTRODUCTION . Stochastic gradient descent ( SGD ) and its variants have become popular optimization methods for training deep neural networks . These methods split a dataset into multiple batches and then optimize them sequentially by gradient descent in each epoch . SGD has two main advantages : not only is it simple to implement , but it can also be applied in online settings where new coming training data are used to train models . However , while many researchers have provided solid theoretical guarantees on the convergence of SGD ( Kingma & Ba ( 2014 ) ; Reddi et al . ( 2018 ) ; Sutskever et al . ( 2013 ) ) , the assumptions of their proofs can not be applied to problems involving deep neural networks , which are highly nonsmooth and nonconvex . Aside from the lack of theoretical guarantees , several additional drawbacks restrict the applications of SGD . It suffers from the gradient vanishing problem , meaning that the error signal diminishes as the gradient is backpropagated , which prevents the neural networks from utilizing further training ( Taylor et al . ( 2016 ) ) , and the gradient of the activation function is highly sensitive to the input ( i.e . poor conditioning ) , so a small change in the input can lead to a dramatic change in the gradient . To tackle these intrinsic drawbacks of gradient descent optimization methods , alternating minimization methods have started to attract attention as a potential way to solve deep learning problems . Here , the loss function of a deep neural network is reformulated as a nested function associated with multiple linear and nonlinear transformations across multi-layers . This nested structure is then decomposed into a series of linear and nonlinear equality constraints by introducing auxiliary variables and penalty hyperparameters . The linear and nonlinear equality constraints generate multiple subproblems , which can be minimized alternately . Some recent alternating minimization methods have focused on applying the Alternating Direction Method of Multipliers ( ADMM ) ( Taylor et al . ( 2016 ) ; Wang et al . ( 2019 ) ) and Block Coordinate Descent ( BCD ) ( Jinshan Zeng ( 2018 ) ) , with empirical evaluations demonstrating good scalability in terms of the number of layers and high accuracy on the test sets , especially for neural networks that are very deep , thanks to parallelism ( Taylor et al . ( 2016 ) ; Wang et al . ( 2019 ) ) . For more information , please refer to Section H in the supplementary materials . These methods also avoid gradient vanishing problems and allow for non-differentiable activation functions such as binarized neural networks ( Courbariaux et al . ( 2015 ) ) , as well as allowing for complex nonsmooth regularization and the constraints that are increasingly important for deep neural architectures that are required to satisfy practical requirements such as interpretability , energy-efficiency , and cost awareness Carreira-Perpinan & Wang ( 2014 ) . However , as an emerging domain , alternating minimization for deep model optimization suffers from a number of unsolved challenges including : 1 . Convergence properties are sensitive to penalty parameters . One recent work by Wang et al . firstly proved the convergence guarantee of ADMM in the fully-connected neural network problem ( Wang et al . ( 2019 ) ) . However , such convergence guarantee is dependent on the choice of penalty hyperparameters : the convergence can not be guaranteed any more when penalty hyperparameters are small . 2 . Lack of unified theoretical frameworks with general conditions . The global convergence of ADMM on deep learning has rarely been explored ( Wang et al . ( 2019 ) ; Zeng et al . ( 2019 ) ) . And existing few works are tailored for and limited to few specific loss functions and activation functions : Zeng et al . proved that the ADMM is convergent for square loss function and twice differentiable activation functions ( e.g . sigmoid ) ( Zeng et al . ( 2019 ) ) ; Wang et al . proved the convergence of ADMM for the Relu activation function ( Wang et al . ( 2019 ) ) . Therefore , there lacks a unified theoretical framework which covers wide range of commonly used losses and activation functions . In order to simultaneously address these technical problems , we propose a new formulation of the deep neural network problem , along with a novel Deep Learning Alternating Minimization ( DLAM ) algorithm . The proposed framework is highly generic and sufficiently flexible to be utilized in common fully-connected deep neural network models , as well as being easily extendable to other models such as convolutional neural networks ( Krizhevsky et al . ( 2012 ) ) and recurrent neural networks ( Mikolov et al . ( 2010 ) ) . Specifically , we , for the first time , transform the original deep neural network optimization problem into an inequality-constrained problem that can be infinitely approximate to the original one . Applying this innovation to an inequality-constraint based transformation ensures the convexity of all subproblems , and hence easily ensures global minima , the inequality-constraint prevents the output of a nonlinear function from changing much and reduces sensitivity to the input . The operation of matrix inversion is avoided by the quadratic approximation technique and a backtracking algorithm . Moreover , while existing methods require typically strict and complex conditions , such as Kurdyka-ojasiewicz ( KL ) properties ( Lau et al . ( 2018 ) ) to prove convergence , our proposed method requires simple and mild conditions to guarantee convergence and covers most of the commonly-used loss functions and activation functions , and the choice of hyperparameters has no effect on the convergence of our DLAM algorithm theoretically . Our contributions include : • We propose a novel formulation for deep neural network optimization . The deeply nested acti- vation functions are disentangled into separate functions innovatively coordinated by inequality constraints that are inherently convex . • We present a novel and efficient DLAM algorithm . A quadratic approximation technique and a backtracking algorithm are utilized to avoid matrix inversion . Every subproblem has a closed-form solution , further boosting efficiency . • We investigate several attractive convergence properties of the DLAM algorithm under mild conditions . The model assumptions are very mild , ensuring that most deep learning problems will satisfy our assumptions . The new DLAM algorithm is guaranteed to converge to a critical point . • We conduct experiments on benchmark datasets to validate our proposed DLAM algorithm . Experiments on two benchmark datasets show that the new algorithm performs well compared with SGD or its variants and ADMM . The rest of paper is organized as follows . In Section 2 , we present the problem formulation and the new DLAM algorithm . In Section 3 , we introduce the main convergence results for the DLAM algorithm . Section 4 reports the results of the extensive experiments conducted to validate the convergence and effectiveness of the new DLAM . Section 5 concludes by summarizing the research . 2 THE DLAM ALGORITHM . In this section , we present our novel DLAM algorithm . Section 2.1 provides the new algorithm ’ s formulation and Section 2.2 shows how the DLAM algorithm solve all the subproblems . 2.1 INEQUALITY APPROXIMATION FOR DEEP LEARNING . The important notations used in this paper are listed in Table 1 . A typical fully-connected deep neural network consists of L layers , each of which are defined by a linear mapping and a nonlinear activation function . A linear mapping is composed of a weight vector Wl ∈ Rnl×nl−1 , where nl is the number of neurons on the l-th layer and an intercept bl ∈ Rnl ; a nonlinear mapping is defined by an activation function hl ( • ) . Given an input al−1 ∈ Rnl−1 from the ( l − 1 ) -th layer , the l-th layer outputs al = hl ( Wlal−1 + bl ) . By introducing an auxiliary variable zl as the temporary result of the linear mapping , the deep neural network problem is formulated mathematically as follows : Problem 1 : minal , Wl , bl , zl R ( zL ; y ) + ∑L l=1 Ωl ( Wl ) s.t . zl= Wlal−1+bl ( l=1 , · · · , L ) , al=hl ( zl ) ( l=1 , · · · , L−1 ) where a0 = x ∈ Rd is the input of the deep neural network , d is the number of feature dimensions , and y is a predefined label vector . R ( zL ; y ) ≥ 0 is the risk function for the L-th layer , which is convex and proper , and Ωl ( Wl ) ≥ 0 is a regularization term on the l-th layer , which is also convex and proper . The equality constraint al = hl ( zl ) is the most challenging to handle here , because common activation functions such as tanh and smooth sigmoid are nonlinear . This makes them nonconvex constraints and hence it is difficult to obtain a global minimum when updating zl ( Taylor et al . ( 2016 ) ) . To deal with this challenge , we innovatively transform the original nonconvex constraints into convex inequality constraints , which can be infinitely approximate to Problem 1 . To do this , we introduce a tolerance εl > 0 and reformulate Problem 1 to reach the following form : minWl , bl , zl , alR ( zL ; y ) + ∑L l=1 Ωl ( Wl ) + ∑L−1 l=1 I ( hl ( zl ) −εl≤al≤hl ( zl ) +εl ) s.t.zl = Wlal−1 + bl ( l=1 , · · · , L ) I ( hl ( zl ) − εl ≤ al ≤ hl ( zl ) + εl ) is an indicator function such that the value is 0 if hl ( zl ) − εl ≤ al ≤ hl ( zl ) + εl and∞ otherwise . For the linear constraint zl= Wlal−1+bl , this can be transformed into a penalty term in the objective function to minimize the difference between zl and Wlal−1+bl . The formulation is shown as follows : Problem 2 : minWl , bl , zl , al F ( W , b , z , a ) = R ( zL ; y ) + ∑L l=1 Ωl ( Wl ) + ∑L l=1 φ ( al−1 , Wl , bl , zl ) + ∑L−1 l=1 I ( hl ( zl ) −εl≤al≤hl ( zl ) +εl ) The penalty term is defined as φ ( al−1 , Wl , bl , zl ) = ( ρ/2 ) ‖zl−Wlal−1−bl‖22 , where ρ > 0 a penalty parameter . W = { Wl } Ll=1 , b = { bl } Ll=1 , z = { zl } Ll=1 , a = { al } L−1 l=1 . The reason for introducing εl is that it allows us to project the nonlinear constraint to a convex εl-ball , thus transforming the nonconvex Problem 1 into the multi-convex Problem 2 , which is much easier to solve . Here , a multi-convex problem means this problem is convex with regard to one variable while fixing others . For example , Problem 2 is convex with regard to z when W , b , and a are fixed . As ρ → ∞ and εl → 0 , Problem 2 approaches Problem 1 .

Summary: This paper proposes Deep Learning Alternating Minimization (DLAM) algorithm. First, deep learning optimization problems are formulated as multi-convex Problem 2, by introducing additional variables, constraints and relaxations. Second, alternating update is then used to solve Problem 2, the analysis shows that weights of update converge to a critical point with O(1/k) rate. Finally, some experiments are conducted on MNIST and Fashion-MNIST to show that DLAM is better than SGD, Adagrad, Adadelta and ADMM.