AN EFFICIENT HOMOTOPY TRAINING ALGORITHM FOR NEURAL NETWORKS

1 INTRODUCTION . Deep learning methods , as the rising star among all machine learning methods in recent years , have already had great success in many applications . Many advancements Cang et al . ( 2018 ) ; Cang & Wei ( 2018 ) ; Yin et al . ( 2018 ; 2016 ) on deep learning have been made in the last few years . However , as the size of new state-of-the-art models keep growing larger and larger , they rely more and more heavily on efficient algorithms for training and making inferences from such models . This clearly puts strong limitations on the application scenarios of nerual netowrks on roboticsKonda et al . ( 2012 ) , auto-pilot automobiles Hammerla et al . ( 2016 ) , and aerial systems Maire et al . ( 2014 ) . At present , there are two big challenges in fundamentally understanding deep neural networks : • How to efficiently solve the highly nonlinear and non-convex optimization problems that arise during the training of a deep learning model ? • How to design a deep neural network structure for specific problems ? In order to solve these challenges , in this paper , we will present a new training algorithm based on homotopy continuation method Bates et al . ( 2011 ; 2008 ) ; Morgan & Sommese ( 1987 ) which has been successfully used to study nonlinear problems such as nonlinear differential equations and optimizations Hao et al . ( 2012 ) ; Hao & Harlim . In order to tackle the nonlinear optimization problem in neural networks , the homotopy training algorithm is designed and shows efficiency and feasibility for both convolutional and fully connected neural networks with complex structures . The idea of the HTA is to start with a simplified optimization problem and construct a continuous path to the original problem . Specifically , let ’ s consider the following general optimization problem min θ f ( θ ) where θ ∈ Rn . ( 1 ) We then divide θ in two groups , namely , θ = θ1∪ θ2 , where θ1 ∈ Rn1 and θ2 ∈ Rn2 ( n1 +n2 = n ) . ( Although that it can be easily extended to any m decompositions , we use m = 2 as an illustration of the HTA idea . ) Then we first solve the following decoupled simplified optimization problems min θ1 f ( θ1 , θ̃2 ) and min θ2 f ( θ̃1 , θ2 ) ( 2 ) where θ̃ = θ̃1 ∪ θ̃2 is a random chosen point in Rn for instance we choose θ̃ = 0 in the dropout Srivastava et al . ( 2014 ) . Second we define the following homotopy function H ( Θ , L ) = f ( θ ) + L‖θ1 − θ̃1‖2 + f ( η ) + L‖η2 − θ̃2‖2 + 1 L ‖θ − η‖2 , ( 3 ) where Θ = ( θ , η ) and L ∈ ( 0 , ∞ ) is the homotopy parameter . When L → ∞ , we have two decoupled optimization problems which we have already solved in the first step ; when L → 0 , we recover the original optimization by forcing θ = η . By gradually increasing parameter L , we are able to construct a continuous path Θ ( L ) between two decoupled low-dimensional optimization problems and the original optimization problem . Remark The HTA can be reviewed in point view of nonlinear equations by considering the necessary condition of the optimization problem ( 1 ) : H ( Θ , L ) = ( ∇θf ( θ ) η − θ ) 1 L + L ∇θ2f ( θ ) θ1 − θ̃1 ∇η1f ( η ) η2 − θ̃2 . When L is large , θ2 and η1 can be solved by ∇θ2f ( θ ) = 0 and ∇η1f ( η ) = 0 respectively . When L is small , we have the solution of ∇θf ( θ ) = 0 by the diagonal homotopy θ − η = 0 . This technique has been widely used in solving systems of nonlinear equations especially for polynomial systems Bates et al . ( 2008 ) ; Wampler & Sommese ( 2005 ) . Since the θ̃ is randomly chosen , the universal approximation theory Park & Sandberg ( 1991 ) can guarantee that the random point θ̃ in Rn is always connected to the solution of ∇θf ( θ ) = 0 . In another word , the solution path Θ ( L ) always exists or the probability that Θ ( L ) is not connected is zero Wampler & Sommese ( 2005 ) . 2 APPLICATIONS OF HTA ON NEURAL NETWORK . 2.1 CONVOLUTIONAL NEURAL NETWORKS . In this subsection , we will specify the variable θ and the objective function f ( θ ) of ( 3 ) in the convolutional neural network ( CNN ) setup . Definition 2.1 . Suppose we have two tensors , Al×m×n = ( ai , j , k ) 1≤i≤l,1≤j≤m,1≤k≤n , Bc×l×m′×n′ = ( bh , i , j , k ) 1≤h≤c,1≤i≤l,1≤j≤m′,1≤k≤n′ , where c , m , n , m′ , n′ , l ∈ N and m′ ≤ m , n′ ≤ n. The convolution of two tensors is defined by the following : A ∗Bh , j , k = l∑ i′=1 m′∑ j′=1 n′∑ k′=1 ai′ , j′+j−1 , k′+k−1 · bh , i′ , j′ , k′ , ( 4 ) where B is called the convolutional kernel . Then with the input x and the output y , the CNN is written as y ( x ; θ ) = σ ( σ ( · · ·σ ( σ ( x ∗W1 + b1 ) ∗W2 + b2 ) · · ·+ bn−1 ) ∗Wn + bn ) ( 5 ) where Wi ∈ Rci , li , mi , ni and σ is the activation function . Then we have θ = { Wi , bi } ni=1 which can be easily divided into two subgroups θj = { Wij , bi } ni=1 where Wij ∈ R ci 2 , li , mi , ni . The objective function f ( θ ) in ( 3 ) is defined by the cross entropy loss function Goodfellow et al . ( 2016 ) and defined as L ( x , y ) = −x ( y ) + log ( ∑ j e x [ j ] ) . Due to the random algorithms in machine learning Bottou ( 2010 ) ; Kingma & Ba ( 2014 ) , the homotopy function in ( 3 ) can be rewritten as H ( Θ ; ξ , L ) where ξ is a random variable due to random algorithms . 2.2 FULLY CONNECTED NEURAL NETWORKS . Similar to the CNN , a fully connected neural network with n hidden layers is written as y ( x , θ ) = σ ( Wnσ ( Wn−1σ ( W2σ ( W1x+ b1 ) + b2 ) · · ·+ bn−1 ) + bn ) , ( 6 ) where Wi ∈ Rn×di , βi ∈ Rdi , n is the dimension of the input x and di is the width of the i-th layer . Then we have θ = { Wi , bi } ni=1 which can be easily divided into two subgroups θ1 = { Wi , bi } n/2 i=1 and θ2 = { Wi , bi } ni=n/2+1 . Or we can use the dropout technique to randomly split θ as θ1 and θ2 . Similarly , the homotopy function in ( 3 ) can be also rewritten as H ( Θ ; ξ , L ) due to the random algorithms Bottou ( 2010 ) ; Kingma & Ba ( 2014 ) . 3 THEORETICAL ANALYSIS . In this section , we will provide a theoretical analysis of the HTA by using stochastic gradient scheme to solve ( 3 ) for each L. We redefine our stochastic gradient scheme for the homotopy process as θk+1 = θk − γkG ( θk ; ξk , Lk ) , ( 7 ) where G ( θ ; ξ , L ) : = ∇θF ( θ ; ξ , L ) , γk is the learning rate , L0 = M , Lk ↗ 0 and M is a given large number . Then we consider the expectation of the homotopy function as Eξ ( H ( θ ; ξ , L ) ) , for simplicity , and denote Eξ [ H ( θ ; ξ , L ) ] as g ( θ ; L ) . 3.1 CONVERGENCE OF THE HTA FOR ANY GIVEN L IN THE NON-CONVEX CASE . Theorem 3.1 . ( Nonconvex Convergence ) If { θk } is contained in a bounded open set , supposing that ( 7 ) is run with a step size sequence satisfying ∞∑ k=1 γk =∞ and ∞∑ k=1 γ2k < ∞ , ( 8 ) then we have E [ 1 Ak K∑ k=1 γk‖∇g ( θk ; L ) ‖22 ] → 0 as K →∞ with Ak : = K∑ k=1 γk . ( 9 ) Proof . First , we prove the Lipschitz-continuous objective gradients condition Bottou et al . ( 2018 ) , which means that g ( θ ; L ) is C1 and ∇g ( θ ; L ) is Lipschitz continuous : • g ( θ ; L ) is C1 . Since y ( xξ ; θ ) ∈ C1 and L ( · , y ) is C1 , we have that H ( θ ; ξ , L ) ∈ C1 or that ∇θH ( θ ; ξ , L ) is continuous . Considering ∇θg ( θ ; L ) = ∇θEξ ( H ( θ ; ξ , L ) ) = Eξ ( ∇θH ( θ ; ξ , L ) ) , ( 10 ) we have that∇θg ( θ , L ) is continuous or that g ( θ ; L ) ∈ C1 . • ∇g ( θ ; L ) is Lipschitz continuous . Since ∇θg ( θ ; L ) = Eξ ( ∇θH ( θ ; ξ , L ) ) , we only need to prove Eξ ( ∇θL ( y ( xξ ; θ ) , yξ ) ) is Lipschitz continuous . In fact ∇θL ( y ( xξ ; θ ) , yξ ) = ∇xL ( y ( xξ ; θ ) , yξ ) · ∇θy ( xξ ; θ ) ( 11 ) we will show that ∇xL ( y ( xξ ; θ ) , yξ ) and ∇θy ( xξ ; θ ) are bounded and Lipschitz continuous . Since that both σ ( x ) = 11+e−x and σ ′ ( x ) = σ ( x ) ( 1−σ ( x ) ) are bounded and Lipschitz continuous , and that { θk } is in an open set ( assumption of Theorem 3.1 ) , ∇θy ( xξ ; θ ) is bounded and Lipschitz continuous . ( xξ is bounded because the size of our dataset is finite . ) By differentiating L ( x , y ) , we have ∇xL ( x , y ) = ( −δ1y + ex1∑ j e xj , · · · , −δny + exn∑ j e xj ) , ( 12 ) where δ is the Kronecker delta . since ∂ ∂xk exi∑ j e xj = exi ∑ j e xj− ( exi ) 2 ( ∑ j e xj ) 2 k = i −exiexk ( ∑ j e xj ) 2 k 6= i , ( 13 ) which implies that ∣∣ ∂ ∂xk exi∑ j e xj ∣∣ ≤ 2 , we have ∇xL ( · , y ) is Lipschitz continuous and bounded . Therefore ∇xL ( y ( xξ ; θ ) , yξ ) is Lipschitz continuous and bounded . Therefore we have that ∇g ( θ ; L ) is Lipschitz continuous . Second , we prove the first and second moment limits condition Bottou et al . ( 2018 ) : a . According to our theorem assumption , { θk } is contained in an open set which is bounded . Since that g is continuous , g is bounded ; b . Since that G ( θk ; ξk , L ) = ∇θF ( θk ; ξk , L ) is continuous , we have Eξk [ G ( θk ; ξk , L ) ] = ∇θEξk [ H ( θk ; ξk , L ) ] = ∇θg ( θk ; L ) . ( 14 ) Therefore ∇g ( θk ; L ) TEξk [ G ( θk ; ξk , L ) ] = ∇g ( θk ; L ) T ·∇g ( θk ; L ) = ‖∇g ( θk ; L ) ‖22 ≥ u‖∇g ( θk ; L ) ‖22 ( 15 ) for 0 < u ≤ 1 . On the other hand , we have ‖Eξk [ G ( θk ; ξk , L ) ] ‖2 = ‖∇g ( θk ; L ) ‖2 ≤ uG‖∇g ( θk ; L ) ‖22 ( 16 ) for uG ≥ 1. c. Since that∇H ( θk ; ξk , L ) is bounded for a given L , we have Eξk [ ‖∇H ( θk ; ξk , L ) ‖22 ] is also bounded . Thus Vξk [ G ( θk ; ξk , L ) ] : = Eξk [ ‖G ( θk ; ξk , L ) ‖22 ] −‖Eξk [ G ( θk ; ξk , L ) ] ‖22 ≤ Eξk [ ‖G ( θk ; ξk , L ) ‖22 ] , ( 17 ) which implies that Vξk [ G ( θk ; ξk , L ) ] is bounded . We have checked assumptions 4.1 and 4.3 in Bottou et al . ( 2018 ) . By theorem 4.10 in Bottou et al . ( 2018 ) , with the diminishing stepsize , namely , ∞∑ k=1 γk =∞ and ∞∑ k=1 γ2k < ∞ , ( 18 ) the following convergence is obtained E [ 1 Ak K∑ k=1 γk‖∇g ( θk ; L ) ‖22 ] → 0 as K →∞ . ( 19 ) 3.2 EXISTENCE OF SOLUTION PATH θ ( L ) FOR THE CONVEX CASE Second we explore the existence of solution path θ ( L ) when t varies fromM to 0 for the convex case theoretically . This is quite difficult for the non-convex case although we may find it numerically . However , theoretical analysis of the convex case would also give us some guidance on how the HTA obtains an optimal solution for L = 0 . Theorem 3.2 . Assume that g ( · , · ) is convex and differentiable and that ‖G ( θ ; ξ , L ) ‖ ≤ M0 . Then for stochastic gradient scheme ( 7 ) , we have lim n→∞ E [ g ( θ̄n , L̄n ) ] = g ( θ1∗ , 0 ) , ( 20 ) where θ̄n = ∑n k=0 γkθk∑n k=0 γk and L̄n = ∑n k=0 γkLk∑n k=0 γk . Proof . E [ ‖θk+1 − θ Lk+1 ∗ ‖2 ] = E [ ‖θk − γkG ( θk ; ξk , Lk ) − θLk∗ ‖2 ] −2E [ 〈θk − γkG ( θk ; ξk , Lk ) − θLk∗ , θ Lk+1 ∗ − θLk∗ 〉 ] +E [ ‖θLk+1∗ − θLk∗ ‖2 ] . ( 21 ) By defining Ak = −2E [ 〈θk − γkG ( θk ; ξk , Lk ) − θLk∗ , θ Lk+1 ∗ − θLk∗ 〉 ] + E [ ‖θ Lk+1 ∗ − θLk∗ ‖2 ] , ( 22 ) and assuming that ∑m k=0Ak ≤ A < ∞ , we obtain E [ ‖θk+1 − θ Lk+1 ∗ ‖2 ] = E [ ‖θk − γkG ( θk ; ξk , Lk ) − θLk∗ ‖2 ] +Ak = E [ ‖θk − θLk∗ ‖2 ] − 2γkE [ 〈G ( θk ; ξk , Lk ) , θk − θLk∗ 〉 ] + γ2kE [ ‖G ( θk ; ξk , Lk ) ‖2 ] +Ak ≤ E [ ‖θk − θLk∗ ‖2 ] − 2γkE [ 〈G ( θk ; ξk , Lk ) , θk − θLk∗ 〉 ] + γ2kM2 +Ak Since that E [ 〈G ( θk ; ξk , Lk ) , θk − θLk∗ 〉 ] = Eξ0 , ··· , ξk−1 [ Eξk [ 〈G ( θk ; ξk , Lk ) , θk − θLk∗ 〉|ξ0 , · · · , ξk−1 ] ] = Eξ0 , ··· , ξk−1 [ 〈∇g ( θk ; Lk ) , θk − θLk∗ 〉|ξ0 , · · · , ξk−1 ] = E [ 〈∇g ( θk ; Lk ) , θk − θLk∗ 〉 ] , ( 23 ) we have E [ ‖θk+1 − θ Lk+1 ∗ ‖2 ] ≤ E [ ‖θk − θLk∗ ‖2 ] − 2γkE [ 〈∇g ( θk ; Lk ) , θk − θLk∗ 〉 ] + γ2kM2 +Ak . ( 24 ) Due to the convexity of g ( · , Lk ) , namely , 〈∇g ( θk , Lk ) , θk − θLk∗ 〉 ≥ g ( θk ; Lk ) − g ( θLk∗ ; Lk ) , ( 25 ) we conclude that E [ ‖θk+1 − θ Lk+1 ∗ ‖2 ] ≤ E [ ‖θk − θLk∗ ‖2 ] − 2γkE [ g ( θk ; Lk ) − g ( θLk∗ , Lk ) ] + γ2kM2 +Ak , ( 26 ) or 2γkE [ g ( θk ; Lk ) − g ( θLk∗ ; Lk ) ] ≤ −E [ ‖θk+1 − θ Lk+1 ∗ ‖2 − ‖θk − θLk∗ ‖2 ] + γ2kM2 +Ak . By summing up k from 0 to n , 2 n∑ k=0 γkE [ g ( θk ; Lk ) − g ( θLk∗ ; Lk ) ] ≤ −E [ ‖θn+1 − θ Ln+1 ∗ ‖2 − ‖θ0 − θ0∗‖2 ] +M2 n∑ k=0 γ2k + n∑ k=0 Ak ≤ D2 +M2 n∑ k=0 γ2k + n∑ k=0 Ak , ( 27 ) where D = ‖θ0 − θ0∗‖ . Dividing 2 ∑n k=0 γk on both sides , we have 1∑n k=0 γk n∑ k=0 γkE [ g ( θk ; Lk ) − g ( θLk∗ ; Lk ) ] ≤ D2 +M2 ∑n k=0 γ 2 k + ∑n k=0Ak 2 ∑n k=0 γk ≤ D2 +M2 ∑n k=0 γ 2 k +A 2 ∑n k=0 γk . According to the convexity of g ( · ; · ) and Jensen ’ s inequality Jensen ( 1906 ) , 1∑n k=0 γk n∑ k=0 γkE [ g ( θk ; Lk ) ] ≤ E [ g ( θ̄n ; L̄n ) ] , ( 28 ) where θ̄n = ∑n k=0 γkθk∑n k=0 γk and L̄n = ∑n k=0 γkLk∑n k=0 γk . Then we have E [ g ( θ̄n ; L̄n ) ] − ∑n k=0 γkg ( θ Lk ∗ ; Lk ) ∑n k=0 γk ≤ D2 +M2 ∑n k=0 γ 2 k +A 2 ∑n k=0 γk , ( 29 ) We choose γk such that ∑n k=0 γk = ∞ and ∑n k=0 γ 2 k < ∞ , for example , γk = 1k . Taking n to infinite , we have 0 ≤ lim n→∞ E [ g ( θ̄n , t̄n ) ] − lim n→∞ ∑n k=0 γkg ( θ Lk ∗ , Lk ) ∑n k=0 γk ≤ 0 . ( 30 ) Because Lk ↗ 0 , g ( · , · ) is continuous and ∑n k=0 γk =∞ , then lim n→∞ ∑n k=0 γkg ( θ Lk ∗ , Lk ) ∑n k=0 γk = g ( θ0∗ , 0 ) , ( 31 ) which implies that lim n→∞ E [ g ( θ̄n , L̄n ) ] = g ( θ0∗ , 0 ) . ( 32 )

Summary: In this paper, the authors propose the Homotopy Training Algorithm (HTA) for neural network optimization problems. They claim that HTA starts with several simplified problems and tracks the solution to the original problem via a continuous homotopy path. They give the theoretical analysis and conduct experiments on the synthetic data and the CIFAR-10 dataset.