An Optimization Principle Of Deep Learning?

1 INTRODUCTION . Deep learning has been successfully applied to various domains such as computer vision , natural language processing , etc , and has achieved state-of-art performance in solving challenging tasks . Although deep neural networks ( DNNs ) have been well-known for decades to have great expressive power Cybenko ( 1989 ) , the empirical success of training DNNs postponed to recent years when sufficient computation power is accessible and effective DNN training techniques are developed . The milestone developments of DNN training techniques can be divided into two categories . First , various techniques have been developed at different levels of neural network design . At the neuron level , various functions have been applied to activate the neurons , e.g. , sigmoid function , hyperbolic tangent ( tanh ) function and the more popular rectified linear unit ( ReLU ) function Nair & Hinton ( 2010 ) . At the layer level , batch normalization ( BN ) has been widely applied to the hidden layers of DNNs to stabilize the training Ioffe & Szegedy ( 2015 ) . Moreover , at the architecture level , skip connections have been introduced to enable successful training of deep networks He et al . ( 2015 ) ; Szegedy et al . ( 2015 ) ; Srivastava et al . ( 2015 ) ; Huang et al . ( 2016 ) . Second , various efficient stochastic optimization algorithms have been developed for DNN training , e.g. , stochastic gradient descent ( SGD ) Robbins & Monro ( 1951 ) ; Rumelhart et al . ( 1986 ) , SGD with momentum Qian ( 1999 ) ; Nesterov ( 2014 ) and Adam Kingma & Ba ( 2015 ) , etc . Table 1 provides a summary of these important DNN training techniques . While these training techniques have been widely applied in practical DNN training , there is limited understanding on how they help facilitate the training and achieve the global minimum of the network . In the existing literature , it is known that the sigmoid and tanh activation functions can cause the vanishing gradient problem , and the ReLU activation function is a popular replacement that avoids this issue Nair & Hinton ( 2010 ) ; Glorot et al . ( 2011 ) . On the other hand , the batch normalization is originally proposed to reduce the internal covariance shift Ioffe & Szegedy ( 2015 ) , and more recent studies show that it allows to use a large learning rate Bjorck et al . ( 2018 ) and improves the loss landscape Santurkar et al . ( 2018 ) . The skip connection has been shown to help eliminate singularities and degeneracies Orhan & Pitkow ( 2018 ) and improve the loss landscape Hardt & Ma ( 2016 ) . Moreover , regarding the optimization algorithm , the momentum scheme has been well-known to accelerate convex optimization Nesterov ( 2014 ) and is also widely applied to accelerate nonconvex optimization Ghadimi & Lan ( 2016 ) , whereas the Adam algorithm normalizes the update in each dimension to accelerate deep learning Kingma & Ba ( 2015 ) . While these studies provide partial explanations to the effectiveness of various DNN training techniques , their reasonings are from very different perspectives and it is unclear whether a general principle can exist that guides these training techniques to facilitate the training process . In particular , these studies do not explain why DNN trainings with different techniques can achieve the global minimum in practice . Moreover , the existing explanations can not quantify the impacts of the training techniques on training deep networks , and a principled quantification metric is still far from clear . This paper attempts to propose an optimization principle that characterizes and quantifies the impacts of various training techniques on DNN training . The proposed principle is applicable to general stochastic algorithms in the nonconvex and over-parameterized regime and leads to guaranteed convergence to a global minimum . We summarize our contributions as follows . 1.1 OUR CONTRIBUTIONS . Theory : We propose an optimization principle that is parameterized by γ > 0 for stochastic algorithms in nonconvex and over-parameterized optimization . The γ-optimization principle ( see Definition 1 ) requires the current update points toward a global minimizer , and the inner product between both directions is lower-bounded by the update norm and the loss gap that is scaled by the factor γ . We show that the γ-optimization principle guarantees the optimization path generated by stochastic algorithms to approach a global minimizer with a monotonically diminishing distance . Moreover , it leads to convergence of the optimization path to the global minimum with a O ( 1/γK ) sub-linear convergence rate . Experiments : We conduct extensive experiments to examine the validity of the γ-optimization principle in DNN training . In specific , we find that all the tested DNN trainings obey the γoptimization principle throughout the training process , and the generated optimization paths satisfy the theoretical properties of the principle that are mentioned above . Moreover , we explore the impacts of the training techniques that are listed in Table 1 on the parameterization γ of the optimization principle . We find that DNN trainings that apply the training techniques achieve fast convergence and obey the optimization principle with a large γ . As a comparison , DNN trainings without the training techniques converge slowly and obey the optimization principle with a small γ . These observations are consistent with the convergence rate characterization of the γ-optimization principle that a larger γ leads to faster convergence . Therefore , the γ-optimization principle captures and quantifies the impacts of these training techniques on DNN training in a unified way through the parameter γ , and sheds light on the underlying mechanism that leads to successful DNN training . 1.2 RELATED WORK . DNN training techniques : Various training techniques have been developed for DNN training . Examples include piece-wise linear activation functions , e.g. , ReLU Nair & Hinton ( 2010 ) , ELU Clevert et al . ( 2015 ) , leaky ReLU Maas et al . ( 2013 ) , batch normalization Ioffe & Szegedy ( 2015 ) , skip connection He et al . ( 2015 ) ; Szegedy et al . ( 2015 ) ; Srivastava et al . ( 2015 ) and advanced optimizers such as SGD with momentum Qian ( 1999 ) , Adagrad Duchi et al . ( 2011 ) , Adam Kingma & Ba ( 2015 ) , AMSgrad Reddi et al . ( 2018 ) . The ReLU activation function and skip connection have been shown to help avoid the vanishing gradient problem and improve the loss landscape Hardt & Ma ( 2016 ) ; Zhou & Liang ( 2017 ) ; Zhang et al . ( 2019 ) ; Zou et al . ( 2018 ) . The batch normalization has been shown to help avoid the internal covariance shift problem Ioffe & Szegedy ( 2015 ) . More recent studies show that batch normalization allows to adopt a large learning rate Bjorck et al . ( 2018 ) and improves the loss landscape in the training Santurkar et al . ( 2018 ) . The convergence properties of the advanced optimizers have been studied in nonconvex optimization Chen et al . ( 2019 ) . Optimization properties of nonconvex ML : Many nonconvex ML models have amenable properties for accelerating the optimization . For example , nonconvex problems such as phase retrieval Zhang et al . ( 2017b ) , low-rank matrix recovery Tu et al . ( 2016 ) , blind deconvolution Li et al . ( 2018b ) and neural network sensing Zhong et al . ( 2017 ) satisfy the local regularity geometry around the global minimum Zhou et al . ( 2016 ) ; Tu et al . ( 2016 ) ; Li et al . ( 2018b ) ; Zhong et al . ( 2017 ) ; Zhou & Liang ( 2017 ) , which guarantees the linear convergence of gradient-based algorithms . More recently , DNN trainings that use SGD have been shown to follow a star-convex optimization path Zhou et al . ( 2019 ) . 2 AN OPTIMIZATION PRINCIPLE FOR OVER-PARAMETERIZED ML . In this section , we introduce the γ-optimization principle that provides convergence guarantees for over-parameterized nonconvex machine learning ( ML ) . We show empirically in the subsequent sections that practical DNN trainings obey this principle . 2.1 GLOBAL MINIMUM OF OVER-PARAMETERIZED MODELS . The goal of a machine learning task is to search for a good ML model θ that minimizes the total loss f on a set of training data samples Z : { zi } ni=1 . The problem is formally written as min θ∈Rd f ( θ ; Z ) : = 1 n n∑ i=1 ` ( θ ; zi ) , ( P ) where ` ( · ; zi ) : Rd → R corresponds to the loss on the i-th data sample zi . For many nonconvex ML models , e.g. , deep neural networks , a prominent feature is the over-parameterization of the model , i.e. , the model capacity is sufficient to over-fit all the training data samples . In another word , the global minimizers of the total loss f under an over-parameterized model are common minimizers of all the individual loss functions ` ( · ; zi ) , i = 1 , ... , n. We summarize this fact formally as follows . Fact 1 . Consider the problem ( P ) with an over-parameterized model θ . Every global minimizer θ∗ of the total loss f is also a global minimizer of each individual loss ` ( · ; zi ) for all i = 1 , ... , n. To elaborate , note that we typically use non-negative loss in DNN training . Therefore , if the total loss f is trained to achieve zero ( i.e. , the global minimum ) , then each individual loss ` ( · ; zi ) must also achieve zero and the obtained model over-fits the training data . Of course , in practice , due to the termination of the optimization process within finite number of iterations , the total loss can only achieve an approximate global minimum that is very close to zero . Throughout the rest of this section , we assume that Fact 1 holds for the problem ( P ) . 2.2 THE γ-OPTIMIZATION PRINCIPLE Consider a generic stochastic algorithm ( SA ) that is initialized with certain model θ0 . In each iteration k , the SA samples a data sample zξk ∈ Z , where ξk ∈ { 1 , ... , n } is obtained via cyclic sampling with reshuffle . Based on the current model θk and the sampled data zξk , the SA generates a stochastic update U ( θk ; zξk ) and applies it to update the model with a learning rate η > 0 according to ( SA ) : θk+1 = θk − ηU ( θk ; zξk ) , k = 0 , 1 , 2 , ... ( 1 ) Equation ( 1 ) covers the update rule of many existing optimizers for DNN training . For example , the stochastic gradient descent ( SGD ) algorithm chooses the update U ( θk ; zξk ) to be the stochastic gradient ∇θ ` ( θk ; zξk ) . In comparison , the SGD with momentum algorithm generates the update using an extra momentum step , and the Adam algorithm generates the update as a moving average of the stochastic gradients normalized by the moving average of their second moments 1 . We next introduce the γ-optimization principle for the SA . Definition 1 ( γ-optimization principle for SA ) . Apply SA to solve the over-parameterized problem ( P ) . For a certain global minimizer θ∗ of the problem ( P ) , the optimization path { θk } k generated by the SA satisfies : for certain γ > 0 and all k = 0 , 1 , 2 , ... , , 〈θk − θ∗ , U ( θk ; zξk ) 〉 ≥ η 2 ‖U ( θk ; zξk ) ‖2 + γ‖θ0 − θ∗‖2 ( ` ( θk ; zξk ) − ` ( θ∗ ; zξk ) ) . ( 2 ) 1The Adam updates depend on the past stochastic samples . To elaborate , the left hand side of eq . ( 2 ) measures the coherence between θk − θ∗ and the current model update U ( θk ; zξk ) . Intuitively , a positive coherence implies that the current model update points toward the global minimizer θ∗ from the current model θk . On the other hand , the right hand side of eq . ( 2 ) regularizes the coherence by two non-negative terms : the square norm of the model update scaled by the learning rate η and the optimality gap of the loss on the sampled data scaled by the constant γ‖θ0 − θ∗‖2 . Hence , a larger value of γ implies that the update U ( θk ; zξk ) is more coherent with the direction towards the minimizer θk − θ∗ and therefore facilitates the convergence . We show in Theorem 1 later that the γ plays a central role in characterizing the convergence rate of the SA . Discussion : The γ-optimization principle is related to other existing ones in nonconvex ML . In specific , nonconvex problems such as phase retrieval Zhang et al . ( 2017b ) and low-rank matrix recovery Tu et al . ( 2016 ) have been shown to satisfy the regularity condition , which replaces the last term in eq . ( 2 ) by γ‖θk − θ∗‖2 and guarantees the linear convergence of { θk } k to θ∗ . However , such a fast convergence rate does not hold in practical DNN training . On the other hand , DNN trainings have been shown in Zhou et al . ( 2019 ) to follow a star-convex optimization path , which corresponds to eq . ( 2 ) with γ = ‖θ0− θ∗‖−2 , U ( θk ; zξk ) = ∇ ` ( θk ; zξk ) and absence of the term η 2‖U ( θk ; zξk ) ‖ 2 . However , such a principle is applicable to SGD updates only and an additional global Lipschitz assumption on the loss function is required in order to have theoretical convergence guarantee . In comparison , the γ-optimization principle is applicable to the more general SA and does not rely on additional assumption to have convergence guarantee as we elaborate below . We obtain the following convergence results of SA under the γ-optimization principle . Theorem 1 ( Convergence of SA ) . Apply SA to solve the over-parameterized problem ( P ) . If the optimization process satisfies the principle in Definition 1 , then the following statements hold .

Summary: This paper proposes a "gamma principle" for stochastic updates in deep neural network optimization. First, the authors propose if Eq. (2) is satisfied then convergence is guaranteed (Thm. 1). Second, they use experimental results of Alexnet and Resnet-18 on cifar10/100 to show that Eq. (2) is satisfied by SGD, SGD with momentum, and Adam, with different activations and tricks like batch normalization, skip connection.