Factor Normalization for Deep Neural Network Models

1 INTRODUCTION . In recent decades , the progress of deep learning , together with advances in GPU devices , has led to a growing popularity of deep neural network ( DNN ) models in both academia and industry . DNN models have been widely used in various fields , such as image classification ( Simonyan & Zisserman , 2014 ; He et al. , 2016a ) , speech recognition ( Hinton et al. , 2012 ; Maas et al. , 2017 ) , and machine translation ( Wu et al. , 2016 ; Vaswani et al. , 2017 ) . However , due to their deep structure , most DNN models are extremely difficult to train . The practical training of a DNN model often highly depends on empirical experience and is extremely time consuming . Therefore , a series of effective optimization methods have been developed for fast DNN training . According to a recent survey paper by Sun et al . ( 2019 ) , most of the optimization methods with explicit derivatives can be roughly categorized into two groups : the first-order optimization methods and the high-order optimization methods . The widely used stochastic gradient descent ( SGD ) algorithm and its variants ( Robbins & Monro , 1951 ; Jain et al. , 2018 ) are typical examples of the first-order optimization methods . The SGD algorithm only computes the first-order derivatives ( i.e. , the gradient ) using a randomly sampled batch . By doing so , the SGD algorithm can handle large-sized datasets with limited computational resources . Unfortunately , the practical feasibility of SGD comes at the cost of sublinear convergence speed ( Johnson & Zhang , 2013a ) . For better convergence speed , various accelerated SGD algorithms have been developed . For instance , the popularly used momentum method ( Polyak , 1964 ; Qian , 1999 ) and the Nesterov Accelerated Gradient Descent ( NAG ) ( Nesterov , 1983 ; Sutskever et al. , 2013 ) method . Both of them took the information from the previous update gradient direction into consideration . Further improvements include AdaGrad ( Duchi et al. , 2011 ) , AdaDelta ( Zeiler , 2012 ) , RMSprop ( Tieleman & Hinton , 2012 ) , Adam ( Kingma & Ba , 2014 ) and others . For a more stable gradient estimation , the stochastic average gradient ( SAG ) ( Roux et al. , 2012 ) and stochastic variance reduction gradient ( SVRG ) ( Johnson & Zhang , 2013b ) methods are also developed . Except for the first-order optimization methods , high-order optimization methods also exist . Popular representatives are the Newton ’ s method and its variants ( Shanno , 1970 ; Hu et al. , 2019 ; Pajarinen et al. , 2019 ) . Compared to the first-order methods , high-order methods might lead to faster convergence speed since they take the information of Hessian matrix into consideration . For example , the Newton ’ s method can have a quadratic convergence speed under appropriate conditions ( Avriel , 2003 ) . However , calculating and storing the Hessian matrix and its inverse is extremely expensive in terms of both time and storage . This leads to the development of some approximation methods , such as Quasi-Newton method ( Avriel , 2003 ) and stochastic Quasi-Newton method ( Luo et al. , 2014 ) . The idea of Quasi or stochastic Quasi Newton method is to approximate the inverse Hessian matrix by a positive definite matrix . For example , DFP ( Fletcher & Powell , 1963 ; Davidon , 1991 ) , BFGS ( Broyden , 1970 ; Fletcher & R , 1970 ; Donald & Goldfarb , 1970 ) and L-BFGS ( Nocedal & Jorge , 1980 ; Liu & Nocedal , 1989 ) methods are popular representatives . Moreover , as a useful technique for fast convergence , various pre-conditioning techniques are also popularly used . ( Huckle , 1999 ; Benzi , 2002 ; Tang et al. , 2009 ) . The basis idea of pre-conditioning is to transform a difficult or ill-conditioned linear system ( e.g. , Aθ = b ) into an easier system with better condition ( Higham & Mary , 2019 ) . As a consequence , the information contained in the feature covariance can be effectively used ( Wang et al. , 2019 ) . Other interesting methods trying to extract useful information from the feature covariance also exist ; see for example Denton et al . ( 2015 ) , Ghiasi & Fowlkes ( 2016 ) , and Lai et al . ( 2017 ) . However , to our best knowledge , there seems no existing models or methods that are particularly designed for high-dimensional features with a factor structure . In the meanwhile , ample amounts of empirical experience suggest that most high-dimensional features demonstrate a strong factor type of covariance structure . In other words , a significant amount of the feature variability can be explained by a latent factor with very low dimensions . As a consequence , we can decompose the original features into two parts . The first is a low dimensional factor part , which accounts for a significant portion of the total volatility . The second is the residual part with factor effects removed . This residual part has the same dimension as the original feature . Consequently , it has a much reduced variability . Moreover , the inter-feature correlation is also reduced substantially . To this end , the original learning problem concerning for the high dimensional features can be decomposed into two sub learning problems . The first one is a learning problem concerning for the latent factor . This is relatively simple since the dimension of the factor is very low . The second problem is related to the residual feature . Unfortunately , this is still a challenging problem due to the high dimensions . However , compared with the original one , it is much easier because the inter feature dependence has been substantially reduced . For a practical implementation , we propose here a novel method called factor normalization . It starts with a benchmark model ( e.g. , VGG or ResNet ) and then slightly modifies the benchmark model into a new model structure . Compared with the benchmark model , the new model takes the latent factor and residuals as different inputs . The benchmark model is remained to process the residuals . The latent factor is then put back to the model in the last layer . This is to compensate for the information loss due to factor extraction . By doing so , the new model allows the factor-related features and residual-related features to be processed separately . Furthermore , different ( i.e. , adaptive ) learning rates can be allowed for factor and residuals , respectively . This leads to adaptive learning and thus fast convergence speed . The rest of this article is organized as follows . Section 2 develops our theoretical motivation with statistical insights . Section 3 provides the details of the proposed new model . Section 4 demonstrates the outstanding performance of the propose model via extensive empirical experiments . Section 5 concludes the article with a brief discussion for future research . 2 THEORETICAL MOTIVATION . To motivate our new model , we provide here a number of interesting theoretical motivations from different perspectives . Since the SGD algorithm is a stochastic version of the GD algorithm , we thus focus on a standard GD algorithm in this section for discussion simplicity . 2.1 THE GD ALGORITHM . Let ( Xi , Yi ) be the observation collected from the i-th instance with 1 ≤ i ≤ N , where Yi is often the class label and Xi = ( Xi1 , ... , Xip ) ⊤ ∈ Rp is the associated p-dimensional feature . The loss function evaluated at i is defined as ℓ ( Yi , X⊤i θ ) , where θ ∈ Rp is the unknown parameter . Then , the global loss is given by LN ( θ ) = N−1 ∑N i=1 ℓ ( Yi , X ⊤ i θ ) . The global gradient is given by L̇N ( θ ) = N−1 ∑N i=1 ℓ̇ ( Yi , X ⊤ i θ ) Xi and ℓ̇ ( y , z ) = ∂ℓ ( y , z ) /∂z . Let θ̂ ( t ) be the estimator obtained in the t-th iteration . Then , the GD algorithm updates the parameter as θ̂ ( t+1 ) = θ̂ ( t ) − αL̇N ( θ̂ ( t ) ) . Here , α is a scalar and is referred as the learning rate ( Robbins & Monro , 1951 ) . Assume that LN ( θ̂ ( t ) ) reaches its minimum at θ̂ such that L̇N ( θ̂ ) = 0 . We then apply Taylor expansion for L̇N ( θ ) at θ̂ . This leads to θ̂ ( t+1 ) − θ̂ = { Ip − αL̈N ( θ̂ ) } ( θ̂ ( t ) − θ̂ ) + o ( ∥θ̂ ( t ) − θ̂∥2 ) = K ( θ̂ ( t ) − θ̂ ) + o ( ∥θ̂ ( t ) − θ̂∥2 ) , where K = Ip−αL̈N ( θ̂ ) . We refer to K as a contraction operator , and it plays a very important role in optimization . Intuitively , all the eigenvalues of K should lie in ( −1 , 1 ) . Otherwise , the algorithm might not converge numerically . Proposition 1 Assume L̈N ( θ̂ ) to be a positive definite matrix . Let λ1 ≥ λ2 ≥ ... ≥ λp > 0 be the eigenvalues of L̈N ( θ̂ ) . To have the GD algorithm converge , we should have 0 < α < 1/λ1 . 2.2 CONDITION NUMBER . By Proposition 1 we know that the learning rate can not be too large . Otherwise the GD algorithm might not numerically converge . The size of the learning rate is controlled by the largest eigenvalue of the Hessian matrix L̈N ( θ̂ ) . The larger λ1 is , the smaller the learning rate must be , and the slower convergence speed should be . This problem is particularly serious if the condition number ( i.e. , λ1/λp ) of the Hessian matrix is very large . In that case , the large λ1 value forces the learning rate α to be very small . In the meanwhile , other small eigenvalues ( λj for j ̸= 1 ) make the convergence speed along the corresponding eigen-directions very slow . Consequently , practitioners should wish the condition number of the Hessian matrix to be as small as possible . On the other hand , as we mentioned previously , most high-dimensional features have a strong factor structure . In other words , the size of the top eigenvalues of the covariance matrix Σ is typically much larger than the rest . Consequently , the condition number of its covariance matrix is typically very large . It is then of great interest to investigate : how would this factor structure affect the condition number of the Hessian matrix ? To address this important problem , we evaluate the expected Hessian matrix as H = E ( L̈ ( θ ) ) = E { ℓ̈ ( Yi , X⊤i θ ) XiX⊤i } , where ℓ̈ ( y , z ) = ∂ℓ̇ ( y , z ) /∂z stands for the second order derivative of ℓ ( y , z ) with respect to z . For illustration purpose , assume Xi is normally distributed with mean 0 . Recall that the covariance matrix is Σ . Define X̃i = Σ−1/2Xi and θ̃ = Σ1/2θ . We then rewrite H as H = Σ1/2E { ℓ̈ ( Yi , X̃⊤i θ̃ ) X̃iX̃⊤i } Σ1/2 = Σ1/2H̃Σ1/2 , where H̃ = E { ℓ̈ ( Yi , X̃⊤i θ̃ ) X̃iX̃⊤i } . Let A be an arbitrary positive definite matrix . Define λmax ( A ) and λmin ( A ) be the maximum and minimum eigenvalues of A , respectively . We then have λmax ( H ) ≥ λmax ( Σ ) λmin ( H̃ ) and λmin ( H ) ≤ λmin ( Σ ) λmax ( H̃ ) . This further suggests that the condition number of H ( i.e. , con ( H ) = λmax ( H ) /λmin ( H ) ) satisfies the following inequality , con ( H ) ≥ con ( Σ ) / con ( H̃ ) , ( 1 ) where con ( A ) stands for the condition number of an arbitrary positive definite matrix . Note that H̃ is the expected Hessian matrix under an extremely ideal situation , where the input feature follows a standard multivariate normal distribution . This is arguable the most ideal situation for numerical optimization . We thus can reasonably expect that con ( H̃ ) should not be very large in this case . Thus , by ( 1 ) we know that con ( Σ ) should play an important role in determining the con ( H ) . In other words , the large condition number of Σ will affect con ( H ) . This makes the numerical optimization by a standard GD algorithm extremely difficult , and thus calls for a novel solution .

Summary: In this paper, a learning method that accelerates the training of DNN is proposed. Given an input X, the proposed method decomposes X as X = BZ + E where BZ is a low-rank approximation of X and E is the residual term. E is used as an input of DNN and Z is used as an additional feature of the input of the last layer. Experiments using MNIST and CIFAR10 show that the proposed method accelerates the speed to reduce the training loss.