Factor Normalization for Deep Neural Network Models

1 INTRODUCTION . In recent decades , the progress of deep learning , together with advances in GPU devices , has led to a growing popularity of deep neural network ( DNN ) models in both academia and industry . DNN models have been widely used in various fields , such as image classification ( Simonyan & Zisserman , 2014 ; He et al. , 2016a ) , speech recognition ( Hinton et al. , 2012 ; Maas et al. , 2017 ) , and machine translation ( Wu et al. , 2016 ; Vaswani et al. , 2017 ) . However , due to their deep structure , most DNN models are extremely difficult to train . The practical training of a DNN model often highly depends on empirical experience and is extremely time consuming . Therefore , a series of effective optimization methods have been developed for fast DNN training . According to a recent survey paper by Sun et al . ( 2019 ) , most of the optimization methods with explicit derivatives can be roughly categorized into two groups : the first-order optimization methods and the high-order optimization methods . The widely used stochastic gradient descent ( SGD ) algorithm and its variants ( Robbins & Monro , 1951 ; Jain et al. , 2018 ) are typical examples of the first-order optimization methods . The SGD algorithm only computes the first-order derivatives ( i.e. , the gradient ) using a randomly sampled batch . By doing so , the SGD algorithm can handle large-sized datasets with limited computational resources . Unfortunately , the practical feasibility of SGD comes at the cost of sublinear convergence speed ( Johnson & Zhang , 2013a ) . For better convergence speed , various accelerated SGD algorithms have been developed . For instance , the popularly used momentum method ( Polyak , 1964 ; Qian , 1999 ) and the Nesterov Accelerated Gradient Descent ( NAG ) ( Nesterov , 1983 ; Sutskever et al. , 2013 ) method . Both of them took the information from the previous update gradient direction into consideration . Further improvements include AdaGrad ( Duchi et al. , 2011 ) , AdaDelta ( Zeiler , 2012 ) , RMSprop ( Tieleman & Hinton , 2012 ) , Adam ( Kingma & Ba , 2014 ) and others . For a more stable gradient estimation , the stochastic average gradient ( SAG ) ( Roux et al. , 2012 ) and stochastic variance reduction gradient ( SVRG ) ( Johnson & Zhang , 2013b ) methods are also developed . Except for the first-order optimization methods , high-order optimization methods also exist . Popular representatives are the Newton ’ s method and its variants ( Shanno , 1970 ; Hu et al. , 2019 ; Pajarinen et al. , 2019 ) . Compared to the first-order methods , high-order methods might lead to faster convergence speed since they take the information of Hessian matrix into consideration . For example , the Newton ’ s method can have a quadratic convergence speed under appropriate conditions ( Avriel , 2003 ) . However , calculating and storing the Hessian matrix and its inverse is extremely expensive in terms of both time and storage . This leads to the development of some approximation methods , such as Quasi-Newton method ( Avriel , 2003 ) and stochastic Quasi-Newton method ( Luo et al. , 2014 ) . The idea of Quasi or stochastic Quasi Newton method is to approximate the inverse Hessian matrix by a positive definite matrix . For example , DFP ( Fletcher & Powell , 1963 ; Davidon , 1991 ) , BFGS ( Broyden , 1970 ; Fletcher & R , 1970 ; Donald & Goldfarb , 1970 ) and L-BFGS ( Nocedal & Jorge , 1980 ; Liu & Nocedal , 1989 ) methods are popular representatives . Moreover , as a useful technique for fast convergence , various pre-conditioning techniques are also popularly used . ( Huckle , 1999 ; Benzi , 2002 ; Tang et al. , 2009 ) . The basis idea of pre-conditioning is to transform a difficult or ill-conditioned linear system ( e.g. , Aθ = b ) into an easier system with better condition ( Higham & Mary , 2019 ) . As a consequence , the information contained in the feature covariance can be effectively used ( Wang et al. , 2019 ) . Other interesting methods trying to extract useful information from the feature covariance also exist ; see for example Denton et al . ( 2015 ) , Ghiasi & Fowlkes ( 2016 ) , and Lai et al . ( 2017 ) . However , to our best knowledge , there seems no existing models or methods that are particularly designed for high-dimensional features with a factor structure . In the meanwhile , ample amounts of empirical experience suggest that most high-dimensional features demonstrate a strong factor type of covariance structure . In other words , a significant amount of the feature variability can be explained by a latent factor with very low dimensions . As a consequence , we can decompose the original features into two parts . The first is a low dimensional factor part , which accounts for a significant portion of the total volatility . The second is the residual part with factor effects removed . This residual part has the same dimension as the original feature . Consequently , it has a much reduced variability . Moreover , the inter-feature correlation is also reduced substantially . To this end , the original learning problem concerning for the high dimensional features can be decomposed into two sub learning problems . The first one is a learning problem concerning for the latent factor . This is relatively simple since the dimension of the factor is very low . The second problem is related to the residual feature . Unfortunately , this is still a challenging problem due to the high dimensions . However , compared with the original one , it is much easier because the inter feature dependence has been substantially reduced . For a practical implementation , we propose here a novel method called factor normalization . It starts with a benchmark model ( e.g. , VGG or ResNet ) and then slightly modifies the benchmark model into a new model structure . Compared with the benchmark model , the new model takes the latent factor and residuals as different inputs . The benchmark model is remained to process the residuals . The latent factor is then put back to the model in the last layer . This is to compensate for the information loss due to factor extraction . By doing so , the new model allows the factor-related features and residual-related features to be processed separately . Furthermore , different ( i.e. , adaptive ) learning rates can be allowed for factor and residuals , respectively . This leads to adaptive learning and thus fast convergence speed . The rest of this article is organized as follows . Section 2 develops our theoretical motivation with statistical insights . Section 3 provides the details of the proposed new model . Section 4 demonstrates the outstanding performance of the propose model via extensive empirical experiments . Section 5 concludes the article with a brief discussion for future research . 2 THEORETICAL MOTIVATION . To motivate our new model , we provide here a number of interesting theoretical motivations from different perspectives . Since the SGD algorithm is a stochastic version of the GD algorithm , we thus focus on a standard GD algorithm in this section for discussion simplicity . 2.1 THE GD ALGORITHM . Let ( Xi , Yi ) be the observation collected from the i-th instance with 1 ≤ i ≤ N , where Yi is often the class label and Xi = ( Xi1 , ... , Xip ) ⊤ ∈ Rp is the associated p-dimensional feature . The loss function evaluated at i is defined as ℓ ( Yi , X⊤i θ ) , where θ ∈ Rp is the unknown parameter . Then , the global loss is given by LN ( θ ) = N−1 ∑N i=1 ℓ ( Yi , X ⊤ i θ ) . The global gradient is given by L̇N ( θ ) = N−1 ∑N i=1 ℓ̇ ( Yi , X ⊤ i θ ) Xi and ℓ̇ ( y , z ) = ∂ℓ ( y , z ) /∂z . Let θ̂ ( t ) be the estimator obtained in the t-th iteration . Then , the GD algorithm updates the parameter as θ̂ ( t+1 ) = θ̂ ( t ) − αL̇N ( θ̂ ( t ) ) . Here , α is a scalar and is referred as the learning rate ( Robbins & Monro , 1951 ) . Assume that LN ( θ̂ ( t ) ) reaches its minimum at θ̂ such that L̇N ( θ̂ ) = 0 . We then apply Taylor expansion for L̇N ( θ ) at θ̂ . This leads to θ̂ ( t+1 ) − θ̂ = { Ip − αL̈N ( θ̂ ) } ( θ̂ ( t ) − θ̂ ) + o ( ∥θ̂ ( t ) − θ̂∥2 ) = K ( θ̂ ( t ) − θ̂ ) + o ( ∥θ̂ ( t ) − θ̂∥2 ) , where K = Ip−αL̈N ( θ̂ ) . We refer to K as a contraction operator , and it plays a very important role in optimization . Intuitively , all the eigenvalues of K should lie in ( −1 , 1 ) . Otherwise , the algorithm might not converge numerically . Proposition 1 Assume L̈N ( θ̂ ) to be a positive definite matrix . Let λ1 ≥ λ2 ≥ ... ≥ λp > 0 be the eigenvalues of L̈N ( θ̂ ) . To have the GD algorithm converge , we should have 0 < α < 1/λ1 . 2.2 CONDITION NUMBER . By Proposition 1 we know that the learning rate can not be too large . Otherwise the GD algorithm might not numerically converge . The size of the learning rate is controlled by the largest eigenvalue of the Hessian matrix L̈N ( θ̂ ) . The larger λ1 is , the smaller the learning rate must be , and the slower convergence speed should be . This problem is particularly serious if the condition number ( i.e. , λ1/λp ) of the Hessian matrix is very large . In that case , the large λ1 value forces the learning rate α to be very small . In the meanwhile , other small eigenvalues ( λj for j ̸= 1 ) make the convergence speed along the corresponding eigen-directions very slow . Consequently , practitioners should wish the condition number of the Hessian matrix to be as small as possible . On the other hand , as we mentioned previously , most high-dimensional features have a strong factor structure . In other words , the size of the top eigenvalues of the covariance matrix Σ is typically much larger than the rest . Consequently , the condition number of its covariance matrix is typically very large . It is then of great interest to investigate : how would this factor structure affect the condition number of the Hessian matrix ? To address this important problem , we evaluate the expected Hessian matrix as H = E ( L̈ ( θ ) ) = E { ℓ̈ ( Yi , X⊤i θ ) XiX⊤i } , where ℓ̈ ( y , z ) = ∂ℓ̇ ( y , z ) /∂z stands for the second order derivative of ℓ ( y , z ) with respect to z . For illustration purpose , assume Xi is normally distributed with mean 0 . Recall that the covariance matrix is Σ . Define X̃i = Σ−1/2Xi and θ̃ = Σ1/2θ . We then rewrite H as H = Σ1/2E { ℓ̈ ( Yi , X̃⊤i θ̃ ) X̃iX̃⊤i } Σ1/2 = Σ1/2H̃Σ1/2 , where H̃ = E { ℓ̈ ( Yi , X̃⊤i θ̃ ) X̃iX̃⊤i } . Let A be an arbitrary positive definite matrix . Define λmax ( A ) and λmin ( A ) be the maximum and minimum eigenvalues of A , respectively . We then have λmax ( H ) ≥ λmax ( Σ ) λmin ( H̃ ) and λmin ( H ) ≤ λmin ( Σ ) λmax ( H̃ ) . This further suggests that the condition number of H ( i.e. , con ( H ) = λmax ( H ) /λmin ( H ) ) satisfies the following inequality , con ( H ) ≥ con ( Σ ) / con ( H̃ ) , ( 1 ) where con ( A ) stands for the condition number of an arbitrary positive definite matrix . Note that H̃ is the expected Hessian matrix under an extremely ideal situation , where the input feature follows a standard multivariate normal distribution . This is arguable the most ideal situation for numerical optimization . We thus can reasonably expect that con ( H̃ ) should not be very large in this case . Thus , by ( 1 ) we know that con ( Σ ) should play an important role in determining the con ( H ) . In other words , the large condition number of Σ will affect con ( H ) . This makes the numerical optimization by a standard GD algorithm extremely difficult , and thus calls for a novel solution .

Summary: The paper describes a training scheme based on decomposing input features into two parts which have different training dynamics: a low rank "factor feature" computed using PCA on the raw features, and a high rank "residual".  The former is processed  by a very shallow network, while the latter passes through the full network, and parameters for each are updated with different learning rates.  Experiments show that the proposed algorithm can speed up wall-time to  a certain accuracy on MNIST and CIFAR10 classification, across several neural network architectures and optimizers.