Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible

1 INTRODUCTION . Whitening is a data preprocessing step that removes correlations between input features ( see Fig . 1 ) . It is used across many scientific disciplines , including geology ( Gillespie et al. , 1986 ) , physics ( Jenet et al. , 2005 ) , machine learning ( Le Cun et al. , 1998 ) , linguistics ( Abney , 2007 ) , and chemistry ( Bro & Smilde , 2014 ) . It has a particularly rich history in neuroscience , where it has been proposed as a mechanism by which biological vision realizes Barlow ’ s redundancy reduction hypothesis ( Attneave , 1954 ; Barlow , 1961 ; Atick & Redlich , 1992 ; Dan et al. , 1996 ; Simoncelli & Olshausen , 2001 ) . Whitening is often recommended since , by standardizing the variances in each direction in feature space , it typically speeds up the convergence of learning algorithms ( Le Cun et al. , 1998 ; Wiesler & Ney , 2011 ) , and causes models to better capture contributions from low variance feature directions . Whitening can also encourage models to focus on more fundamental higher order statistics in data , by removing second order statistics ( Hyvärinen et al. , 2009 ) . Whitening has further been a direct inspiration for deep learning techniques such as batch normalization ( Ioffe & Szegedy , 2015 ) and dynamical isometry ( Pennington et al. , 2017 ; Xiao et al. , 2018 ) . 1.1 WHITENING DESTROYS INFORMATION USEFUL FOR GENERALIZATION . In the high dimensional setting , for any model with a fully connected first layer , we show theoretically and experimentally that whitening the data and then training with gradient descent or stochastic gradient descent ( SGD ) results in a model with poor or nonexistent generalization ability , depending on how the whitening transform is computed . We emphasize that , analytically , this result applies to any model whose first layer is fully connected , and is not restricted to linear models . Empirically , the results hold in an even larger context , including in convolutional networks . Here , the high dimensional setting corresponds to a number of input features which is comparable to or larger than the number of datapoints . While this setting does not usually arise in modern neural network applications , it is of particular relevance in fields where data collection is expensive or otherwise prohibitive ( Levesque et al. , 2012 ) , or where the data is intrinsically high dimensional ( Stringer et al. , 2019 ; Fusi et al. , 2016 ; Shyr , 2012 ; Martínez-Ramón et al. , 2006 ; Bruce et al. , 2002 ) , and is also the focus of increasing interest in statistics ( Wainwright , 2019 ) . The loss of generalization ability for high dimensional whitened datasets is due to the fact that whitening destroys information in the dataset , and that in high dimensional datasets whitening destroys all information which can be used for prediction . This is related to investigations of information loss due to PCA projection ( Geiger & Kubin , 2012 ) . Our result is not restricted to neural networks , and applies to any model in which the input is transformed by a dense matrix with isotropic weight initialization . 1.2 SECOND ORDER OPTIMIZATION HARMS GENERALIZATION SIMILARLY TO WHITENING . Second order optimization algorithms take advantage of information about the curvature of the loss landscape to take a more direct route to a minimum ( Boyd & Vandenberghe , 2004 ; Bottou et al. , 2018 ) . There are many approaches to second order or quasi-second order optimization ( Martens & Grosse , 2015 ; Dennis Jr & Moré , 1977 ; Broyden , 1970 ; Fletcher , 1970 ; Goldfarb , 1970 ; Shanno , 1970 ; Liu & Nocedal , 1989 ; Schraudolph et al. , 2007 ; Sunehag et al. , 2009 ; Martens , 2010 ; Byrd et al. , 2011 ; Vinyals & Povey , 2011 ; Lin et al. , 2008 ; Hennig , 2013 ; Byrd et al. , 2014 ; Sohl-Dickstein et al. , 2014 ; Desjardins et al. , 2015 ; Grosse & Martens , 2016 ; Martens et al. , 2018 ; George et al. , 2018 ; Zhang et al. , 2017 ; Botev et al. , 2017 ; Bollapragada et al. , 2018 ; Berahas et al. , 2019 ; Gupta et al. , 2018 ; Agarwal et al. , 2016 ; Duchi et al. , 2011 ; Shazeer & Stern , 2018 ; Anil et al. , 2019 ; Agarwal et al. , 2019 ; Lu et al. , 2018 ; Kingma & Ba , 2014 ; Zeiler , 2012 ; Tieleman & Hinton , 2012 ; Osawa et al. , 2020 ) , and there is active debate over whether second order optimization harms generalization ( Wilson et al. , 2017 ; Zhang et al. , 2018 ; 2019 ; Amari et al. , 2020 ; Vaswani et al. , 2020 ) . The measure of curvature used in these algorithms is often related to feature-feature covariance matrices of the input , and of intermediate activations ( Martens & Grosse , 2015 ) . In some situations , it is already known that second order optimization is equivalent to steepest descent training on whitened data ( Sohl-Dickstein , 2012 ; Martens & Grosse , 2015 ) . The similarities between whitening and second order optimization allow us to argue that pure second order optimization also prevents information about the input distribution from being leveraged during training , and can harm generalization ( see Figs . 3 , 4 ) . We do find , however , that when strongly regularized and carefully tuned , second order methods can lead to superior performance ( Fig . 5 ) . 2 THEORY OF WHITENING , SECOND ORDER OPTIMIZATION , AND GENERALIZATION . Consider a dataset X ∈ Rd×n consisting of n independent d-dimensional examples . We write F for the feature-feature second moment matrix and K for the sample-sample second moment matrix : F = XX > ∈ Rd×d , K = X > X ∈ Rn×n . ( 1 ) We assume that at least one of F or K is full rank . We omit normalization factors of 1/n and 1/d in the definitions of F and K , respectively , for notational simplicity in later sections . Note that as defined , K is also the Gram matrix of X . Definition 2.0.1 ( Whitening ) . Any linear transformation M s.t . X̂ = MX maps the eigenspectrum of F to ones and zeros , with the multiplicity of ones given by rank ( F ) . We consider the two cases n ≤ d and n ≥ d ( when n = d both cases apply ) . n ≥ d : F̂ = Id×d , K̂ = d∑ i=1 ûiû > i . n ≤ d : F̂ = n∑ j=1 v̂j v̂ > j , K̂ = I n×n . ( 2 ) Here , F̂ and K̂ denote the whitened second moment matrices , and the vectors ûi and v̂j are orthogonal unit vectors of dimension n and d respectively . Eq . 2 follows directly from the fact that X > X and XX > share nonzero eigenvalues . We are interested in understanding the effect of whitening on the performance of a trained model when evaluated on a test set . We will see that for models with a dense first layer ( eg , fully connected neural networks ) , the trained model depends on the training inputs only through K. In general , training dynamics and generalization performance can depend non-trivially on K. However , whitening trivializes K , and so eliminates the ability of the network and training algorithm to take advantage of information contained in it . 2.1 TRAINING DYNAMICS DEPEND ON THE TRAINING DATA ONLY THROUGH ITS SECOND MOMENTS . Consider a model f with a dense first layer Z : f ( X ) = gθ ( Z ) , Z = WX , ( 3 ) where W denotes the first layer weights and θ denotes all remaining parameters ( see Fig . 2 ( a ) ) . The structure of gθ ( · ) is unrestricted . W is initialized from an isotropic distribution . We study a supervised learning problem , in which each vector Xi corresponds to a label Yi.1 We adopt the notation Xtrain ∈ Rd×ntrain and Ytrain for the training inputs and labels , and write the corresponding second moment matrices as Ftrain and Ktrain . We consider models with loss L ( f ( X ) ; Y ) trained by SGD . The update rules are θt+1 = θt − η ∂L t ∂θt and W t+1 = W t − η ∂L t ∂W t = W t − η ∂L t ∂Zttrain X > train , ( 4 ) where t denotes the current training step , η is the learning rate , and Lt is the loss evaluated only on the minibatch used at step t. As a result , the activations Ztrain evolve as Zt+1train = Z t train − η ∂Lt ∂Zttrain X > trainXtrain = Z t train − η ∂Lt ∂Zttrain Ktrain . ( 5 ) Treating the weights , activations , and function predictions as random variables , with distributions induced by the initial distribution over W 0 , the update rules ( Eqs . 4-5 ) can be represented by the causal diagram in Fig . 2 ( b ) . We can now state one of our main results . Theorem 2.1.1 . Let f ( X ) be a function as in Eq . 3 , with linear first layer Z = WX , and additional parameters θ . Let W be initialized from an isotropic distribution . Further , let f ( X ) be trained via gradient descent on a training dataset Xtrain . The learned weights θt and first layer activations Zttrain are independent of Xtrain conditioned on Ktrain and Ytrain . In terms of mutual information I , we have I ( Zttrain , θt ; Xtrain | Ktrain , Ytrain ) = 0 ∀t . ( 6 ) Proof . To establish this result , we note that the first layer activation at initialization , Z0train , is a random variable due to random weight initialization , and only depends on Xtrain through Ktrain : I ( Z0train ; Xtrain | Ktrain ) = 0 . ( 7 ) This is a consequence of the isotropy of the initial weight distribution , explained in detail in Appendix A . Combining this with Eqs . 4-5 , the causal diagram for all of training is given by ( the black part of ) Fig . 2 ( c ) . The conditional independence of Eq . 6 follows from this diagram . 2.2 TEST SET PREDICTIONS DEPEND ON TRAIN AND TEST INPUTS ONLY THROUGH THEIR SECOND MOMENTS . Let Xtest ∈ Rd×ntest and Ytest be the test data . The test predictions ftest = f ( Xtest ) are determined by Zttest = W tXtest and θt . So far we have discussed the evolution of Ztrain . To identify sources of data dependence , we can write the evolution of the test set predictions Ztest over the course of training in a similar fashion : Zt+1test = Z t test − η ∂Lt ∂Zttrain Ktrain×test , ( 8 ) where Ktrain×test = X > trainXtest ∈ Rntrain×ntest . The initial first layer activations are independent of the training data , and depend on Xtest only through Ktest : I ( Z0test ; X | Ktest ) = 0 , ( 9 ) where X is the combined training and test data . If we denote the second moment matrix over this combined set by K , then the evolution of the test predictions is described by the ( purple part of the ) causal diagram in Fig . 2 ( c ) , from which we conclude the following . Theorem 2.2.1 . For a function f ( X ) as in Eq . 3 , trained with the update rules Eqs . 4-5 from an isotropic weight initialization , test predictions depend on the training data only through K and Ytrain . This is summarized in the mutual information statement I ( ftest ; X | K , Ytrain ) = 0 . ( 10 ) We emphasize that Theorem 2.2.1 applies to any model with a dense first layer , and is not limited to linear models . 1Our results also apply to unsupervised learning , which can be viewed as a special case of supervised learning where Yi contains no information .

Summary: The authors analyse the training dynamics of a machine learning model consisting in a linear unit, followed by any parametrized function. The authors in particular focus on the impact of whitening  the data beforehand or using second order methods. They show that the learned parameters of the model only depend on the training data through its Gram matrix. Since whitening trivializes the Gram matrix, the authors argue that whitening destroys important information.