Traditional and Heavy Tailed Self Regularization in Neural Network Models

1 INTRODUCTION . The inability of optimization and learning theory to explain and predict the properties of NNs is not a new phenomenon . From the earliest days of DNNs , it was suspected that VC theory did not apply to these systems ( 1 ) . It was originally assumed that local minima in the energy/loss surface were responsible for the inability of VC theory to describe NNs ( 1 ) , and that the mechanism for this was that getting trapped in local minima during training limited the number of possible functions realizable by the network . However , it was very soon realized that the presence of local minima in the energy function was not a problem in practice ( 2 ; 3 ) . Thus , another reason for the inapplicability of VC theory was needed . At the time , there did exist other theories of generalization based on statistical mechanics ( 4 ; 5 ; 6 ; 7 ) , but for various technical and nontechnical reasons these fell out of favor in the ML/NN communities . Instead , VC theory and related techniques continued to remain popular , in spite of their obvious problems . More recently , theoretical results of Choromanska et al . ( 8 ) ( which are related to ( 4 ; 5 ; 6 ; 7 ) ) suggested that the Energy/optimization Landscape of modern DNNs resembles the Energy Landscape of a zero-temperature Gaussian Spin Glass ; and empirical results of Zhang et al . ( 9 ) have again pointed out that VC theory does not describe the properties of DNNs . Martin and Mahoney then suggested that the Spin Glass analogy may be useful to understand severe overtraining versus the inability to overtrain in modern DNNs ( 10 ) . We should note that it is not even clear how to define DNN regularization . The challenge in applying these well-known ideas to DNNs is that DNNs have many adjustable “ knobs and switches , ” independent of the Energy Landscape itself , most of which can affect training accuracy , in addition to many model parameters . Indeed , nearly anything that improves generalization is called regularization ( 11 ) . Evaluating and comparing these methods is challenging , in part since there are so many , and in part since they are often constrained by systems or other not-traditionally-ML considerations . Motivated by this situation , we are interested here in two related questions . Under review as a conference paper at ICLR 2019 • Theoretical Question . Why is regularization in deep learning seemingly quite different than regularization in other areas on ML ; and what is the right theoretical framework with which to investigate regularization for DNNs ? • Practical Question . How can one control and adjust , in a theoretically-principled way , the many knobs and switches that exist in modern DNN systems , e.g. , to train these models efficiently and effectively , to monitor their effects on the global Energy Landscape , etc. ? That is , we seek a Practical Theory of Deep Learning , one that is prescriptive and not just descriptive . This theory would provide useful tools for practitioners wanting to know How to characterize and control the Energy Landscape to engineer larger and betters DNNs ; and it would also provide theoretical answers to broad open questions as Why Deep Learning even works . Main Empirical Results . Our main empirical results consist in evaluating empirically the ESDs ( and related RMT-based statistics ) for weight matrices for a suite of DNN models , thereby probing the Energy Landscapes of these DNNs . For older and/or smaller models , these results are consistent with implicit Self-Regularization that is Tikhonov-like ; and for modern state-of-the-art models , these results suggest novel forms of Heavy-Tailed Self-Regularization . • Self-Regularization in old/small models . The ESDs of older/smaller DNN models ( like LeNet5 and a toy MLP3 model ) exhibit weak Self-Regularization , well-modeled by a perturbative variant of MP theory , the Spiked-Covariance model . Here , a small number of eigenvalues pull out from the random bulk , and thus the MP Soft Rank and Stable Rank both decrease . This weak form of Self-Regularization is like Tikhonov regularization , in that there is a “ size scale ” that cleanly separates “ signal ” from “ noise , ” but it is different than explicit Tikhonov regularization in that it arises implicitly due to the DNN training process itself . • Heavy-Tailed Self-Regularization . The ESDs of larger , modern DNN models ( including AlexNet and Inception and nearly every other large-scale model we have examined ) deviate strongly from the common Gaussian-based MP model . Instead , they appear to lie in one of the very different Universality classes of Heavy-Tailed random matrix models . We call this HeavyTailed Self-Regularization . The ESD appears Heavy-Tailed , but with finite support . In this case , there is not a “ size scale ” ( even in the theory ) that cleanly separates “ signal ” from “ noise. ” Main Theoretical Results . Our main theoretical results consist in an operational theory for DNN Self-Regularization . Our theory uses ideas from RMT—both vanilla MP-based RMT as well as extensions to other Universality classes based on Heavy-Tailed distributions—to provide a visual taxonomy for 5+1 Phases of Training , corresponding to increasing amounts of Self-Regularization . • Modeling Noise and Signal . We assume that a weight matrix W can be modeled as W ' Wrand + ∆sig , where Wrand is “ noise ” and where ∆sig is “ signal. ” For small to medium sized signal , W is well-approximated by an MP distribution—with elements drawn from the Gaussian Universality class—perhaps after removing a few eigenvectors . For large and strongly-correlated signal , Wrand gets progressively smaller , but we can model the non-random strongly-correlated signal ∆sig by a Heavy-Tailed random matrix , i.e. , a random matrix with elements drawn from a Heavy-Tailed ( rather than Gaussian ) Universality class . • 5+1 Phases of Regularization . Based on this , we construct a practical , visual taxonomy for 5+1 Phases of Training . Each phase is characterized by stronger , visually distinct signatures in the ESD of DNN weight matrices , and successive phases correspond to decreasing MP Soft Rank and increasing amounts of Self-Regularization . The 5+1 phases are : RANDOM-LIKE , BLEEDINGOUT , BULK+SPIKES , BULK-DECAY , HEAVY-TAILED , and RANK-COLLAPSE . Based on these results , we speculate that all well optimized , large DNNs will display Heavy-Tailed Self-Regularization in their weight matrices . Evaluating the Theory . We provide a detailed evaluation of our theory using a smaller MiniAlexNew model that we can train and retrain . • Effect of Explicit Regularization . We analyze ESDs of MiniAlexNet by removing all explicit regularization ( Dropout , Weight Norm constraints , Batch Normalization , etc . ) and characterizing how the ESD of weight matrices behave during and at the end of Backprop training , as we systematically add back in different forms of explicit regularization . • Exhibiting the 5+1 Phases . We demonstrate that we can exhibit all 5+1 phases by appropriate modification of the various knobs of the training process . In particular , by decreasing the batch size from 500 to 2 , we can make the ESDs of the fully-connected layers of MiniAlexNet vary continuously from RANDOM-LIKE to HEAVY-TAILED , while increasing generalization accuracy along the way . These results illustrate the Generalization Gap pheneomena ( 12 ; 13 ; 14 ) , and Under review as a conference paper at ICLR 2019 they explain that pheneomena as being caused by the implicit Self-Regularization associated with models trained with smaller and smaller batch sizes . 2 BASIC RANDOM MATRIX THEORY ( RMT ) In this section , we summarize results from RMT that we use . Several overviews of RMT are available ( 15 ; 16 ; 17 ; 18 ; 19 ; 20 ; 21 ; 22 ) . Here , we will describe a more general form of RMT . 2.1 MARCHENKO-PASTUR ( MP ) THEORY FOR RECTANGULAR MATRICES . MP theory considers the density of singular values ρ ( νi ) of random rectangular matrices W. This is equivalent to considering the density of eigenvalues ρ ( λi ) , i.e. , the ESD , of matrices of the form X = WTW . MP theory then makes strong statements about such quantities as the shape of the distribution in the infinite limit , it ’ s bounds , expected finite-size effects , such as fluctuations near the edge , and rates of convergence . To apply RMT , we need only specify the number of rows and columns of W and assume that the elementsWi , j are drawn from a distribution that is a member of a certain Universality class ( there are different results for different Universality classes ) . RMT then describes properties of the ESD , even at finite size ; and one can compare perdictions of RMT with empirical results . Most well-known is the Universality class of Gaussian distributions . This leads to the basic or vanilla MP theory , which we describe in this section . More esoteric—but ultimately more useful for us—are Universality classes of Heavy-Tailed distributions . In Section 2.2 , we describe this important variant . Gaussian Universality class . We start by modeling W as an N ×M random matrix , with elements from a Gaussian distribution , such that : Wij ∼ N ( 0 , σ2mp ) . Then , MP theory states that the ESD of the correlation matrix , X = WTW , has the limiting density given by the MP distribution ρ ( λ ) : ρN ( λ ) N→∞−−−−→ Q fixed Q2πσ2mp √ ( λ+ − λ ) ( λ− λ− ) λ if λ ∈ [ λ− , λ+ ] 0 otherwise . Here , σ2mp is the element-wise variance of the original matrix , Q = N/M ≥ 1 is the aspect ratio of the matrix , and the minimum and maximum eigenvalues , λ± , are given by λ± = σ2mp ( 1± 1√ Q ) 2 . ( 1 ) Finite-size Fluctuations at the MP Edge . In the infinite limit , all fluctuations in ρN ( λ ) concentrate very sharply at the MP edge , λ± , and the distribution of the maximum eigenvalues ρ∞ ( λmax ) is governed by the TW Law . Even for a single finite-sized matrix , however , MP theory states the upper edge of ρ ( λ ) is very sharp ; and even when the MP Law is violated , the TW Law , with finitesize corrections , works very well at describing the edge statistics . When these laws are violated , this is very strong evidence for the onset of more regular non-random structure in the DNN weight matrices , which we will interpret as evidence of Self-Regularization . 2.2 HEAVY-TAILED EXTENSIONS OF MP THEORY . MP-based RMT is applicable to a wide range of matrices ; but it is not in general applicable when matrix elements are strongly-correlated . Strong correlations appear to be the case for many welltrained , production-quality DNNs . In statistical physics , it is common to model strongly-correlated systems by Heavy-Tailed distributions ( 32 ) . The reason is that these models exhibit , more or less , the same large-scale statistical behavior as natural phenomena in which strong correlations exist ( 32 ; 19 ) . Moreover , recent results from MP/RMT have shown that new Universality classes exist for matrices with elements drawn from certain Heavy-Tailed distributions ( 19 ) . We use these Heavy-Tailed extensions of basic MP/RMT to build an operational and phenomenological theory of Regularization in Deep Learning ; and we use these extensions to justify our analysis of both Self-Regularization and Heavy-Tailed Self-Regularization . Briefly , our theory for simple Self-Regularization is insipred by the Spiked-Covariance model of Johnstone ( 33 ) and it ’ s interpretation as a form of Self-Organization by Sornette ( 34 ) ; and our theory for more sophisticated Heavy-Tailed Self-Regularization is inspired by the application of MP/RMT tools in quantitative finance by Bouchuad , Potters , and coworkers ( 35 ; 36 ; 37 ; 23 ; 25 ; 19 ; 22 ) , as well as the relation of Heavy-Tailed phenomena more generally to Self-Organized Criticality in Nature ( 32 ) . Here , we Under review as a conference paper at ICLR 2019 Generative Model w/ elements from Universality class Finite-N Global shape ρN ( λ ) Limiting Global shape ρ ( λ ) , N →∞ Bulk edge Local stats λ ≈ λ+ ( far ) Tail Local stats λ ≈ λmax Basic MP Gaussian MP , i.e. , Eqn . ( 1 ) MP TW No tail . SpikedCovariance Gaussian , + low-rank perturbations MP + Gaussian spikes MP TW Gaussian Heavy tail , 4 < µ ( Weakly ) Heavy-Tailed MP + PL tail MP Heavy-Tailed ∗ Heavy-Tailed∗ Heavy tail , 2 < µ < 4 ( Moderately ) Heavy-Tailed ( or “ fat tailed ” ) PL∗∗ ∼ λ− ( aµ+b ) PL ∼ λ− ( 1 2 µ+1 ) No edge . Frechet Heavy tail , 0 < µ < 2 ( Very ) Heavy-Tailed PL∗∗ ∼ λ− ( 1 2 µ+1 ) PL ∼ λ− ( 1 2 µ+1 ) No edge . Frechet Under review as a conference paper at ICLR 2019 Identifying the Universality class . Given α , we identify the corresponding µ and thus which of the three Heavy-Tailed Universality classes ( 0 < µ < 2 or 2 < µ < 4 or 4 < µ , as described in Table 1 ) is appropriate to describe the system . The following are particularly important points . First , observing a Heavy-Tailed ESD may indicate the presence of a scale-free DNN . This suggests that the underlying DNN is strongly-correlated , and that we need more than just a few separated spikes , plus some random-like bulk structure , to model the DNN and to understand DNN regularization . Second , this does not necessarily imply that the matrix elements of Wl form a Heavy-Tailed distribution . Rather , the Heavy-Tailed distribution arises since we posit it as a model of the strongly correlated , highly non-random matrix Wl . Third , we conjecture that this is more general , and that very welltrained DNNs will exhibit Heavy-Tailed behavior in their ESD for many the weight matrices .

Summary: This manuscript studies the implicit regularization of neural networks from the perspective of random matrix theory. The authors provide both empirical and theoretical results that aim to show that the empirical spectral density of weights of DNN captures the implicit regularization phenomenon. However, the results are far from rigorous theory and it is not clear how recent results in MP theory yields the statements made in the paper.