Variability of Neural Networks and Han-Layer: A Variability-Inspired Model

What makes an artificial neural network easier to train or to generalize better than its peers ? We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is , in general , a dominant cost-factor . Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants , which is related but not identical to vanishing gradient . Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks . Guided by variability considerations , we propose a new architecture called Householder-absolute neural layers , or Han-layers for short , to build high variability networks with a guaranteed immunity to gradient vanishing or exploding . On small stylized models , Han-layer networks exhibit a far superior generalization ability over fully-connected networks . Extensive empirical results demonstrate that , by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers , Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance . We will also briefly discuss current limitations of the proposed Han-layer architecture . 1 INTRODUCTION . 1.1 OVERVIEW . Deep neural networks ( DNNs ) have greatly advanced the state of the arts in many machine learning tasks such as image classification , text categorization , speech recognition , to name just a few out of a long list . Despite their tremendous successes , it remains to be fully understood why DNNs work so well on so many tasks in machine learning . In this paper , we take an intuitive approach to providing a new angle from which one can qualitatively investigate certain behaviors of DNNs and provide explanations to some critical issues . Our new angle is based on a notion of variability for DNNs which seems to have not been specifically examined in the past . Our study of variability focuses primarily on function ( forward propagation ) values rather than on the gradient ( back-propagation ) values , offering a different insight from gradient-based views . In essence , the proposed variability is a qualitative surrogate of expressivity or expressiveness for neural networks ( see a recent survey paper ( Gühring et al. , 2020 ) and references thereof ) . Instead of giving a precise , quantitative analysis on what kinds of functions can or can not be expressed by certain neural network models which in general can be exceedingly difficult , we opt to give a qualitative characterization that , to a degree , reflects the levels of expressivity , thus sidestepping hurdle in analyses while still providing useful information on the expressiveness of neural networks . In doing so , we aim to find useful guidelines for developing new network architectures . We show that for fully connected and ReLU-activated DNNs with a fixed number of model parameters , the higher variability is ( within a reasonable range ) , the easier it is to train the model to reach highquality solutions . In particular , variability rises with a quantity called activation ratio , or AR ( see ( 5 ) for definition ) , and falls with the occurrence of a phenomenon called Collapse to Constant ( C2C ) , which is not identical ( but related ) to gradient vanishing . Guided by the variability viewpoint , we propose a novel neural-layer design aimed to increase AR and decrease the chance of C2C at the same time . 1.2 MAIN CONTRIBUTIONS . The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models . We study the notion of variability that reflects DNN ’ s expressivity and trainability . Guided by variability , we construct a new architecture called Householder-absolute neural layer or network ( Han-layer or HanNet ) to attain higher variability with fewer parameters . As the name suggests , in a fully connected layer or network ( FC-layer or FCNet ) , we replace square weight matrices with Householder reflectors and use the absolute-value function for activations . To sum up , our contributions are both conceptual and practical , consisting of the following aspects . We study the notion of variability , along with a measurement for it , for DNNs that adds a new angle to view and explain the behaviors of DNNs . We construct HanNet to achieve a high variability and at the same time guarantee an immunity to vanishing or exploding gradient . The chance of Collapse to Constants , yet not completely eliminated in theory , has also been greatly diminished . On stylized small problems , our experiments suggest that variability is indeed a key performance indicator for ReLU-activated FCNets and that HanNets possess an unusually high level of generalization ability . As is illustrated in Figure 4 in Section 5.2.1 , a HanNet outperforms FCNets by a huge margin , producing nearly perfect results . On several standard datasets in regression and image classification , our experiments indicate that comparing to FCNets , HanNets can greatly reduce the number of model parameters , sometimes by orders of magnitude , while still maintaining and often improving generalization performance . This property represents a promising feature with potential impacts on large-scale applications . Many important issues remain to be investigated in order to better understand the power and the limits of HanNets , and to unlock their potentials in real-world applications . 2 RELATED WORK . As is well known , initialization is a critical step towards the success or failure of neural network training . We start our investigation by visualizing landscapes of DNNs with properly initialized parameters with different activation functions . Landscapes of DNNs have been examined from various angles and for different purposes , for example see ( Kawaguchi , 2016 ; Li et al. , 2017 ; Laurent & Brecht , 2018 ; Kuditipudi et al. , 2019 ; Fort et al. , 2020 ; Sun et al. , 2020 ) . In our case , visualization results lead to the concept of variability . The two key building blocks in our proposed HanNets are ( 1 ) the absolute-value ( ABS ) function as an activation function and ( 2 ) the Householder reflectors as the orthogonal weight matrices — two topics that previously appeared in literature within quite different contexts from ours ; for example , ABS functions in ( Batruni , 1991 ; Lin & Unbehauen , 1992 ; Karnewar , 2018 ) and Householder matrices in ( Mhammedi et al. , 2017 ; Tomczak & Welling , 2016 ; Vorontsov et al. , 2017 ; Wang et al. , 2020 ; Wisdom et al. , 2016 ; Zhang et al. , 2018 ) . In addition , other related works will be referenced throughout the paper . 2.1 NOTATIONS . Given an activation function φ ( · ) and a sequence of weight-bias parameters { ( Wi , bi ) } of suitable sizes , we first define layer functions ψi ( · ) ≡ ψi ( · , Wi , bi ) : = Wiφ ( · ) + bi , i = 1 , 2 , · · · , ( 1 ) each of which is the composition of an affine function with the activation φ ( · ) . As mentioned above , we often drop the dependence of ψi on the parameter pair ( Wi , bi ) whenever no confusion arises . For any positive integer k , define Fk ( x ) : = ( ψk ◦ · · · ◦ ψ1 ) ( x ) , ( 2 ) which is the composition of ψ1 , . . . , ψk , parameterized by the weight-bias pairs ( Wi , bi ) for i = 1 , · · · , k. In general , Wi ∈ Rni+1×ni , bi ∈ Rni+1 and φi : Rni+1 → Rni+1 , thus function Fk ( x ) is from Rn1 to Rnk . For an L-layer DNN , the model output is FL ( x , W , b ) , where W = { W1 , ... , WL } and b = { b1 , ... , bL } . For any given parameter pair , the network maps an input x to an output FL ( x , W , b ) , which can be computed through the forward propagation : s0 = x ; zk = Wksk−1 + bk , sk = φ ( zk ) , k = 1 , · · · , L. ( 3 ) 3 RISE AND FALL OF VARIABILITY . We start with a set of simple experiments in Section 3.1 to observe the landscapes of neural network functions FL ( x ) as the network depth L increases while the total number of model parameters is kept approximately unchanged . 3.1 LANDSCAPE EXPERIMENTS . To facilitate visualization , we add an input layer and an output layer , both of dimension 2 , to the L hidden layers . For convenience , we continue to use FL ( x ) to denote the extended network . The data set is a set of grid points on the square [ −1 , 1 ] 2 ⊂ R2 over which we compute and plot the surface z = ‖FL ( x , W , b ) ‖2 for randomly sampled ( W , b ) with proper scalings . Three activation functions , Sigmoid φ ( t ) = 1/ ( 1 + e−t ) , ReLU φ ( t ) = max ( 0 , t ) and ABS φ ( t ) = |t| , are used . For ReLU we use Kaiming initialization , and for other two functions we use Xavier initialization ( Glorot & Bengio , 2010 ; He et al. , 2015 ) . We keep the total number of parameters in ( W , b ) at around 10,000 . For each given depth value L , we calculate the corresponding width d ( rounded to the nearest integetr ) , then compute and plot z over the grid on [ −1 , 1 ] for 5 different samples for ( W , b ) . Figure 1 is the result of one single random sample , while the complete plots are given in Appendix ( see Figures 12 , 13 and 14 , where each row consists of 5 plots corresponding to 5 random parameter samples ) . Based on these plots , we make the following observations . For Sigmoid , the surfaces are rather monotonous with few variations either in the x-space ( within each plot ) or in the parameter space ( across the 5 plots ) . In particular , in the x-space , we hardly see any peaks and valleys . Most remarkably , with slightly larger L values , the surfaces become constants . We call this curious phenomenon “ Collapse to Constants ” or simply C2C . For ReLU , the landscapes have much richer expressions from the very beginning ( L = 2 ) . The amount of variations increases as L grows deeper from 2 to 10 and even to 20 . However , at L = 40 and 60 , the phenomenon of C2C again shows up . ABS Activation seems more C2C-resistive than ReLU . Many peaks and valleys still exist at deep depths ( L = 40 , 60 ) . Upon further examinations , it is clear that for large L not only the scalar function ‖FL ( x , W , b ) ‖ tends to constants in x-space , but in fact the vector-valued function FL ( x , W , b ) itself tends to constant vectors for all x ∈ [ −1 , 1 ] 2 . We will explain this C2C phenomenon later .

Summary: **Contributions**: - Introduces the notion of variability to explain the better trainability and generalization of certain deep neural architectures. This is backed by a concrete definition of the variability in terms of the third order derivative information of the loss landscape.  - Empirical experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants (C2C).  - Inspired by the notion of variability, authors propose Householder-absolute neural layer aka Han-layer that replaces square weight matrices with Householder reflectors, and use absolute-value function for activations.   -  HanNet achieves a high variability as well as guarantee an immunity to vanishing or exploding gradients, and chance for collapse to constants has been diminished.  - HanNets achieve better generalization with lesser parameters than fully connected networks.