Variability of Neural Networks and Han-Layer: A Variability-Inspired Model

What makes an artificial neural network easier to train or to generalize better than its peers ? We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is , in general , a dominant cost-factor . Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants , which is related but not identical to vanishing gradient . Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks . Guided by variability considerations , we propose a new architecture called Householder-absolute neural layers , or Han-layers for short , to build high variability networks with a guaranteed immunity to gradient vanishing or exploding . On small stylized models , Han-layer networks exhibit a far superior generalization ability over fully-connected networks . Extensive empirical results demonstrate that , by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers , Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance . We will also briefly discuss current limitations of the proposed Han-layer architecture . 1 INTRODUCTION . 1.1 OVERVIEW . Deep neural networks ( DNNs ) have greatly advanced the state of the arts in many machine learning tasks such as image classification , text categorization , speech recognition , to name just a few out of a long list . Despite their tremendous successes , it remains to be fully understood why DNNs work so well on so many tasks in machine learning . In this paper , we take an intuitive approach to providing a new angle from which one can qualitatively investigate certain behaviors of DNNs and provide explanations to some critical issues . Our new angle is based on a notion of variability for DNNs which seems to have not been specifically examined in the past . Our study of variability focuses primarily on function ( forward propagation ) values rather than on the gradient ( back-propagation ) values , offering a different insight from gradient-based views . In essence , the proposed variability is a qualitative surrogate of expressivity or expressiveness for neural networks ( see a recent survey paper ( Gühring et al. , 2020 ) and references thereof ) . Instead of giving a precise , quantitative analysis on what kinds of functions can or can not be expressed by certain neural network models which in general can be exceedingly difficult , we opt to give a qualitative characterization that , to a degree , reflects the levels of expressivity , thus sidestepping hurdle in analyses while still providing useful information on the expressiveness of neural networks . In doing so , we aim to find useful guidelines for developing new network architectures . We show that for fully connected and ReLU-activated DNNs with a fixed number of model parameters , the higher variability is ( within a reasonable range ) , the easier it is to train the model to reach highquality solutions . In particular , variability rises with a quantity called activation ratio , or AR ( see ( 5 ) for definition ) , and falls with the occurrence of a phenomenon called Collapse to Constant ( C2C ) , which is not identical ( but related ) to gradient vanishing . Guided by the variability viewpoint , we propose a novel neural-layer design aimed to increase AR and decrease the chance of C2C at the same time . 1.2 MAIN CONTRIBUTIONS . The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models . We study the notion of variability that reflects DNN ’ s expressivity and trainability . Guided by variability , we construct a new architecture called Householder-absolute neural layer or network ( Han-layer or HanNet ) to attain higher variability with fewer parameters . As the name suggests , in a fully connected layer or network ( FC-layer or FCNet ) , we replace square weight matrices with Householder reflectors and use the absolute-value function for activations . To sum up , our contributions are both conceptual and practical , consisting of the following aspects . We study the notion of variability , along with a measurement for it , for DNNs that adds a new angle to view and explain the behaviors of DNNs . We construct HanNet to achieve a high variability and at the same time guarantee an immunity to vanishing or exploding gradient . The chance of Collapse to Constants , yet not completely eliminated in theory , has also been greatly diminished . On stylized small problems , our experiments suggest that variability is indeed a key performance indicator for ReLU-activated FCNets and that HanNets possess an unusually high level of generalization ability . As is illustrated in Figure 4 in Section 5.2.1 , a HanNet outperforms FCNets by a huge margin , producing nearly perfect results . On several standard datasets in regression and image classification , our experiments indicate that comparing to FCNets , HanNets can greatly reduce the number of model parameters , sometimes by orders of magnitude , while still maintaining and often improving generalization performance . This property represents a promising feature with potential impacts on large-scale applications . Many important issues remain to be investigated in order to better understand the power and the limits of HanNets , and to unlock their potentials in real-world applications . 2 RELATED WORK . As is well known , initialization is a critical step towards the success or failure of neural network training . We start our investigation by visualizing landscapes of DNNs with properly initialized parameters with different activation functions . Landscapes of DNNs have been examined from various angles and for different purposes , for example see ( Kawaguchi , 2016 ; Li et al. , 2017 ; Laurent & Brecht , 2018 ; Kuditipudi et al. , 2019 ; Fort et al. , 2020 ; Sun et al. , 2020 ) . In our case , visualization results lead to the concept of variability . The two key building blocks in our proposed HanNets are ( 1 ) the absolute-value ( ABS ) function as an activation function and ( 2 ) the Householder reflectors as the orthogonal weight matrices — two topics that previously appeared in literature within quite different contexts from ours ; for example , ABS functions in ( Batruni , 1991 ; Lin & Unbehauen , 1992 ; Karnewar , 2018 ) and Householder matrices in ( Mhammedi et al. , 2017 ; Tomczak & Welling , 2016 ; Vorontsov et al. , 2017 ; Wang et al. , 2020 ; Wisdom et al. , 2016 ; Zhang et al. , 2018 ) . In addition , other related works will be referenced throughout the paper . 2.1 NOTATIONS . Given an activation function φ ( · ) and a sequence of weight-bias parameters { ( Wi , bi ) } of suitable sizes , we first define layer functions ψi ( · ) ≡ ψi ( · , Wi , bi ) : = Wiφ ( · ) + bi , i = 1 , 2 , · · · , ( 1 ) each of which is the composition of an affine function with the activation φ ( · ) . As mentioned above , we often drop the dependence of ψi on the parameter pair ( Wi , bi ) whenever no confusion arises . For any positive integer k , define Fk ( x ) : = ( ψk ◦ · · · ◦ ψ1 ) ( x ) , ( 2 ) which is the composition of ψ1 , . . . , ψk , parameterized by the weight-bias pairs ( Wi , bi ) for i = 1 , · · · , k. In general , Wi ∈ Rni+1×ni , bi ∈ Rni+1 and φi : Rni+1 → Rni+1 , thus function Fk ( x ) is from Rn1 to Rnk . For an L-layer DNN , the model output is FL ( x , W , b ) , where W = { W1 , ... , WL } and b = { b1 , ... , bL } . For any given parameter pair , the network maps an input x to an output FL ( x , W , b ) , which can be computed through the forward propagation : s0 = x ; zk = Wksk−1 + bk , sk = φ ( zk ) , k = 1 , · · · , L. ( 3 ) 3 RISE AND FALL OF VARIABILITY . We start with a set of simple experiments in Section 3.1 to observe the landscapes of neural network functions FL ( x ) as the network depth L increases while the total number of model parameters is kept approximately unchanged . 3.1 LANDSCAPE EXPERIMENTS . To facilitate visualization , we add an input layer and an output layer , both of dimension 2 , to the L hidden layers . For convenience , we continue to use FL ( x ) to denote the extended network . The data set is a set of grid points on the square [ −1 , 1 ] 2 ⊂ R2 over which we compute and plot the surface z = ‖FL ( x , W , b ) ‖2 for randomly sampled ( W , b ) with proper scalings . Three activation functions , Sigmoid φ ( t ) = 1/ ( 1 + e−t ) , ReLU φ ( t ) = max ( 0 , t ) and ABS φ ( t ) = |t| , are used . For ReLU we use Kaiming initialization , and for other two functions we use Xavier initialization ( Glorot & Bengio , 2010 ; He et al. , 2015 ) . We keep the total number of parameters in ( W , b ) at around 10,000 . For each given depth value L , we calculate the corresponding width d ( rounded to the nearest integetr ) , then compute and plot z over the grid on [ −1 , 1 ] for 5 different samples for ( W , b ) . Figure 1 is the result of one single random sample , while the complete plots are given in Appendix ( see Figures 12 , 13 and 14 , where each row consists of 5 plots corresponding to 5 random parameter samples ) . Based on these plots , we make the following observations . For Sigmoid , the surfaces are rather monotonous with few variations either in the x-space ( within each plot ) or in the parameter space ( across the 5 plots ) . In particular , in the x-space , we hardly see any peaks and valleys . Most remarkably , with slightly larger L values , the surfaces become constants . We call this curious phenomenon “ Collapse to Constants ” or simply C2C . For ReLU , the landscapes have much richer expressions from the very beginning ( L = 2 ) . The amount of variations increases as L grows deeper from 2 to 10 and even to 20 . However , at L = 40 and 60 , the phenomenon of C2C again shows up . ABS Activation seems more C2C-resistive than ReLU . Many peaks and valleys still exist at deep depths ( L = 40 , 60 ) . Upon further examinations , it is clear that for large L not only the scalar function ‖FL ( x , W , b ) ‖ tends to constants in x-space , but in fact the vector-valued function FL ( x , W , b ) itself tends to constant vectors for all x ∈ [ −1 , 1 ] 2 . We will explain this C2C phenomenon later .

Summary: The paper introduces an estimator of expressivity or trainability of deep networks, that they call "variability", that characterizes the variations in a scalar output of the network, when the input change, and seems correlated to trainability of the network, and to its depth (to an extent).  The paper proposes to parametrize weight matrices by using Householder reflectors, parametrized by a single vector each, which has two benefits: - A reduced number of free parameters, which enables more depth (more layers and non-linearities) for the same model size - Orthogonality of the matrices.  Experiments on a synthetic 2D benchmark, two small-scale regression datasets, and small-resolution image datasets show improved results compared to dense nets.