HAN-LAYER: A VARIABILITY-INSPIRED MODEL
Year: 2021

Abstract:
What makes an artificial neural network easier to train or to generalize better than its peers? We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is, in general, a dominant cost-factor. Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants, which is related but not identical to vanishing gradient. Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks. Guided by variability considerations, we propose a new architecture called Householder-absolute neural layers, or Han-layers for short, to build high variability networks with a guaranteed immunity to gradient vanishing or exploding. On small stylized models, Han-layer networks exhibit a far superior generalization ability over fully-connected networks. Extensive empirical results demonstrate that, by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers, Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance. We will also briefly discuss current limitations of the proposed Han-layer architecture.

text: What makes an artificial neural network easier to train or to generalize better than its peers? We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is, in general, a dominant cost-factor. Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants, which is related but not identical to vanishing gradient. Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks. Guided by variability considerations, we propose a new architecture called Householder-absolute neural layers, or Han-layers for short, to build high variability networks with a guaranteed immunity to gradient vanishing or exploding. On small stylized models, Han-layer networks exhibit a far superior generalization ability over fully-connected networks. Extensive empirical results demonstrate that, by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers, Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance. We will also briefly discuss current limitations of the proposed Han-layer architecture.

heading: 1 INTRODUCTION

heading: 1.1 OVERVIEW
text: Deep neural networks (DNNs) have greatly advanced the state of the arts in many machine learning tasks such as image classification, text categorization, speech recognition, to name just a few out of a long list. Despite their tremendous successes, it remains to be fully understood why DNNs work so well on so many tasks in machine learning. In this paper, we take an intuitive approach to providing a new angle from which one can qualitatively investigate certain behaviors of DNNs and provide explanations to some critical issues. Our new angle is based on a notion of variability for DNNs which seems to have not been specifically examined in the past. Our study of variability focuses primarily on function (forward propagation) values rather than on the gradient (back-propagation) values, offering a different insight from gradient-based views.
In essence, the proposed variability is a qualitative surrogate of expressivity or expressiveness for neural networks (see a recent survey paper (Gühring et al., 2020) and references thereof). Instead of giving a precise, quantitative analysis on what kinds of functions can or cannot be expressed by certain neural network models which in general can be exceedingly difficult, we opt to give a qualitative characterization that, to a degree, reflects the levels of expressivity, thus sidestepping hurdle in analyses while still providing useful information on the expressiveness of neural networks. In doing so, we aim to find useful guidelines for developing new network architectures.
We show that for fully connected and ReLU-activated DNNs with a fixed number of model parameters, the higher variability is (within a reasonable range), the easier it is to train the model to reach highquality solutions. In particular, variability rises with a quantity called activation ratio, or AR (see (5)
for definition), and falls with the occurrence of a phenomenon called Collapse to Constant (C2C), which is not identical (but related) to gradient vanishing. Guided by the variability viewpoint, we propose a novel neural-layer design aimed to increase AR and decrease the chance of C2C at the same time.

heading: 1.2 MAIN CONTRIBUTIONS
text: The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models. We study the notion of variability that reflects DNN’s expressivity and trainability. Guided by variability, we construct a new architecture called Householder-absolute neural layer or network (Han-layer or HanNet) to attain higher variability with fewer parameters. As the name suggests, in a fully connected layer or network (FC-layer or FCNet), we replace square weight matrices with Householder reflectors and use the absolute-value function for activations. To sum up, our contributions are both conceptual and practical, consisting of the following aspects.
We study the notion of variability, along with a measurement for it, for DNNs that adds a new angle to view and explain the behaviors of DNNs. We construct HanNet to achieve a high variability and at the same time guarantee an immunity to vanishing or exploding gradient. The chance of Collapse to Constants, yet not completely eliminated in theory, has also been greatly diminished.
On stylized small problems, our experiments suggest that variability is indeed a key performance indicator for ReLU-activated FCNets and that HanNets possess an unusually high level of generalization ability. As is illustrated in Figure 4 in Section 5.2.1, a HanNet outperforms FCNets by a huge margin, producing nearly perfect results.
On several standard datasets in regression and image classification, our experiments indicate that comparing to FCNets, HanNets can greatly reduce the number of model parameters, sometimes by orders of magnitude, while still maintaining and often improving generalization performance. This property represents a promising feature with potential impacts on large-scale applications.
Many important issues remain to be investigated in order to better understand the power and the limits of HanNets, and to unlock their potentials in real-world applications.

heading: 2 RELATED WORK
text: As is well known, initialization is a critical step towards the success or failure of neural network training. We start our investigation by visualizing landscapes of DNNs with properly initialized parameters with different activation functions. Landscapes of DNNs have been examined from various angles and for different purposes, for example see (Kawaguchi, 2016; Li et al., 2017; Laurent & Brecht, 2018; Kuditipudi et al., 2019; Fort et al., 2020; Sun et al., 2020). In our case, visualization results lead to the concept of variability. The two key building blocks in our proposed HanNets are (1) the absolute-value (ABS) function as an activation function and (2) the Householder reflectors as the orthogonal weight matrices — two topics that previously appeared in literature within quite different contexts from ours; for example, ABS functions in (Batruni, 1991; Lin & Unbehauen, 1992; Karnewar, 2018) and Householder matrices in (Mhammedi et al., 2017; Tomczak & Welling, 2016; Vorontsov et al., 2017; Wang et al., 2020; Wisdom et al., 2016; Zhang et al., 2018). In addition, other related works will be referenced throughout the paper.

heading: 2.1 NOTATIONS
text: Given an activation function φ(·) and a sequence of weight-bias parameters {(Wi, bi)} of suitable sizes, we first define layer functions
ψi(·) ≡ ψi(·,Wi, bi) := Wiφ(·) + bi, i = 1, 2, · · · , (1) each of which is the composition of an affine function with the activation φ(·). As mentioned above, we often drop the dependence of ψi on the parameter pair (Wi, bi) whenever no confusion arises. For
any positive integer k, define Fk(x) := (ψk ◦ · · · ◦ ψ1)(x), (2)
which is the composition of ψ1, . . . , ψk, parameterized by the weight-bias pairs (Wi, bi) for i = 1, · · · , k. In general, Wi ∈ Rni+1×ni , bi ∈ Rni+1 and φi : Rni+1 → Rni+1 , thus function Fk(x) is from Rn1 to Rnk . For an L-layer DNN, the model output is FL(x,W,b), where W = {W1, ...,WL} and b = {b1, ..., bL}. For any given parameter pair, the network maps an input x to an output FL(x,W,b), which can be computed through the forward propagation:
s0 = x; zk = Wksk−1 + bk, sk = φ(zk), k = 1, · · · , L. (3)

heading: 3 RISE AND FALL OF VARIABILITY
text: We start with a set of simple experiments in Section 3.1 to observe the landscapes of neural network functions FL(x) as the network depth L increases while the total number of model parameters is kept approximately unchanged.

heading: 3.1 LANDSCAPE EXPERIMENTS
text: To facilitate visualization, we add an input layer and an output layer, both of dimension 2, to the L hidden layers. For convenience, we continue to use FL(x) to denote the extended network. The data set is a set of grid points on the square [−1, 1]2 ⊂ R2 over which we compute and plot the surface z = ‖FL(x,W,b)‖2 for randomly sampled (W,b) with proper scalings. Three activation functions, Sigmoid φ(t) = 1/(1 + e−t), ReLU φ(t) = max(0, t) and ABS φ(t) = |t|, are used. For ReLU we use Kaiming initialization, and for other two functions we use Xavier initialization (Glorot & Bengio, 2010; He et al., 2015).
We keep the total number of parameters in (W,b) at around 10,000. For each given depth value L, we calculate the corresponding width d (rounded to the nearest integetr), then compute and plot z over the grid on [−1, 1] for 5 different samples for (W,b). Figure 1 is the result of one single random sample, while the complete plots are given in Appendix (see Figures 12, 13 and 14, where each row consists of 5 plots corresponding to 5 random parameter samples).
Based on these plots, we make the following observations. For Sigmoid, the surfaces are rather monotonous with few variations either in the x-space (within each plot) or in the parameter space
(across the 5 plots). In particular, in the x-space, we hardly see any peaks and valleys. Most remarkably, with slightly larger L values, the surfaces become constants. We call this curious phenomenon “Collapse to Constants” or simply C2C. For ReLU, the landscapes have much richer expressions from the very beginning (L = 2). The amount of variations increases as L grows deeper from 2 to 10 and even to 20. However, at L = 40 and 60, the phenomenon of C2C again shows up. ABS Activation seems more C2C-resistive than ReLU. Many peaks and valleys still exist at deep depths (L = 40, 60).
Upon further examinations, it is clear that for large L not only the scalar function ‖FL(x,W,b)‖ tends to constants in x-space, but in fact the vector-valued function FL(x,W,b) itself tends to constant vectors for all x ∈ [−1, 1]2. We will explain this C2C phenomenon later.

heading: 3.2 VARIABILITY AND A MEASUREMENT FOR IT
text: From the above experimental results, we see unmistakable differences in the outputs of network functions FL(x,W,b) from one activation function to another and as the depth L grows. All the differences are rooted in the richness of landscape variations in the data space where x varies, as well as in the parameter space where (W,b) varies. We will refer to such richness of landscape variations as variability of network functions.
Intuitively speaking, higher variability means not only higher expressiveness in data space, but also better responsiveness to parameter changes. On the other hand, a network with low variability means that it may have a poor approximation power and consequently may be difficult to train. In particular, a deep network would have lost all variability, in certain regions of the parameter space, when it becomes a constant in the entire data space for parameters from those regions, which we refer to as Collapse to Constant (C2C). In Appendix A.4, we treat C2C and its characteristics with more details. Variability may serve as a predictive indicator on trainability of suitable DNNs. We present numerical evidence in Appendix B, verifying that the pattern of variability is highly correlated to the performance of corresponding DNNs.
In general, it is nontrivial to develop effective and computable measures for variability, which should be a subject of investigations on its own right. Here we present a variability measurement that seems to have worked well with two-dimensional data spaces. We define
V3 := E(W,b) ( ‖f‖−1∞ ∫ Ω ∥∥∥∥∆3f∆x3i ∥∥∥∥ 1 dx ) (4)
where f ≡ f(FL(x; W,b)) is a scalar-valued function, the mean is taken over a certain region of interest in the parameter space, and ∆3f/∆x3i is a finite-difference approximation of the third derivative over a grid. The phenomenon of C2C occurs when V3 is close to zero. For example, if f is a least squares loss function and FL(·) is linear, then V3 vanishes. On the other hand, when FL(·) has high nonlinearity, then the value V3 will be relatively large.
In Figure 2, we plot the variability of FL(x; W,b) on Ω = [−1, 1]2, computed as described in Appendix A.2, for network depth L varying from 3 to 45 (with increment 3). As we see on the left plot for ReLU, the V3-variability initially increases with L until it reaches its peak at around L = 16. Afterwards, it starts to decline until it vanishes suddenly at L = 25 because the network output becomes constant at least at one parameter sample, making the geometric mean zero. On the middle plot for absolute-value, besides a slightly higher profile, after the peak variability declines but does not vanish up to L = 60.

heading: 3.3 RISE WITH ACTIVATION RATIO AND FALL WITH C2C
text: It should be clear that variability comes from nonlinear activations in the model. As we see from the previous variability experiments, variability initially always rises as the network depth increases until it reaches a peak. There is a simple explanation for this. That is, when the total number of parameters is fixed, the number of activations always increases with the depth. The activation ratio (AR) for a
model is defined below
AR = total number of individual activations
total number of model parameters . (5)
The activation ratio is monotonically increasing with depth L. In particular, the rate of increase is the largest at L = 1, which shows that the activation ratio increases most rapidly at the beginning, explaining why variability rises quickly at the very beginning, see Appendix A.3 for a formula and more details. In this work, we will also use AR as one of the metrics for variability. However, we caution against the simplistic view that the higher the AR or variability is, the better. In fact, experiments indicate that excessive nonlinearity tends to increase training difficulties.
It should be clear that variability declines and eventually vanishes as C2C develops and materializes. A detailed treatment, including a quantitative characterization of C2C, is given in Appendix A.4.

heading: 4 HANNET: A HIGH VARIABILITY MODEL
text: A path to enhancing and maintaining variability of a DNN is to raise its resistance level to C2C and increase its AR simultaneously. Our construction of Han-layers follows exactly this path.

heading: 4.1 HOUSEHOLDER WEIGHTING: ORTHOGONAL AND HIGH AR
text: It is well-known that Householder reflection matrix associated with a nonzero vector u ∈ Rn is
H(u) = I − 2uuᵀ/‖u‖2, (6)
which is symmetric and orthogonal. As is well known, orthogonality is a desirable property for weight matrices in DNNs. In addition, the degree of freedom in H(u) is n, one order of magnitude smaller than O(n2) for a generic orthogonal matrix.
Evidently, the activation ratio of a DNN can be increased by reducing the number of model parameters without changing the network depth and width, such as by parametrizing an n × n weight matrix with far less than O(n2) parameters (similar to what is done in CNNs). This motivates us to use a Householder reflection matrix to replace a general matrix or a generic orthogonal matrix in a neural layer so that the AR value associated with this layer is enlarged by a factor of O(n). In natural language processing areas, it has been proposed to use a product of multiple Householder matrices to parameterize weight matrices in recursive neural networks, for example (Mhammedi et al., 2017).

heading: 4.2 HAN-LAYERS AND HIGH VARIABILITY
text: A Householder-absolute neural layer, or Han-layer, is composed of a Householder matrix followed by ABS activation (see Appendix C.1 for an algorithmic form), and a HanNet structure is created by using multiple Han-layers. We have already visualized the landscape of a deep HanNet in Figure 1, which has more peaks and valleys than the others implying a higher level of variability. In Figure
2, we calculate the variability measure V3 defined in (4), where the geometric mean is used that vanishes whenever C2C occurs. We observe that the variability of HanNets remains at a high level without deterioration in the entire tested range.

heading: 4.3 IMMUNITY TO GRADIENT VANISHING OR EXPLODING
text: Let the neural network function FL(x,W,b) be defined in (2). The spectrum of a so-calledGL-matrix (see (12) in Appendix A.4) characterizes whether vanishing or exploding gradient would happen or not. Roughly speaking, GL-matrix is the dominant part of the gradient. As L → ∞, vanishing gradient corresponds to GL → 0 and exploding gradient to GL →∞. For HanNet, with the notation u = {uk}Lk=1, the GL-matrix takes the form
GL(x,u,b) = L∏ k=1 H(uk)∇φ(zk), (7)
where H(uk) is defined by (6) and ∇φ(zk) is the diagonal, Jacobian matrix of φ(t) = |t| evaluated component-wise at a vector zk. Evidently, the diagonal matrices ∇φ(zk) are, with a high probability, all orthogonal matrices with diagonal entries equal to ±1. Hence, the following result holds implying that gradient can never vanish or explode (in a probability sense). Proposition 1. Under a mild distribution assumption, the G-matrix for HanNet, that is, GL(x,u, b) defined in 7, remains orthogonal with probability one for any x, any ui 6= 0 in u, any b, and any integer L > 0.

heading: 4.4 HAN/MLP-MIXER
text: In literature, the term Multi-Layer Perceptron (MLP) is often used exchangeably with FCNet. Recently, MLP-dominated models have seen a wave of revivals for image recognition tasks (Tolstikhin et al., 2021; Liu et al., 2021). MLP-dominated models (without multi-head attentions) are much more concise than Transformer-based models (Dosovitskiy et al., 2020; Touvron et al., 2021) but can still maintain testing performances on very large-scale datasets. The motivation of MLP-Mixer (Tolstikhin et al., 2021) is to use the purely fully-connected layers to remove attention architectures. An MLP-Mixer block is the elementary unit in MLP-Mixer models that consists of several FC-layers and skip-connections to form the following map from input X to output Y ,
Z = X + GELU (W2 Layer Norm(W1X)) , (8) Y = Z + GELU (Layer Norm(ZW3) W4) , (9)
where the first row (8) is called token-mixing for cross-token communication and the second row (9) is called channel-mixing for cross-channel communication, both being of MLP structure. Here we form our Han-Mixer block by replacing all weight matrices Wi by Householder matrices Hi, i = 1, 2, 3, 4, and all activation functions by the absolute function ABS, that is,
Z = ABS(H2ABS(H1X)), Y = ABS(ABS(ZH3)H4), (10) where we remove skip-connections and layer-normalizations (since HanNets does not suffer from gradient problems). See Appendix C.2 for figurative illustrations of MLP-Mixer vs. Han-Mixer blocks. The resulting Han/MLP-Mixer models are shown in Figure 3, where we arrange some HanMixer blocks after MLP-Mixer blocks (which may be empty). The main reason for us to combine the two types of Mixer blocks is that, short of drastically increasing network width, Han-Mixers alone cannot always provide enough model parameters for large-scale datasets. In addition, we use the convolution stem recommended by (Xiao et al., 2021) instead of the one in (Tolstikhin et al., 2021).

heading: 5 EXPERIMENTS

heading: 5.1 DATASETS
text: To empirically investigate the efficacy of HanNets in comparison to FCNets, we use multiple datasets including a synthetic dataset Checkerboard, 2 classic regression datasets, Elevators (Dua
& Graff, 2017) and Cal Housing (Pace & Barry, 1997), and 4 widely used image classification datasets CIFAR10, CIFAR100, STL10, and ImageNet32 (Krizhevsky et al., 2009; Coates et al., 2011; Chrabaszcz et al., 2017), where the last one is a down-sampled version of ImageNet (out of affordability considerations). All dataset settings and training details are given in Appendix D.1.

heading: 5.2 RESULTS

heading: 5.2.1 SYNTHETIC DATASET: CHECKERBOARD
text: The ground truth and the training set, along with a couple of typical results, are plotted in Figure 4.
On this dataset, we compare a range of FCNets with HanNets (in fact, ResNets were also compared, without better results than those from FCNets). We generate more than 200 pairs of FCNets and HanNets with width varying from 20 to 100 (increment 10) and depth from 2 to 30. Figure 5 (a, b) gives the test results in the heat-map form. We observe from Figure 5 that overall there exists a large gap in testing accuracy between the FCNets (around 85%) and HanNet (over 99%) over a wide range of test cases. The unexpected, near-perfect results obtained by HanNets, without any explicit regularization, look quite stunning and are stable in the sense that similar results can be trained from other random 25% vs. 75% data-splittings. We observe that over-parametrization is not a necessary condition for near perfect solutions. In fact, 400 parameters are sufficient to achieve near-zero testing error in one of the cases in Figure 5(b).
An ablation study in Appendix D.2 suggests that Householder weighting and ABS activating be equally critical to the surprising performance of HanNets on the Checkerboard dataset. In addition, A stability study in Appendix D.3 is done in which a portion of training labels is randomly flipped, and the results show that with seriously damaged labels HanNets still maintain a clear advantage over
FCNets in testing performance. In addition, extensive experiments show that variability is closely and positively related to trainability (see Appendix B for details).

heading: 5.2.2 REGRESSION DATASETS: ELEVATORS AND CAL HOUSING
text: On the two regression Datasets, we compare the performance of a HanNet with two FCNets. Table 1 lists the relevant statistics for the 3 DNNs where depth refers to the number of hidden layers (there exist additional, data-size-dependent input/output layers). We see that in terms of parameter numbers FCNet1 and HanNet are comparable peers, while FCNet2 has about 15 times more parameters.
We present testing NRMSE (normalized root mean square error) loss values in Figure 11 in Appendix D.4, where the curves are average values over 5 trials with error bars. Clearly, HanNet outperforms its “peer” FCNet1 by a notable margin, especially when fewer training samples are used, and is competitive with (on 80% training data) or better than (on 20% training data) FCNet2 which uses 15 times more parameters. We mention that the best testing performance of HanNet is statistically the same as that reported in (Tsang et al., 2018) with an FCNet model that uses over 5 million parameters. Additionally, with fewer parameters, HanNet appears less influenced by overfitting.

heading: 5.2.3 IMAGE CLASSIFICATION DATASETS
text: In this section, we investigate HanNets’ generalization ability in image classification by comparing MLP-Mixers with our Han/MLP-Mixers presented in Subsection 4.4. The comparisons are done with 3 standard image datasets and a down-sampled version of ImageNet (due to computing source limitation). More experimental details are given in Appendix D.5.
Empirical evidences from MLP-based models (Tolstikhin et al., 2021) suggest that these models can achieve the state-of-the-art results on very large-scale datasets. Since the datasets used in our tests are not large enough to play to the full strengths of Mixer models, our aim here is not to observe how close our results can approach the state-of-the-art, but how Han-layer structures can impact the performance of Mixer models.
Tables 2 and 3 report test results from various MLP-Mixer and H/M-Mixer models. On the smaller dataset STL10, HanMixers alone can already produce the best result. On larger datasets CIFAR10 and CIFAR100, enhanced performance from HanMixers are achieved either by adding a relatively few parameters (e.g., MLPMixer (2 or 4) vs MLPMixer (2 or 4) + HanMixer (16)), or even with
far fewer parameters (e.g., MLPMixer (4) vs MLPMixer (1)+HanMixer (16)). On ImageNet32, our Han/MLP-Mixer combination model clearly outperforms pure MLPMixers and offers a competitive performance with WideResNet while using only 40% parameters. In summary, the benefits of using HanMixers have been made evident in this set of experiments.

heading: 6 CONCLUSIONS
text: As a qualitative surrogate for expressivity and trainability, variability provides a useful angle to view certain critical behaviors of DNNs. Empirical evidence suggests that activation ratio and Collapse to Constants are two major contributing factors to the gain and loss of variability, respectively. Based on such insights, we propose a Han-layer architecture consisting of Householder weighting and absolute-value activating. This new architecture greatly raises activation ratio and provides a guaranteed immunity to vanishing or exploding gradient.
Extensive experimental results indicate that when used judiciously as replacements of fully connected layers, Han-layers can significantly reduce the number of model parameters (thus potentially the associated computing costs), while maintaining or improving the level of generalization performance, as is shown by our results on MLP-Mixer models for image classification.
On the synthetic Checkerboard dataset, HanNets consistently exhibit a surprisingly high level of generalization ability. It remains to be investigated that under what conditions and to what extent such an ability could be realized on large-scale datasets.
Several limitations in our results are worth mentioning. So far, our measurement of variability is limited to low-dimensional data spaces. It remains to be seen whether or not a computable extension to high-dimensional spaces can be fruitfully done. As is currently defined, a Han-layer is not as flexible as a fully connected layer since a Householder matrix must be square. Finally, we offer two points of perspectives: (1) measured variability should be viewed within a reasonable range (chaotic landscapes are generally bad); and (2) too many Han-layers, even though immune to gradient problems, may still increase the difficulty in training due to high nonlinearity.

heading: A.1 EXISTING TECHNIQUES THAT ENHANCE VARIABILITY
text: Several existing techniques in deep learning can be interpreted from the viewpoint of enhancing the variability of neural networks. For example, convolutional neural networks (CNN) use far fewer parameters, in comparison to fully connected networks, at each layer, thus significantly increasing activation ratios and subsequently enhancing variability. In this view, achieving higher activation ratios should be a significant contributing factor to the great success of CNN.
Since C2C has a close relationship with vanishing gradient, it is not surprising that existing techniques designed to alleviate the latter can also help with the former. Specifically, since G-matrices are limits of C-matrices, techniques that slow down the size decrease of G-matrices usually also slow down the size decrease of C-matrices. Such techniques include Residual Networks (or ResNet) (He et al., 2016) and Batch Normalizations (Ioffe & Szegedy, 2015). A particularly simple and widely utilized technique is to initialize weight matrices by orthogonal matrices (Hu et al., 2019; Huang et al., 2018).

heading: A.2 SETTINGS OF V3 IN FIGURE 4
text: In our experiment, we use the least-squares function: f(·) := ‖ · ‖22 and Ω = [−1, 1]2. The mean over parameters will be computed as a sample mean. In particular, we use the geometric mean which vanished if a single sampled value is zero. Therefore, once FL(x; W,b) becomes a constant (or linear for that matter) on Ω, variability as measured by V3 vanishes.
A.3 INCREASE OF ACTIVATION RATIO
For simplicity, let us consider fully connected (FC) network of L layers for which the number of parameters in (W,b) is Nw = (d2 + d)L where d is the width of the layers. In this case, the number of activations is Ld. The activation ratio (AR) ρ for this network is the total number of activations divided by the total number of parameters, i.e., ρ = Ld/Nw, which can be written as a function of L. By solving the equation Nw = (d2 + d)L for d, we obtain d = (1 + √ 1 + 4Nw/L)/2. Therefore, the activation ratio is ρ(L) = 1
Nw
( L+ √ L2 + 4LNw ) , L ≥ 1. (11)
Furthermore, since
ρ′(L) = 1
Nw
( 1 +
1 + 2Nw/L√ 1 + 4Nw/L
) > 0,
the activation ratio is monotonically increasing with L. In particular, the rate of increase is the largest at L = 1. This simple calculation shows that the activation ratio increases most rapidly at the beginning, explaining why variability arises quickly at the very beginning.

heading: A.4 COLLAPSE TO CONSTANTS AND ITS CHARACTERIZATION
text: To compute the derivative of FL(x,W,b) with respect to the parameters, one uses the chain-rule to obtain so-called back-propagation formulas, such as
∂
∂b1 FL(x,W,b) = ∂sL ∂b1 = ∇φ(z1)WT2 ∇φ(z2) · · ·WTL∇φ(zL),
where zk are computed in (3),∇φ(zk) are diagonal matrices with scalar-valued φ′ applied componentwise. It is well-known that the behavior of the derivatives is critically determined by the properties of the above matrix product. For convenience and without loss of generality, we add W1 to the product and define
GL ≡ GL(x,W,b) := L∏ k=1 WTk ∇φ(zk), (12)
which we will simply call the G-matrix associated with the network FL(x,W,b). It is well known in deep learning that excessively small (or large) size of GL causes vanishing (or exploding) gradient, which is a major source of difficulty in training.
Now we define another matrix product called the C-matrix, by replacing the derivative ∇φ(zk) in (12) by the finite difference at two points zk and z̄k; that is,
CL ≡ CL(x, x̄,W,b) := L∏ k=1 WTk Dφ(zk, z̄k), (13)
where Dφ(·, ·) ∈ Rd×d is diagonal defined by
[Dφ(u, v)]ii = [φ(u)− φ(v)]i
[u− v]i , i = 1, · · · , d, (14)
with the convention 0/0 = 1, {zk} and {z̄k} are computed via the recursion (3) starting from x and x̄, respectively. By their definitions, it is clear that G-matrices are limits of C-matrices.
The following proposition shows that C-matrices characterize the C2C phenomenon.
Proposition 2. Let network FL(x,W, b) : Rd → Rd be defined as in (2). For any two distinct points x, x̄ ∈ Rd, there holds FL(x)− FL(x̄) = [CL(x, x̄)]T (x− x̄). (15)

heading: Consequently,
text: lim L→∞ CL(x, x̄) = 0 =⇒ lim L→∞ (FL(x)− FL(x̄)) = 0. (16)
The verification of this proposition is straightforward, so we omit it.
We note that the difference going to zero in (16) does not imply that each individual sequence goes to the same limit. On the contrary, such limits generally do not exist if the bias sequence {bk} is bounded away from zero.
Regarding the C2C phenomenon, the following remarks are in order.
• Wherever CL(x, x̄) is sufficiently small, the output values of the network for these two inputs x and x̄ will be close to each other for large L.
• CL(x, x̄) will be small if, for all or many k, ‖WTk Dφ(zk, z̄k)‖ are sufficiently smaller than 1 in some norm which occurs when one or both of the matrices is sufficiently small.
• If CL(x, x̄) is sufficiently small for all x in some region, then the corresponding outputs of the network will be like a constant in this region. In particular, this can happen when weight matrices in {Wk} are small for all or many k.
If the weight matricesWk, k = 1, · · · , L, are properly normalized (for example, allWk are orthogonal matrices), then the size of CL will be solely determined by that of Dφ(zk, z̄k) for a given point pair (x, x̄), which in turn depends on activation φ in use. We consider the popular ReLU activation φ(t) = max(0, t) and, again for the purpose of contrast, the absolute-value function φ(t) = |t|. In both cases, the diagonal entries in (14) satisfy [Dφ(u, v)]ii ∈ [−1, 1]. Proposition 3. Suppose that u, v ∈ Rd be i.i.d. random variables with
Prob([u]i ≥ 0) = Prob([v]i ≥ 0) = p ∈ (0, 1), i = 1, 2, · · · , d.

heading: Then
text: Prob (|φ(u)− φ(v)| = |u− v|) = { p2d, φ(t) = max(0, t)(
p2 + (1− p)2 )d , φ(t) = |t|
(17)
where the absolute values are taken component-wise.
Proof. Consider the scalar case d = 1 with u 6= v. For ReLU function φ(t) = max(0, t),∣∣∣∣φ(u)− φ(v)u− v ∣∣∣∣ { = 1, u, v ≥ 0< 1, otherwise
where the probability for the first case (ratio equal to 1) is p2. For absolute value φ(t) = |t|,∣∣∣∣φ(u)− φ(v)u− v ∣∣∣∣ { = 1, uv ≥ 0< 1, otherwise
where the probability for the first case (ratio equal to 1) is p2 + (1− p)2.
Since all the components are i.i.d., by raising the above probabilities to their d-th power, we obtain the corresponding probabilities for the vector case for d > 1.
The proposition indicates that the probability for ReLU to preserve distances in Rd is much smaller than that for absolute-value. For instance,
• when p = 1/2 the above two probabilities in (17) become 1/4d and 1/2d, respectively; this is, the latter is 2d times larger than the former;
• when p = 1/4 the above two probabilities in (17) become (1/16)d and (10/16)d, respectively; this is, the latter is 10d times larger than the former.
Remark 4. When d is large, the probability is small for either function to maintain diagonals |[Dφ(u, v)]ii| = 1 for all indices i. That is, probabilistically speaking, the two functions are contractive, i.e., with a high probability, they shrink the distance between points after each application, while ReLU is a more forceful contraction than absolute-value is, especially when the positive probability p of input is small.
In Figure 6, we present a couple of typical experimental results on C- and G-matrices. The left plot is for ReLU activations only with random weight matrices and bias vectors specified, then multiplied by the Xavier/Kaiming initialization constant √ 2/d. For a randomly select point pair (x, x̄), we compute the corresponding C-matrix, as well as the G-matrices at these two points, for depth L from 1 to 1000. The curves depict the spectral norms of the matrices CL and GL (two of those) at each depth value L. As we can see, although for L three curves roughly follow a similar pattern of up-and-downs, the C-matrix still differs significantly from the two G-matrices in that it becomes about two-orders of magnitude smaller than the two G-matrices at the same depth. This signifies an important point that, at least in some cases, C2C should be considered as the leading cause for loss of trainability, instead of vanishing gradient as commonly perceived.
The right plot of Figure 6 is for the comparison of two different activation functions, ReLU, and absolute-value. Starting from a random point-pair, we plot the C-matrices computed on a set of random weight-bias pairs as specified with initialization multipliers √ 2/d and √ 1/d for ReLU and absolute-value, respectively. The plot shows clearly that ReLU, with considerably smaller C-matrices, is much more prone to the negative influence of C2C than the absolute-value is. At L = 1000, the C-matrix for ReLU is more than three orders of magnitude smaller than that for absolute-value.
B VARIABILITY VS. TRAINABILITY
As we have seen, for fully connected neural nets with a fixed number of parameters, variability is initially small, due to low activation ratio values, and then it rises to a peak with the growth of the depth; after that, it starts to fall due to the progression of C2C phenomenon. In this section, we present numerical evidence verifying that the pattern of variability change is highly correlated to the performance of corresponding neural nets. This suggests that variability may well serve as a predictive indicator of the performance of suitable neural nets.

heading: B.1 EXPERIMENT SETTING
text: Our experiments are done on a styled synthetic model checkerboard in which the data points are the 6561 mesh points of an 81× 81 grid over the square [−1, 1]2 in R2. These mesh points are divided into two sets, one corresponding to 0-labels and another to 1-labels, so that together they form an 8 by 8 checkerboard blocks, as is shown in Figure 7, where each of the 64 squares contains 81 grid points and the surrounding edges contain 1377 points. The blocks take either 0 or 1 (blue or red) label in an alternating pattern, and the surrounding edges all take the 0-label. In essence, we aim to approximate the piecewise linear, non-smooth function shown in the right plot of Figure 7.
We train fully connected, rectangular neural nets (FCNets) with the same number of nodes or neurons (but see below for exceptions) at each hidden layer. In each of our experiments, when we vary the number of hidden layers L, we adjust the width d so that the total number of model parameters (i.e., weights and biases) is kept to a prescribed constant (though, in order to do so we sometimes have to add or delete one or two nodes from some hidden layers). It is worth noting that as the depth grows deeper, the width becomes narrower.
We randomly split the dataset of 6561 points into two parts: 25% as a training set containing m = 1640 samples, and the remaining 75% as a testing set. Denoting the training set by {xi}mi=1, we minimize the least squares loss function,
min W,b fL(W,b) ≡ m∑ i=1 ‖F̂L(xi; W,b)− yi‖22, (18)
where F̂L(x,W,b) : R2 → R2 is constructed from the neural net function FL(x,W,b) in (2) plus an input layer and an output layer, and each label vector yi ∈ R2 is either (0, 0)T or (1, 1)T , representing
binary labels. We refer to the objective function value as the training loss and the percentage of correctly classified data points as the training accuracy. We note that a 100% training accuracy generally does not imply a 0-training loss.
To ensure that the optimization calculation is done sufficiently, we apply the gradient descent method (instead of SGD) with 40000 iterations without a stopping criterion. For each run, we always try 10 different initial learning rates (step-sizes) as in
{0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1.0}
and then pick the best result for output. During the 40000 iterations, learning rates are reduced by a factor of 5 three times at the junctures corresponding to iterations 20000, 28000, and 36000, respectively.
We initialize weight matrices and bias vectors with i.i.d. standard normal random numbers, and the weight matrices are then scaled by the Xavier/Kaiming initialization constants. The activation function is ReLU and ABS. All trial instances are run with 10 different samples of random initial parameters. For final output, we always record and output the best result in terms of the training error regardless of where it occurs during training (in the middle or at the end).

heading: B.2 COMPUTATIONAL RESULTS
text: We show the results of training FCNets by GD of varying depths (x-axis) and parameters (from left to right in each row). For each network, we compute the mean value of training/testing loss/accuracy. All result is reasonably stable as shown by the std of 10 runs. Once over-parameterization happens, there exists a bottom of the valley in Figure 8 to get or approach to the global optimum.
We fix the total number of parameters at Nw = 1600, 2400 and 3200. For each Nw value, we solve Problem (18) with FCNets-ReLU and FCNets-ABS, following the procedure described in the previous subsection, with the number of hidden layers L varying from 2 to 31 (with increment 1 up to 20 then increment 2 afterwards) for ReLU and extending to 41 for ABS. The results, including training and testing losses and accuracies, are summarized in Figure 8 containing 6 plots in 3 columns corresponding to the 3 values of Nw.
As previously shown, when networks grow deeper, variability first increases and then decreases. As we observe from Figure 8, there are remarkably similar patterns in all curves in Figure 8 for either training and test accuracies or losses (for which the rise-and-fall pattern is flipped).
In particular, we compare the left plot of Figure 2 with the third column of Figure 8. For these two cases, the numbers of total model parameters are sufficiently close (3300 vs. 4000) to roughly examine the locations of interests. Indeed, the peak of variability occurs at L = 12 in the left plot of Figure 2, while the best performance of the FCNet happens between L = 10 and L = 15, as can be seen in the third column of Figure 8.
We offer the following interpretations of the experimental results, as pertinent to the relationship between network variability and trainability.
• Variability in the data space indicates a model’s expressivity, and in turn, it implies sensitivity in the parameter space. A well-designed and computable measure of variability, which remains a topic of further research, could serve well as an indicator of trainability.
• With low variability, models apparently have more local traps, making training difficult. On the other hand, near or around variability peaks, there appears to exist few or no local traps, as evidenced by the middle sections in the plots of Figure 8 where the training process seems to reach global optima with few or no exceptions, at least so in over-parametrized models.
• As expected, over-parametrization helps trainability. However, it may not always benefit generalizations much or at all, as is evidenced by the curves of testing loss and accuracy in Figure 8.
• FCNet-ReLU is unable to reach near-zero loss values with more than 20 hidden layers, but FCNet-ABS still succeeds even after the hidden-layer number exceeds 30, confirming that ABS works better than ReLU in deeper DNNs.

heading: C HANNET

heading: C.1 HANNET ALGORITHM
text: Algorithm 1 HanLayer. Input: vector x and parameters (u, b). Output: y = HanLayer(x;u, b) 1: function HANLAYER(x, u, b) 2: z = x− 2 u
ᵀx ‖u‖2u+ b . Householder Reflection plus bias
3: y = ABS(z) . Absolute-value Function Activation 4: return y 5: end function

heading: C.2 MIXER BLOCK
text: Figure 9 presents two Mixer structures using FC-Layers and Han-Layers.

heading: D EXPERIMENT

heading: D.1 EXPERIMENT SETTINGS
text: Our Checkerboard dataset is similar to the one in Section 4 but more complicated, shown as Figure 4 with 6561 mesh points in a 12× 12 block checkerboard over the square [−1, 1]2. We consider, and our experiments have confirmed, that this dataset is a highly challenging one for standard DNNs.
{0.001, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 1}
and select the best result as the output. In addition, the initial learning rate is annealed by a factor of 5 at the fractions 1/2, 7/10 and 9/10 of the training durations. Other parameter settings for SGD are listed in Table 5 below.
For the two regression datasets, we choose Adam (Kingma & Ba, 2015) as the training method which seems to be the method of choice for several works in that area including (Tsang et al., 2018). For these image classification datasets, we choose a modified Adam called AdamW (Loshchilov & Hutter, 2018). Detailed settings are in Table 6 below.
All the training is done using PyTorch (Paszke et al., 2019) running on a shared cluster. Finally, we mention that we always run multiple times for each test instance, starting from different random initializations of model parameters. All the reported values are the average of at least 5 runs. It is noted that the results in each block on Figure 5 are the average of 9 runs. We randomly manufacture the train set 3 times and do 3 trials on each set.

heading: D.2 ABLATION STUDY ON CHECKERBOARD
text: To better grasp how much the two HanNets components, Householder weighting and ABS activating, contribute to the surprising results on the Checkerboard dataset, we conduct an ablation study with three different configurations in a 100 × 17 network framework. The results are listed in Table 7 where the layer type is either Householder (H) or fully connected (FC) and the activation type is either ABS or ReLU. Table 7 suggests that Householder weighting and ABS activating be equally critical to the 99% generalization accuracy of the HanNet. We investigate the heavily lacking of variability in the networks with Householder matrix plus ReLU function affects the model performance.

heading: D.3 FLIPPING TRAINING LABELS ON CHECKERBOARD
text: To check the stability of the near-perfect result of the HanNet, we randomly flip training labels from 0 to 20% on Problem 4. Figure 10 (a) presents the testing accuracy in HanNet and FCNet, where the network architecture is 100× 17 for HanNet and 100× 6 for FCNet. The training result for FCNet is given in Figure 10 (c), while Figure 10 (d) are, the best results for the HanNet in terms of testing accuracy. The best result shows that HanNet can still learn well the original pattern from seriously damaged labels.

heading: D.4 REGRESSION DATASETS
text: Table 1 presents the model specifications and statistics of datasets (Elevators and Cal Housing). Figure 11 shows the NRMSE values on two FCNets and HanNet.

heading: D.5 SETTINGS ON IMAGE CLASSIFICATION DATASETS
text: We summarize various configurations of each block on different data sets in Table 9. We run Mixers using the following number of layers {1, 2, 4, 8, 12, 16} and select the best one on each dataset. We use the convolution stem recommended by (Xiao et al., 2021) instead of the one in (Tolstikhin et al., 2021). Our convolutional stem designs use five layers, including four 3× 3 convolutions and a single 1 × 1 final layer. The output channels are [64, 128, 256, 512, 512] on CIFAR10, CIFAR100 and STL10, and [128, 256, 512, 1024, 1024] on ImageNet32.
E PLOTS FROM LANDSCAPE EXPERIMENTS
1e −7
+1
1e −8
+1

Review summaries:
The paper introduces an estimator of expressivity or trainability of deep networks, that they call "variability", that characterizes the variations in a scalar output of the network, when the input change, and seems correlated to trainability of the network, and to its depth (to an extent).  The paper proposes to parametrize weight matrices by using Householder reflectors, parametrized by a single vector each, which has two benefits: - A reduced number of free parameters, which enables more depth (more layers and non-linearities) for the same model size - Orthogonality of the matrices.  Experiments on a synthetic 2D benchmark, two small-scale regression datasets, and small-resolution image datasets show improved results compared to dense nets. 

**Contributions**: - Introduces the notion of variability to explain the better trainability and generalization of certain deep neural architectures. This is backed by a concrete definition of the variability in terms of the third order derivative information of the loss landscape.  - Empirical experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants (C2C).  - Inspired by the notion of variability, authors propose Householder-absolute neural layer aka Han-layer that replaces square weight matrices with Householder reflectors, and use absolute-value function for activations.   -  HanNet achieves a high variability as well as guarantee an immunity to vanishing or exploding gradients, and chance for collapse to constants has been diminished.  - HanNets achieve better generalization with lesser parameters than fully connected networks.  

This paper introduces some concepts, i.e., variability, activation ratio, and collapse to constants, to help understand deep neural networks, and proposes a new layer called Householder-absolute layers to improve network variability. Experiments are conducted on both synthetic datasets and empirical datasets. The synthetic experiments show some interesting results and visualizations, and for the real datasets, improvements over baseline architectures can be observed.

The paper introduces a new notion of _variability_ of a DNN, that intuitively is the "richness of landscape variation" wrt data and parameters. The paper attributes the _variability_ to the generalization performance of fully connected DNNs. Inspired by the increasing _variability_ the work proposes _Han-layer_ that is a linear layer with a Householder matrix and abs(.) activation function. The empirical results show that _Han-layer_ can improve the performance of MLP-Mixer on various datasets.