Variability of Neural Networks and Han-Layer: A Variability-Inspired Model
Authors: Yueyao Yu
Yin Zhang
Venue: ICLR 2022 Submitted (2022)

Abstract: What makes an artificial neural network easier to train or to generalize better than its peers?  We introduce a notion of variability to view such issues under the setting of a fixed number of parameters which is, in general, a dominant cost-factor.  Experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants, which is related but not identical to vanishing gradient.  Further experiments on stylized problems show that variability is indeed a key performance indicator for fully-connected neural networks.  Guided by variability considerations, we propose a new architecture called Householder-absolute neural layers, or Han-layers for short, to build high variability networks with a guaranteed immunity to gradient vanishing or exploding.
On small stylized models, Han-layer networks exhibit a far superior generalization ability over fully-connected networks.  Extensive empirical results demonstrate that, by judiciously replacing fully-connected layers in large-scale networks such as MLP-Mixers, Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance.  We will also briefly discuss current limitations of the proposed Han-layer architecture.


Under review as a conference paper at ICLR 2022
VARIABILITY OF NEURAL NETWORKS AND HAN-LAYER: A VARIABILITY-INSPIREE MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
What makes an artificial neural network easier train or to generalize better than its peers? We introduce notion of variability to view such issues under the setting of fixed numher of parameters which is, in general, dominant cost-factor. Experiments verify that variability correlates positively to the number of activations and negatively to phenomenon called Collapse to Constants, which is related but not identical to vanishing gradient. Further experiments on stylized problems show that variability is indeed key performance indicator for fully-connected neural networks. Guided by variability considerations, we propose a new architecture called Houscholder-absolute neural layers, or Han-layers for short, to build high variability networks with a guaranteed immunity to gradient vanishing or exploding. On small stylized models, Han-layer networks exhibit a far superior generalization ability over fully-connected networks. Extensive empirical results demonstrate that, by judiciously replacing fully-connected layers in large-scale networks such MLP-Mixers, Han-layers can greatly reduce the number of model parameters while maintaining or improving generalization performance. We will also briefly discuss current limitations of the proposed Han-layer architecture.
INTRODUCTION
1.1 OVERVIEW
Deep neural networks (DNNs) have greatly advanced the state of the arts in many machine learning tasks such image classification, text categorization, speech recognition, to name just few out of long list. Despite their tremendous successes, it remains to be fully understood why DNNs work so well on so many tasks in machine learning. In this paper, we take an intuitive approach to providing a new angle from which one can qualitatively investigate certain behaviors of DNNs and provide explanations to some critical issues. Our new angle is based on a notion of variability for DNNs which seems to have not been specifically examined in the past. Our study of variability focuses primarily on function (forward propagation) values rather than on the gradient (back-propagation) values, offering a different insight from gradient- madde views.
In essence, the proposed variability is qualitative surrogate of expressivity or expressiveness for neural networks (see recent survey paper (Gühring et al., 2020) and references thereof). Instead of giving precise, quantitative analysis on what kinds of functions can or cannot be expressed by certain neural network models which in general can be exceedingly difficult, we opt to give qualitative characterization that, to a degree, reflects the levels of expressivity, thus sidestepping hurdle in analyses while still providing useful information on the expressiveness neural networks. In doing so, we aim to find useful guidelines for developing new network architectures.
We show that for fully connected and Rel.U-activated DNNs with a fixed number of model parameters, the higher variability (within reasonable range), the easier is train the model to reach high- quality solutions. In particular, variability rises with a quantity called activation ratio, AR (see (5)

---

Under review as a conference paper at ICLR 2022
for definition), and falls with the occurrence of phenomenon called Collapse to Constant (C2C), which is not identical (but related) to gradient vanishing. Guided by the variability viewpoint, we propose novel neural-layer design aimed to increase AR and decrease the chance of C2C at the same time.
1.2 MAIN CONTRIBUTIONS
The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models. We study the notion of variability that reflects DNNN' expressivity and trainability, Guided by variability, we construct a new architecture called Householder-absouute neural layer or network (Han-layer or HanNet) to attain higher variability with fewer parameters. As the name suggests, a fully connected layer or network (FC-layer or FCNet), we replace square weight matrices with Householder reflectors and use the absolute- function for activations. To sum up, our contributions are both conceptual and practical, consisting of the following aspects.
We study the notion of variability, along with measurement for it, for DNNs that adds new angle to view and explain the behaviors of DNNs. We construct HanNet to achieve high variability and at the same time guarantee an immunity vanishing or exploding gradient. The chance of Collapse to Constants, yet not completely eliminated in theory, has also been greatly diminished.
On stylized small problems, our experiments suggest that variability is indeed a key performance indicator for ReLU-activated FCNets and that HanNets possess an unusually high level of generaliza- tion ability. As is illustrated in Figure 4 in Section 5.2.1 HanNet outperforms FCNets by huge margin, producing nearly perfect results.
On several standard datasets in regression and image classification, our experiments indicate that comparing FCNets, HanNets greatly reduce the number of model parameters, sometimes by orders of magnitude, while still maintaining and often improving generalization performance. This property represents promising feature with potential impacts on large-scale applications.
Many important issues remain to be investigated in order to better understand the power and the limits of HanNets, and to unlock their potentials worldworld applications.
2 RELATED WORK
As is well known, initialization is critical step towards the success or failure of neural network training. We start our investigation by visualizing landscapes of DNNs with properly initialized parameters with different activation functions. Landscapes of DNNs have been examined from various angles and for different purposes, for example see (Kawaguchi) 2016 Li et al. 2017 Laurent & Brecht 2018 Kuditipudi et al. 2019: Fort et al.) 2020 Sun et al. 2020). In our case, visualization results lead to the concept variability. The two key building blocks our proposed HanNets are (1) the absolute-val valuue (ABS) function as an activation function and (2) the Householder reflectors the orthogonal weight matrices two topics that previously appeared in literature within quite
different contexts from ours; for example, ABS functions in (Batruni 1991 Lin & Unbehauen 1992
Karnewar 2018) and Householder matrices in (Mhammedi et al. 2017 Tomczak & Welling 2016
Vorontsov et al. 2017 Wang et al., 2020 Wisdom et al. 2016 Zhang et al. 2018) In addition, other
related works will be referenced throughout the paper.
2.1 NOTATIONS
Given activation function Φ(-) and sequence of weight-bias parameters b:)) of suitable sizes, we first define layer functions
(/a(-) (.. b.) == W:o(-) + b;, i 1,2,
each of which is the composition of affine function with the activation o(-). As mentioned above, often drop the dependence of v/s the parameter pair (W, by) whenever confusion arises. For
2

---

Under review as a conference paper at ICLR 2022
any positive integer k. define
Fx(x) == (th ..... ob))
(2)
which is the composition of Wi,..., Pk. parameterized by the weight-bias pairs (W, b.) for i
1, 1. k. In general, W; E IR*+*xm 6, E R and Φ; R - I thus function (x) is
from R to _ For an L-layer DNN, the model output is Fr, W, b). where W = (W1, Wr and b = fb,,...,by b. For any given parameter pair, the network maps an input x to an output (z, W, b). which can be computed through the forward propagation:
Sg z;
2k Wk Sk = o(zk), 1.......4
(3)
3 RISE AND FALL OF VARIABILITY
We start with set of simple experiments in Section 3.1 observe the landscapes of neural network functions (2) as the network depth L increases while the total number of model parameters is kept approximately unchanged.
3.1 LANDSCAPE EXPERIMENTS
To facilitate visualization, add input layer and output layer, both of dimension 2, the L hidden layers. For convenience, continue use (x)) to denote the extended network. The data set is set of grid points the square [-1, 1] C IR2 over which we compute and plot the surface = F2 (I, W, b) for randomly sampled (W, b) with proper scalings. Three activation functions, Sigmoid ((t) = 1/(1 + e) ReLU @(() = max(0, t) and ABS Φ(t) = |t]. are used. For ReLU we use Kaiming initialization, and for other two functions we use Xavier initialization (Glorot & Bengio
2010
He et al. 2015).
We keep the total number of parameters in (W. b) around 10,000. For each given depth value L, caleulate the corresponding width d (rounded to the nearest integetr), then compute and plot over the grid on [-1, 1] for 5 different samples for (W, b). Figure 1 is the result of one single random sample, while the complete plots are given in Appendix (see Figures consists of 5 plots corresponding to 5 random parameter samples).
13 and 14 where each row
Sigmoid
ReLU
ABS
HanNet
Figure 1: Landscape of (x, W. 2. Rows I to 3 are for FCNets with 3 activations: Sigmoid
(L = 2, 4, 6, 10, 15). ReLU and ABS (L- = 2, 10, 20, 40, 60). Row 4 is for HanNets (to be defined) with the same L values as for ReLU and ABS.
Based on these plots, we make the following observations. For Sigmoid, the surfaces are rather monotonous with few variations either in the z-space (within each plot) in the parameter space
3

---

Under review as a conference paper at ICLR 2022
(across the 5 plots). In particular, in the x-space, we hardly see any peaks and valleys. Most remarkably, with slightly larger I. values, the surfaces become constants. We call this curious phenomenon "Collapse to Constants" or simply C2C. For ReLU, the landscapes have much richer expressions from the very beginning (L = 2). The amount of variations increases as L grows deeper from to 10 and even to 20. However, at I. == 40 and 60, the phenomenon of C2C again shows up. ABS Activation seems more C2C-resistive than ReLU. Many peaks and valleys still exist at deep depths (L = 40, 60).
Upon further examinations, clear that for large L not only the scalar function [z W, W)]l tends to constants in r-space, but in fact the vector-valued function (2, W, b) itself tends to constant vectors for all [-1,1]². We will explain this C2C phenomenon later.
3.2 VARIABILITY AND A MEASUREMENT FOR
From the above experimental results, we see unmistakable differences in the outputs of network functions Fr (z, W, b) from one activation function another and the depth L grows. All the differences are rooted in the richness of landscape variations in the data space where æ varies, as well as in the parameter space where (W, b) varies. We will refer to such richness of landscape variations as variability of network functions.
Intuitively speaking, higher variability means not only higher expressiveness in data space, but also better responsiveness to parameter changes. On the other hand, a network with low variability means that it may have a poor approximation power and consequently may be difficult to train. In particular, a deep network would have lost all variability, in certain regions of the parameter space, when becomes constant in the entire data space for parameters from those regions, which we refer to as Collapse to Constant (C2C). In Appendix A.4 we treat C2C and its characteristics with more details. Variability may serve predictive indicator on trainability of suitable DNNs. We present numerical evidence in Appendix B verifying that the pattern of variability is highly correlated to the performance of corresponding DNNs.
In general, it is nontrivial to develop effective and computable measures for variability, which should be a subject of investigations on its own right. Here we present a variability measurement that seems to have worked well with wwo-dimensional data spaces. We define
V3 = (WW)) (I111" /
A Ar
dr
where f = (a; b)) scalar-valued function, the mean is taken over a certain region of interest in the parameter space, and A / is a finite-difference approximation of the third derivative over grid. The phenomenon of C2C occurs when V3 is close to zero. For example, if is least squares loss function and (-)-) is linear, then V/3 vanishes. On the other hand, when F-).) has high nonlinearity, then the value V33 will be relatively large.
In Figure we plot the variability of Fz; W) on Ω = [-1,1]² computed as described in Appendix A.2 for network depth I. varying from 3 to 45 (with increment 3). As we see on the left plot for ReLU, the Vg-variability initially increases with L until it reaches its peak at around = 16. Afterwards, it starts to decline until it vanishes suddenly at = 25 because the network output becomes constant at least at one parameter sample, making the geometric mean zero. On the middle plot for absolute-value, besides slightly higher profile, after the peak variability declines but does not vanish up to L = 60.
3.3 RISE WITH ACTIVATION RATIO AND FALL WITH C2C
should be clear that variability comes from nonlinear activations in the model. As we see from the previous variability experiments, variability initially always rises the network depth increases until reaches a peak. There is a simple explanation for this. That is, when the total number of parameters is fixed, the number of activations always increases with the depth. The activation ratio (AR) for
4

---

Under review as a conference paper at ICLR 2022
200
Dgg
Biohh
(a) FCNet-RRe.U
(b) FCNet-ABS
(c) HanNet
Figure 2: Variability measured by V/3 in (4) for FCNet-ReLU (left) and FCNet-ABS (middle) and HanNets (right). Each bar represents a geometric mean 3000 parameter samples. As depth grows, the width d decreases that the total number of model parameters is approximately 4000. The structure of HanNets is the same as that of FCNets with far fewer model parameters.
model is defined below
total numbee of individual octivations AR
(5)
total number of model parameters
The activation ratio is monotonically increasing with depth L. In particular, the rate of increase is the largest at L 1, which shows that the activation ratio increases most rapidly at the beginning. explaining why variability rises quickly at the very beginning, see Appendix A.3 for a formula and more details. In this work, we will also use AR one of the metrics for variability. However, we caution against the simplistic view that the higher the AR or variability is, the better. In fact, experiments indicate that excessive nonlinearity tends to increase training difficulties.
should be clear that variability declines and eventually vanishes as C2C develops and materializes. detailed treatment, including a quantitative characterization of C2C, is given in Appendix [.44
4 HANNET: A HIGH VARIABILITY MODEL
path to enhancing and maintaining variability of a DNN is raise its resistance level to C2C and increase its AR simultaneously. Our construction of Han-layers follows exactly this path.
4.1 HOUSEHOLDER WEIGHTING: ORTHOGONAL AND HIGH AR
is well-known that Householder reflection matrix associated with a nonzero vector u E R is
(uu = - 2wu SuuuT|len /||u
(6)
which is symmetric and orthogonal. As is well known, orthogonality is desirable property for weight matrices in DNNs. In addition, the degree of freedom in H (u) is n, one order of magnitude smaller than (n²²) for generic orthogonal matrix.
Evidently, the activation ratio of a DNN can be increased by reducing the number of model parameters without changing the network depth and width, such as by parametrizing an n X n weight matrix with far less than (n²²) parameters (similar to what is done in CNNs). This motivates us to use Householder reflection matrix to replace a general matrix or a generic orthogonal matrix in a neural layer that the AR value associated with this layer is enlarged by factor of (n)). In natural language processing areas, it has been proposed to product of multiple Householder matrices to parameterize weight matrices in recursive neural networks, for example (Mhammedi et al. 2017
4.2 HAN-LAYERS AND HIGH VARIABILITY
Householder-absolute neural layer, or Han-layer, composed of Householder matrix followed by ABS activation (see Appendix C.I for an algorithmic form), and a HanNet structure created by using multiple Han-layers. We have already visualized the landscape of deep HanNet in Figure [ which has more peaks and valleys than the others implying higher level of variability. In Figure
5

---

Under review as a conference paper at ICLR 2022
2
calculate the variability measure V3 defined in 4 where the geometric mean is used that vanishes whenever C2C occurs. We observe that the variability of HanNets remains at a high level without deterioration in the entire tested range.
4.3 IMMUNITY TO GRADIENT VANISHING OR EXPLODING
Let the neural network function (z, W, b) be defined in 2 The spectrum of so-called Gz- (see (12) in Appendix (A.4) characterizes whether vanishing or exploding gradient would happen or not. Roughly speaking, G -matrix is the dominant part of the gradient. As L + co, vanishing gradient corresponds to GL - 0 and exploding gradient to GL - oo. For HanNet, with the notation [ux]k=> the GL Gm-mariix takes the form
L Gz(r, u, II V(()))
where (uu) is defined by (6) and Vo((z) is the diagonal, Jacobian matrix d(() = |4] evaluated component-wil at a vector zg. Evidently, the diagonal matrices Vø(z+) ) are, with high probability, all orthogonal matrices with diagonal entries equal +1. Hence, the following result holds implying that gradient never vanish or explode (in probability sense).
Proposition 1. Under a mild distribution assumption, the G-matrix for HanNet, that is, GL u, b) defined in 7 remains orthogonal with probability one for any I, any us + 0 in u, any b, and any integer 0.
4.4 HAN/MLP-MIXER
In literature, the term Multi-Layer Perceptron (MLP) is often used exchangeably with FCNet. Recently, ML--nommnated models have seen a wave revivals for image recognition tasks (Tolstikhin) et al. 2021: Liu et al. 202T). MI.P-dominated models (without multi-head attentions) are much more concise than Transformer-basee models (Dosovitskiy al., 2020; Touvron et al.) (202)) but can still maintain testing performances on very large-scale datasets. The motivation of MLP-Mixer (Tolstikhin al. 2021 is use the purely fully-comnected layers to remove attention architectures. An MLP-Mixer block is the elementary unit in MLP-Mixe models that consists of several FC-layers and skip-connections to form the following map from input X to output Y.
Z X + GELU (W₂ Layer Norm(W, (V)) y= Z + GELU (Layer Norm (ZW;) Ws),
(9)
where the first row 8 is called token- biennmixing for cross token communication and the second row (9) is called channel-mixing for cross-channel communication, both being of MLP structure. Here we form our Han-Mixer block by replacing all weight matrices W, by Householder matrices H,, i = 1, 2, 4, and all activation functions by the absolute function ABS, that is, Y
(10) where we remove skip-connections and layer-normalizations (since HanNets does not suffer from gradient problems). See Appendix C.2 for figurative illustrations MLP-Mixer vs. Han-Mixer blocks. The resulting Han/MLP-Min models are shown Figure where we arrange some Han- Mixer blocks after MIP-Mixer blocks (which may be empty). The main reason for to combine the two types of Mixer blocks that, short of drastically increasing network width, Han-Mixers alone cannot always provide enough model parameters for large-scale datasets. In addition, convolution stem recommended by (Xiao et al. (02I) instead of the one in (Tolstikhin et al. 202T).
Z ABS( (22HHHHH
the
EXPERIMENTS
5.1 DATASETS
To empirically investigate the efficacy of HanNets in comparison FCNets, use multiple datasets including synthetic dataset Checkerboard, 2 classic regression datasets, Flevators (Dual

---

Under review as a conference paper at ICLR 2022
convolution stem
Mixer Block Lu
Hon Block x
Linsar Classifier
3,30o00 COIY sinde of
Global Avg
stide
Pooking
class
conv
Figure 3: Overall structure of the tested Han/MLP-Mixer model (Las can be zero).
Graff 2017) and Cal Housing (Pace & Barry 1997). and 4 widely used image classification
datasets CIFARIO, CIFAR STL10, and ImageNet32 (Krizhevsky al.] 2009; Coates al.
2011
Chrabaszez et al. 2017), where the last is down-sampled version of ImageNet (out of
affordability considerations). All dataset settings and training details are given in Appendix D.]
5.2
RESULTS
5.2.1
SYNTHETIC DATASET: CHECKERBOARD
The ground truth and the training set, along with a couple of typical results, are plotted in Figure
050
A09
025
835
300
(a) Training set (25%) (b) FCNet: 85.2% test acc (c) HanNet: 99.5% test acc
Figure 4: The top row consists functions (from left to right): the target binary function, a function trained from FCNet and one from a HanNet, respectively. The bottom row: (a) the training set with 25% data points, (b) and (c) are top views of the two trained solutions (rounded to 0 or 1), along with their test accuracies. The training accuracies are nearly 100% for both.
On this dataset, we compare a range of FCNets with HanNets (in fact, ResNets were also compared, without better results than those from FCNets). HanNets with width varying from 20 to 100 (increment 10) and depth from 30. Figure (a, b) gives the test results in the heat-map form. We observe from Figure 5 that overall there exists large gap in testing accuracy between the FCNets (around 85%) and HanNet (over 99%) range of test cases. The unexpected, near-perfect results obtained by HanNets, without any explicit regularization, look quite stunning and are stable in the sense that similar results can be trained from other random 25% vs. 75% data-splittigss. We observe that over over-paramen parametrization is not necessary condition for near perfect solutions. In fact, 400 parameters are sufficient to achieve near-zero testing error in one of the cases in Figure ()...
generate more than 200 pairs FCNets and
wide
ablation study in Appendix D.2 suggests that Householder weighting and ABS activating be equally critical the surprising performance HanNets on the Checkerboard dataset. In addition, D.3ddixX.33p stability study in Appendix D.3 is done in which portion of training labels randomly flipped, and the results show that with seriously damaged labels HanNets still maintain clear advantage
7

---

Under review as a conference paper at ICLR 2022
0.85
-.88
9050
0.50
0.83
0.72
0.72
0.03
0.54
0.54
0.75
Widd
Depah
Figure 5: (Left) FCNet: the best testing accuracy is up to 87%. (Mid) HanNet: the best testing accuracy is over 99%. (Right) FCNet vs HanNet: the number of model parameters in each line is fixed, respectively, at 4000, 8000 and 20000 for FCNet-s, FCNer-m and FCNet-I, and at 1000, 3000, 5000 for the lines for HanNets. all experiments, the training errors are near zero. The plot shows the existence of significant gaps generalization ability between FCNets and HanNets.
FCNets in testing performance. In addition, extensive experiments show that variability is closely and positively related to trainability (see Appendix for details).
5.2.2 REGRESSION DATASETS: ELEVATORS AND CAL HOUSING
On the two regression Datasets, compare the performance of HanNet with FCNets. Table T lists the relevant statistics for the DNNs where depth refers the number of hidden layers (there exist additional, data-sized sete-iize-deenneent input/output layers). We see that in terms of parameter numbers FCNetl and HanNet are comparable peers, while FCNet2 has about 15 times more parameters.
Model
FCNet1
FCNet2
HanNet
Depth x Width (#param)
5 x 50 (10.9K) 5 x 200 (165K)
20 x 200 (10.6K)
Table I: Essential statistics of DNNs compared (more details in Appendix D.4.
We present testing NRMSE (normalized root mean square error) loss values in Figure in Ap- pendix D.4, where the curves are average values over trials with error bars. Clearly, HanNet outperforms its "peer" FCNetl by notable margin, especially when fewer training samples are used, and competitive with (on 80% training data) or better than (on 20% training data) FCNet2 which uses 15 times more parameters. We mention that the best testing performance of HanNet statistically the same as that reported in (Tsang et al., 2018) with an FCNet model that uses over 5 million parameters. Additionally, with fewer parameters, HanNet appears less influenced by overfitting.
5.2.3 IMAGE CLASSIFICATION DATASETS
In this section, we investigate HanNets' generalization ability in image classification by comparing MI.P-Mixers with our Han/MI.P-Mixers presented in Subsection 4.4 The comparisons are done with standard image datasets and a down-sampled version of ImageNet (due to computing source limitation). More experimental details given in Appendix D.5
Empirical evidences from MLP-based models (Tolstikhin et al. 2021) suggest that these models can achieve the state-of-the-ard results on very large-scale datasets. Since the datasets used in our tests are not large enough to play to the full strengths of Mixer models, our aim here is not to observe how close our results can approach the state-of-the-art but how Han-layer structures can impact the performance of Mixer models.
Tables 2 and 3 report test results from various MI.e-Mxxrr and H/M-Mixer models. On the smaller dataset STL10, HanMixers alone can already produce the best result. On larger datasets CIFAR10 and CIFAR 100, enhanced performance from HanMixers are achieved either by adding relatively few parameters (e.g., MI.PMixer (2 or ML.Miixr (2 4) HanMixer (16)), with
8

---

Under review as a conference paper at ICLR 2022
#Param
STL10
CIFARIO
CIFARI00
CNN stem
1.82 M
18.2
7.2
27.8
+MLPMixer (2)
2.89 M
18.2
5.3 %
27.2 %
+MLPMixr (4)
3.96 M
18.7 %
5.5 %
27.1 %
+MLPMixer (0) HanMixer (16)
1.86 M
15.3 %
5.9 %
26.7 %
+MLPMixer (1) HanMixer (16)
2.39 M
16.7 %
5.0 %
24.6 %
+MLPMixer (2) + HanMixer (16)
2.93 M
17.1 %
4.7 %
24.4 %
+MLPMixer (4) + HanMixer (16)
4.00 M
17.8 %
4.6 %
24.7 %
Table Error rates (%) on CIFAR and STL10 datasets.
#Param
Topl emror
Top5 error
CNN stem
8.28 M
55.7 %
31.2 %
+MLPMixer (4)
16.7 M
44.2 %
21.2 %
+MLPMixer (8)
27.2 M
42.6 %
20.2 %
+ +ML.Mixer (0) HanMixer (16)
8.35 M
48.9 %
26.1 %
+MLPMixer (4) HanMixer (16)
16.8 M
41.1 %
19.2
WideResNet (Chrabaszez et 2017
37.1 M
41.0
18.9
Table The top-1 and top-5 error rates on ImageNet32. On this dataset, among MLPMixer (i) for i 4,8, 12, 16, gives the best results,
far fewer parameters (e.g., MLPMixer (4) MLPMixer (1)+HanMixer (16)). On ImageNet32, our Han/MLP-Mixer combination model clearly outperforms pure MLPMixers and offers a competitive performance with WideResNet while using only 40% parameters. In summary, the benefits of using ManMixers have been made evident this set of experiments.
CONCLUSIONS
As qualitative surrogate for expressivity and trainability, variability provides useful angle to view certain critical behaviors of DNNs. Empirical evidence suggests that activation ratio and Collapse Constants two major contributing factors to the gain and loss of variability, respectively. Based on such insights, propose Han-layer architecture consisting of Householder weighting and absolute-value activating. This new architecture greatly raises activation ratio and provides guaranteed immunity to vanishing or exploding gradient.
Extensive experimental results indicate that when used judiciously as replacements of fully connected layers, Han-layers can significantly reduce the number of model parameters (thus potentially the associated computing costs), while maintaining or improving the level of generalization performance, as is shown by our results on MLP-Mixer models for image classification.
On the synthetic Checkerboard dataset, HanNets consistently exhibit surprisingly high level of generalization ability. It remains be investigated that under what conditions and to what extent such ability could be realized large-scale datasets.
Several limitations in our results are worth mentioning, So far, our measurement of variability is limited to low-dimensional data spaces. It remains to be seen whether or not a computable extension to high-diig figh--iimnsional spaces can be fruitfully done. As is currently defined, Han-layer is not as flexible as fully connected layer since a Householder matrix must be square. Finally, we offer two points of perspectives: (1) measured variability should be viewed within reasonable range (chaotic landscapes generally bad); and (2) too many Han-layers, even though immune to gradient problems, may still increase the difficulty in training due high nonlinearity.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Roy Batruni. A multilayer neural network with piecewise-lineal structure and back-propagation learning. IEEE Transactions Neural Networks, (3)):395-403 1991.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint urXiv:1707.08819 2017.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-222222 223. JMLR Workshop and Conference Proceedings, 2011.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale, In International Conference Learning Representations, 2020.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL ics. uus.ucii.edum..
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. arXiv preprint arXiv:2010.15110, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ingo Gühring, Mones Raslan, and Gitta Kutyniok. Expressivity of deep neural networks. arXiv preprint arXiv:2007.04775 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Conference on Learning Representations, 2019.
Lei Huang, Xianglong Liu, Bo Lang, Adans Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR.
Animesh Karnewar. Aann: Absolute artificial neural network. In 2018 3rd International Conference for Convergence in Technology (I2CT), pp. 1-6. IEEE, 2018.
Kenji Kawaguchi. Deep learning without poor local minima. arXiv preprint arXiv:1605.07111 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster),
2015.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, and Rong Ge. Explaining landscape connectivity of low-cost solutions for multilayer nets. arXiv preprint urXiv::9006 06247, 2019.
Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International conference on machine learning, pp. 2902-2907. PMLR, 2018.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv: 17 12.0991, 2017.
J-N Lin and Rolf Unbehauen. Canonical piecewise-linear approximations. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 39(8):697-699, 1992.
Hanxiao Liu, Zihang Dai, David R So, and Quoe V Le. Pay attention to mips. arXiv preprint arXiv:2105.08050, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2018.
Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal
parametrisation of recurrent neural networks using householder reflections. In International Conference on Machine Learning, pp. 2401-2409 PMLR, 2017.
Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Letters, 33 (3):291-297, 1997.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 2019.
Ruoyu Sun, Dawei Li, Siang, Tian Ding, and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5):95-108, 2020.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, al. Mip-mixer: all-mip architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
Jakub M Tomezak and Max Welling. Improving variational auto-encoders using householder flow. arXiv preprint arXiv:6611.06633 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Michael Tsang, Hanpeng Liu, Sanjay Purushotham, Pavankumar Murali, and Yan Liu. Neural interaction transparency (nit): Disentangling learned interactions for improved interpretability. In NeurlPS, pp. 5809-5818, 2018
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning, pp. 3570-3578. PMLR, 2017.
Jaayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11505-11515. 2020.
11

---

Under review as a conference paper at ICLR 2022
Scott Wisdom, Thomas Powers, John R Hershey, Jonathan Le Roux, and Les E Atlas. Full-capacity unitary recurrent neural networks. In NIPS, 2016.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106. 14881, 2021.
Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning, pp. 5806-5814, PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2022
A VARIABILITY
A.1 EXISTING TECHNIQUES THAT ENIIANCE VARIABILITY
Several existing techniques in deep learning can be interpreted from the viewpoint of enhancing the variability of neural networks. For example, convolutional neural networks (CNN) use far fewer parameters, in comparison to fully connected networks, at each layer, thus significantly increasing activation ratios and subsequently enhancing variability. In this view, achieving higher actívation ratios should be a significant contributing factor to the great success of CNN.
Since C2C has close relationship with vanishing gradient, is not surprising that existing techniques designed to alleviate the latter can also help with the former. Specifically, since G-matrices are limits of C-matrices, techniques that slow down the size decrease of G-matrices usually also slow down the size decrease of C-matrices. Such techniques include Residual Networks (or ResNet) (He et al. 2016) and Batch Normalizations (loffe & Szegedy 2015). A particularly simple and widely utilized technique is to initialize weight matrices by orthogonal matrices (Hu et al. 2019
Huang et al. 2018).
A.2 SETTINGS OF V3 IN FIGURE
In our experiment, we use the least-squares function: f(-) := -1 and S 1-1, The mean over parameters will be computed vanished if single sampled value is zero. Therefore, once (2; W, b) becomes constant (or linear for that matter) on Q variability as measured by V3 vanishes.
sample mean. In particular, we use the geometric mean which
A.3 INCREASE OF ACTIVATION RATIO
For simplicity, let us consider fully connected (FC) network of layers for which the number of parameters (W. b) is Nw = (&f + d where d is the width of the layers. this case, the number of activations is Ld. The activation ratio (AR) p for this network is the total number of activations divided by the total number of parameters, i.e., p Ld/Nw> which can be written function of L. By solving the equation N (d + L for d, we obtain d = (1 + VI + 4N L)/2. Therefore, the activation ratio is
P(L) = N₂ (L+ LL+VL2 12 + ALNN) L21.
(11)
Furthermore, since
1
P'(L) = No V1+ 4N
0,
the activation ratio is monotonically increasing with L. particular, the rate of increase is the largest at L = 1. This simple calculation shows that the activation ratio increases most rapidly the beginning, explaining why variability arises quickly at the very beginning.
A.4 COLLAASE TO CONSTANTS AND ITS CHARACTERIZAATIO
To compute the derivative of Fz (x. W, b) with respect to the parameters, one uses the chain-rule to obtain so-called back pack-propagation formulas, such as
i Fz(Z. W. b)
ŌSL 0b,
Vo(z)) -VVVO(z))
where zy are computed in 3 VΦ(zz) are diagonal matrices with scalar-valued ø' applied component- wise. It is well-known that the behavior of the derivatives is critically determined by the properties of the above matrix product. For convenience and without loss of generality, we add W, to the product and define
GL = GL W,b) = I W Vo(z))
(12)
13

---

Under review as a conference paper at ICLR 2022
which will simply call the G-matrix associated with the network (z,z, W. is well known in deep learning that excessively small (or large) size of Gz, causes vanishing (or exploding) gradient, which is major source of difficulty in training.
Now we define another matrix product called the C-matrix, by replacing the derivative V(((z)) in by the finite difference at two points 2 and žgi that is,
Cr. = Cr,a,, W. II W Do(=E, k=1
(13)
where Do(s.) E dxd is diagonal defined by
[Da(u, = [Ø(u) -()] [u-o], vj.
i i= 1,-....
(14)
with the convention 0/0 = 1, {=x} and {Ex} } computed via the recursion starting from æ and 3. respectively. By their definitions, it is clear that G-matrices are limits of C-matrices.
The following proposition shows that C-matrices characterize the C2C phenomenon.
Proposition 2. Let network FL W, Rd R& be defined in For any two distinct points F. E RR, there holds
FL(x) (Z) (x, (x T).
(15)
Consequently.
= lim CL(I,T) () = lim (F)(z) FL(2)) = 0.
(16)
1000
The verification of this proposition is straightforward, so we omit
We note that the difference going zero in (16) does not imply that each individual sequence goes to the same limit. On the contrary, such limits generally do not exist if the bias sequence {bu} is bounded away from zero.
Regarding the C2C phenomenon, the following remarks are in order.
Wherever Cr 7) is sufficiently small, the output values of the network for these two inputs and E will be close to each other for large L.
(r, Ø) will be small if, for all many k. I/ Do(=u, Ex)ll are sufficiently smaller than in some norm which occurs when one or both of the matrices is sufficiently small.
If Cr. (ar, .)) sufficiently small for all x in some region, then the corresponding outputs of the network will be like constant this region. In particular, this can happen when weight matrices in }W&} are small for all many k.
the weight matrices WE, & 11 I., are properly normalized (for example, all W; are orthogonal matrices), then the size of Ct will be solely determined by that of D Σx) for given point pair (1r, .). which in turn depends ((t) ma 1) and, again for the purpose of contrast, the absolute-value function ((t) |t]- In both cases, the diagonal entries in (14) satisfy [Dou,, v)]u [-1,
activation in use. We consider the popular ReLU activation
Proposition
Suppose that u, Rdd be i...d. random variables with
Prob[[u], 2 0) = Prob ([u) 0) = E (0, 1), i= 1,2,,....
Then
2d
(t)) = max t)
Prob (lo(u) - [(v)) = 1u=/)) =
(17)
(p" +11 2)
((t) = |t|
where the absolute values taken component-wise.
14

---

Under review as a conference paper at ICLR 2022
Proof. Consider the scalar case d = with + v. For ReLU function <(t) max((0 t),
((u) {
1, u,
1, otherwise
where the probability for the first case (ratio equal to 1) is p². For absolute value Φ(t) = |t],
Φ(u) - = {
we 2 0 < 1, otherwise
1,
where the probability for the first case (ratio equal to is p² (1 p)
Since all the components are i.i.d., by raising the above probabilities to their d-th power, we obtain the corresponding probabilities for the vector case for d > 1.
The proposition indicates that the probability for ReLU to preserve distances in R4 is much smaller than that for absolute- value. For instance,
when p 1/2 the above two probabilities in [17 become 1/44 and 1/2", respectively, this is, the latter is 2d times larger than the former;
when p = 1/4 the above two probabilities
become (1/16)4 and (10/16)4, respec-
tively; this is, the latter is 10d times larger than the former.
Remark 4. When d is large, the probability is small for either function to maintain diagonals [[Do(u, v)]al = for all indices i. That is, probabilistically speaking, the two functions are contractive, i.e., with a high probability, they shrink the distance between points after each application. while ReLU more forceful contraction than absolute-valee is, especially when the positive probability p of input is small.
In Figure we present a couple of typical experimental results on C- and G-matrices. The left plot for ReLU activations only with random weight matrices and bias vectors specified, then multiplied by the Xavier/Kaiming initialization constant /2/d. For randomly select point pair (zr, compute the corresponding C-matrix, well as the G-matrices these two points, for depth L from to 1000. The curves depict the spectral norms of the matrices Ct. and Gy. (two of those) at each depth value L. As we can see, although for L three curves roughly follow similar pattern of up-and-downs, the C-matrix still differs significantly from the two G-matrices in that it becomes about two-orders magnitude smaller than the two G-matrices at the same depth. This signifies an important point that, at least in some cases, C2C should be considered as the leading cause for loss of trainability, instead of vanishing gradient as commonly perceived.
Normal Weight Matrices (d = 150) ReLUU
Weight Matrices 50)
ReLU C-nati
G-mminl
ABS C-merris
munik2
10
(omi()
10a
1
10a
Corn(()
10
2
10"5
10"
10:10
200
400
600
800
1000
200
400
600
800
1000
number layers
layers
Figure 6: Sizes of C- and G-matrices for ReLU and absolute-value activations
15

---

Under review as a conference paper at ICLR 2022
The right plot of Figure absolute-vale value. Starting from random point-pair, we plot the C-matrices computed on set of random weight-bias pairs as specified with initialization multipliers 2 and v1/d for ReLU and absolute-value, respectively. The plot shows clearly that ReLU, with considerably smaller C-matrices, much more prone to the negative influence C2C than the absolute-value is. At I. = 1000, the C-matrix for ReLU is more than three orders of magnitude smaller than that for absolute-valuc.
6 for the comparison of two different activation functions, ReLU, and
B VARIABILITY VS. TRAINABILITY
As have seen, for fully connected neural nets with fixed number of parameters, variability is initially small, due to low activation ratio values, and then it rises to peak with the growth of the depth; after that, it starts to fall due to the progression of C2C phenomenon. In this section, present numerical evidence verifying that the pattern of variability change is highly correlated to the performance of corresponding neural nets. This suggests that variability may well serve as predictive indicator of the performance of suitable neural nets.
B. EXPERIMENT SETTING
Our experiments done styled synthetic model checkerboard in which the data points the 6561 mesh points of an 81 81 grid over the square 1]1, in IR2. These mesh points are divided into two sets, one corresponding to O-labels and another to 1-labels, so that together they form an 8 by 8 checkerboard blocks, as is shown in Figure 7 where each of the 64 squares contains 81 grid points and the surrounding edges contain 1377 points. The blocks take either O or (blue or red) label in an alternating pattern, and the surrounding edges all take the O-label. In essence, we aim to approximate the piecewise linear, non-smooth function shown the right plot of Figure
475
64
64
62
-12
00
-335
-100
Figure 7: Checkerboard: left plot for data points; right plot for labels; colors match binary labels.
We train fully connected, rectangular neural nets (FCNets) with the same number of nodes (but see below for exceptions) at each hidden layer. In each of our experiments, when we vary the number of hidden layers L, we adjust the width d so that the total number of model parameters (i.e., weights and biases) is kept to prescribed constant (though, in order to do so we sometimes have to add or delete one or two nodes from some hidden layers). It is worth noting that as the depth grows deeper, the width becomes narrower.
We randomly split the dataset 6561 points into two parts: 25% as a training set containing m 1640 samples, and the remaining 75% as testing set. Denoting the training set by (m:),", minimize the least squares loss function,
min fr b) W.b
Σ (f - -31133
(18)
where (x, W,b):
R R2 is constructed from the neural net function Fg (z; W, in (2) plus an input layer and output layer, and each label vector y IR2 is either (0, 0) (1, 1) representing
16

---

Under review as a conference paper at ICLR 2022
binary labels. We refer to the objective function value as the training loss and the percentage of correctly classified data points as the training accuracy. We note that a 100% training accuracy generally does not imply a O-training loss.
To ensure that the optimization calculation is done sufficiently, we apply the gradient descent method (instead of SGD) with 40000 iterations without a stopping criterion. For each run, we always try 10 different initial learning rates (step-sizes) as in
{0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1.0}
and then pick the best result for output. During the 40000 iterations, learning rates are reduced by factor of 5 three times at the junctures corresponding to iterations 20000, 28000, and 36000, respectively.
We initialize weight matrices and bias vectors with i.i.d. standard normal random numbers, and the weight matrices are then scaled by the Xavier/Kalt Kaming initialization constants. The activation function is ReLU and ABS. All trial instances are run with different samples of random initial parameters. For final output, we always record and output the best result in terms of the training error regardless of where occurs during training (in the middle or at the end).
B.2 COMPUTATIONAL RESULTS
We show the results of training FCNets by GD of varying depths (x-axis) and parameters (from left to right in each row). For each network, we compute the mean value of training/testing loss/accuracy. All result is reasonably stable as shown by the std of 10 runs. Once vee-parramttrrization happens, there exists bottom of the valley in Figure to get or approach to the global optimum.
We fix the total number of parameters at N = 1600, 2400 and 3200. For each Nw value, we solve Problem (18) with FCNets-RelLs and FCNets-ABS, following the procedure described in the previous subsection, with the number of hidden layers T. varying from to 31 (with increment up to 20 then increment 2 afterwards) for ReLU and extending to 41 for ABS. The results, including training and testing losses and accuracies, are summarized in Figure containing 6 plots in 3 columns corresponding to the values of N...
As previously shown, when networks grow deeper, variability first increases and then decreases. As we observe from Figure 8, there are remarkably similar patterns in all curves in Figure 8 for either training and test accuracies or losses (for which the rise-and-fall pattern is flipped).
In particular, we compare the left plot of Figure 2 with the third column of Figure For these two cases, the numbers of total model parameters are sufficiently close (3300 vs. 4000) to roughly examine the locations of interests. Indeed, the peak of variability occurs 12 in the left plot of Figure 2 while the best performance of the FCNet happens between 10 and 15, as can be seen in the third column of Figure
We offer the following interpretations of the experimental results, pertinent to the relationship between network variability and trainability.
Variability in the data space indicates model's expressivity, and in turn, it implies sensitivity in the parameter space. A well-designed and computable measure of variability, which remains a topic of further research, could serve well as an indicator of trainability.
With low variability, models apparently have more local traps, making training difficult. On the other hand, evidenced by the middle sections in the plots of Figure 8 where the training process seems to reach global optima with few or no exceptions, at least so in pver-parametrized models.
around variability peaks, there appears to exist few local traps,
As expected, over-param peeemetaamtiinntionnn helps trainability. However, it may not always benefit generalizations much at all, as is evidenced by the curves of testing loss and accuracy in Figure g
17

---

Under review as a conference paper at ICLR 2022
unnt love
pig 2p1
Taxs
-n-ed Ag..0oi
Layars
Number Layata
Layers
Nomber
Stmmber
Numbera
Numter Layers
1200
Layes
Layere
Numberal of Layaz
Lagers
on Checkerboard. The 1st-row contains computed training and testing losses, and the 2nd-row training and testing accuracies. Columns 1 to 3 (from left to right) are for results corresponding to the number of model parameters equal to 1600, 2400, and 3200, respectively (recall that the training set contains 1640 samples). Each group of curves depicts the mean and variance of random runs.
Figure 8: UCN vs FCNet-ABS
FCNet-ReLU is unable to reach near-zero loss values with more than 20 hidden layers, but FCNet-ABS still succeeds even after the hidden- number exceeds 30, confirming that ABS works better than ReL.U in deeper DNNs.
C HANNET
C.I HANNET ALGORITIM
Algorithm 1 HanLayer.
Input: vector æ and parameters (u, b).
Output: HaanLayr(rr;,,
1: function HANLAYER(2, b)
2: = I 2
|a 3: ABBS(2)
w'T 2 +
> Householder Reflection plus bias
Absolute-vale Function Activation
4: return y
5: end function
C.2 MIXER BLOCK
Figure presents two Mixer structures using FC-Layers and Han-Layers.
18

---

Under review as a conference paper at ICLR 2022
Layer
HH
SRY
Norm
Jobier
Figure 9: Two implementations of the channel mixing module using MLP-Mixer and Han-Mixer, respectively. HH denotes Houscholder.
D EXPERIMENT
D.1 EXPERIMENT SETTINGS
Our Checkerboard dataset similar to the one in Section but more complicated, shown as Figure with 6561 mesh points in 12 12 block checkerboard over the square [-1, consider, and our experiments have confirmed, that this dataset a highly challenging one for standard DNNs.
Datasets
dim
Training set N
Testing set N
Synthetic
Checkerboard
1640
4921
Elevators
18
3320 or 13279
13279 or 3320
Regression
Cal Housing
8
4128 or 16512
16512 or 4128
CIFAR-10
3072
50000
10000
Image Classification
CIFAR-100
3072
50000
10000
STL-10
27648
5000
8000
ImageNet32
3072
1281167
50000
Table 4: Dataset statistics: N is the number of samples and dim the dimension of data vectors. We use the mean-square етгоr loss function for the checkerboard and the two regression datasets, and the cross-entropy loss function to classify images. We use stochastic gradient descent (SGD) with momentum to train Checkerboard. To solve each test instance, we run SGD using each of the following initial learning rates:
(0.001, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 1}
and select the best result as the output. In addition, the initial learning rate is annealed by a factor of at the fractions 1/2, 7/10 and 9/10 of the training durations. Other parameter settings for SGD are listed in Table below..
Dataset
Iteration
batch size
weight decay
momentum
Checkerboard
40000
100
0.0
0.9
Table 5: SGD parameters.
For the two regression datasets, choose Adam (Kingma & Ba 2015) as the training method which seems to be the method of choice for several works in that area including (Tsang et al. 2018). For these image classification datasets, we choose modified Adam called AdamW (Loshchilov & Hutter 2018). Detailed settings are in Table 6 below.
All the training is done using Py PoTch (Paszke et al. 2019) running on a shared cluster. Finally, we mention that we always run multiple times for each test instance, starting from different random initializations of model parameters. All the reported values are the average of least 5 runs. is noted that the results in each block on Figure 5 are the average of runs. We randomly manufacture the train set 3 times and do 3 trials on each set.
19

---

Under review as a conference paper at ICLR 2022
Dataset
Optimizer
epochs
Ir
batch size
weight decay
Elevators/ Cal Housing
Adam
300
0.001
100
0.0
CIFAR-10/ CIFAR-I00
Adam W
600
0.001
256
0.1
STY.-10
AdamW
300
0.001
64
0.1
Imagenet-32
AdamW
300
0.001
512
0.1
Table 6: Adam/Adami parameters.
D.2 ABLATION STUDY CHECKERBOARD
Layer type
Activation type
Test accuracy
H
FC
ABS
ReLU
(a)
X
V
X
99.2%
(b)
/
X
X
V
66.2%
(c) X
/
/
X
85.3%
Table Ablation study on 100 17 network framework: Effects of layer and activation types on performance. denotes Householder and FC denotes fully-connected layers.
To better grasp how much the two HanNets components, Householder weighting and ABS activating, contribute to the surprising results on the Checkerboard dataset, we conduct an ablation study with three different configurations in a 100 17 network framework. The results are listed Table 7 where the layer type either Householder (H) or fully connected (FC) and the activation type either ABS ReL.U. Table 7 suggests that Householder weighting and ABS activating be equally critical to the 99% generalization accuracy of the HanNet. We investigate the heavily lacking of variability in the networks with Householder matrix plus ReLU function affects the model performance.
D.3 FLIPPING TRAINING LABELS ON CHECKERBOARD
PONet N0/1
25 53
Random Label Fllppaig (96)
Li5 150
375
4.00
-10
(a) FCNet vs HanNet
(b) Training set (25%)
(c) FCNet: test 86.2%
(d) HanNet: test 95.2%
Figure 10: (a) Testing accuracy FCNet and HanNet. (b) Training set: violet-colored points are those whose 10% labels are flipped (background color represents their original label). (c) FCNet: the best testing accuracies 86.5% in the model trained from (b). (d) HanNet: the best testing accuracy is 95.2%
To check the stability of the near-perfee result of the HanNet, we randomly flip training labels from to 20% on Problem Figure 10 (a) presents the testing accuracy HanNet and FCNet, where the network architecture is 100 x given in Figure 10 (c), while Figure (d) are, the best results for the HanNet in terms of testing accuracy. The best result shows that HanNet still learn well the original pattern from seriously damaged labels.
HanNet and 100 for FCNet. The training result FCNet
20

---

Under review as a conference paper at ICLR 2022
D.4 REGRESSION DATASETS
Table
presents the model specifications and statistics of datasets (Elevators and Cal Housing).
Figure
shows the NRMSE values on two FCNets and HanNet.
Elevators
Cal Housing
Model
Depth x Width
Parameters
AR(%)
Parameters
AR (%)
FCNetl
5 x 50
11201
2.23
10701
2.33
FCNe12
5 x 200
164801
0.60
162801
0.61
HanNet
20 x 200
11601
34.47
9601
41.66
Table 8: Model specifications and statistics of datasets (Elevators and Cal Housing)
Hakhi kottng
Hallie SOVRR
Lia tie corkig
Haacia taning
Liacile tanking
Haabia
Thalki -Chhd
011
Number Eppccs
Manther Upoddo
Narmbbrr Epeccs
Eppocts
Figure 11: All figures show testing performance where the top row is for the Elevators dataset; and the bottom row for Cal Housing. From left right: (1) HanNet (red) vs. FCNetl (blue) on 80% training data, (2) then 20% training data; (3) HanNet (red) FCNet2 (blue) on 80% training data, (4) then 20% training data. Note that HanNet testing results repeated.
D.5 SETTINGS ON IMAGE CLASSIFICATION DATASETS
We summarize various configurations of each block on different data sets in Table
run Mixers using the following number of layers {1, 8, 12, 16) and select the best one each dataset. We the convolution stem recommended by (Xiao al., 2021) instead of the
in (Tolstikhin et al.) 2021). Our convolutional stem designs use five layers, including four 3 x 3 convolutions and single 1 X final layer. The output channels are [64, 128, 256, 512, 512] on CIFARIO, CIFAR 100 and STL10, and [128, 256, 512, 1024, 1024] on ImageNet32.
CIFAR
STL10
ImageNet32
Patch size
4 x 4
8 X
4 4
Sequence Length
64
144
64
Channel number
512
512
1024
Table 9: The architecture on one MLP-block or Han-block. For simplification, in our experiments, all hidden MLP weights are square matries.
E PLOTS FROM LANDSCAPE EXPERIMENTS
21

---

Under review as a conference paper at ICLR 2022
Figure 12: Visualization variability on FCNet-Sigmoid with 2,4.6,10,15 hidden layers.
Figure 13: Visualization variability on FCNet-ReLU with 2,10,,0,40.60 hidden layers.
22

---

Under review as a conference paper at ICLR 2022
Figure 14: Visualization of variability on FCNet-ABS with 2,10,20,40,60 hidden layers.
lon
Figure 15: Visualization of variability on HanNer with 2,10,20,40,60 hidden layers.
23


Reviews:
[{"review_id": "yvXyouzFUiJ", "reviewer": "Reviewer_xnGa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n========\nThe parametrization of weight matrices as Householder reflectors makes sense as a way to reduce the number of free parameters, and force orthogonality, like was demonstrated in a different context (recurrent networks) by Mhammedi et al. (2017).\n\nThe HanMixer layer indeed seems to help improving performance of the CNN stem on small-resolution image datasets.\n\nWeaknesses\n==========\nThe theoretical justification or inspiration is tenuous and flimsy for many reasons.\n- The intuition of section 3.1, Fig. 1, and Appendix E comes from a 2D to 1D mapping, which is not a usual setting for neural networks, and is probably not representative of high-dimensional geometry (which can be deceptive).\n- Even the definition of \"variability\" in Eq. (4) does not scale well with the dimension of the input space, so it would be hard to assess its validity (although no attempts are made).\n- That definition depends on a distribution over the parameters W and b, which is arbitrary.\n- The use of the 3rd derivative is not explained or justified. It is strange as, for instance, linear models are somewhat expressive, and well trainable, though their variability would be 0.\n- Empirical correlations in the 2D case between depth and that measure (for a given number of free parameters) does not mean the \"activation ratio\" is a meaningful measure.\n- \"variability\" is qualified as a surrogate of \"expressivity\" as defined by Gühring et al. (2020), but expressivity refers to which functions can be expressed by a given class of function approximators, or how \"wide\" that class is. However, \"variability\" is then compared to the ease of training / optimizing a network, which would correspond to the training or optimization error of Gühring et al.\n- the so-called \"collapse-to-constant\" phenomenon is not anything new, especially for activation functions like sigmoid (I believe it was addressed by Glorot & Bengio (2010)), and the issue of \"dead neurons\" with ReLU has been studied. Activation functions less prone to that issue should have been compared: tanh, ELU... Comparing with established techniques like batch normalization and layer normalization could have been interesting, instead of jumping to the conclusion that absolute value was the answer.\n- In general, there is no evidence that the specific heuristics and metrics described here generalize beyond the un-trained networks operating on a 2D [-1, 1] grid, or to settings where the total number of parameters (weights) is not fixed.\n\nExperiments are lacking ablations or comparative studies of which elements of the Householder matrix formulation help with the generalization. For instance, orthogonal matrices with more parameters could have been used, or other low-rank approximations with the same number of parameters.\n\nOther\n=====\nSection 4.1, the statement that \"orthogonality is a desirable property for weight matrices in DNNs\" requires more discussion, justifications, and citations than \"As is well known\". I can't find a specific reference, but for instance there has been an argument that reversible/unitary transformations hurt generalization because of the inability to \"throw away\" irrelevant information.", "summary_of_the_paper": "The paper introduces an estimator of expressivity or trainability of deep networks, that they call \"variability\", that characterizes the variations in a scalar output of the network, when the input change, and seems correlated to trainability of the network, and to its depth (to an extent).\n\nThe paper proposes to parametrize weight matrices by using Householder reflectors, parametrized by a single vector each, which has two benefits:\n- A reduced number of free parameters, which enables more depth (more layers and non-linearities) for the same model size\n- Orthogonality of the matrices.\n\nExperiments on a synthetic 2D benchmark, two small-scale regression datasets, and small-resolution image datasets show improved results compared to dense nets.\n", "summary_of_the_review": "The parametrization of weight matrices by Householder matrices is interesting, although the experiments are quite preliminary.\nHowever, the whole derivation of proxies for expressivity or trainability is not adequately grounded theoretically, nor demonstrated empirically beyond a small contrived example.\nI do not think this paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "niLgkCFKQMF", "reviewer": "Reviewer_8jvQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strengths**\n- Notion of variability is novel.\n- Han-layers outperform fully connected layers on variety of tasks with lesser parameters.\n- Han networks have better variability than typical fully connected layers.\n\n\n**Weaknesses**:\n\n\n- Use of Householder matrix is not novel and they have been used in the RNNs as well as DNNs. For ex.  https://arxiv.org/pdf/1803.09327.pdf  uses Householder matrices to construct MLP equivalents (see Spectral-MLP and Spectral-Resnet architectures). \n\n- Although Householder matrices mitigate the vanishing or exploding gradient issues, the added non-linearity and higher depth, would mean additional training time and inference cost. As such a trade off exists, and that has not been explored in this work.\n\n- Parameter count is one way to compare architectures, but more importantly one should add additional metrics such as inference time, floating point operations, training time, etc. This is simply due to the fact that parameter count alone is somewhat misleading. Consider the case of convolutional operations vs matrix multiplication. The number of parameters in the convolution would be very small compared to a similar sized FC operation, while the FLOPs / compute required for the convolution would in general be much higher than the FC operation. At the very least, one should FLOPs comparison for MLPMixer and MLPMixer+HanMixer operations for architecture resulting in similar performance. \n\n\n- One important critic of the variability definition is that it uses third order gradient information and this would easily fail to scale to even problems where the DL community has tried to use second order gradient information. It would be good to explain the reasoning for this choice of the variability and did the authors consider any alternatives to this? \n\n\n- Increasing Activation Ratio is not something one should promote in general. Point being, when two models with same parameters are compared, if one has more activations than the other, this architecture will be cost more compute. Increasing AR can only be promoted when you have cheap activation functions, for ex. it would be much more costly to evaluate an exponential unit than a ReLU.\n\n\n- One trend that seems to be constant in the empirical evaluations is the substantially increased depth of the network. Are there scenarios where these deep networks are cost efficient? Since any increase in depth comes with added overhead of working memory and inference time due to the sequential dependency on the hardware.\n\n- Its somewhat clear from the paper that Han-layers do loose some of the representational power which the FC layers have, in the sense that to achieve similar results the Han-networks require more depth.\n\n\n\n**Questions for Authors**\n\n- In Appendix D.4, Fig.11, baseline graphs (blue lines) have a trend where they reach a minima very quickly and then suddenly keep on increasing forever. Is there any explanation for this behaviour? Were the baselines implemented corectly? Are there any difference in the experimental setups such as learning rate, weight decay, optimizer for baseline and HanNet?\n\n- Have you compared your architecture with the baselines in terms of metrics such as training time, inference time, floating point operations, working memory usage, etc.? Is there any reason why these metrics have been omitted in the paper?\n\n- In Sec.3.1, Why draw random networks in this space? In reality, one should expect some structure in the data and similarly the networks should be learnt to explore such structure.\n\n- Why would one not expect the sigmoid plot in the Figure~1 to be flat? Is it not possible that simply due to the fact that the network weights are initialized randomly, the activation is mostly dead or operating in a saturated regime?\n\n\n\n\n**Writing Clarity**:\n- Spelling error: Sec.3.1, para 2, ... nearest integetr.. \n- Simplification: Sec.4.3, para 1, so-called G_L matrix is nothing but the gradients of the function w.r.t. parameters. \n- Introduction is very hard to understand due to lack of following definitions. Since these are the basic issues the paper is dealing with, one would expect simple and intuitive definitions in the introduction. (a) Does not explicitly define variability, (b) Does not define collapse to constant.\n\n", "summary_of_the_paper": "**Contributions**:\n- Introduces the notion of variability to explain the better trainability and generalization of certain deep neural architectures. This is backed by a concrete definition of the variability in terms of the third order derivative information of the loss landscape.\n\n- Empirical experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants (C2C).\n\n- Inspired by the notion of variability, authors propose Householder-absolute neural layer aka Han-layer that replaces square weight matrices with Householder reflectors, and use absolute-value function for activations. \n\n-  HanNet achieves a high variability as well as guarantee an immunity to vanishing or exploding gradients, and chance for collapse to constants has been diminished.\n\n- HanNets achieve better generalization with lesser parameters than fully connected networks.\n\n", "summary_of_the_review": "Notion of variability maybe a step in the right direction to explain generalization and trainability of DNNs. While the Han-layers do seem to outperform FCNets, many questions remain unanswered in this work, namely, \n- Are Han-nets compute hungry (train, inference) when compared to FCnets?\n- Is the additional depth necessary?\n- Is it harder to train Han-nets in comparison to FCnets, due to orthogonality and additional non-linearity?\n- Are there simpler definitions of the notion of variability that extend to higher dimensions than the toy examples illustrated in the paper?\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e3jQGIjaHau", "reviewer": "Reviewer_7Vdi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Clarity:__ The paper is clearly written and easy to follow.\n\n__Comments:__\n- The __variability__ is only loosely defined as \"richness of landscape variations\". It seems that it is understandable but a more accurate definition is needed.\n- The connection of the above definition to equation 4 (measurable property) is not clear. The only explanation provided in the paper is \"seems to have worked well with two-dimensional data spaces\". Which is not specific.\n- Equation 4 does not specify the distribution over W and b. Paper is silent about that except \"for randomly sampled (W, b) with proper scalings\". That does not specify the form of distribution.\n- The motivation for the proposed _Han-layer_ seems to be based only on the nice behavior of equation 4 resented in Figure 2. That does not look solid for me.\n- The empirical improvements are tested only on MLP-Mixer models. While all the datasets are different from those used in the original MLP-Mixer paper. \n\n__Conclusions__:\n- _Variability_ is indeed an interesting concept, but it is neither explained clearly enough nor compared accurately with other generalization measures. \n- _Han-layer_ does not have a solid connection to _variability_.\n- The empirical results are nice but do not seem to be solid enough in isolation from other contributions (e.g., tested only on one architecture, gains are relatively small).\n\nThe motivation (__The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models.__) provided in the work is nice. However,  my feelings are that the paper delivers on the opposite. It proposes a new layer and uses it to build new variants of MLP-Mixer. While all the parts of the work feel half-baked.", "summary_of_the_paper": "The paper introduces a new notion of _variability_ of a DNN, that intuitively is the \"richness of landscape variation\" wrt data and parameters. The paper attributes the _variability_ to the generalization performance of fully connected DNNs. Inspired by the increasing _variability_ the work proposes _Han-layer_ that is a linear layer with a Householder matrix and abs(.) activation function. The empirical results show that _Han-layer_ can improve the performance of MLP-Mixer on various datasets.", "summary_of_the_review": "The paper aims to propose new a measure associated with generalization, use it to define new models, and gain some insights into the behavior of DNNs.  However, all the parts of the work feel half-baked.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N8jdmxuusp0", "reviewer": "Reviewer_QXHA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- This paper is very well written and easy to follow.\n- The synthetic experiments are well designed and visualized, and the corresponding results and conclusions are interesting.\n\n[Weaknesses]\n1. On the new concepts\n- The newly proposed concepts are inspiring but not rigorous, thus producing more questions than it answers. \n- Variability seems to have a close relationship with the expressiveness of a neural network. Since few theoretical or rigorous discussions are provided for variability, it remains conceptual and abstract except for the visualization in Figure 2. Moreover, as stated by the authors, it is not the case that the higher variability the better. Then what could be the practical use of the proposed concept?\n- For the activation ratio, discussions are even less sufficient. For example, keeping the number of activations to be the same, then reducing network parameters would improve AR. Suppose we reduce the network parameters by introducing low-rankness to the model weights. Then AR increases as rank reduces, and there seems to be a trade-off between parameter efficiency and network expressiveness. Can AR contribute any more precise / more widely-applicable insight?\n\n2. On the synthetic experiments\n- The landscape experiments compared FCNets with HanNet. However, it is well-known that deep FCNet has the gradient vanishing problem, and thus it is not surprising that the collapse-to-constant phenomenon exists. Currently, Figure 1 shows that HanNet is a solution to this problem. However, existing solutions are mostly based on residual connections and batch normalization. Shouldn't the solutions be compared against each other? That is, what does the landscape look like for FCNets with residual connections and/or batch normalization?\n\n3. On the image classification datasets\n- The datasets and models are too small to shed light on the effectiveness of the proposed method in the more practical scenarios. Because Han-layer has only a few parameters, it is possible that Han-layer is beneficial for toy examples but not on much more complicated real applications.\n- Scaling down the resolution of ImageNet samples makes training less costly but this also makes it difficult to compare the proposed method to existing benchmarks. Thus, it is difficult to judge the significance of the improvements. Besides, WideResNet could be a bit outdated. For a more SOTA ResNet-based result, authors may refer to \n\nBello, I., Fedus, W., Du, X., Cubuk, E. D., Srinivas, A., Lin, T. Y., ... & Zoph, B. (2021). Revisiting ResNets: Improved training and scaling strategies. NeurIPS 2021.\n\n[Other details]\n- typo in Section 3.1: \"rounded to the nearest integetr\"\n- Since geometric mean is actually used for equation (4) instead of the arithmetic mean, which is the sample version of expectation, I recommend changing the expectation to a notion of the geometric mean.", "summary_of_the_paper": "This paper introduces some concepts, i.e., variability, activation ratio, and collapse to constants, to help understand deep neural networks, and proposes a new layer called Householder-absolute layers to improve network variability.\nExperiments are conducted on both synthetic datasets and empirical datasets. The synthetic experiments show some interesting results and visualizations, and for the real datasets, improvements over baseline architectures can be observed.", "summary_of_the_review": "This paper is easy to read and provides some interesting discussions and visualizations. However, concerns exist regarding\n- the usefulness of the proposed concepts;\n- the design of the landscape experiments; and\n- the effectiveness of the proposed model on more complicated applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the ideas presented in the paper are interesting and novel. However, all reviewers also agreed that the paper is quite preliminary in its current form: the particular approach, while sensible, appears to be somewhat heuristic, and the evaluations are not as complete as necessary to fully evaluate the proposed approach.

Generally, my sense is that there is something quite interesting in this work, but the present paper is too preliminary for publication. I would encourage the authors to take the reviewer comments into account and improve the work into a more complete submission for a future venue.
Decision: Reject