Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces an estimator of expressivity or trainability of deep networks, that they call "variability", that characterizes the variations in a scalar output of the network, when the input change, and seems correlated to trainability of the network, and to its depth (to an extent).

The paper proposes to parametrize weight matrices by using Householder reflectors, parametrized by a single vector each, which has two benefits:
- A reduced number of free parameters, which enables more depth (more layers and non-linearities) for the same model size
- Orthogonality of the matrices.

Experiments on a synthetic 2D benchmark, two small-scale regression datasets, and small-resolution image datasets show improved results compared to dense nets.

main_review: Strengths
========
The parametrization of weight matrices as Householder reflectors makes sense as a way to reduce the number of free parameters, and force orthogonality, like was demonstrated in a different context (recurrent networks) by Mhammedi et al. (2017).

The HanMixer layer indeed seems to help improving performance of the CNN stem on small-resolution image datasets.

Weaknesses
==========
The theoretical justification or inspiration is tenuous and flimsy for many reasons.
- The intuition of section 3.1, Fig. 1, and Appendix E comes from a 2D to 1D mapping, which is not a usual setting for neural networks, and is probably not representative of high-dimensional geometry (which can be deceptive).
- Even the definition of "variability" in Eq. (4) does not scale well with the dimension of the input space, so it would be hard to assess its validity (although no attempts are made).
- That definition depends on a distribution over the parameters W and b, which is arbitrary.
- The use of the 3rd derivative is not explained or justified. It is strange as, for instance, linear models are somewhat expressive, and well trainable, though their variability would be 0.
- Empirical correlations in the 2D case between depth and that measure (for a given number of free parameters) does not mean the "activation ratio" is a meaningful measure.
- "variability" is qualified as a surrogate of "expressivity" as defined by Gühring et al. (2020), but expressivity refers to which functions can be expressed by a given class of function approximators, or how "wide" that class is. However, "variability" is then compared to the ease of training / optimizing a network, which would correspond to the training or optimization error of Gühring et al.
- the so-called "collapse-to-constant" phenomenon is not anything new, especially for activation functions like sigmoid (I believe it was addressed by Glorot & Bengio (2010)), and the issue of "dead neurons" with ReLU has been studied. Activation functions less prone to that issue should have been compared: tanh, ELU... Comparing with established techniques like batch normalization and layer normalization could have been interesting, instead of jumping to the conclusion that absolute value was the answer.
- In general, there is no evidence that the specific heuristics and metrics described here generalize beyond the un-trained networks operating on a 2D [-1, 1] grid, or to settings where the total number of parameters (weights) is not fixed.

Experiments are lacking ablations or comparative studies of which elements of the Householder matrix formulation help with the generalization. For instance, orthogonal matrices with more parameters could have been used, or other low-rank approximations with the same number of parameters.

Other
=====
Section 4.1, the statement that "orthogonality is a desirable property for weight matrices in DNNs" requires more discussion, justifications, and citations than "As is well known". I can't find a specific reference, but for instance there has been an argument that reversible/unitary transformations hurt generalization because of the inability to "throw away" irrelevant information.
summary_of_the_review: The parametrization of weight matrices by Householder matrices is interesting, although the experiments are quite preliminary.
However, the whole derivation of proxies for expressivity or trainability is not adequately grounded theoretically, nor demonstrated empirically beyond a small contrived example.
I do not think this paper should be accepted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: **Contributions**:
- Introduces the notion of variability to explain the better trainability and generalization of certain deep neural architectures. This is backed by a concrete definition of the variability in terms of the third order derivative information of the loss landscape.

- Empirical experiments verify that variability correlates positively to the number of activations and negatively to a phenomenon called Collapse to Constants (C2C).

- Inspired by the notion of variability, authors propose Householder-absolute neural layer aka Han-layer that replaces square weight matrices with Householder reflectors, and use absolute-value function for activations. 

-  HanNet achieves a high variability as well as guarantee an immunity to vanishing or exploding gradients, and chance for collapse to constants has been diminished.

- HanNets achieve better generalization with lesser parameters than fully connected networks.


main_review: 
**Strengths**
- Notion of variability is novel.
- Han-layers outperform fully connected layers on variety of tasks with lesser parameters.
- Han networks have better variability than typical fully connected layers.


**Weaknesses**:


- Use of Householder matrix is not novel and they have been used in the RNNs as well as DNNs. For ex.  https://arxiv.org/pdf/1803.09327.pdf  uses Householder matrices to construct MLP equivalents (see Spectral-MLP and Spectral-Resnet architectures). 

- Although Householder matrices mitigate the vanishing or exploding gradient issues, the added non-linearity and higher depth, would mean additional training time and inference cost. As such a trade off exists, and that has not been explored in this work.

- Parameter count is one way to compare architectures, but more importantly one should add additional metrics such as inference time, floating point operations, training time, etc. This is simply due to the fact that parameter count alone is somewhat misleading. Consider the case of convolutional operations vs matrix multiplication. The number of parameters in the convolution would be very small compared to a similar sized FC operation, while the FLOPs / compute required for the convolution would in general be much higher than the FC operation. At the very least, one should FLOPs comparison for MLPMixer and MLPMixer+HanMixer operations for architecture resulting in similar performance. 


- One important critic of the variability definition is that it uses third order gradient information and this would easily fail to scale to even problems where the DL community has tried to use second order gradient information. It would be good to explain the reasoning for this choice of the variability and did the authors consider any alternatives to this? 


- Increasing Activation Ratio is not something one should promote in general. Point being, when two models with same parameters are compared, if one has more activations than the other, this architecture will be cost more compute. Increasing AR can only be promoted when you have cheap activation functions, for ex. it would be much more costly to evaluate an exponential unit than a ReLU.


- One trend that seems to be constant in the empirical evaluations is the substantially increased depth of the network. Are there scenarios where these deep networks are cost efficient? Since any increase in depth comes with added overhead of working memory and inference time due to the sequential dependency on the hardware.

- Its somewhat clear from the paper that Han-layers do loose some of the representational power which the FC layers have, in the sense that to achieve similar results the Han-networks require more depth.



**Questions for Authors**

- In Appendix D.4, Fig.11, baseline graphs (blue lines) have a trend where they reach a minima very quickly and then suddenly keep on increasing forever. Is there any explanation for this behaviour? Were the baselines implemented corectly? Are there any difference in the experimental setups such as learning rate, weight decay, optimizer for baseline and HanNet?

- Have you compared your architecture with the baselines in terms of metrics such as training time, inference time, floating point operations, working memory usage, etc.? Is there any reason why these metrics have been omitted in the paper?

- In Sec.3.1, Why draw random networks in this space? In reality, one should expect some structure in the data and similarly the networks should be learnt to explore such structure.

- Why would one not expect the sigmoid plot in the Figure~1 to be flat? Is it not possible that simply due to the fact that the network weights are initialized randomly, the activation is mostly dead or operating in a saturated regime?




**Writing Clarity**:
- Spelling error: Sec.3.1, para 2, ... nearest integetr.. 
- Simplification: Sec.4.3, para 1, so-called G_L matrix is nothing but the gradients of the function w.r.t. parameters. 
- Introduction is very hard to understand due to lack of following definitions. Since these are the basic issues the paper is dealing with, one would expect simple and intuitive definitions in the introduction. (a) Does not explicitly define variability, (b) Does not define collapse to constant.


summary_of_the_review: Notion of variability maybe a step in the right direction to explain generalization and trainability of DNNs. While the Han-layers do seem to outperform FCNets, many questions remain unanswered in this work, namely, 
- Are Han-nets compute hungry (train, inference) when compared to FCnets?
- Is the additional depth necessary?
- Is it harder to train Han-nets in comparison to FCnets, due to orthogonality and additional non-linearity?
- Are there simpler definitions of the notion of variability that extend to higher dimensions than the toy examples illustrated in the paper?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces some concepts, i.e., variability, activation ratio, and collapse to constants, to help understand deep neural networks, and proposes a new layer called Householder-absolute layers to improve network variability.
Experiments are conducted on both synthetic datasets and empirical datasets. The synthetic experiments show some interesting results and visualizations, and for the real datasets, improvements over baseline architectures can be observed.
main_review: [Strengths]
- This paper is very well written and easy to follow.
- The synthetic experiments are well designed and visualized, and the corresponding results and conclusions are interesting.

[Weaknesses]
1. On the new concepts
- The newly proposed concepts are inspiring but not rigorous, thus producing more questions than it answers. 
- Variability seems to have a close relationship with the expressiveness of a neural network. Since few theoretical or rigorous discussions are provided for variability, it remains conceptual and abstract except for the visualization in Figure 2. Moreover, as stated by the authors, it is not the case that the higher variability the better. Then what could be the practical use of the proposed concept?
- For the activation ratio, discussions are even less sufficient. For example, keeping the number of activations to be the same, then reducing network parameters would improve AR. Suppose we reduce the network parameters by introducing low-rankness to the model weights. Then AR increases as rank reduces, and there seems to be a trade-off between parameter efficiency and network expressiveness. Can AR contribute any more precise / more widely-applicable insight?

2. On the synthetic experiments
- The landscape experiments compared FCNets with HanNet. However, it is well-known that deep FCNet has the gradient vanishing problem, and thus it is not surprising that the collapse-to-constant phenomenon exists. Currently, Figure 1 shows that HanNet is a solution to this problem. However, existing solutions are mostly based on residual connections and batch normalization. Shouldn't the solutions be compared against each other? That is, what does the landscape look like for FCNets with residual connections and/or batch normalization?

3. On the image classification datasets
- The datasets and models are too small to shed light on the effectiveness of the proposed method in the more practical scenarios. Because Han-layer has only a few parameters, it is possible that Han-layer is beneficial for toy examples but not on much more complicated real applications.
- Scaling down the resolution of ImageNet samples makes training less costly but this also makes it difficult to compare the proposed method to existing benchmarks. Thus, it is difficult to judge the significance of the improvements. Besides, WideResNet could be a bit outdated. For a more SOTA ResNet-based result, authors may refer to 

Bello, I., Fedus, W., Du, X., Cubuk, E. D., Srinivas, A., Lin, T. Y., ... & Zoph, B. (2021). Revisiting ResNets: Improved training and scaling strategies. NeurIPS 2021.

[Other details]
- typo in Section 3.1: "rounded to the nearest integetr"
- Since geometric mean is actually used for equation (4) instead of the arithmetic mean, which is the sample version of expectation, I recommend changing the expectation to a notion of the geometric mean.
summary_of_the_review: This paper is easy to read and provides some interesting discussions and visualizations. However, concerns exist regarding
- the usefulness of the proposed concepts;
- the design of the landscape experiments; and
- the effectiveness of the proposed model on more complicated applications.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a new notion of _variability_ of a DNN, that intuitively is the "richness of landscape variation" wrt data and parameters. The paper attributes the _variability_ to the generalization performance of fully connected DNNs. Inspired by the increasing _variability_ the work proposes _Han-layer_ that is a linear layer with a Householder matrix and abs(.) activation function. The empirical results show that _Han-layer_ can improve the performance of MLP-Mixer on various datasets.
main_review: __Clarity:__ The paper is clearly written and easy to follow.

__Comments:__
- The __variability__ is only loosely defined as "richness of landscape variations". It seems that it is understandable but a more accurate definition is needed.
- The connection of the above definition to equation 4 (measurable property) is not clear. The only explanation provided in the paper is "seems to have worked well with two-dimensional data spaces". Which is not specific.
- Equation 4 does not specify the distribution over W and b. Paper is silent about that except "for randomly sampled (W, b) with proper scalings". That does not specify the form of distribution.
- The motivation for the proposed _Han-layer_ seems to be based only on the nice behavior of equation 4 resented in Figure 2. That does not look solid for me.
- The empirical improvements are tested only on MLP-Mixer models. While all the datasets are different from those used in the original MLP-Mixer paper. 

__Conclusions__:
- _Variability_ is indeed an interesting concept, but it is neither explained clearly enough nor compared accurately with other generalization measures. 
- _Han-layer_ does not have a solid connection to _variability_.
- The empirical results are nice but do not seem to be solid enough in isolation from other contributions (e.g., tested only on one architecture, gains are relatively small).

The motivation (__The purpose of this work is to gain more insights into the behaviors of DNNs and then use them to build new DNN models.__) provided in the work is nice. However,  my feelings are that the paper delivers on the opposite. It proposes a new layer and uses it to build new variants of MLP-Mixer. While all the parts of the work feel half-baked.
summary_of_the_review: The paper aims to propose new a measure associated with generalization, use it to define new models, and gain some insights into the behavior of DNNs.  However, all the parts of the work feel half-baked.
review_title: null

Meta-review: The reviewers generally agreed that the ideas presented in the paper are interesting and novel. However, all reviewers also agreed that the paper is quite preliminary in its current form: the particular approach, while sensible, appears to be somewhat heuristic, and the evaluations are not as complete as necessary to fully evaluate the proposed approach.

Generally, my sense is that there is something quite interesting in this work, but the present paper is too preliminary for publication. I would encourage the authors to take the reviewer comments into account and improve the work into a more complete submission for a future venue.
Decision: Reject