Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an interesting approach to normalizing the signals in a deep network both in the forward pass and in the backward pass with the goal of preventing vanishing and exploding gradients for better training convergence (and perhaps general elegance). 

Previously, this has been partly solved by the use of orthogonal weight matrices (and this completely solves the problem for linear networks). The authors show that additionally a specific modification of the non-linearity (essentially a scale and shift) is enough to guarantee good bi-directional normalization with high probability.  

I did not check the math. It would be good if the authors could provide some intuition for the main contribution: that an appropriate scaling and shifting of the nonlinearity is what is needed. 

Empirical results are demonstrated in essentially toy settings (synthetic, and mnist/cifar with fully connected networks) but these experiments are thorough (but see question below). 

My positive review of the paper is mostly based on viewing this as a natural completion of an existing line of work (around the use of orthogonal weight matrices for normalization), but I worry that this line of work may not lead to practically useful results since the orthogonality constraint is hard to enforce exactly, and it is not clear if this would work for convolution given the requirement for sufficient width. This is why I am hesitant to give it a higher score. 

Questions, mostly around the practicality of the proposed techniques.

1. How would this work on convolutional networks where one does not typically expect large “width” i.e. is this work only of theoretical interest?

2. How does this impact generalization? There is reason to believe that mucking with gradients can hurt generalizations. This has been discussed for  adaptive methods (Duchi et al., etc. alluded to the in the introduction) in https://arxiv.org/abs/1705.08292.

3. How would you compare with residual networks? Particularly in terms of run-time and generalization.

Minor:

a. What is the significance of Theorem 1?

b. Theorem 2 statement should say $\phi$ is Gaussian-Poincare normalized?

review_title: A completion of sorts for an interesting line of work 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The authors introduce a variant
of self-normalizing networks (SNNs) that also maintain
the norm of the errors during backpropagation. The presented
Bidirectional self-normalizing networks (BSNN)
have interesting theoretical properties, but are hardly 
assessed empirically. There are severe flaws in 
the method evaluation. 


Pros: 
a) The mathematical formulations are very clear 
and sound.
b) The connection between SNNs and works in
mean-field theory (Poole, Schoenholz) are very
well put in place. 
c) Making both the forward- and the backward pass 
self-normalizing appears desirable for NNs.
d) The work is well embedded into related works. 


Cons: 
a) Unclear relevance due to lacking and small-scale experiments, furthermore,
due to missing error bars and significance tests. 

i) Firstly, SNNs were introduced with relatively large-scale experiments on fully-connected networks. 
In order to demonstrate improvements due to bidirectional normalization, 
the suggested activation functions should be compared in that set of experiments. 
The presented experiments are done with architectures that are extremely far from the current SOTA on CIFAR10, 
with 46% versus 90% accuracy already reported in 2013 (see ref[2]). The authors
should perform experiments on large scale, e.g. CIFAR100 and ImageNet or voice recognition [3], 
with architectures at or close to the state-of-the-art to demonstrate 
the broad applicability and relevance of their method. 

ii) It is unclear whether BSNNs represent an advance at all because 
all presented performance metrics come without repetitions, error bars
and significance tests. Purported improvements could be just by chance. 
The authors should use repetitions, present all metrics with confidence
intervals and perform statistical tests. 

iii) Conclusions of the experiments are not justified due to limited
hyperparameter search. The setting of the experiments that activation
functions are compared with the same architecture, same learning rate, 
momentum term, and batch size strongly limits the conclusions since
activation functions could work with slightly different settings. 
The authors should allow each method (here: activation function) 
to optimize hyperparameters. 

iv) It is not necessary that almost exact preservation 
of the norm in the backward pass must improve learning, especially,
because it is traded against a bias shift. 
For standard SNNs, with the SELU activation, there is 
a small exploding gradient, which, however leads
to lower layers learning faster [1]. This could even assist learning.
This point should make clear that stringent empirical
assessment is necessary. Therefore, the authors should provide
a much wider empirical assessment of BSNNs.

b) Strong uncommented assumtions. The presented work requires
orthogonal matrices, and several other assumptions. Those assumptions
are left mostly uncommented and leave unclear how applicable this 
theory is and how strong it restricts the results and conclusion.
The authors should comment on all assumptions and relate it to learned 
neural networks. 
i) The assumption of orthogonal weight matrices is a strong one. 
This is computationally extremely costly, even in the relaxed way
of the authors (Section 3.2). The authors should elaborate on this.  

c) Neglection of learning dynamics. Whereas, the derivation of SNNs
involves learning dynamics by showing that even with changes in the
weights, a fixed points remains, BSNNs neglect learning dynamics. 
Even definition of self-normalization requires exact preservations
of the norm. The authors should aim at requiring Eq (4) and (5) to 
be kept on expectation and not exactly. Otherwise, no learned 
network is self-normalizing. 


Minor:


Questions:
a) The expectation over E[\phi(x)] is not more equal zero, which introduces
a bias shift. How does this bias shift interact with learning? 
Can you state this difference to SNNs clearer in the manuscript?
b) Lemma 3: Can you comment on how this bound arises from (Ball, 1997)?


Justification of score:
Despite my many points of concerns, the theory in this paper and the main 
idea are good. I would be willing 
to vote for acceptance if the experimental part is strongly improved. 


References:
[1] Hoedt, P.J., Hochreiter, S. and Klambauer, G. Characterising activation functions by their backward dynamics around forward fixed points. Critiquing and Correcting Trends in Machine Learning workshop at NeurIPS 2018. 
[2] Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013, May). Maxout networks. In International conference on machine learning (pp. 1319-1327). PMLR.
[3] Huang, Z., Ng, T., Liu, L., Mason, H., Zhuang, X., & Liu, D. (2020, May). SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6854-6858). IEEE.


review_title: Good idea and theory; substandard experimental part.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

In this paper, the authors introduce the bidirectional self-normalizing neural networks (BSNN) that preserve the norms in both forward and backward passes. To serve such purpose, a new class of activation functions, GPN, is proposed, which can be obtained via the affine transform of existing activation functions like tanh and SELU. The authors prove that under orthogonal weights and GPN, the norm for both forward and backward passes can be well preserved. Besides, the conclusions are also supported by experiments on synthetic data and real-world data like MNIST and CIFAR 10.


Strength:

--The paper is well organized and easy to follow.

--The proofs seem rigorous.

--The experiments on synthetic data and real-world data are consistent with the theoretical prediction.


Weakness:

--One major strength of the self-normalizing neural network powered by SELU as well as batch normalization is that, even if the statistical properties deviate from the ideal point at some layers, the succeeding layers can gradually fix the deviation. This property greatly stabilizes the training process even under a high learning rate. On the other hand, methods like dedicated initialization and weight normalization/standardization do not have this property, and previous studies have shown that they are less effective compared with SELU and BN. While GPN can be achieved via adding a simple affine transform to the existing activation functions like tanh, does it mean that it is more like an initialization technique? Or it still has the property of fixing deviations in preceding layers?

--For the experiments on real-world data, the depth 200 may be too deep for these small datasets, and the accuracy reported is less satisfying compared with networks with 30-50 layers. Although I understand that the authors want to demonstrate the performance of self-normalization in very deep models, it would be more convincing if the proposed activation functions are also compared with existing ones like SELU under a more practical depth (e.g., 30-50 layers).

review_title: Good Work but Need more Analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper tackles the problem of vanishing/exploding gradient in neural networks by imposing constraints on the weights and activation functions that ensure consistent Frobenius norm across updates to the weight matrices in all layers of the network.  

The paper is very well written, straight to follow, theory is well laid out and it is interesting.  The only (minor) issue I have on the presentation side is the name “bidirectional self-normalizing neural networks”.  This phrasing initially made me think the the network is bidirectional…but the network itself is not, it’s the self-normalizing that is bidirectional.   Wouldn’t it be more precise to say “bidirectionally self-normalized neural networks”?

However, despite a well outlined theory, I think that in the end the experimental evaluation demonstrates a major weakness of the proposed approach.  Credit to the authors for including Table 2 with evaluation on real datasets, but the generalisation performance there seems to show quite clearly that -GPN variants of the networks are equivalent to linear classifiers.  A quick check of the performance of linear classifier on MNIST and CIFAR10 without augmentation reveals 93% and 41% testing accuracy - pretty close to the performance shown in Table 2.  Sure, -GPNising allows for training a 200-layer model, but it seem to come at the cost of reducing the model to barely more than a linear classifier.  The sacrifice of all non-trivial representational power in order to gain stable learning dynamics is just not worth it.

review_title: Nice theoretical treatment, but results equivalent to linearising deep models

Meta-review: Three knowledgeable referees rate this paper ok but not good enough or borderline positive (4,4,6), and one fairly confident referee rates it borderline positive 6. The referees discussed the authors' responses and, while they considered the idea and some of the theory good, they remain concerned, in particular about the experimental part and generalization discussion. The scores remained unchanged after the discussion. Hence I must reject the article. 
Decision: Reject