Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes normalizing the stable rank (ratio of the Frobenius norm to the spectral norm) of weight matrices in neural networks. They propose an algorithm that provably finds the optimal solution efficiently and perform experiments to show the effectiveness of this normalization technique.

Stable rank of the weight matrix is an interesting quantity that shows up in several generalization bounds. Therefore, regularizing such measure could potentially help with generalization. Authors discuss this clearly and they provide an algorithm that provably finds the projection. I enjoyed reading this part of the paper. The only question that I have from this part is the role of partitioning index. It looks like it is not really being used in the experiments later. Is that right? What is the importance of adding it to the paper if it is not being used?

My main issue is with the empirical evaluation of the normalization technique. I am not an expert in GANs so I leave that to other reviewers to judge. Experiments on random labels and looking at different generalization measures are all nice but they are not sufficient for showing that this normalization technique is actually useful in practice. Therefore, I suggest authors to put more emphasize at showing how their regularization can improve generalization in practice. My suggestions:

- Authors only provided experiments on CIFAR100 dataset to support their claim on improving generalization. I suggest adding at least one other dataset (CIFAR10, or even better imagenet) to improve their empirical results. 

- Unfortunately, there are two major issues with the current CIFAR100 results: 1) the accuracies reported for ResNet and DenseNet are too low compare what is reported in the literature. Please resolve this issue. 2) The current result is with training with a fixed number of epochs. Instead, train with a stopping criterion based on the cross-entropy loss on the training set and use the same stopping criterion for all models. Also, add the plots that show training and test errors based on the #epochs.


Overall, I think the paper is interesting but the empirical results are not sufficient to support the main claim of the paper (improving generalization). I'm willing to increase my score if authors apply the above suggestions. 


*************************

After author rebuttals:

Authors have address my concerns adequately in the last revision and improved the experiment section. Therefore, I increase my score to 6.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Stable Rank Normalization for Improved Generalization in Neural Networks and GANs

Summary:

This paper proposes to normalize network weights using the Stable Rank, extending the method of spectral normalization. The stable rank is the ratio of the squared frobenius norm to the squared spectral norm (the top singular value). The authors motivate this decision in the context of lipschitz constraints and noise sensitivity. The proposed method (combined with spectral norm as SRN alone does not explicitly subsume SN) is tested for both classification (using a wide range of popular models on CIFAR) and GAN training. Performance (classification accuracy and FID/IS) is measured and several auxiliary investigations are performed into generalization bounds, sample complexity, and sensitivity to hyperparameters.

My Take:

This paper motivates and presents an interesting extension of spectral norm, and evaluates it quite well with thorough experiments in a range of settings. The method looks to be reasonably accessible to implement, although its compute cost is not properly characterized and some details (like the orthogonalization step necessary in power-iteration for more than one SV) seem to be omitted. My two main concerns are that the results, while good, are not especially strong (the relative improvement is not very high) and that the paper could be made substantially more concise to fit within the 8 page soft limit (I felt there was plenty of material that could be moved to the appendix). All in all this is a reasonably clear accept to me (7/10) that with some cleanup could be a more solid 8, and I argue in favor of acceptance.

Notes

-The paper should characterize the runtime difference between SRN and SN. It is presently unclear how computationally intensive the method is. What is the difference in time per training iteration? The authors should also indicate their hardware setup and overall training time.

-I found table 1 confusing as it lacks the test error. Are the test errors the same for all these models and the authors are just showing that for certain settings the SRN models have higher training error? If there is a difference in testing error, then this table is misleading, as one cares little about the training error if the test errors vary. If the test errors are approximately the same, then why should I care if the training error is higher? This would just be a way to decrease the stated “generalization gap,” which is not necessarily indicative of a better model (vis-à-vis the commonly held misconception that comparing training error between models is properly indicative of relative overfitting). 

-Nowhere (that I could spot) in the body of the paper is it explained what “Stable-50”, “SRN-50”, and “SRN-50%” are. I assume these all mean the same thing and it refers to the choice of the c hyperparameter, but this should be explicitly stated so that the reader knows which model corresponds to which settings.

Minor

-The footnotes appear to be out of order,  footnote 1 appears on page 9

-There are typos such as “simiplicity,” please proofread thoroughly.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: While spectral normalization is often used to improve generalization by
directly bounding the Lipschitz constant of linear layers, recent works have
highlighted alternate methods that aim to reduce generalization error.  This
paper shows how to implement these "stable rank" normalizations with little
computational overhead.  The authors then apply the method to a wide variety of
classification and GAN problems to show the benefits of stable rank
normalization.

This is a good paper and can be accepted.  The added value comes from their
Thm. 1, where they detail precisely how to project a real matrix onto one of
lower srank while preserving the largest k eigenvalues.  The spectral preservation
k seems to be a new feature of their method. Full proofs and additional results are
provided in appendices.  There seems to be enough information to implement
the described methods.

The paper is carefully written and introductory sections do a great job of putting
the problem in perspective.  Very few typos ("calssification", run a spell check).

--------- Fun to think about ------ here are some extra comments

Some related older introductory approaches could also be quickly mentioned:
 - linear layers represented as "bottlenecks" to enforce low rank explicitly
 - or solving in manifold of reduced-rank matrices directly

For simplicity, they target the same srank r=c*min(m,n) for all layers, even though only the sum
of sranks is important.  For CNNs with only a few linear layers is there any observable
difference by lightly deviating from this?  Does the first linear layer typically contribute
the lion's share to the sum of sranks?

It is interesting that by only addressing the linear layers of deep CNNs they
are able to see consistent improvements. [i.e. 3 linear layers after 101 CNN layers].
 This makes me wonder whether future work will also address how "stable rank"
concepts might be extended to the convolutional layers.  As a starting point, spectral
values of the block-circulant matrices corresponding to convolutions have been
described [ Sedghi et al. "Singular Values of Convolutional Layers" ].

review_title: Official Blind Review #2

Meta-review: The authors propose stable rank normalization, which minimizes the stable rank of a linear operator and apply this to neural network training. The authors present techniques for performing the normalization efficiently and evaluate it empirically in a range of situations. The only issues raised by reviewers related to the empirical evaluation. The authors addressed these in their revisions. 
Decision: Accept (Spotlight)