Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the instability of Transformers by studying the progression of attention entropy: first a decrease followed by a quick increase then a long stable phase. A strong correlation exists between the minima of attention entropy and training stability. In the study, decreasing learning rate and increasing the warmup epoch leads to lower training loss. In addition, the following observations were made: Increasing the learning rate can lead to attention entropy collapse.To lower attention entropy,  reparametrizement of all linear layers with spectral normalization is considered.  The proposed algorithm is evaluated on 4 tasks: image classification, image self-supervised learning, ASR, and language modeling.
strength_and_weaknesses: (-) The readability of the paper needs much to be desired. It seems that instability is measured by the entropy of the attention head. Decreasing the learning rate or increasing warmup epoch leads to more stability. Without providing any explanation, the paper mentions that the training loss decreases with increase in learning rate. The paper should state whether decrease in training loss comes from stability. What is the consequence of entropy collapse? Explain how entropy collapse relates with the stability and accuracy? Is  stability and accuracy related? It is not clear how the lower bound of the attention entropy is related to stability.

(-) The paper is proposing a re-parameterization of the weights of the linear layer using a spectral norm. It is not clear how this is related to regularization and performance increase? Many normalization methods have been proposed in the past, and it is not clear how the proposed is different previous normalization methods- a table should compare the performance of previous methods.
clarity,_quality,_novelty_and_reproducibility: (-) The paper is not clear. It introduces a spectral normalization, but the explanation of how it differs from past spectral normalization is not convincingly given.

(-) Many normalization methods have been proposed, and it is not clear how this differs from the past such as Eq(2). The paper discusses briefly the differences, but the constant gamma in the normalization does not seem to be a major difference.  

(+) The paper should be reproducible but it is not certain.
summary_of_the_review: See strength and weakness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the $\sigma$-reparameterised transformer. The authors note a phenomenon in during training of transformers where the 'attention entropy' (essentially, how sharply peaked the attention scores are) decreases and then increases again. In the cases where the transformer fails to learn, this attention entropy does not recover. In order to combat this, the $\sigma$-reparameterisation is introduced, essentially a version of spectral normalization where the matrices are rescaled by a learned parameter (instead of having unit norm). This provides impressive gains in stability of training and empirical performance on a variety of tasks. 
strength_and_weaknesses: Strengths
+ The empirical results are strong, showing increased robustness to changing hyperparameters and similar or better final performance. This is a good result for democratisation of ML, since it reduces the amount of time and money that must be spent on fiddly hyperparameter tuning. Figure 4 is particularly compelling. 
+ The method is extremely simple to implement, yet has (to my knowledge) not been proposed before.
+ The experimental results are fairly comprehensive. 

Weaknesses
+ While I believe the final experimental results stand on their own, on a scientific level I am not convinced by the argument relating the failure to train to the collapse of the attention-entropy. In the case in figure 1, the two phenomena are definitely correlated, but this doesn't appear to continue in figure 2, where the baseline attention entropy 'collapses' but still gives good downstream performance. Similarly, none of the successful baseline models in figure 3 seem to have this 'dip' phenomenon where the entropy decreases then increases as the model begins to learn properly. Similarly, the connection between the singular values and attention entropy doesn't seem to hold practically: look at figure 3 where we have the green solid and dotted line have similar singular values but completely different attention entropy. It seems plausible that the collapse of the attention entropy and the failure of downstream performance could both be symptoms of some other problem, which the $\sigma$-reparameterisation avoids. I realise the authors are careful to not say that entropy collapse causes training instability, but I believe it could be possible to get some direct evidence for this claim, which would strengthen the scientific contribution of the paper. 
  
  + Nitpicks:
    + Could you try a baseline of dividing the initalization by a factor of the spectral norm of a random matrix?
    + In the theory you use $\|W_KW_Q^\top\|_2$, but in the implementation it seems that you apply the reparameterisation on all of the weight matrices. Did you try only reparameterising the $W_KW_Q^\top$ matrix directly?
    + How does the overall memory usage compare when using LARS and fp32 on attention compared to adam and full mixed precision?

clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is quite clear

Quality

The paper is high quality in terms of the experiments. The argument motivating of $\sigma$-reparameterisation from the standpoint of fixing the problem of entropy collapse could be improved. However I wouldn't classify this as the main contribution. 

Novelty

As far as I am aware, the method is novel. However, I am not an expert on transformer regularization, and this approach could well be discussed in an appendix of previous work or similar. 

Reproducibility

The code does not seem to be available, but the method is so simple it could be implemented and reproduced quite quickly (modulo the expense of training a large transformer model nowadays)
summary_of_the_review: The paper is a good empirical contribution to the state of the art of training transformer-based models, giving a new form of parameterisation of the weight matrices that dramatically improves the robustness to hyperparameter choices. This will lead to improved reliability of training transformers. The scientific argument analysing the attention collapse is a bit weak, and could be improved. But since that is not the focus of the paper, it shouldn't be required for an accept. 

Update after rebuttal:
After reading the authors' rebuttal, I believe they have addressed my queries and some of the insightful remarks from the other reviewers.

Update after end of rebuttal period: 
Given the lack of response by the reviewers to the rebuttal to the experimental queries from the other reviewers, I'm unsure to what extent the issues raised by the other reviewers have been satisfied. Since I'm not completely familiar with the state of the art in stable transformer architectures, I've reduced my confidence.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Setting: Transformer model, test on speech recognition and other tasks.

Measure attention entropy.
Find correlation between the minima of attention entropy and the model’s training stability.
(How is training stability measured?)
Reparametrize all linear layers with Spectral Normalization and an additional learned scalar.

Experiments with σReparam on image classification, image self supervised learning, automatic speech recognition and language modeling.

strength_and_weaknesses: Strength:
- Somewhat novel reparameterization of the weights.
- Some analysis on attention entropy and its correlation to the training stability.

Weaknesses:
- Very similar to weight norm, so the novelty is limited.
- No actual comparisons to weight norm?
- Source code not released?
- Results are not good?
- Librispeech train-clean-100 is not representative as a speech recognition task.

clarity,_quality,_novelty_and_reproducibility: Weight norm is mentioned, which sounds very similar from the motivation to decouple the magnitude of the weights. It is claimed that its effectiveness in Transformers is limited. Was that tested? Where are the results? It would be interesting to see such a comparison.

I'm not so familiar with image classification. ImageNet-1k current SOTA seems to be 88.3% for Top1 accuracy (https://paperswithcode.com/sota/image-classification-on-imagenet?tag_filter=171). The provided 82.2% are quite far away from that?

The table 1 should contain some recent SOTA results.

Why only the train-clean-100 subset of Librispeech? This subset is way too easy, way too small, and not really used often in the literature, so not good for comparison. It should be tested on the real standard Librispeech, and potential other relevant speech corpora.

"We use SpecAugment (Park et al., 2019) only activated right at the beginning of training." - What does this mean? Later in training it is deactivated?

"Training We use Adagrad (Duchi et al., 2011) if not specified otherwise," - why Adagrad? This is quite unusual. How does it compare to SGD or Adam?

In Table 3, there should be other results from the literature for comparison. Esp, it is said that the model is used from (Likhomanenko et al., 2021a). Where is that in the table? The numbers in (Likhomanenko et al., 2021a) all look better?

I don't exactly understand the speech recognition results. So it seems that in general, the proposed method yields worse results than the baseline?
summary_of_the_review: Unfortunately there are too many weaknesses which should be addressed. If those are addressed, and it can indeed yield improvements over good SOTA models, on actual relevant benchmarks, then this would be very interesting. But this requires much more work.

If the argument is only about more stable training and not about better performance, this anyway needs to be better justified and tested.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The author has discussed the transformers models' training stability on various tasks. The author has investigated the sharpness of attention concerning attention entropy on each attention head during training. The author observes that the attention entropy first decreases then increases, and then enters into the long stable stage. To address the issue, the author has proposed a simple technique \sigma_reparametrization, which reparametrize all the linear layers with spectral normalization and learnable scalar multiplication. Finally, the author has evaluated the proposed techniques on several tasks, such as supervised image classification, self-supervised learning, automatic speech recognition, and language modeling tasks, and claimed better stability and robustness concerning the choice of hyperparameters.
strength_and_weaknesses: Strength:
The main reason to accept this paper is empirical results and showing performance o various tasks. The author has provided more quantitative results on various tasks.


Weaknesses:

The current literature survey does not discuss the advantages and disadvantages of the paper. The author should validate why I should include this paper in this current work. 

How did you get this equation-2? Is there any intuition of this equation?


The generalization of the method. The equation-2 is valid for the linear layer. What happened to the convolution layer? Is this valuable reparametrization for the convolution network?

The paper does not explain the relation between attention entropy and \sigma_reparameterisation.

How does this learnable parameter contribute to the stability of the transformer?

Instead of evaluating the proposed method on multiple tasks, the author could have done an ablation analysis on different kinds of networks and various types of transformer architecture. The author should provide ablation analysis results to clarify the proposed method's work principle.

Why do we need such reparametrization, learning another parameter? Did you analyze other kinds of techniques to tackle the attention entropy collapse problem? This requires a substantial literature survey and motivation for the proposed method. Can you justify this? How is the attention entropy collapse solved by the proposed reparametrization technique?


The author should compare the stability analysis results of this transformer with the existing state-of-the-art stable transformer. Also need to add a small paragraph in the related work section about stability/robustness in the transformers.

Can you elaborate on the number of parameters and FLOPS required for this? What about the complexity of the proposed method?

The name spectral reparametrization in the title makes it confusing here. It seems it works on frequency domain reparametrization. 

clarity,_quality,_novelty_and_reproducibility: Please refer to the weaknesses section.
summary_of_the_review: 
Justification :
The overall writing quality is ok, and the proposed method is simple and beneficial. However, the experiment comparison lacks justification, and the technical contribution is plain.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewer 3X2m. Rating: 4, Conf: 4.
Reviewer janJ. Rating: 6, Conf: 2.
Reviewer fFS8. Rating: 3, Conf: 4.
Reviewer qd4k. Rating: 3, Conf: 4.

The paper shows during transformer training, the attention entropy typically decreases significantly, before increasing significantly again. They hypothesize that the lowest point of entropy correlates with training stability (and back this with an experiment that shows a correlation between dips in entropy and instability). They then propose to lower bound the entropy through a new method, sigma-reparam. Experiments show that the method results in impoved training stability and robustness.

Reviewers note that the empirical results are fairly comprehensive and strong, and that the method is simple to implement.
Reviewers also note that readability should be improved, and that more comparisons should be done with existing regularizers. As a response, the authors include comparisons to Weight Norm and Spectral Norm. Mot importantly, the authors failed to provide convinving proof that attention entropy causes optimization instabilities.
In general, the authors did a good job responding to the reviewers' critiques. Unfortunately the reviewers declined to further engage, and did change their scores, except for Reviewer janJ who reduced his score from 8 to 6 and confidence from 3 to 2 after the rebuttal period.

On Nov 17, the authors wrote that they found the reviews by reviewers fFS8 and qd4k to be low quality and unfair. However, even when ignoring these two reviews, the score would IMO still be too low for acceptance.
justification_for_why_not_higher_score: Three out of four ratings are reject, readability should be improved, unclear whether training instabilities really caused by attention entropy.
Decision: Reject