Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper consists of two parts. First, it conducts the analysis of an ASR model. Specifically, it measures the diagonality of the self-attention map and notes that the diagonality for the top layers is higher than for the lower layers. Based on this, the paper suggests that the lower layers are responsible for the phonetic information and the higher layers are responsible for the linguistic information. Then, the paper confirms this hypothesis by using each layer for a phoneme classifier.

Second, based on the above observations, the paper proposes a method to exploit this division into the phonetic-linguistic layers. The proposed method is to reuse the attention maps across several layers. This modification allows for more efficient architecture both in terms of latency in train and test time. In some cases, the proposed architecture outperforms the baseline.
main_review: # Correctness and Significance

The paper is logical and methodical in the investigation. The proposed measurements and experiments make sense. The CAD metric makes sense. This won't change the result of the work much, but CAD looks unnormalized: the double sum seems to require 1/T^2 term, not 1/T.

The phoneme analysis is conducted well. Both the phoneme classification accuracy curve and the phoneme attention relationship metrics confirm the paper's hypothesis on the phonetic-linguistic division. Nevertheless, I would like to see a discussion on alternative hypotheses. I don't have a concrete suggestion here, but is it possible that the result obtained in Figure 2 is spurious or can be explained by something else.

Finally, the proposed architecture for the attention map reuse is a logical step to improve the model. The paper reports multiple architecture variations and compares them. I found the most inspiring that this technique provides a way to reduce latency.

# Clarity

The paper is very well structured and well written. It was a pleasure to read it.

# Minor things

- The graphs are hard read when printed. Especially in black and white.
- Session -> Section
summary_of_the_review: In summary, this is a good exploration into the transformer architecture for speech recognition. The paper provides valuable insights into the transformer's attention map structure and connects it with the phonetics. Then the paper uses these insights to improve the model.

I would like to request the alternative hypotheses on why the attention becomes diagonal in higher layers. Or some discussion on this.

Then, the figures need to be improved for reading when printed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores how self attention maps in conformer based ASR models are distributed across layers. The authors note that the lower layers of the model capture phonetic information, and the variance of the attention map distribution for the same class of phones across the utterance goes down. The higher layers do not display this behavior. The authors hypothesize that the higher layers capture linguistic information that likely combines information across phones.

Based on the observation that the variance of the attention map distribution goes down, the authors propose reusing the attention map from one layer across multiple layers. This reduces the amount of computation needed for inference, and reduces the training time. Tying attention maps also results in small improvements in performance.

main_review: I thank the authors for the additional results and clarifications. Overall, I still feel that the analysis is interesting, but the experimental results are somewhat lacking. It will be interesting to see whether the analysis holds for more popular end-to-end models.

---------------------

Strengths:
* The authors present an excellent analysis of how attention maps are distributed across layers, and find interesting patterns – that lower layers present phonetic information and upper layers more local linguistic information. The figures present enough support for the claim.
* The authors are able to reduce the inference computation by a factor of 1.96 for a 30 second audio, with marginal improvements in WER.

Weaknesses:
* Increasing diagonality of the attention maps has been observed in prior work as well (as the authors have cited and discussed). While the present work improves this understanding (mainly the phonetic information of the lower layers part), the analysis is not entirely novel.
* There are several approaches to improving inference speed by directly addressing the attention map computation. For example, for ASR, prior work has looked at using masked attention where-in each frame only attends to L frames to left and R frames to the right [1, 2, 3]. Therefore, each frame is computing attention only using at (L + 1+ R) frames. This addresses the quadratic increase in computation. 
* Alternatively, Performers [4] also proposes linear time and space complexity for attention computation. It would be useful to have some of these comparisons to the presented method.
* The authors’ analysis suggests that the attention map in lower layers is reusable, but those in the upper layers is not (Fig. 5(b)). But their best strategy ties maps for both lower and upper layers. This feels counterintuitive. Is there a reason why that is so?
* How much of the analysis is tied to the CTC-ASR model used? What about LAS [5] or RNN-T [6] models, which are widely used as well? 
* The results on Librispeech are much worse than other state-of-the-art (SOTA) end-to-end models (e.g., [7]), most likely because the authors use CTC. It would be useful to see if the techniques result in SOTA performance when using LAS or RNN-T.

Other remarks:
* Figure 1 has too much information and is a little hard to follow. Consider either improving the figure description, or splitting into multiple parts.
* Sec. 3.2: The diagonality analysis is not that different from Zhang et al., and arrives at very similar conclusions. For the models authors consider, are the plots different when using the metric in Zhang et al.?
* Sec. 3.2: Furthermore, it’s not very clear from the description what r(T - 1) is. 
* For the plots, is PAR and CAD summarized per utterance or per corpora?
* Figure 6 and the description is hard to follow. What does it mean to set averaged PAR of the baseline to 1.0? Consider moving some of the details from the Appendix to the Sec. 3.4 since PAR is one of the main proposals of the paper.
* Figure 6: The authors claim that a coverage of 0.974 is bad. It’s unclear why. 0.974 seems quite high.
* Does adding more heads to the baseline improve performance? It seems like using 8 heads works the best for the shared attention map model, but the baseline only uses 4.

Minor:
* Abstract: “We propose a novel metric …” The sentence is too long and a little hard to follow, especially with so little context since it is appearing in the abstract.
* Abstract: standardizes the phonetic variance -> reduces the phonetic variance?
* Introduction: 30 seconds … 750 frames: Will be worth adding what is the processing window size (40 msec?).
* Sec. 2.1: (d_h)^2: Consider using special chars like \dag or \ddag for footnotes to avoid confusion. As of now, it looks like d_h is squared.
* Sec. 3.2: What does h stand for in CAD_h?
* Sec 3.3: (1) First, … (2) Second, .. : First and Second are redundant.
* Consider using just 1 decimal point for results. 2 decimal points does not really add a lot of value, comparison-wise.


References:

[1] Zhang, Qian, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. "Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss." In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833. IEEE, 2020.

[2] Audhkhasi, Kartik, Tongzhou Chen, Bhuvana Ramabhadran, and Pedro J. Moreno. "Mixture Model Attention: Flexible Streaming and Non-Streaming Automatic Speech Recognition." Proc. Interspeech 2021 (2021): 1812-1816.

[3] Tripathi, Anshuman, Jaeyoung Kim, Qian Zhang, Han Lu, and Hasim Sak. "Transformer transducer: One model unifying streaming and non-streaming speech recognition." arXiv preprint arXiv:2010.03192 (2020).

[4] Choromanski, Krzysztof, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins et al. "Rethinking attention with performers." arXiv preprint arXiv:2009.14794 (2020).

[5] Chan, William, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. "Listen, attend and spell: A neural network for large vocabulary conversational speech recognition." In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960-4964. IEEE, 2016.

[6] Graves, Alex. "Sequence transduction with recurrent neural networks." arXiv preprint arXiv:1211.3711 (2012).

[7] Gulati, Anmol, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han et al. "Conformer: Convolution-augmented transformer for speech recognition." arXiv preprint arXiv:2005.08100 (2020).

summary_of_the_review: Overall, the authors present an interesting analysis of the attention maps, which in the reviewer’s opinion is the strongest part of the work. It is interesting to categorize lower layers as phonetic, and upper layers as linguistic. That being said, similar analysis has been done in prior work. The presented results are also lacking; it is unclear if the method is very specific to CTC and whether it generalizes to other more popular end-to-end techniques. The authors have also not considered alternative strategies to minimize computation that directly addresses the quadratic computation of attention on long sequences.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper analyses the multi-head self attention behaviour in CTC based ASR models.
They identify the working regimes, namely, phonetic and linguistic localisation. 
The paper argues that lower MHSA layers build phonetic attentions patterns while upper layers do that linguistically showing some diagonal pattern similarly to NLP tasks. Based on these findings the authors propose to share attention maps across layers.
This yields some modest improvements in terms of WER but obtains larger speeds-up with respect to a basic Conformer model.
During the study they introduce new measurements such as cumulative attention diagonality (CAD) and phoneme attention relationship (PAR). 
main_review: The paper is very well presented with a large battery of carefully designed experiments and conclusions.
In my opinion, It can naturally be divided into 2 main parts, the study of the MHSA and the layer-wise attention map reuse.

The first part, namely, the analysis of the different emerging patterns of the multi-head self-attention on each layer is very well supported and very interesting. For instance, the layer-wise analysis of phonetic attention was carefully designed to avoid the diagonal contributions of the same phoneme, and the CAD was proposed to account for the cumulative diagonally of layers. The authors found and provide supporting evidence of 2 different regimes in the MHSA. In the upper layer they show that there is a strong diagonal attention which they attribute to linguistic localisation. The evidence for such an intuitive hypothesis is vague in this specific work, and leverages conclusions from other papers, however, they don't provide a strong experimental evidence for that claim. In practice, this is just a semantic issue as there is strong evidence for the emergence of such diagonal pattern in the paper and previous works. 

 The lower layers that show a lesser diagonal behaviour are studied from the phonetic perspective, which this is typically ignored. They then hypothesise that those lower layers are performing phonetic localisation and provide evidence for that by:
 1 - adding probes of clarifiers after each conformer layer and showing that there is a peak of classifier performance for phoneme classification at the middle layer.
 2 - computing the phoneme attention relationship (PAR) per attention heads. I found figure 3 confusing, as there are many white dots --maybe without enough attention probability ?-- not commented in the attention pattern. In contrast, I found Fig 5,  Fig 11 or Fig 12 much more illustrative and clear. In addition,  Figure 5 is an average and it would be interesting to contrast the phonetic properties of each head to understand whether there are emerging patterns. It is not very clear whether the selected output vocabulary is affecting in any way to these patterns as they are using 128 sub-word tokens. Would this pattern hold for 512 or more sub-words tokens ? 
3 - computing the PAR sharing the attention maps.

In the second part of the paper, the authors propose to reuse attention maps in these 2 regimes, and they prove essentially that those maps can be reuse while keeping the number of parameters the same. This yields small WER improvements and large speed-ups. 
I found the constraint on the number of parameters necessary but I would also have liked to see which is the performance when the maps are shared but the hidden value dimension is not increased, as that would have provided a very interesting point for the prior analysis of the first part of the paper. In addition and while the selected pattern make sense for optimising the search space, looking at the results of the phoneme classifications probes, one would have expected maybe additional non-symmetric share patterns for lower layers, like 1(H4)+1(H4)+6(H4) for instance to help to build the phonetic attention from the convolutional front-end.

Finally, the architecture used in the paper is a Conformer architecture which includes a convolutional layers in between MHSA layers. 
Although the authors discuss this potential issue in the appendix, I think it is valuable to discuss it in the main text (maybe I missed it).
Having performed one of the experiments turning off the convolutions at least for the phonetic layers would have been really interesting and would have provided a stronger conclusion. 

I really enjoyed the paper. 
Nice Job.


summary_of_the_review: The paper is well-written and provides a lot of insights into how CTC models with MHSA learn. 
I think this is a very interesting experimental analysis of the properties of the Conformer/Transformer models that can help to understand non-autoregressive end2end ASR models. The claims are very well supported and the practical conclusions and implications are very interesting. The code is provided and it seems possible to reproduce the paper.
I would have liked to see more experiments on the paper but just because the results are very interesting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper develops a mechanism to analyze self-attention for speech recognition, and further use the resulting insight to facilitate attention reuse for reducing inference cost. The analysis identifies a distinct pattern between lower half and upper half layers on the ASR encoder through cumulative attention diagonality and characterizes the lower half as finding phoneme localization and upper half as finding linguistic localization. The paper justifies the characterization of phoneme localization by illustrating an example of such a phoneme localization, and also shows that the output of the layers in the middle provide better phoneme classification accuracy than the upper layers. An additional analysis, called phoneme attention relationship, further quantifies the correlation between phonemes and attention, and shows that the lower layers have higher correlation. These insights are used to design attention reuse across layers, and shown to maintain the quality while improving the speed on Librispeech.


main_review: Strength

The analysis part of the paper can be a good contribution to the speech recognition community. The distinct increase after the first half of layers on cumulative attention diagonality brings an interesting insight on the behavior of the speech recognition model, and the characterization of phonetic and linguistic localization is convincing.

The phoneme analysis authors conduct in the paper provide a good insight for readers to understand the correlation between attention and phoneme classes.
The phoneme attention relationship is a good metric to understand the correlation between attention and phoneme, and provide a good measurement on the attention coverage for speech recognition

Overall the proposed analysis mechanism can benefit beyond improving inference speed, for example, it will be a great analysis tool for understanding representation learning which is beneficial for speech self-supervised learning.

Weakness

The downstream task this paper focuses on, improving speed with attention reuse, does not bring out the full potential of the proposed analysis mechanism. In fact, it seems the connection between the analysis and the task is not very strong, as one can design those configurations listed in Table 2 without the insight of phoneme attention relationship. It would serve authors better to identify other more correlated tasks.

summary_of_the_review: The analysis proposed in this paper can be a good contribution to the community. The task focused in this paper, however, is on the weak side. Overall the analysis is beneficial to the community, and can be considered for acceptance.
review_title: null

Meta-review: The paper conducted a thorough experimental analysis of the attention map in the Conformer models for CTC based speech recognition models and connected it with phonetic and linguistic information in the speech. Using these insights, the paper presented some computation improvement and marginal quality gains. The authors actively conducted additional experiments to further justify the claims. The paper is strong in terms of the systematic way of in-depth analysis and further development (i.e. sharing the attention map across layers for speedup). But as pointed out by the reviewers, it lacks some comparisons with other alternatives to justify the importance of sharing attention maps in reducing computations.  Also it would be better if there's justifications on how the observations generalize to other types of models (such as LAS, RNN-T). 

The decision is mainly because of the thorough analysis conducted in the paper which can be a good contribution to the community.
Decision: Accept (Spotlight)