Understanding the Role of Self Attention for Efficient Speech Recognition
Authors: Kyuhong Shim
Jungwook Choi
Wonyong Sung
Venue: ICLR 2022 Spotlight (2022)
Keywords: transformer
self attention
speech recognition

Abstract: Self-attention (SA) is a critical component of Transformer neural networks that have succeeded in automatic speech recognition (ASR). In this paper, we analyze the role of SA in Transformer-based ASR models for not only understanding the mechanism of improved recognition accuracy but also lowering the computational complexity. We reveal that SA performs two distinct roles: phonetic and linguistic localization. Especially, we show by experiments that phonetic localization in the lower layers extracts phonologically meaningful features from speech and reduces the phonetic variance in the utterance for proper linguistic localization in the upper layers. From this understanding, we discover that attention maps can be reused as long as their localization capability is preserved. To evaluate this idea, we implement the layer-wise attention map reuse on real GPU platforms and achieve up to 1.96 times speedup in inference and 33% savings in training time with noticeably improved ASR performance for the challenging benchmark on LibriSpeech dev/test-other dataset.


Published as a conference paper at ICLR 2022
UNDERSTANDING THE ROLE OF SELF ATTENTION FOR EFFICIENT SPEECH RECOGNITION
Kyuhong Shim' Jungwook Choi², Wonyong Sung'
Department of Electrical and Computer Engineering, Seoul National University! Department of Electrical Engineering, Hanyang University? skhu200 snu kr. choi jhhnjahay kr, wysung@snu.
ABSTRACT
Self-attention (SA) is critical component of Transformer neural networks that have succeeded in automatic speech recognition (ASR). In this paper, we analyze the role of SA in Transformer-based ASR models for not only understanding the mechanism of improved recognition accuracy but also lowering the computational complexity. We reveal that SA performs distinct roles: phonetic and linguistic localization. Especially, we show by experiments that phonetic localization in the lower layers extracts phonologically meaningful features from speech and reduces the phonetic variance in the ufterance for proper linguistic localization in the upper layers. From this understanding, we discover that attention maps can be reused long their localization capability is preserved. To evaluate this idea, plement the layer-wise attention map reuse on real GPU platforms and achieve up 1.96 times speedup in inference and 33% savings in training time with notice- ably improved ASR performance for the challenging benchmark on LibriSpeech dev/test-other dataset.
im-
INTRODUCTION
Recent advances in end-to-ent automatic speech recognition (ASR) have been driven by Trans- former models (Vaswani al. 2017). Transformer was first introduced for natural language process- ing (NLP) tasks such neural machine translation (Vaswani et al., 2017 Ott et al. (018),)) language modeling (Dai et al. 2019 Rae et al. 2019), and text generation (Raffel al. 2020). Thanks to its superior performance in processing sequence input, Transformer has heen widely adopted in vari- ous state-of-art ASR models (Zhang et al. 2020b Ng et al. 2021 Guo et al. 202I). Self-attention (SA) is a core component of bransformer-based ASR, which dynamically collects information from multiple frames of an audio sequence. However, the computation and memory costs of SA increase quadratically with the length of sequence, which is particularly problematic for ASR. For exam- ple, just 30-second utterance corresponds to about 750 frames with a widely used window stride of 40ms.
Understanding the role of SA may provide essential insights for the efficient design of Transformer- based ASR models. Extensive studies have examined the behavior of SA in the field of NLP (Koval-
eva et al. 2019: Park et al. 2019: Gong et al. 2019: Rogers et al. 2020). Recently, several studies Turther attempted discover the characteristics of SA the speech domain. Yang al. (2020) revealed that self-attention features in the self-supervised audio Transformer are categorized into global, vertical, and diagonal patterns. Zhang al.) (2021b) focused on the diagonality of upper SA layers in ASR models for improving efficiency. However, these prior works revealed limited insights on the patterns discovered SA, constraining use for improving model efficiency. Thus, providing holistic view of the role of SA is desirable for efficient ASR model design.
In this work, we reveal that SA plays two distinct roles in the success of Transformer- ASR models: phonetic and linguistic localization, illustrated in Figure [ First, phonetic localization of lower SA layers attends to the phonologically meaningful global context. Second, linguistic lo- calization of upper SA layers mainly attends to the local context of near-diagonal attention map. We hypothesize that the phonetic variance in utterances such as variations in pronunciation is stan- dardized in the lower SA layers that the upper SA layers identify local linguistic features

---

Published as a conference paper at ICLR 2022
Different Attention Maps
Hhraud-
Text script
Audio features
Phonetic
Linguistic localization
localization
1.96x faster inference 33% reduced training time
#hoods
WER 8.40 - 8.05 % LibriSppech rest-other
Audio features
Layer-wise Attention Map Reusing
Figure I: Illustration of the role of SA layers in Transformer-based ASR models and the proposed layer-wise attention map reuse. We discover that lower layers and upper layers show different be- havior.
for accurate transcription. To investigate the behavior of SA layers, we propose phoneme attention relationship (PAR) to explain how phoneme localization works quantitatively. Interestingly, we dis- cover that the phonetic localization represents the traditional knowledge on phonetics; for example, labial, velar, nasal phonemes tend attend each other.
Based on this understanding, we propose a practical method for efficient ASR model design. We reuse the attention maps of SA layers while preserving the phonetic localization capability of the lower SA layers, resulting in up to 1.96x speedup in inference and 33% savings of training time with considerably improved performance in challenging ASR tasks (LibriSpeech dev/test-other dataset).
Our contributions can be summarized as follows:
We reveal that SA layers contribute to ASR with two distinct roles: phonetic localization in the lower layers and linguistic localization in the upper layers. This unique distinction leads to an in-depth analysis of phonetic SA for the first time. We further propose phoneme attention relationship (PAR) quantitatively identify the role of phonetic localization.
We propose layer-wise attention map reuse for efficient Transformer-based ASR models. In particular, we discover that attention map reuse is possible in lower SA layers as long the phonetic localization property quantified by PAR is preserved. We demonstrate with the popular ASR model and dataset that ASR performance can be maintained or slightly improved even if the attention map is reused.
We implement the attention map reuse real GPU platforms and achieve up to 1.96x in- ference speedup and 33% savings in training time, demonstrating that the proposed method is practical.
2 BACKGROUND
2. ASR ENCODER AND SELF-ATTENTION
SA is usually utilized as a module inside the ASR model consisting of stacked Transformer encoder layers. ASR encoder takes sequence of short-time
(STFT) audio features, known as a 'famme, as input and extracts high-level feature of each frame through multiple layers. As illustrated in Figure I the extracted high-level feature changes for each layer; stacked SA layers
2

---

Published as a conference paper at ICLR 2022
first extract phonetic features from audio features and utilize these features build linguistic features for the output transcription.
We briefly review the SA computation procedure. Consider sequence of d-dimensional column vectors X = { 22, ...Tr as input. Each vector corresponds to each frame of speech where the total number of T frames (K), and value (V) of h-th attention head as follows:
included. The input feature vector X is projected to query (Q), key
4h = Wh Ti: Kn,1 = W Es, Uh = -W" (WR. W W E &R (Rdxdd
(1)
WQ.K.V indicates projection matrices for Q. K, V, respectively. dn d/H is the dimension of each attention head where H is the number attention heads. The attention map (A₂), which represents how much frames attend to each other, is computed by scaled dot-product operation followed by softmax. The resulting attention map takes a form of a 2D matrix where each row probability vector. A single element of the attention map (A) represents how much i-th frame attends to j-th frame. The attention head (dx) is weighted sum of V using the attention map weight. Note that each attention head corresponds to different attention map A&; this multi-head design enables focusing on various perspectives within single SA layer. The output O {01, 02, .2.r. is computed by the projection (WO E Rdxd) on the concatenated attention h
ghaskinis ddn
Aa Softmax
dn,i = Σ Anli, Oz = W Concat(da.))
h
SA layer contains (dd2) parameters.Ar shown in equations, SA requires quadratic computation and memory complexity 2)T²), in exchange for the ability to access any location in the sequence. When N layers of SA are stacked, the burden proportionally increases.
2.2 PREVIOUS WORK ON SELF-ATTENTION ANALYSIS
ASR considers both phonetic and linguistic aspects transform audio input to text output. However, the studies on NLP mostly analyze the linguistic characteristics of SA, and the studies on self- supervised audio representation learning (SSAL) mainly focus the phonetic behaviors of SA. The valuable findings from both domains cannot be directly applied to ASR.
The behavior of SA has been widely studied in the NLP domain (Rogers et al. 2020), mostly focused on BERT (Devlin et al. 2019). a self-supervised language representation learning model. Kovaleva et al. (219)) and Guan et al. (2020) suggested that attention patterns can be clustered into several groups and the pattern may change depending on the fine-tuning task. Clark et al. (2019) and Tenney et al. (2019) observed attention maps that correspond to linguistic concepts of the language. Voita al. (2019) also characterized linguistic attention heads and connected the knowledge to efficient model structure. However, studies on NLP only provide analysis on linguistic attention.
NLPP
SSAL Recently, several studies have been introduced to understand how the audio information is encoded in SSAL models, such as CPC (Oord et al. 2018), Wav2Vec 2.0 (Baevski et al. 2020 Mockingjay (Liu al., 20200) HuBERT (Hsu al.) 2021). and Audio ALBERT (Chi et al. 2021
Ma et al. (2021) and Shah et al. (202T) demonstrated that wide spectrum of phonetic information included in these models. Especially, Yang et al. (2020) categorized attention maps into three cate- gories: global, vertical, and diagonal, where diagonal heads attend to local frames and vertical heads either focus or neglect specific phonemes. However, Yang et al.] (2020) only discovered attention patterns without explanation how phonetic feature extraction is achieved with these patterns.
ASR Previous works have investigated the redundancy of attention maps mainly based on di- agonality. From the observation that attention maps upper layers show highly diagonal patterns, Zhang et al. (2021b) proposed replacing upper SA layers to feed-forward layers without performance loss. Zhang al. (2021a) removed SA heads of high diagonality during training as a regularization
In this paper, SA is equivalently used as multi-hcad SA. "The two terms 'head" and "attention head' are used interchangeably.
3

---

Published as a conference paper at ICLR 2022
0.9
Cumulated Attention
Diagonality (CAD)
0.85
Figure 2: Cumulative at- tention diagonality (CAD) of each attention head. Four points for each layer correspond to the CAD of four attention heads. The black line connects the median across the layers.
0.8
0.75
0.65
0.6
Index
but keep every head for the test time. Similarly, Chang et al. (220)) introduced adaptive attention span where each attention head equips different attention span width to reduce the sequence length for the computation, starting from the intuition that some heads only attend to neighboring frames. These approaches mainly focus on reducing the burden of diagonal and concentrated attention, how- ever, diagonality- based analysis has limitations in optimizing phonetic attention. We distinguish SA into two groups and provide proper analysis for each.
UNDERSTANDING THE ROLE OF SELF-ATTENTION IN ASR
3.1 ANALYSIS SETUP
We train and evaluate the model on the LibriSpeech-960 (Panayotov et al. 2015) dataset. The dataset include two types of data, clean and other, where other contains more challenging utterances. We extract the 30-dimensional log-Mel filterbank feature from a 25ms window with a stride of 10ms. We use 128 sub-wort tokens as vocabulary, built on SentencePiece (Kudo & Richardson, 20188 library using the byte-pair encoding (Sennrich et al. 2016). The analyses are performed on LibriSpeech test-clean dataset unless specified.
We use Conformer (anformer-M(medium) (Gulati et al. 2020) as the baseline ASR encoder, trained with CTC (Graees et al. 2006) loss. Conformer variant of Transformer. which combines an additional convolution module to enhance the ability to collect local neighboring features. We follow recently
introduced state-of-the--at ASR studies (Zhang et al. 2020b; Ng et al. 202T Guo et al.] 2021) that have adopted Conformer their ASR encoder. We train the model with AdamW (Loshchilov) & Hutter, 2018) optimizer for 200K iterations. Please see Appendix A. and A.2 for the model configuration and training details.
3.2 DISTINGUISHING BETWEEN PHONETIC AND LINGUISTIC SELF ATTENTIONS
To understand the role of SA, we start by examining the attention map AG, which characterizes the functionality of SA. The attention map indicates how frame attends the other frames in terms of probabilistic distribution for each attention head. Thus, analysis on attention maps across the heads of a SA layer would discover important characteristies of SA. To measure the diagonality of the attention map, we introduce cumulative attention diagonality (CAD) defined as the integral of the sum of attention probabilities constrained by distance below:
CADA = -f" ΣΣ i=
Aulin . IlIi STT 1)]dr
(3)
where is the number frames in a sequence, determines the range of distribution in the tention map under test, and h is an index of the attention head. Appendix B.I.I provides detailed explanations of above equation and visualizes some CAD examples.
Figure 2 shows the attention diagonality analysis for the SA layers. There is clear transition of CAD from lower layers (layer 1-8) to upper layers (layer 9-16). We emphasize this is a unique trend observed in ASR models compared to other domains: Kovaleva et al. (2019) classified attention maps in BERT but did not report the grouping of the same types. Yang et al. (2020) also categorized attention maps in SSAL but those categories broadly appear through layers. In ASR, Zhang et al.
(2021b) observed a similar diagonality pattern for SA layers, but they considered as an increase of diagonality over layers and did not separate the distinct patterns.
4

---

Published as a conference paper at ICLR 2022
index (i)
Phoneme
MISSUS
BIRDS
FISHES
TARTS
MYSELF
TASK
SUCCESS
BEASTS
NEVER
BIRDS
TARTS
DINNER
THOROUGH
ER
R
ER
R
K
FISHES
0
IDEA
BE
SHE
YY D
THROUGH PROVIDING
2
BEASTS
Frame indox (1)
Figure 3: Visualization the phonetic localization. Each element corresponds to Ali. where j indicates the frame index. Several that correspond certain phoneme, give higher attention to similar phonemes across the columns. For better visualization, we selectively draw boxes on three representative patterns (S, ER, and IY).
From the diagonality analysis, we categorize the role of SA layers into two parts: phonetic and linguistic localization. In ASR, linguistic localization refers to the behavior of the attention map that focuses on local (near in distance) frames and aggregates the information for text transcription. Since these local frames are particularly important in audio to text transcription in ASR, the attention map is characterized with a diagonal pattern. As shown in Figure 2 the upper layers tend to exhibit diagonally dominant patterns, implying their role as linguistic localizer. We visualize two cases of diagonal attention patterns in Figure | extracted from layers 12 and 15 of the baseline model. Our observation is consistent with the prior work such as Zhang al. (2021b). On the other hand, there has been little discussion about the role of lower layers in ASR in the context of phoneme localization. We discuss this in detail in the next section.
3.3 CHARACTERISTIC OF PHONETIC LOCALIZATION
Phonetic localization denotes the role of the attention map that focuses on similar (near in content) frames and extracts the phonologically meaningful features. We observe two characteristics of pho- netic localization. First, phonetic localization is realized as attention to similar phonemes across the sequence. Second, the localization transforms each corresponding frame more likely others.
To analyze phonetic attention, we exploit the phoneme information to find out the relationship be- tween attention and phonemes. For the LibriSpeech dataset, we use the frame-level phoneme align- ments obtained from Montreal Forced Aligner (McAuliffe et al. 2017). Table [ lists all phoneme classes. Please visit Appendix
A.3 for details on phoneme pre-pree peeprocesssig.
Idx. Phn.
2
3
9
10
12
15
16
Idx. AW
AY
All
EY
111
UW
NG
20
21
22
24
25
27
30
31
32
33
34
35
Phn.
DH
G
K
SH
TH
JH
W
Table 1: Phoneme index for the analysis. Phonemes are extracted from the LibriSpeech lexicon and collapsed into 36 classes. Phonemes are reordered according their phonological properties.
Figure demonstrates the first characteristic of phoneme localization. The attention map presents that the same or similar phonemes tend to assign high attention weight to each other, for example, ('S' to 'Z'), ('R to RR'), and ('IY' to 'IH').
We evaluate our second statement by phoneme classification on hidden layer representations (out- put of layers, also known as a hidden activation hidden vector), similar to previous approaches Baevski et al. 2021 Liu et al. 2021). We extract hidden layer representations and train a soft- max classifier for each layer. Input is 256-dimensional vector and output contains total 37 output classes (36 phonemes "silence"), See Appendix A.5 for the details on training the classifier and visualization of the confusion matrix.
5

---

Published as a conference paper at ICLR 2022
0.83
Phoneric Classification
Figure 4: Phoneme classi- fication accuracy Lib- riSpeech test datasets. The zeroth entry implies the classifier is trained from hidden representations ob- tained before the first SA layer.
Accuracy (%))
0.7
0.55
test-clean
test-odker
0.45
10
13
16
Luyer Index
Segureesshone shows the phoneme classification accuracy for different layers. The accuracy increases for the lower layers (layer 1-8) where phonetic localization dominates. Specifically, the accuracy of layer (before SA) is only 53.8%, but consistently increases to 81.7% at layer 8. The phoneme classification accuracy indicates how well hidden vectors can be distinguished according to their phoneme classes. Therefore, the accuracy increase on lower layers implies that hidden layer repre- sentations
standardized by phonetic localization. We assume that the accuracy decreases for the upper layers because they perform the linguistic localization and convert phoneme- in- formation output text.
3.4 LAYER-WEL WIRE ANALYSIS OF PHONETIC ATTENTIONS
In phonetic attention, we observed that similar phonemes attend each other and the phonetic features are clustered through layers. For the next step, we investigate the layer-wise behavior of phonetic attention understand the contribution of each phonetic attention layer. We introduce phoneme attention relationship (PAR) to understand how SA processes phonological information by exposing how much each phoneme class attends the other phonemes on average. Specifically, we directly map attention probability As [i. to [p, g] where i-th frame and j-th frame correspond to phoneme p and q, respectively. two phoneme classes p and g are different, we simply transport the probability from A, j] to Pa g]. On the other hand, if two frames are within the same phoneme class p. we exclude consecutive frames of the same class to avoid unnecessarily emphasize the effect of diagonal attention maps. PAR is computed as below
pp
T
Σ
Σ
An(i,i (x+1)
(4)
1Cpl [Cgl VEC, JEC, 1
T C
Px(p,p)
Σ
Anli,J]
(p =
(5)
C
(1)
SEC,
jEC,-E,
Cp. C2 indicates the set of frame indices that correspond to phoneme class p and q. (1) indicates the number of frames that satisfies conditions: belong to the same class i-th frame and all frames between itself and i-th frame also belong to the same class. In other words, Cp Ei) indicates the subset of C, that are not connected i-th frame by consecutive class p frames. For the Ps [r p] calculation, we do not include consecutive frames of the same class. We discuss the purpose of this exclusion in Appendix B.2.1
We visualize the average PAR of the lower half and upper half of layers in Figure 5 A prominent diagonal component appears, representing that phonemes put high attention to themselves. Inter- estingly, we also discover well-known phonological characteristics in the lower half of layers. For example, labial (B, P), velar (G, K), and alveolar (S, Z.) consonants highly attend each other. Nasal phonemes (M, N, NG) also show a high correlation. From the empirical observations, we denote that phonetic attention map creates heterogeneous patterns. Thanks to the multi-head structure, single SA layer can capture multiple phonetic relationships. Please see Figure T2 in Appendix B.2.2 for various PAR examples for each head.
If each phonetic attention map corresponds to different relationships. reuse phonetic attention maps across multiple layers? We answer this question by introducing the PAR coverage, which
PAR as well CAD averaged through the corpora.

---

Published as a conference paper at ICLR 2022
(AA, AE, AW,
(ER, R
(EY,
(UH, UW)
(M,N, NG
(B,
(DH, TH
(G,K)
(CH,
(S.Z)
30
(CH. SH, JH)
10
15
20
25
30
35
(a) Lower half of layers
(b) Upper half of layers
Figure 5: Averaged phoneme attention relationship of the lower half (Ist, 8th) and upper half (9th, ... 16th) layers. The result averaged through layers and heads test-clean dataset. Each row and column corresponds to the phoneme index. Brighter (yellow) values indicate stronger attention between phonemes. Elements that stand out are highlighted, where phonemes with similar properties tend to attend to each other.
Reusing Config.
Avg. Per-layer Coverage (%)
Accumulated Coverage (%)
Accumulated Coverage (%)
0.975
8
0.95
4
1x16
0.925
2x8
0.875
4x4
0.968
0.016
0.992
0.85
1x
-x-2x8
8x2
0.973 0.000
0.974
0.825
2
Layer Index
Figure 6: Accumulated PAR coverage of lower layers (1st, 8th). Averaged PAR of the baseline (Figure 5)a) set to 1.0, which considered be desirable reference. The left plot on the accumulated coverage shows how each layer participates in covering the strength of the relationship. The right table summarizes the coverage of different reuse configurations. higher average per- layer coverage ratio implies that each layer performs more similarly to the baseline.
indicates how much each layer covers the phonetic relationship represented in the averaged PAR from the baseline (Figure(5(a)) The averaged PAR the baseline is considered represent all the essential phoneme relationships. Please refer to Appendix A.4 for details.
To investigate the effect of reuse, we plot the accumulated PAR coverage for different reuse configu- rations in Figure In calculating the accumulated coverage ratio, we take the average every PAR under certain layer and compare with the reference PAR. The accumulated coverage consistently increases to 1, which means that the missing relationships are fulfilled through layers. We test four configurations (will be introduced in detail the next Section) x Y, where layers share the same attention map. As the number of reuse increases x 4x4 8 x 2), the average per-layer coverage also grows, which implies that each SA layer tries to capture more phonetic relationships to recover the performance. However, for 8 2, the model fails to fully cover the reference, repre- sented as low accumulated coverage ratio of 0.974. In Appendix B.3 we visualize the effect the reuse PAR coverage where phonetic features captured in 2 8 and 4 x are missed in 8 2.

---

Published as a conference paper at ICLR 2022
Table 2: Word rate (%) for different attention map reuse configurations. "HX" indicates that the number of attention heads in the self self-attention layer is set to X. All configurations carry almost the same number of parameters, No external language model is used.
Configuration
#Heads
Head dim.
dev-clean
dev-other
text-clean
test-other
1(H4) x 16 (baseline)
64
64
3.1
8.3
3,2
8.4
2(14)
32
3.0
8.2
3.3
4 + 4(88)
32
3.1
8.1
3.2
4(H2)+44113 +(H4))4()))))))) 4(14)
12
128/64
3.1
8.2
3.4
8.3
4(14))((14)++ +((44 +
16
64
3.0
8.2
3.3
8.2
4HH))+((8))+
4(H4)
24
32/64
3.1
8.3
3.2
8.4
4(H4)+4(H4) (0)++4(2)) (/((H2) 4(H2)
12
64/128
3.1
8.5
3.4
8.5
4(H4)+4(14) +(444) +-
16
64
3,0
8.2
3,3
8.2
4(14)+4(14)+++ (1A(H44++4(18) 4(H88) 4(H8)
24
64/32
3.1
8.2
3.3
8.1
(114)+((H4)) 8(H4)
12
3.1
8.3
3,3
8.2
4(44)+4(HH) (-44))4(HH) 4(H4)
12
64
3.1
8.5
3.3
8.6
8(H4) 8(H4) 8(13) 8(88)
8
3.3
8.8
3.6
8.7
16
8.5
3,4
4 LAYER- WISE ATTENTION MAP REUSE
We propose layer-wise attention map a method to design efficient Transformer-based ASR encoder by reducing the heavy SA computation. The core idea is to reuse the computed attention map from the previous layer. More specifically,
reuse attention map of (-th SA layer to (1+1), (I+ 2), M - 1) consecutive SA layers. If a single attention map is shared through M layers, the computation burden of SA be reduced by M times. During training, the reused attention map receives gradients from M layers. This layer-wise reuse is easy to implement and fully supported by modern accelerator hardware. The idea of reuse attention map through layers have been proposed for NLP (iiao et al. 2019 Ying el al. 202I) but not tested for ASR. We discuss the difference in Section For SA layers that receive the pre-computed attention map, query and key are used and can be removed. To compensate the parameter size for those layers, we simply double the output dimension of (WV) R2dd Wo Rdx2d).
Table shows the word error rate (WER) on LibriSpeech dev and test dataset. Configuration "X x y" indicates that X successive layers are grouped to share the same attention map and total Y groups are built. Therefore, there exist HXY unique attention heads for each model. We also use the notation '+', for example, 4 x is identical + + 4. For each configuration, we train the model from scratch with the same training setup the baseline. Note that the increased number of heads comes with the decreased per-head head dimension to keep the parameter size comparable.
Best and Worst We first compare the best (4(H8) 4) and the worst configuration (8(H4) x The worst is the most naive setting that just applies very aggressive attention map reuse. Although the speed about the same, performance can be improved by increasing the number of heads (8(H8) x 2). In contrast, the best working setting is carefully designed to maximize performance. For example, equipping the same number heads (2(H4) 8) does not show similar performance compared to the best configuration.
Sensitivity to Reuse We examine which of the phonetic or linguistic localization is more sensitive attention map reuse. Comparing two configurations with the identical number of heads and head dimensions (4+4+88 vs. 844++)) 4)) in the 5th block of Table we conclude the phonetic localization suffers more from increasing the reuse of layers. In other words, the linguistic localization seems to be more robust to the reuse. We conjecture that too few phonetic localization heads fail to capture every essential relationship.
Number of Heads in Phonetic Localization To better understand the trade-off between the num- ber of heads and head dimension, we conduct three experiments that only differ on the number of heads in lower layers. As shown in the 3rd block of Table 2 among the three configurations,
8

---

Published as a conference paper at ICLR 2022
Table 3: Effect of different configurations speed. The numbers inside of the parentheses indicate the speed-up ratio. The front convolutional sub-sampling is not included. Changing the number of heads does not make much difference to the speed.
#Param (M)
(atency(ms) by sequence length (approx. sec)
Training cost(h)
Config.
128 (5.1s)
256 (10.2s)
512 (20.5s)
768 (30.7s)
16
25.45
1.43 (x1.00)
3.74 (x1.00)
11.11 (x1.00)
22.32 (x1.00)
430.0
2 x 8
24.92
1.25 (x1.15)
2.98 (x (x1.26)
7.92 (x1.40)
15.05 (x1.48)
337.5
4 x 4
24.66
1.14 (x1.25)
2.56 (x1.46)
6.29 (x1.77)
11.38 (x1.96)
288.4
8 x 2
24.52
1.08 (x1.32)
2.35 (x1.59)
5.47 (x2.03)
9.55 (x2.34)
268.8
(4(H4) x 4) surpasses the other two in over 4 benchmarks. We expect a trade-off between the number of phonetic localization heads and per-head dimension; the former enables more various aspects to be covered while the latter helps richer representation for each head.
Number of Heads in Linguistic Localization We perform the same experiment above for the upper layers, shown in the 4th block of Table Interestingly, we found that increasing the number of heads for linguistic localization tends improve the overall performance. In addition, considerable performance loss is detected when the number of heads decreased to 2(H2). This observation implies that the previous studies that only reduce linguistic attention may face limitations when the remaining linguistic heads are too few.
Inference and Training Speed Table compares different reuse configurations. Both training and inference speed can be greatly improved by reducing the number of attention computation. The impact becomes more significant for longer sequences as T increases. Our best config configuration (4(H8) x 4) accelerates the inference by 1.96x times (for 30-second utterance) and reduces training cost by 33%. Note that the number of parameters is almost equivalent for all configurations because of the expansion of V dimension. Inference speed is evaluated on a single RTX- CTX-Titan(24GDD GPU and training cost is measured in GPU-hours on A A100(40GB) GPU.
RELATED WORK
Attention Map Reuse in NLP Xiao et al. (2019) proposed sharing of attention map through con- secutive layers for neural machine translation. They determine the reuse policy by Jensen-Shanmn divergence (JSD) values between two attention maps. Ying et al. (202T) propose similar approach for BERT but with manual reuse configurations. In addition to the critical difference in the do- main (NLP vs. ASR), both works depend on the similarities of the attention map, however, do not investigate why the similarity is developed.
Efficient Attention Map Computation for ASR Several studies have been proposed to reduce the cost of attention computation for ASR. Wang et al. (2021) proposed prob-sparse SA that only computes the top-k queries that are less uniform. For the streaming purpose, block processing of input sequence has been widely adopted (Ych et al.) 2021 Shi et al., 2021). Masked attention, which restricts the attention range to local neighbors, have also been used (Zhang et al.] 2020a Tripathi al. 2020 Audhkhasi et al. 2021). While these approaches focus reducing the effective sequence length, our method directly reduces SA computation.
CONCLUSION
In this paper, we analyze the role of self-attention in Transformer-based ASR and show that the role can be distinguished into two types: phonetic and linguistic localization. Especially, we showed that the phonetic localization captures various phonetic relationships and contributes to the performance by standardizing the features over similar phonemes, verified by the increasing phoneme classifica- tion accuracy over lower layers. The distinguished roles of SA in lower and upper layers also lead to an efficient ASR model that reuses the attention map for multiple SA layers. The proposed method has achieved a significant 1.96 times of speedup inference and 33% reduced training time, with the reduction of word error rate from 8.40% to 8.05% on the LibriSpeech test-other dataset.

---

We explain extra details for the model architecture, training procedure, pre-processing steps, and We also provide the source code for the experiments in Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
experiments for analyses in Appendix A supplemental materials.
ACKNOWLEDGMENTS
This work supported by the National Research Foundation of Korea (NRF) grant funded by Korea government (MSIT) (No. 202IRIA2C1013513), This work was also partly supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by MSIT (No. 2020-0-01373, No. 2021-0-00020-001). This work was also supported in part by Samsung Advanced Institute of Technology, Samsung Electronies Co., Ltd. This work was also partly supported by the Google AI Focused Research Awards Program awarded to Wonyong Sung. We gratefully acknowledge the GCP credit support from Google AI and the GPU server support from the Artificial Intelligence Cluster Agency (AICA, Korea).
REFERENCES
Kartik Audhkhasi, Tongzhou Chen, Bhuvana Ramabhadran, and Pedro Moreno. Mixture model attention: Flexible streaming and non-streaming automatic speech recognition. Proc. Interspeech 2021, pp. 1812-1816, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances Neural Information Processing Systems, 33, 2020.
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recogni- tion. arXiv preprint 1rXiv2205..1088. 2021.
Xuankai Chang, Aswin Shanmugam Subramanian, Pengeheng Guo, Shinji Watanabe, Yuya Fujita, and Motoi Omachi. End-to-end asr with adaptive span self-attention. In INTERSPEECH, pp- 3595-3599, 2020.
Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-War Li, and Hung-yi Lee. Audio albert: A lite bert for self-supervised learning of audio representation. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 344-350. IEEE, 2021.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models heyond a fixed-length context. In ACL (I), 2019.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HITI (1), 2019.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Learning, pp. 2337-2346. PMLR, 2019.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Yue Guan, Jingwen Leng, Chao Li, Quan Chen, and Minyi Guo. How far does bert look at: Distance- based clustering and analysis of bert' attention. arXiv preprint 2.00::/00/0009433 2020.
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: wwwolution-augmentee transformer for speech recognition. In Proc. Interspeech 2020, pp. 5036-5040 2020.
10

---

Published as a conference paper at ICLR 2022
Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi In- aguma, Naoyuki Kamo, Chenda Li, Daniel ments on espnet toolkit boosted by conformer. In ICASSP 2021-2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp- 5874-5878. IERE, 2021.
Jiatong Shi, et al. Recent develop-
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2 (10.2200007744 2021.
Mingkun Huang, Jun Zhang, Meng Cai, Yang Zhang, Jiali Yao, Yongbin You, Yi He, and Zejun Ma. Improving transducer with normalized jointer network. arXiv preprint arXiv:201 1.01576, 2020.
P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 876-885, 2018.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, 2020.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4365-4374, 2019.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 66-71, 2018
Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Un- supervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. (419-6423. IEEE, 2020.
Andy T Liu, Shang- Wen Li, and Hung-yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:2351-2366, 2021.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Danni Ma, Neville Ryant, and Mark Liberman. Probing acoustic representations for phonetic prop- erties. In ICASSP 202 202/-2021 / IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 311-315. IEEE, 2021.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pp. 498-502, 2017.
Edwin G Ng. Chung-Cheng Chiu, Yu Zhang, and William Chan. Pushing the limits of non- autoregressive speech recognition. arXiv preprint 1rXiv:2104.03416, 2021.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 2018
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, 2018.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 2206-5210. IEEE, 2015.
11

---

Published as a conference paper at ICLR 2022
Cheonbok Park, Inyoup Na, Yongjang Jo, Sungbok Shin, Jachyo Yoo, Bum Chul Kwon, Jian Zhao, Hyungjong Noh, Yeonsoo Lee, and Jaegul Choo. Sanvis: Visual analytics for understanding self-attention networks. In 2019 IEEE Visualization Conference (VIS), pp. 146-150. IEEE, 2019.
Daniel S Park, Yu Zhang, Chung- Cheng Chiu, Youzheng Chen. Bo Li, William Chan, Quoe V Le, and Yonghui Wu. Specaugment large scale datasets. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6879-6883. IEEE, 2020.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181 1-18-6 2018.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih. Sinong Wang. and Jie Tang. Blockwise self- attention for long document understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 2555-2565, 2020.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1715-1725, 2016.
Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah. What all do audio trans- former models hear? probing acoustic representations for language delivery and its structure. arXiv preprint arXiv:2101.00336 2021.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self- attention for fast and memory-efficient sequence modeling. In International Conference on Learn- ing Representations, 2018
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Christian Fuegen, Frank Zhang, Due Le, Ching- Feng Yeh, and Michael L Seltzer. Weak-attention suppression for transformer based speech recog- nition. In Interspeech, 2020.
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching- Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6783-6787. IEEE, 2021.
Kaiser Sun and Ana Marasović. Effective attention sheds light interpretability. arXiv preprint
arXiv:2105.08855, 2021.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 45934459444 4601, 2019.
Anshuman Tripathi, Jaeyoung Kim, Qian Zhang, Han Lu, and Hasim Sak. Transformer trans- ducer: One model unifying streaming and non-streaming speech recognition. arXiv preprint arXiv:2010.03199 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
12

---

Published as a conference paper at ICLR 2022
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, 2019.
Xiong Wang, Sining Sun, Lei Xie, and Long Ma. Efficient conformer with prob-sparse attention mechanism for end-to-endspeech recognition. In INTERSPEECH, 2021.
Tong Xiao, Yingiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), 2019.
Shuwen Yang, Andy T. Liu, and Hung yi Lee. Understanding self-attention of self-supervised audio transformers. In Proc. Interspeech 2020, pp. 3785-3789, 2020. doi: 10.21437/nnerspeecch. 2020-2231.
Ching-Feng Yeh, Yongqiang Wang, Yangyang Shi, Chunyang Wu, Frank Zhang, Julian Chan, and Michael L Seltzer. Streaming attention-base models with augmented memory for end-to-end speech recognition. 202/ IEEE Spoken Language Technology Workshop (SLT), pp. 8-14. IEEE, 2021.
Chengxuan Ying, Guolin Ke, Di He, and Tie- Yan Liu. Lazyformer: Self attention with lazy update. arXiv preprint wrXiv:2102.127000 2021.
Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer coders and mn-t In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833, IEEE, 2020a.
Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. Stochastic attention head removal: A simple and effective method for improving transformer based asr models. In Proc. Interspeech 2021, 2021a.
Shucong Zhang, Erfan Loweimi, Peter Bell, and Steve Renals. On the usefulness of self-attention for automatic speech recognition with transformers. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 89-96. IEEE, 2021b.
Yu Zhang, James Qin, Daniel Park, Wei Han, Chung-Cheme Chiu, Ruoming Pang, Quoc V Le. and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010. 10504, 2020b.
13

---

Published as a conference paper at ICLR 2022
A DETAILS
A.1
MODEL
Our baseline model is Conformer-M(medinn Gulati et al.
2020) trained with CTC (Graves al.
2006) loss. Table 4 shows the configuration of the model.
Unspecified details follow the original
Conformer paper. We observed that SyneBN (Peng 2018 critical for the overall perfor- mance. We employ weak attention suppression (WAS) (Shi et al. 2020) of y = 0.5 for faster convergence and improved performance.
Encoder
#Layers #Heads
16
Hidden dim.
256
4
Feed-forward dim.
1024
Conv. kernel size
31
Conv. normalization
SyncBN
BN momentum
0.005
BN epsilon
le-5
Hidden drop prob.
0.1
Attention drop prob.
0.1
Conv. Subsampling
#Layers Conv. kernel size
2
#Channels
256 SyncBN ReLU
3
Conv. normalization
Conv. stride
2
Activation func.
Table 4: Conformer-M implementation details.
We decide to use Conformer as our baseline, following recent ofate-of-the-arr ASR models, because these models can benefit the most from the proposed efficient model design. However, there may be several concerns on the clarity of our analysis on SA because the convolution module is jointly used with SA inside Conformer. Because the ability to gather information from the entire sequence is only equipped in SA, the analysis results the role of phonetic heads could not be presented without SA. The convolution kernel size of 31, which covers about 1.2 seconds, too short to gather long-ran oongrange information. We believe that the convolution module may guide the model to focus on local information first at the early stage of the training, however, the role of SA not much affected by the difference between Conformer and Transformer.
A.2 TRAINING
Optimizer Scheduler
Maximum LR
1.5e-3
Weight decay
le-5
Adam epsilon
le-8
Adam betas
(0.9, 0.99)
IR warm-up iters
5K
IR keep iters
95K
Total iters
200K
Batch size
480
Additional Details
#Frequency masking
2
Frequency mask width
27
#'Time masking #Models for SWA
10
Time mask width
0.05 (5%)
45
Variational noise
0.02
CTC beam size
32
Gradient norm clip
20
Table 5: Training details including optimizer, scheduler, augmentation and other hyper-parameters
We use AdamW (Loshchilov Hutter 2018) optimizer with the inverse square-root learning rate schedule (Vaswani et al.) (017). Table 4 shows the training configuration. We linearly increase the learning rate (LR) to the maximum value for 5K iterations and keep LR at maximum for 95K iterations, followed by 100K iterations of LR decrease. We use 4x A A100(40GB) GPUs for the experiments. To fit the batch size of 480 in these GPUs, we assign 40 samples per GPU and ac- cumulate the gradient of batches. We don't use bucketing for generating the mini-batch during training. We also employ adaptive SpecAugment (Park et al.) 2020). stochastic weight averaging (SWA) (Izmailov al. (008)), and variational noise.
14

---

Published as a conference paper at ICLR 2022
A.3 PHONEME PRE-PRRCESSIIG
For phonetic analyses, We employ the collapsed list of phonemes that are included in the Lib- riSpeech lexicon. We collapse ('AA, AO' to 'AA'), ('OW, OY' to 'O'), and ('SH, ZH' to 'SH', which leads the phoneme classes in Table for better understanding the characteristics. Because the LibriSpeech dataset does not provide frame-wist wise phoneme alignments, we extract the phoneme alignment from MFA (McAuliff et al., 2017) and map these alignments to each frame. Especially, exploit the fact that each frame corresponds to 40ms interval after passing through two convolu- tional layers (convolutional sub-samm suml-sagl of stride 2 in front of the model. We assign the phoneme class to each frame if the center of the frame is within the phoneme duration, including the 'silence' phoneme.
Except for the phoneme classification, analysis excludes 'silence" frames by exploiting frame- level phoneme alignments. Then, we re-normalize the remaining attention probability to preserve the probability sum 1. We observe that these silence frames sometimes consume much probability mass for both linguistic and phonetic heads, which makes our analysis difficult. Note that this phenomenon of assigning strong attention ambiguous tokens, such [CLS] [SEP], has been also reported in NLP (Kobayashi et al. 2020; Sun & Marasović (022)) Recently, Kobayashi et al. (2020) introduced the concept of norm- based analysis and reported that those [CLS) and [SEP] does not contribute much to the output even though their attention weight is large. We leave analysis using effective attention as future work.
A.4 PHONEME ATTENTION RELATIONSHIP COVERAGE
20
Figure 7: Top-10 phoneme classes (column) for each phoneme (row) in the reference phoneme atten- tion relationship of Figure 5]a).
35
20
The coverage ratio R of the target PAR compared to the reference PAR is calculated as below:
PARA(P, g) PAR p, (..)) 1): R. P pEP
Minimum
Σ
Ralp. 4]
(6)
Qp
QEQp
P indicates every phoneme class and Qp indicates top- phoneme classes in the order of the largest
PAR elements for the phoneme (Rx :). Top-10 classes are visualized in Figure 7 topmost phoneme classes because our interest the important relationships.
We exploit
A.5 PHONEME CLASSIFICATION
For the phoneme classification, we train a single fully-connected layer as classifier. We choose the simplest architecture as a classifier to more directly correlate the phoneme accuracy and hidden layer representations. These representations are extracted from dev-clean and dev-other dataset and evaluated on test-clean and test-other dataset. We use SGD with learning rate of 0.11 momentum of 0.9, and weight decay le-3. The training takes 15 epochs, where the learning rate multiplied by 0.1 for every 3 epochs. We visualize confusion matrices of the phoneme classification in Figure As layer proceeds, wrongly classified phonemes (non- -on-niigonal) disappear and leave a clear diagonal line the confusion matrix.
15

---

Published as a conference paper at ICLR 2022
Aceuricy (%)
30
01
(a) Layer (48.4%)
(b) Layer (68.0%)
(c) Layer 8 (75.5%)
Figure 8: Confusion matrix and phoneme accuracy for selected layers. Visualized the result from the LibriSpeech test-other dataset. Each row and column corresponds to the 37 phoneme classes, including 'silence" zeroth class.
Accumalated Attemtion Probubiiity
0.8
0.
CAD:
0.678
CAD: 0.812
0.6%
0.816
0.701
0.832
02
0,705
02-
0.865
Layer (avg. CAD: 0.695)
Layer (avg. CAD: 0.843)
Figure 9: Examples of typical cumulative attention diagonality (CAD). (a) and (b) visualizes curves (before the integral) of four attention heads in each layer. 22-axis and y-axis depict the relative distance and accumulated attention probability, respectively. CAD is represented as the area under curve. CAD values of each head are also listed.
B ADDITIONAL RESULTS AND DISCUSSIONS
B.1 DIAGONALITY
B.1.1 CAD RESULTS
The CAD good indicator of how fast the accumulated attention increases over the distance, directly represents the diagonality of the attention weight. The CAD is interpreted as the area under function (rr), where D(r) calculates the amount of total attention weight within restricted range j r(T 1), i r(T 1)]- (T - 1) is the maximum possible distance between two frames where T is the number frames.
min(T LT
CAD [
Anli.5l dr dr= f D(r)dr
Σ m1,
I-r(T-1))
If the is same, larger D(r) means that the attention is more concentrated near the diagonal. Please note that D(r) is a monotonically increasing function whose output is always in the range [0, 1].
To help understand the concept of CAD, we visualize two typical examples of the cumulative atten- tion diagonality in Figure [ For layer 2, where attention heads perform the phonetic localization, CAD values are low. In contrast, for layer 14 that concentrates linguistic localization, CAD values are much higher.
Figure 10 plots the sorted CAD values. We determined the threshold (0.75) where the curve of sorted CAD values changes from convex to concave. The higher CAD value represents that more probability mass is concentrated near the diagonal in the attention map, while the lower CAD value
16

---

Published as a conference paper at ICLR 2022
0.9
#Layer
2
3
Curulazivve Attention
0.75
4
4
4
0.8
#Layer
5
6
7
22
Diagonaitty
0.75
42
0.75
4
#Laaer
2
0.85
Mlayeer
13
14 15
16
32
56
0.75
0
Sorted Attention Iendd Index
Figure 10: Sorted CAD over attention heads in the baseline model. Over 64 heads, 42 heads belong to the CAD value under 0.75. The table on the right side indicates the number of linguistic attention heads that are of CAD value under 0.75.
implies that the distribution is more uniformly distributed. We observe that diagonally concentrated heads take large portion in upper layers.
B.1.2 PREVIOUS DIAGONALITY ANALYSIS
Previous studies utilized a different metric to calculate the diagonality of an attention map (Zhang et al. 2021b Yang et al. 2020). In this version of diagonality, the metric is interpreted as the negative normalized average attention distance (span-length). In Yang et (2020), the diagonality calculated below:
T
T
D3 - T2
ΣE
-i
==
We introduced the cumulative attention diagonality (CAD) because the previous diagonality metric lacks information about how attention is distributed by distance. In other words, CAD is more com- prehensive because it provides the overall diagonality value as well as the tendency of the attention according to the distance. Our CAD metric dearly captures the flat region in lower layers (Figure In contrast, in Zhang et al. (2021b), the diagonality increases from the lower layers to the upper layers, which may not be sufficient observation in ASR.
B.2 PHONEME ATTENTION RELATIONSHIP
B.2.1 EXCLUDING CONSECUTIVE FRAMES
We treat ql * q) and Ps(p, pl differently because we want to separate the effect of the diagonal (position-biue based) attention map. An attention map that highly focuses on the surroundings (diagonal-like) can unintentionally disturb the purpose of PAR in measuring Pa(p, because it will give high value to p-p relationship not because the contents are similar, but because the location close. Our purpose on PAR is to examine the phonetic (content-based) behavior, excluding consecutive frames of the same phoneme class better represents SA in lower layers that correspond to the phonetic localization.
B.2.2 PAR RESULTS
Figure II visualizes how averaged PAR changes through layers. Lower layers (first row in Figure) show noticeable regions that represent the important phoneme relationships, including the diagonal. Note that the diagonal stands out even though we excluded the consecutive same phoneme classes. Each lower layer focuses on different aspects, supporting our analysis on the PAR coverage (Figure 6) that the accumulated coverage ratio continuously increases through layers. In other words, each layer covers a certain part of the reference PAR with less overlap between layers. On the other hand, the upper layers (second row in Figure) do not show the emphasized pattern. This is expected for linguistic localization heads that generate highly diagonal attention maps because their attention is mainly assigned to near frames regardless of their phoneme classes.
For the finer understanding, we also visualize how attention heads compose the averaged PAR. Figure 12 shows that each head corresponds to different phonological properties but their averaged
17

---

Published as a conference paper at ICLR 2022
2
Layer
Layer
(d) Layer
Layer 10
Layer 12
(g) Layer
(h) Layer 16
Figure 11: Phoneme attention relationships (PAR) in SA layers. Averaged PAR of even-numbered layers (2nd, 4th, 16th) visualized. Lower layers show high correlations between similar phonetic features, but these relationships are weakened in upper layers.
19
Head
Head
25
30
Average PAR over heads
(Layer
Head
Head
Figure 12:
Phoneme attention relationships of self-attention heads. Each head see different aspects, enriching the overall information the layer captures.
interests mimic the reference PAR. This emphasizes the importance of multiple attention heads in phonetic localization; each head specializes in capturing the specific phoneme relationship and contributes differently.
B.2.3 PREVIOUS PHONEME RELATIONSHIP ANALYSIS
The idea of phoneme relationship analysis first introduced in Yang et al. (2020) as phoneme relation map (PRM), but our PAR is different from PRM in two ways. First and the most difference that PRM do not distinguish consecutive frames and discontinuous frames that corresponds to same phoneme class. Therefore, attention heads that only focus neighbors would also present heavy self-to- self-to-self phoneme relationship in PRM, which hinders the clarity of the analysis. Second, PRM do not apply correction according to the sequence length. We multiply T to reduce the effect of sequence length, inspired by the fact that the expectation of (averaged) probability is potentially smaller for longer sequences.
18

---

Published as a conference paper at ICLR 2022
20-
25
30
50
23
35
25
(1)
(b)
8 x
Figure 13: Phoneme attention relationship (PAR) for three attention map reuse configurations (2x x 4, 8 2). PAR(a)(bX() are obtained by averaging PAR of lower layers. Key missing relationship highlighted in white circles, such as (B, P), (G, K), and (DH, TH) in Figure
B.3 ATTENTION MAP REUSE AND PAR
For the understanding of the performance loss for extreme reuse cases, we visualize the PAR of different configurations in Figure 13 We observe that most of the patterns resemble the baseline model, which implies that heads learn similar roles during training. However, for x 2, several information lost; diagonal became unclear and highlighted correlations disappeared.
B.4 COMPARISON TO MASKED ATTENTION
We compare the masked attention Zhang et al. 2020a Tripathi et al. 2020 Audhkhasi et al. 2021 Huang et al., 2020) with the proposed attention map reuse. Masked attention, similar to block-based attention (Shen et al. 2018 Qiu et al.] (020), is method to reduce the computational burden of self-attention by restricting
length the accessible context. masked attention adopted for ASR, each frame only attends to local neighbors; L frames to the left and It frames to the right, denoted as [-1, R].
Table 6: Comparison of the word error rate between the proposed attention reuse and the masked attention. s_" indicates that the attention range is not restricted (unlimited).
Model
Lower layers
Upper layers
dev-clean
dev-other
test-clean
text-other
Baseline Low64
3.1
8.3
3.2
8.4
[464
3.1
8.4
3.4
8.5
Up64
64, 64]
3.1
8.2
3.3
8.2
4(H8)< x 4
3.1
8.1
3.2
8.1
We train the models with the same setting the Conformer-M M baseline (as in Table 2 but with a limited attention range for either lower or upper layers. Among the 16 layers the baseline, lower and upper layers consist of 8 layers each and correspond phonetic localization and linguistic localization, respectively. For selected layers, we restrict each frame to only attend to near neighbors within the distance of 64 frames. Table shows the performance on the LibriSpeech dataset.
For lower layers, attention range restriction causes clear degradation of the recognition accuracy. We consider that phonetic localization lower layers demands a wide range of attention. On the other hand, for upper layers, attention range restriction shows almost comparable performance to the baseline, and even better in some subsets. We hypothesize that the upper layers for CTC-based ASR may not require very wide context, because their attention pattern is highly diagonal. In addition, the restriction-bar based computational savings seems to no larger than the proposed attention reuse. A context range of 128 (64+64) frames corresponds to about 5.1 seconds, where the average utter- ance length of the corpora is about 7.4 seconds. Therefore, we expect approximately 30% reduction in attention calculation when masked attention is applied to every layer. Our attention reuse (4 4) reduces about 75% of the attention computation without any degradation in the performance.
19


Reviews:
[{"review_id": "yTwmbxog51", "reviewer": "Reviewer_Vyqx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Correctness and Significance\n\nThe paper is logical and methodical in the investigation. The proposed measurements and experiments make sense. The CAD metric makes sense. This won't change the result of the work much, but CAD looks unnormalized: the double sum seems to require 1/T^2 term, not 1/T.\n\nThe phoneme analysis is conducted well. Both the phoneme classification accuracy curve and the phoneme attention relationship metrics confirm the paper's hypothesis on the phonetic-linguistic division. Nevertheless, I would like to see a discussion on alternative hypotheses. I don't have a concrete suggestion here, but is it possible that the result obtained in Figure 2 is spurious or can be explained by something else.\n\nFinally, the proposed architecture for the attention map reuse is a logical step to improve the model. The paper reports multiple architecture variations and compares them. I found the most inspiring that this technique provides a way to reduce latency.\n\n# Clarity\n\nThe paper is very well structured and well written. It was a pleasure to read it.\n\n# Minor things\n\n- The graphs are hard read when printed. Especially in black and white.\n- Session -> Section", "summary_of_the_paper": "The paper consists of two parts. First, it conducts the analysis of an ASR model. Specifically, it measures the diagonality of the self-attention map and notes that the diagonality for the top layers is higher than for the lower layers. Based on this, the paper suggests that the lower layers are responsible for the phonetic information and the higher layers are responsible for the linguistic information. Then, the paper confirms this hypothesis by using each layer for a phoneme classifier.\n\nSecond, based on the above observations, the paper proposes a method to exploit this division into the phonetic-linguistic layers. The proposed method is to reuse the attention maps across several layers. This modification allows for more efficient architecture both in terms of latency in train and test time. In some cases, the proposed architecture outperforms the baseline.", "summary_of_the_review": "In summary, this is a good exploration into the transformer architecture for speech recognition. The paper provides valuable insights into the transformer's attention map structure and connects it with the phonetics. Then the paper uses these insights to improve the model.\n\nI would like to request the alternative hypotheses on why the attention becomes diagonal in higher layers. Or some discussion on this.\n\nThen, the figures need to be improved for reading when printed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YUDG0RwL5-K", "reviewer": "Reviewer_ki2s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thank the authors for the additional results and clarifications. Overall, I still feel that the analysis is interesting, but the experimental results are somewhat lacking. It will be interesting to see whether the analysis holds for more popular end-to-end models.\n\n---------------------\n\nStrengths:\n* The authors present an excellent analysis of how attention maps are distributed across layers, and find interesting patterns – that lower layers present phonetic information and upper layers more local linguistic information. The figures present enough support for the claim.\n* The authors are able to reduce the inference computation by a factor of 1.96 for a 30 second audio, with marginal improvements in WER.\n\nWeaknesses:\n* Increasing diagonality of the attention maps has been observed in prior work as well (as the authors have cited and discussed). While the present work improves this understanding (mainly the phonetic information of the lower layers part), the analysis is not entirely novel.\n* There are several approaches to improving inference speed by directly addressing the attention map computation. For example, for ASR, prior work has looked at using masked attention where-in each frame only attends to L frames to left and R frames to the right [1, 2, 3]. Therefore, each frame is computing attention only using at (L + 1+ R) frames. This addresses the quadratic increase in computation. \n* Alternatively, Performers [4] also proposes linear time and space complexity for attention computation. It would be useful to have some of these comparisons to the presented method.\n* The authors’ analysis suggests that the attention map in lower layers is reusable, but those in the upper layers is not (Fig. 5(b)). But their best strategy ties maps for both lower and upper layers. This feels counterintuitive. Is there a reason why that is so?\n* How much of the analysis is tied to the CTC-ASR model used? What about LAS [5] or RNN-T [6] models, which are widely used as well? \n* The results on Librispeech are much worse than other state-of-the-art (SOTA) end-to-end models (e.g., [7]), most likely because the authors use CTC. It would be useful to see if the techniques result in SOTA performance when using LAS or RNN-T.\n\nOther remarks:\n* Figure 1 has too much information and is a little hard to follow. Consider either improving the figure description, or splitting into multiple parts.\n* Sec. 3.2: The diagonality analysis is not that different from Zhang et al., and arrives at very similar conclusions. For the models authors consider, are the plots different when using the metric in Zhang et al.?\n* Sec. 3.2: Furthermore, it’s not very clear from the description what r(T - 1) is. \n* For the plots, is PAR and CAD summarized per utterance or per corpora?\n* Figure 6 and the description is hard to follow. What does it mean to set averaged PAR of the baseline to 1.0? Consider moving some of the details from the Appendix to the Sec. 3.4 since PAR is one of the main proposals of the paper.\n* Figure 6: The authors claim that a coverage of 0.974 is bad. It’s unclear why. 0.974 seems quite high.\n* Does adding more heads to the baseline improve performance? It seems like using 8 heads works the best for the shared attention map model, but the baseline only uses 4.\n\nMinor:\n* Abstract: “We propose a novel metric …” The sentence is too long and a little hard to follow, especially with so little context since it is appearing in the abstract.\n* Abstract: standardizes the phonetic variance -> reduces the phonetic variance?\n* Introduction: 30 seconds … 750 frames: Will be worth adding what is the processing window size (40 msec?).\n* Sec. 2.1: (d_h)^2: Consider using special chars like \\dag or \\ddag for footnotes to avoid confusion. As of now, it looks like d_h is squared.\n* Sec. 3.2: What does h stand for in CAD_h?\n* Sec 3.3: (1) First, … (2) Second, .. : First and Second are redundant.\n* Consider using just 1 decimal point for results. 2 decimal points does not really add a lot of value, comparison-wise.\n\n\nReferences:\n\n[1] Zhang, Qian, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. \"Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss.\" In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7829-7833. IEEE, 2020.\n\n[2] Audhkhasi, Kartik, Tongzhou Chen, Bhuvana Ramabhadran, and Pedro J. Moreno. \"Mixture Model Attention: Flexible Streaming and Non-Streaming Automatic Speech Recognition.\" Proc. Interspeech 2021 (2021): 1812-1816.\n\n[3] Tripathi, Anshuman, Jaeyoung Kim, Qian Zhang, Han Lu, and Hasim Sak. \"Transformer transducer: One model unifying streaming and non-streaming speech recognition.\" arXiv preprint arXiv:2010.03192 (2020).\n\n[4] Choromanski, Krzysztof, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins et al. \"Rethinking attention with performers.\" arXiv preprint arXiv:2009.14794 (2020).\n\n[5] Chan, William, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. \"Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.\" In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4960-4964. IEEE, 2016.\n\n[6] Graves, Alex. \"Sequence transduction with recurrent neural networks.\" arXiv preprint arXiv:1211.3711 (2012).\n\n[7] Gulati, Anmol, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han et al. \"Conformer: Convolution-augmented transformer for speech recognition.\" arXiv preprint arXiv:2005.08100 (2020).\n", "summary_of_the_paper": "The paper explores how self attention maps in conformer based ASR models are distributed across layers. The authors note that the lower layers of the model capture phonetic information, and the variance of the attention map distribution for the same class of phones across the utterance goes down. The higher layers do not display this behavior. The authors hypothesize that the higher layers capture linguistic information that likely combines information across phones.\n\nBased on the observation that the variance of the attention map distribution goes down, the authors propose reusing the attention map from one layer across multiple layers. This reduces the amount of computation needed for inference, and reduces the training time. Tying attention maps also results in small improvements in performance.\n", "summary_of_the_review": "Overall, the authors present an interesting analysis of the attention maps, which in the reviewer’s opinion is the strongest part of the work. It is interesting to categorize lower layers as phonetic, and upper layers as linguistic. That being said, similar analysis has been done in prior work. The presented results are also lacking; it is unclear if the method is very specific to CTC and whether it generalizes to other more popular end-to-end techniques. The authors have also not considered alternative strategies to minimize computation that directly addresses the quadratic computation of attention on long sequences.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XYxf2mtoAYH", "reviewer": "Reviewer_kKif", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well presented with a large battery of carefully designed experiments and conclusions.\nIn my opinion, It can naturally be divided into 2 main parts, the study of the MHSA and the layer-wise attention map reuse.\n\nThe first part, namely, the analysis of the different emerging patterns of the multi-head self-attention on each layer is very well supported and very interesting. For instance, the layer-wise analysis of phonetic attention was carefully designed to avoid the diagonal contributions of the same phoneme, and the CAD was proposed to account for the cumulative diagonally of layers. The authors found and provide supporting evidence of 2 different regimes in the MHSA. In the upper layer they show that there is a strong diagonal attention which they attribute to linguistic localisation. The evidence for such an intuitive hypothesis is vague in this specific work, and leverages conclusions from other papers, however, they don't provide a strong experimental evidence for that claim. In practice, this is just a semantic issue as there is strong evidence for the emergence of such diagonal pattern in the paper and previous works. \n\n The lower layers that show a lesser diagonal behaviour are studied from the phonetic perspective, which this is typically ignored. They then hypothesise that those lower layers are performing phonetic localisation and provide evidence for that by:\n 1 - adding probes of clarifiers after each conformer layer and showing that there is a peak of classifier performance for phoneme classification at the middle layer.\n 2 - computing the phoneme attention relationship (PAR) per attention heads. I found figure 3 confusing, as there are many white dots --maybe without enough attention probability ?-- not commented in the attention pattern. In contrast, I found Fig 5,  Fig 11 or Fig 12 much more illustrative and clear. In addition,  Figure 5 is an average and it would be interesting to contrast the phonetic properties of each head to understand whether there are emerging patterns. It is not very clear whether the selected output vocabulary is affecting in any way to these patterns as they are using 128 sub-word tokens. Would this pattern hold for 512 or more sub-words tokens ? \n3 - computing the PAR sharing the attention maps.\n\nIn the second part of the paper, the authors propose to reuse attention maps in these 2 regimes, and they prove essentially that those maps can be reuse while keeping the number of parameters the same. This yields small WER improvements and large speed-ups. \nI found the constraint on the number of parameters necessary but I would also have liked to see which is the performance when the maps are shared but the hidden value dimension is not increased, as that would have provided a very interesting point for the prior analysis of the first part of the paper. In addition and while the selected pattern make sense for optimising the search space, looking at the results of the phoneme classifications probes, one would have expected maybe additional non-symmetric share patterns for lower layers, like 1(H4)+1(H4)+6(H4) for instance to help to build the phonetic attention from the convolutional front-end.\n\nFinally, the architecture used in the paper is a Conformer architecture which includes a convolutional layers in between MHSA layers. \nAlthough the authors discuss this potential issue in the appendix, I think it is valuable to discuss it in the main text (maybe I missed it).\nHaving performed one of the experiments turning off the convolutions at least for the phonetic layers would have been really interesting and would have provided a stronger conclusion. \n\nI really enjoyed the paper. \nNice Job.\n\n", "summary_of_the_paper": "The paper analyses the multi-head self attention behaviour in CTC based ASR models.\nThey identify the working regimes, namely, phonetic and linguistic localisation. \nThe paper argues that lower MHSA layers build phonetic attentions patterns while upper layers do that linguistically showing some diagonal pattern similarly to NLP tasks. Based on these findings the authors propose to share attention maps across layers.\nThis yields some modest improvements in terms of WER but obtains larger speeds-up with respect to a basic Conformer model.\nDuring the study they introduce new measurements such as cumulative attention diagonality (CAD) and phoneme attention relationship (PAR). ", "summary_of_the_review": "The paper is well-written and provides a lot of insights into how CTC models with MHSA learn. \nI think this is a very interesting experimental analysis of the properties of the Conformer/Transformer models that can help to understand non-autoregressive end2end ASR models. The claims are very well supported and the practical conclusions and implications are very interesting. The code is provided and it seems possible to reproduce the paper.\nI would have liked to see more experiments on the paper but just because the results are very interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "44MV6VUimYz", "reviewer": "Reviewer_mbFx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\nThe analysis part of the paper can be a good contribution to the speech recognition community. The distinct increase after the first half of layers on cumulative attention diagonality brings an interesting insight on the behavior of the speech recognition model, and the characterization of phonetic and linguistic localization is convincing.\n\nThe phoneme analysis authors conduct in the paper provide a good insight for readers to understand the correlation between attention and phoneme classes.\nThe phoneme attention relationship is a good metric to understand the correlation between attention and phoneme, and provide a good measurement on the attention coverage for speech recognition\n\nOverall the proposed analysis mechanism can benefit beyond improving inference speed, for example, it will be a great analysis tool for understanding representation learning which is beneficial for speech self-supervised learning.\n\nWeakness\n\nThe downstream task this paper focuses on, improving speed with attention reuse, does not bring out the full potential of the proposed analysis mechanism. In fact, it seems the connection between the analysis and the task is not very strong, as one can design those configurations listed in Table 2 without the insight of phoneme attention relationship. It would serve authors better to identify other more correlated tasks.\n", "summary_of_the_paper": "This paper develops a mechanism to analyze self-attention for speech recognition, and further use the resulting insight to facilitate attention reuse for reducing inference cost. The analysis identifies a distinct pattern between lower half and upper half layers on the ASR encoder through cumulative attention diagonality and characterizes the lower half as finding phoneme localization and upper half as finding linguistic localization. The paper justifies the characterization of phoneme localization by illustrating an example of such a phoneme localization, and also shows that the output of the layers in the middle provide better phoneme classification accuracy than the upper layers. An additional analysis, called phoneme attention relationship, further quantifies the correlation between phonemes and attention, and shows that the lower layers have higher correlation. These insights are used to design attention reuse across layers, and shown to maintain the quality while improving the speed on Librispeech.\n\n", "summary_of_the_review": "The analysis proposed in this paper can be a good contribution to the community. The task focused in this paper, however, is on the weak side. Overall the analysis is beneficial to the community, and can be considered for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper conducted a thorough experimental analysis of the attention map in the Conformer models for CTC based speech recognition models and connected it with phonetic and linguistic information in the speech. Using these insights, the paper presented some computation improvement and marginal quality gains. The authors actively conducted additional experiments to further justify the claims. The paper is strong in terms of the systematic way of in-depth analysis and further development (i.e. sharing the attention map across layers for speedup). But as pointed out by the reviewers, it lacks some comparisons with other alternatives to justify the importance of sharing attention maps in reducing computations.  Also it would be better if there's justifications on how the observations generalize to other types of models (such as LAS, RNN-T). 

The decision is mainly because of the thorough analysis conducted in the paper which can be a good contribution to the community.
Decision: Accept (Spotlight)