SlowGuess/ACL_2020_OCR huggingface dataset:
Archiki Prasad
Department of EE, IIT Bombay, India
archikiprasad@gmail.com
Preethi Jyothi
Department of CSE, IIT Bombay, India
pjyothi@cse.iitb.ac.in
In this work, we present a detailed analysis of how accent information is reflected in the internal representation of speech in an end-to-end automatic speech recognition (ASR) system. We use a state-of-the-art end-to-end ASR system, comprising convolutional and recurrent layers, that is trained on a large amount of US-accented English speech and evaluate the model on speech samples from seven different English accents. We examine the effects of accent on the internal representation using three main probing techniques: a) Gradient-based explanation methods, b) Information-theoretic measures, and c) Outputs of accent and phone classifiers. We find different accents exhibiting similar trends irrespective of the probing technique used. We also find that most accent information is encoded within the first recurrent layer, which is suggestive of how one could adapt such an end-to-end model to learn representations that are invariant to accents.
Traditional automatic speech recognition (ASR) systems, consisting of independently-trained acoustic, pronunciation and language models, are increasingly being replaced by end-to-end ASR systems (Chiu et al., 2018; Hori et al., 2017). An end-to-end ASR system refers to a single model that subsumes all the traditional ASR components and directly translates a speech utterance into a sequence of graphemes. Such models benefit from jointly training acoustic and language models and eliminating the need for a pronunciation dictionary. While end-to-end ASR models have clear merits and are elegant in their formulation, they tend to be opaque in their predictions and difficult to interpret.
In order to understand better what is encoded in the layers of an end-to-end ASR system, prior work has explored the use of phone probes (classifiers)
to analyze the phonetic content of representations at each layer (Belinkov and Glass, 2017; Belinkov et al., 2019). This analysis was restricted to a single accent of English. In this paper, we work with multiple accents of English and propose a number of different tools (other than phone probes) to investigate how accent information is encoded and propagated within an end-to-end ASR system.
Why accented speech? We have witnessed impressive strides in ASR performance in the last few years. However, recognizing heavily accented speech still remains a challenge for state-of-the-art ASR systems. An end-to-end ASR model trained on a standard speech accent significantly underperforms when confronted with a new speech accent. To shed more light on why this happens, a systematic investigation of how such models behave when evaluated on accented speech might be useful. The insights from such an investigation might also come in handy when trying to adapt end-to-end neural architectures to be more accent-agnostic.
We tackle the following specific questions of interest in this work:
1. How do the gradients of an end-to-end ASR model behave when subject to varying accents?
2. How do we directly measure the amount of accent information encoded within hidden representations of an end-to-end model?
3. How do accents impact phone accuracy across different layers in an end-to-end model?
While the analyses of black-box models in computer vision and natural language processing have received a considerable amount of attention, prior work on the analysis of end-to-end ASR models are notably few in number. With presenting various analysis techniques that are applicable to speech,
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3739-3753
(a) Phonetic coverage histogram
(b) Phonetic duration histogram
Figure 1: Phonetic coverage and duration histograms for the US accent. X-axis labels refer to individual phones.
we hope this work can serve as a starting point for further studies and spur more analysis-driven investigations into end-to-end ASR models. The code used in our work is publicly available. $^{1}$
In this section, we first introduce the dataset of accented speech samples used in our experiments, along with details of the phone-level alignments that were necessary for our subsequent analyses. We also provide a detailed description of the specific end-to-end ASR model that we use in this work, along with important implementation details.
2.1 Dataset
We extracted accented speech samples from the Mozilla Common Voice speech corpus (Mozilla). The Voxforge corpus (Voxforge.org) was another potential source for accented speech samples. However, we preferred the Mozilla corpus as the dataset is relatively cleaner, has larger diversity in speech across accents and more importantly contains the same content rendered in different speech accents (which we exploited in our experimental analysis). We considered accented speech samples from seven different English accents: African, Australian, Canadian, England, Indian, Scotland and US. These were chosen to span the gamut of accents in terms of how they differ from the primary accent that was used to train the ASR system (US). US and Canadian serve as native accents; African, Australian and England accents are sufficiently different from the native accents while Indian and Scotland accents vary substantially.
We created a dataset of utterances in each accent using the following heuristic. First, we chose sentences that appeared in speech samples corresponding to five or more accents (including US). For African and Scotland accents that contained
very few speech samples overall, we chose transcripts that had an utterance with the same text spoken by a US-accented speaker. This finally led to 3500 samples being chosen for each accent containing text that appeared in at least two accents, at most six accents and 3.24 different accents on average. We chose the utterances to largely overlap in text so that differences in ASR performance could be mostly attributed to acoustic differences and not language model-related differences.
Alignments: For our empirical investigation, we require phone alignments for all the accented speech samples. We used an existing Kaldi-based forced aligner, gentle², to align the speech samples. The aligner uses the CMU dictionary and accommodates multiple pronunciations for a word which is important for accented speech. Although the aligner was trained on US-accented speech, we found the alignments assigned to various accented speech samples to be fairly robust as determined by a manual check of the alignments for a random set of Indian-accented utterances. The aligner failed to produce outputs on samples of poor quality; these samples were omitted from our analysis.
Figure 1(a) shows the coverage across phones for the US-accented speech samples and Figure 1(b) shows the total duration of phones for US-accented speech samples. Phone coverage and phone duration distributions for all the other accents are almost identical in shape to the US accent. Aggregate plots visualizing these distributions across the remaining accents are shown in Appendix A.
2.2 End-to-end ASR: Deep Speech 2
We chose DeepSpeech2 (Amodei et al., 2016) as our end-to-end ASR model. This is a widely-used architecture that directly maps speech features to graphemes and is trained using the Connectionist Temporal Classification (CTC) loss (Graves et al.,
'https://github.com/archiki/ ASR-Accent-Analysis/
Available at lowerquality/gentle
2006). The input to the model is a sequence of frequency magnitude spectrograms (henceforth referred to as SPEC), obtained using a 20ms Hamming window and a stride of 10ms. With a sampling rate of  $16\mathrm{kHz}$ , we end up with 161-dimensional input features. The first two layers are 2D-convolutions with 32 kernels at each layer with sizes  $41 \times 11$  and  $21 \times 11$ , respectively. Both convolutional layers have a stride of 2 in the frequency domain while the first layer and second layer have a stride of 2 and 1, respectively, in the time domain. This setting results in 1312 features per time frame after the second convolutional layer which we will henceforth refer to as CONV. The convolutional layers are followed by 5 bidirectional LSTMs (Hochreiter and Schmidhuber, 1997), each with a hidden state size of 1024 dimensions. These layers are henceforth referred to as  $\mathrm{RNN}_0$ ,  $\mathrm{RNN}_1$ ,  $\mathrm{RNN}_2$ ,  $\mathrm{RNN}_3$  and  $\mathrm{RNN}_4$ . The implementation of this model is adapted from Naren (2016). This model is trained on 960 hours of US-accented speech obtained from the Librispeech corpus (Panayotov et al., 2015). All subsequent experiments use this pretrained model, which we will refer to as DS2.
Table 1 shows the performance of DS2 when evaluated on speech samples from different accents. Both word error rates (WER) and character error rates (CER) on the test sets are reported for each accent. As expected, US and Canadian-accented samples perform best. DS2 has the most trouble recognizing Indian-accented samples, incurring a high WER of  $49.1\%$ , followed by Scotland-accented samples with a WER of  $36.7\%$ .
The next three sections are grouped based on the probing techniques we adopt to examine the effect of accents on the internal representations learned by the model:
Gradient-based analysis of the model (§3).
- Information-theoretic measures to directly quantify accent information in the learned representations (§4).
- Outputs of phone and accent classifiers at each layer (\$5).
Accent	Utterances	Duration	Error
Train	Test	Train	Test	WER	CER
African	2500	1000	3	1	28.7	16.2
Australia	2500	1000	2	1	28.7	16.6
Canada	2500	1000	2	1	18.7	9.9
England	2500	1000	2	1	29.0	16.4
Indian	2500	1000	2	1	49.1	31.6
Scotland	2500	1000	2	1	36.7	22.3
US	2500	1000	3	1	20.4	10.9
Table 1: Data statistics of accented speech datasets. Duration is approximated to hours and WER/CER refer to the test error rates for each accent using DS2.
3 Gradient-based Analysis
Gradient-based techniques have been widely adopted as an explainability tool in both computer vision and NLP applications. In this section, we adapt some of these techniques to be used with speech and derive insights based on how accents modify gradient behavior.
3.1 Attribution Analysis
A simple gradient-based explanation method considers the gradient of the output  $f_{j}$  from a neural network (where  $j$  denotes a target class) with respect to an input  $x_{i}$  (where  $i$  refers to the  $i^{\text{th}}$  input time-step used to index the input sequence  $\mathbf{x}$ ):
$$
\operatorname {g r a d} (j, i, \mathbf {x}) = \frac {\partial f _ {j}}{\partial x _ {i}}
$$
Here,  $\operatorname{grad}(j,i,\mathbf{x})$  serves as an approximate measure of how much  $x_{i}$  contributes to  $f_{j}$  (Simonyan et al., 2014). For speech as input,  $x_{i}$  would be an acoustic feature vector (e.g. spectral features). Thus,  $\operatorname{grad}(j,i,\mathbf{x})$  would be a vector of elementwise gradients with respect to  $x_{i}$ . For each  $x_{i}$ , we use the L2 norm to reduce the gradient vectors to scalars:  $a_{i,j} = \| \operatorname{grad}(j,i,\mathbf{x})\|_2$ . We refer to  $a_{i,j}$  as an attribution. We note here that instead of using the L2 norm, one could use the dot product of the gradient  $\operatorname{grad}(j,i,\mathbf{x})$  and the input  $x_{i}$  as an alternate gradient-based method (Denil et al., 2014). For our task, this attribution method seemed less suitable (compared to computing the L2 norm) as dot products would have the undesirable effect of being sensitive to prosodic variations in speech and speech sounds like fricatives or stop onsets which have sparse spectral distributions. (We refer interested readers to Appendix C for visualizations using the dot product-based attribution method.)
We compute character-level attribution from the DS2 system using the following two-step approach.
$^{3}$ US-accented samples are drawn from various parts of the US and are more diverse in accent, compared to the Canadian-accented samples. We suspect this could be why US underperforms compared to Canada.
Figure 2: Example illustrating gradient attribution corresponding to the word "FIRE" across different accents.
First, we consider the output character with the highest softmax probability at each output timestep. Next, we consider only non-blank characters produced as output and sum the gradients over all contiguous repetitions of a character (that would be reduced to a single character by the CTC algorithm). Word-level attribution can be similarly computed by summing the character-level attributions corresponding to each character that makes up the word.
Figure 2 illustrates how attribution changes for a specific word, "FIRE", across different accents. We consider speech samples from all seven accents corresponding to the same underlying reference text, "The burning fire had been extinguished". Each subplot also shows the phonetic alignment of the text on its x-axis. We observe that the attributions for "FIRE" are fairly well-aligned with the underlying speech in the US and Canadian samples; the attributions appear to deviate more in their alignments
for all the other accents.
To quantify the differences in alignment across accents suggested by the visualization in Figure 2, we measure the alignment accuracy using the earth mover's distance (EMD). For each accent, we compute the EMD between two distributions, one derived from the attributions and the other from the reference phonetic alignment. The EMD between two distributions  $p$  and  $q$  over the set of frames (or rather, frame sequence numbers)  $T$  is defined as
$$
\operatorname {E M D} (p, q) = \inf  _ {Z} \sum_ {i, j \in T} | i - j | \cdot Z (i, j)
$$
where the infimum is over all "transportation functions"  $Z: T \times T \to \mathbb{R}_+$  such that  $\sum_{j \in T} Z(i, j) = p(i)$  (for all  $i$ ) and  $\sum_{i \in T} Z(i, j) = q(j)$  (for all  $j$ ).
Given a correctly predicted word, we define the distribution  $p$  as the uniform distribution over the frames that are aligned with the word, and  $q$  as the distribution obtained by normalizing the word-level attribution of the word in the utterance. For each accent, we sample a set of words that were correctly predicted (equally many for all accents) and compute the average of the EMD between the dis
4The CTC algorithm produces output probabilities for observing a "blank", signifying no label. Excluding the blank symbol from our analysis helped with reducing gradient computation time. We also confirmed that including the blank symbol did not change the results from our analysis.
Accent	EMD
C0	C1	C2	Overall
US	43.54	42.42	39.55	42.6
Canada	42.17	39.68	40.47	40.94
Indian	53.07	47.47	49.63	50.34
African	46.63	42.61	41.05	44.3
England	47.0	41.52	43.44	44.3
Scotland	45.34	41.38	41.65	43.26
Australian	46.91	44.24	47.45	45.87
Table 2: EMD trends quantifying the difference in attributions across accents.  $C_0$ ,  $C_1$  and  $C_2$  are clusters of words containing {1-2}, 3 and {4-5} phones, respectively
tributions  $p$  and  $q$  corresponding to each word. This average serves as an alignment accuracy measure for the accent. For the EMD analysis, we restrict ourselves to a set of 380 sentences that have corresponding speech utterances in all accents. This way, the content is mostly identical across all accents. Table 2 shows the averaged EMD values for each accent computed across all correctly predicted words. Larger EMD values signify poorer alignments. The overall values clearly show that the alignments from US and Canadian-accented samples are most accurate and the alignments from the Indian-accented samples are most inaccurate. We also cluster the words based on the number of phones in each word, with  $C_0$ ,  $C_1$  and  $C_2$  referring to words with {1-2}, 3 and {4-5} phones, respectively. As expected, words in  $C_0$ , being smallest in size, deviate most from the reference distribution and incur larger EMD values (compared to  $C_1$  and  $C_2$ ). The overall trend across accents remains the same for each cluster.
3.2 Information Mixing Analysis
Another gradient-based analysis we carried out is to check if accents affected how, at various levels, the representation at each frame is influenced by the signal at the corresponding input frame. One can expect that, in layers higher up, the representation at each frame mixes information from more and more input frames. However, it is reasonable to expect that most of the contribution to the representation should still come from the frames in a window corresponding to the same phone. (We examine the contribution of neighboring phones in Appendix B)
As detailed below, we devise quantities that measure the extent of information mixing and apply them to our systems. Not surprisingly, as shown below, we do observe that mixing increases as one
Figure 3: Comparison of phone focus across layers for various accents.
climbs through the layers. But somewhat surprisingly, we find that there is little variation of these trends across accents. This suggests that information mixing is largely dictated by the network itself, rather than by the details of the data.
The quantities we use to measure information mixing are inspired by Brunner et al. (2020). We define the contribution of the  $i^{\mathrm{th}}$  input frame  $x_{i}$  to the final output of the network  $f$  via the representation  $e_j^l$  in a given layer  $l$  corresponding to frame  $j$  as:
$$
\hat {g} _ {i, j} ^ {l} = \left\| \sum_ {k = 1} ^ {d} \left(\frac {\partial f}{\partial e _ {j} ^ {l} (k)}\right) \left(\frac {\partial e _ {j} ^ {l} (k)}{\partial x _ {i}}\right) \right\| _ {2} \tag {1}
$$
where  $e_j^l$  is a  $d$ -dimensional vector  $(e_j^l (k)$  refers to the  $k^{th}$  dimension of  $e_j^l$ ), and  $f$  consists of the non-blank characters in the maximum probability output (after the softmax layer). We use a normalized version of  $\hat{g}_{i,j}^l$  to compare the contribution to  $e_j^l$  from different  $x_{i}$ :
$$
g _ {i, j} ^ {l} = \frac {\hat {g} _ {i , j} ^ {l}}{\sum_ {n = 1} ^ {T} \hat {g} _ {n , j} ^ {l}}
$$
For this analysis, we used a subset of 250 utterances for each accent that have almost the same underlying content.
A measure of "focus" of an input phone at level  $l$  - how much the frames at level  $l$  corresponding to that phone draw their contributions from the corresponding frames in the input - is obtained by summing up  $g_{i,j}^{l}$  over  $i,j$  corresponding to the phone. Figure 3 shows this quantity, averaged over all phones in all the utterances for each accent. We observe that the focus decreases as we move from CONV to  $\mathrm{RNN}_4$ , with the largest drop appearing between CONV and  $\mathrm{RNN}_0$ . This is intuitive as we expect some of the focus to shift from individual
This smaller sample was chosen for faster gradient computations and gave layer-wise phone accuracies similar to what we obtained for the complete test set of 1000 utterances. A plot showing these consistent trends is included in Appendix D
Figure 4: Variation in binary focus measure, averaged over all the phones, across layers for various accents.
phones to their surrounding context, as we move to a recurrent layer (from the CONV layer). This trend persists in moving from  $\mathrm{RNN}_0$  to  $\mathrm{RNN}_4$  with the focus on individual phones steadily dropping. We also see a consistent albeit marginal trend across accents with US/Canadian-accented samples showing the lowest focus.
For each input phone, one can also define a binary measure of focus at level  $l$ , which checks that the focus of the frames at that level has not shifted to an input phone other than the one whose frames it corresponds to. That is, this binary focus measure is 1 if the focus of the phone at a level as defined above is larger than the contribution from the input frames of every other phone. Figure 4 shows how this measure, averaged across all phones for each accent, varies across layers. Again, we see that focus is highest in the first CONV layer, dropping to  $70\%$  at  $\mathrm{RNN}_1$  and  $45\%$  at  $\mathrm{RNN}_3$ . Further, again, we observe very similar trends across all accents.
From both the above analyses of focus, we observe that there is a pronounced drop in focus through the layers, but this trend is largely independent of the accent. We also plot variations for the well-known TIMIT database (Garofolo, 1993) in both Figures 3 and 4 to confirm that the same trend persists. For TIMIT, we used the samples from the specified test set along with the phonetic alignments that come with the dataset. We conclude that information mixing, and in particular, the measures of focus we used, are more a feature of the network than the data.
4 Information-Theoretic Analysis
In the previous section, we used gradient-based methods to examine how much an input frame (or a set of frames corresponding to a phone or a word) contributes to the output and how these measures change with varying accents. Without computing gradients, one could also directly measure how
Figure 5: Mutual Information between hidden representations and accents across layers.
much information about accents is encoded within the representations at each layer. Towards this, motivated by Voita et al. (2019), we compute the mutual information (MI) between random variables  $e_{\mathbf{x}}^{l}$  and  $\alpha$ , where  $e_{\mathbf{x}}^{l}$  refers to a representation at layer  $l$  corresponding to input  $\mathbf{x}$  and  $\alpha \in [0,6]$  is a discrete random variable signifying accents. We define a probability distribution for  $e_{\mathbf{x}}^{l}$  by discretizing the space of embeddings via  $k$ -means clustering (Sajjadi et al., 2018). We use mini-batched  $k$ -means to cluster all the representations corresponding to files in the test sets mentioned in Table 1 across accents and use the cluster labels thereafter to compute MI.
Figure 5 shows how MI varies across different layers for three different values of  $k$ . Increasing  $k$  would naturally result in larger MI values. (The maximum possible value of MI for this task would be  $\log_2(7)$ .) We observe a dip in MI going from spectral features SPEC to CONV, which is natural considering that unprocessed acoustic features would contain most information about the underlying accent. Interestingly, we observe a rise in MI going from CONV to  $\mathrm{RNN}_0$  signifying that the first layer of RNN-based representations carries the most information about accent (not considering the acoustic features). All subsequent RNN layers yield lower MI values.
Apart from the MI between representations and accents that capture how much accent information is encoded within the hidden representations, we also compute MI between representations and a discrete random variable signifying phones. The MI computation is analogous to what we did for accents. We will now have a separate MI plot across layers corresponding to each accent. Figure 6 shows the MI values across layers for each accent when  $k = 500$  and  $k = 2000$ . We see an overall trend of increasing MI from initial to later layers. Interestingly, the MI values across ac
(a) Cluster Size: 500
(b) Cluster Size: 2000
Figure 6: Mutual Information between representations and phones for different clusters sizes and accents.
Figure 7: t-SNE plot for representations of top 10 phones across US and Indian-accented samples, with the following layers on the X-axis (from left to right): SPEC, CONV, RNN $_0$ , RNN $_1$ , RNN $_2$ , RNN $_3$ , RNN $_4$
cents at  $\mathrm{RNN_4}$  exhibit a familiar ordering where US/Canadian accents receive the highest MI value while Indian and Scotland's accents receive the lowest MI value.