Accented Speech Recognition With Accent-specific Codebooks
Authors: Darshan Deepak Prabhu
Preethi Jyothi
Sriram Ganapathy
Vinit Unni
Venue: EMNLP 2023 Main (2023)
Keywords: accented speech recognition
cross-attention
codebooks
conformer
domain adaptation
TL;DR: We propose a new technique based on quantized accent codebooks to improve accent adaptation for ASR.

Abstract: Speech accents pose a significant challenge to state-of-the-art automatic speech recognition (ASR) systems. Degradation in performance across underrepresented accents is a severe deterrent to the inclusive adoption of ASR. In this work, we propose a novel accent adaptation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks. These learnable codebooks capture accent-specific information and are integrated within the ASR encoder layers. The model is trained on accented English  speech, while the test data also contained accents which were not seen during training.  On the Mozilla Common Voice multi-accented dataset, we show that our proposed approach yields significant performance gains not only on the seen English accents (up to 37% relative improvement in word error rate) but also on the  unseen accents (up to 5% relative improvement in WER). Further, we  illustrate benefits for a zero-shot transfer setup on the L2Artic dataset. We also compare the performance with other approaches based on accent adversarial training.

Accented Speech Recognition With Accent-specific Codebooks
Darshan Prabhu-, Preethi Jyothi', Sriram Ganapathy Vinit Unnit
Indian Institute Technology Bombay, Mumbai, India
Indian Institute of Science, Bangalore, India
(darshanp, picaaan,,piyyyto pjyothi, viniyyotiiiiiitteeee... venie iitb ac. int, sriramgeiisc.ac.in
Abstract
2004), ASR systems show substantial degradation in performance when dealing with new accents that are unseen during training (Chu et al., 2021).
Speech accents pose significant challenge to state-of-the-an automatic speech recognition (ASR) systems. Degradation performance across underrepresented accents severe de- terrent to the inclusive adoption of ASR. In this work, we propose a novel accent adap- tation approach for end-to-end ASR systems using cross-attention with a trainable set of codebooks. These learnable codebooks capture accent-specific information and are integrated within the ASR encoder layers. The model is trained on accented Finglish speech, while the test data also contained accents which were not seen during training. On the Mozilla Com- mon Voice multi-accented dataset, we show that our proposed approach yields significant performance gains not only on the seen English accents (up to 37% relative improvement in word error rate) but also on the unseen accents (up to 5% relative improvement WER). Fur- ther, we illustrate benefits for zero-shot trans- fer setup on the L2Artic dataset. We also com- pare the performance with other approaches based on accent ad adversarial training.
Prior works attempting to address accent-related challenges for ASR can be categorized into three groups: i) multi-accent training (Huang et al., 2014; Elfeky et al.. 2016), ii) accent-aware training us- ing accent embeddings (Jain et al., 2018 8) or adver- sarial learning (Sun et al., 2018), and iii) accent adaptation using supervised (Rao and Sak, 2017; Winata al., 2020) or unsupervised techniques (Tu- ran et al., 2020). While partial success has been achieved using most of these approaches, the devel- opment of robust speech recognition systems that are invariant to accent differences in training and test remains a challenging problem.
In this work, propose a new codebook based technique for accent adaptation of state-of-the- art Conformer-based end-to-end (F2E) ASR mod- cls (Gulati et al., 2020). For cach of the accents observed in the training data, we define a codebook with a predefined number of vandomly-initialized vectors. These accent codes are integrated with the self-attended representations in each encoder layer via the cross- aatentttttnioon mechanism, similar the perceiver framework (Jaegle et al., 2021). The ASR model is trained on multi-accenice data with standard end-to-end (E2E) ASR objectives. The codes capture accent-specific information as the training progresses. During inference, we propose ream-search decoding algorithm that searches over a combined set of hypotheses obtained by us- ing each set of accent-specific codes (once for each seen accent) with the trained ASR model. On the Mozilla Common Voice (MCV) corpus, observe significant improvements on both seen and new ac- cents at test-time compared to the baseline and existing supervised accenttadaptationn techniques. Our main contributions are:
1 Introduction
Accents in speech typically refer to the distinctive way in which the words are pronounced by diverse speakers. While speaker's accent may be pri- marily derived from their native language, speech accents are also influenced by various other fac- tors related to the geographic location, educational background, socio-economic and liniu-lin factors like race, gender and cultural diversity (Ben- zeghiba et al., 2007). It is therefore infeasible to build automatic speech recognition (ASR) systems which comprehensively cover speech accents dur- ing training. In such scenarios, novel speech ac- cents continue to have an adverse effect on ASR performance (Beringer et al., 1998; Aksënova et al., 2022). While humans effectively recognize speech from new and unseen accents (Clarke and Garrett,
We propose a new accent adaptation technique for Conformer based end-to-end ASR models

---

using cross-altention over of learnable codebooks. Our technique comprises learn- ing accent-specific codes during training and a new beam-search decoding algorithm to per- form an optimized combination of the codes from the seen accents. We demonstrate signif- icant performance improvements on both seen and unseen accents over competitive baselines the MCV dataset.
agnostic approaches use cosine losses (Unni al., 2020) or contrastive losses (Khosla et al., 2020; Han et al., 2021) to make the model accent neutral. These losses force the model to output similar rep- resentations for inputs with the same underlying transcript.
Accent-aware ASR. Accent-aware approaches feed the model additional information about the accent of the input speech. Early work in this category focused on using the multi-task learning (MTL) paradigm (Zheng et al., 2015; Jain et al., 2018; 8; Das et al., 2021a) that jointly trains accent- specific auxiliary tasks with ASR. Different types of embeddings like i-vectors (Saon et al., 2013; Chen al., 2015), dialect symbols (Li et al., 2017), embeddings extracted from TDNN models (Jain et al., 2018) or from wav2vec2 models trained as classifier (Li et al., 2021 la; Deng et al., 2021) have also been explored for accented ASR. Many simple ways of fusing accent information with the input speech have been previously investigated. This fu- sion can either be sum (Jain et al., 2018; Viglino et al., 2019; Li et al., 202 weighted sum (Deng concatenation (Li et al., 2021a,b).
Even on a zero-shot setting involving new accented evaluation set, L2-Aretic (Zhao et al., 2018), we show significant improvements us- ing our codebooks trained using MCV.
We publicly release train/development/estt splits spanning different seen and unseen ac- cents in the MCV corpus. Reproducible splits on MCV have been entirely missing in prior work and we hope this will facilitate fair comparisons across existing and new accent- adaptation techniques."
2 Related Work
et al., 2021) Few works also explore the possibility of merg- ing both accent-aware and accent-agnostic tech- niques within the same model (Zhou et al., 2023). Our work also proposes an accent-aware approach. However, unlike prior work that focuses on pre- fetched accent information, we learn accent infor- mation embedded within codebooks during train- ing. Additionally, instead of simply concatenating input speech with accent embeddings, we propose a learned fusion of accent information with speech representations using cross-attention. Prior work by Deng et al. (2021) demonstrates une-grained integration of accent information. However, our proposed framework integrates accent information as part of end-to-end training resulting in robust adaptation.
Traditional cascaded ASR systems (Gareía-Moral et al., 2007) handled accents by either modify- ing the pronunciation dictionary (Humphries and Woodland, 1997; Weninger et al., 2019) modi- fying the acoustic model (Fraga-Silva et al., 2014; Yoo et al., 2019). More recent work on accented ASR has focused on building end-to-end accent- robust ASR models. Towards this, there are two sets of prior works: Accent-agnosse approaches and Accent-aware approaches.
Accent-agnostic ASR. Such approaches force the model to disregard the accent information present in the speech and focus only on the under- lying content. Prior work based on this approach uses adversarial training (Ganin et al., 2015) or sim- ilarity losses. Using domain adversarial training, with the diseriminator being an accent classifier, has shown significant improvements over standard ASR models (Sun et al., 2018). Pre-training the accent classifier (Das et al., 2021b) and clustering- based accent relabelling (Hu et al., 2020) have also led to further performance improvements. The use of generative adversarial networks for this task has also been explored (Chen et al., 2019). Rather than being explicitly domain adversarial, other accent
3 Methodology
Base model. Our base architecture uses the stan- dard joint CTC-Attention framework (Kim et al., 2016) with an encoder (ENC), a decoder (DEC- ATT), and Connectionist Temporal Classification (CTC) (Graves et al., 2006) module (DEC-CTO) CTC). For a given speech input x {œ1...., TT), the encoder ENC generates contextualized representa- tions h ENC(x) = (hi....,hr) The encoder representations are further used by DEC-ATT
'The MCV data splits and codehase are available at: wwwpe//cccccccdddddcooo-tt-tttt.

---

CTC Loss
Attention Loss
deterministic gating scheme select the codebook specific the underlying accent of the training ex- ample. To support the selection of a single accent codebook during inference, when the accent labels for the test utterances are unknown, we modify the beam-search scarch decoder to search across all seen ac- cents. We found such a hard gating to be critical to achieve ASR performance improvements. In Section 6, compare the proposed model with soft gating mechanism that works with a standard beam-search decoding.
ho
CTC
Decoder
Add Norm
Accent
Specific
Linear layer
Codebooks
Architecture
Add Norm
Layer 12
Convelution
Ercoder Layer
Add Norm
Embedding Layer
Cross Attention i
Encoder ht
[s, 2pring
Each codebook contains P d-dimensional vec- tors that we refer to as codebook entries. The en- tries belonging to the 7th codebook are generated as:
Encoder h?
AAcd Norm
c² - (ct.... dpp) Embedding([1,. PI.) PJ)
Convolution Sub-sampling
where Embedding is a standard embedding layer. In the following sections, we use fct.....c0. cp) to refer to the codebook corresponding to the un- derlying accent label for given training example.
(e%, Gu.., ct)
Figure I: Overview of our proposed architecture inte- grating accent codebooks into encoder layers via cross- attention. A represents the accent label for training instance and {cl ee....eMM Aa) } is the collection of accent-
3.2 Encoder with Accent Codebooks
Figure illlustrates the overall architecture with the proposed integration of a codebook into each encoder layer via a cross-attention sub-layer. We will refer to this new ule as ENC, [ENC),.... ... ENCy that consists of stack of L identical Conformer layers. The 7th encoder layer ENC₁ takes both h and inputs, and produces h' output. Codebook c is shared across all the encoder layers. All the vectors involved in the computation attention scores are 256 dimensional.
specific codebooks.
and DEC-CTC to jointly predict the output token sequence y ../1....,.2... 9j.............................. y...... w. DEC-ArT is an autoregressive decoder that maximizes the con- ditional likelihood of producing an output token Yj given and the previous labels y1,.... 1/-1- In contrast, DEC-CTC uses CTC to maximize the like- lihood of given h by marginalizing over all align- ments. The encoder is implemented using Con- former layers (Gulati et al., 2020) and the decoder is implemented using Transformer layers (Vaswani al., 2017).
encoder mod-
A cross-attention sub-layer integrates accent in- formation from codebook into each encoder layer. This sub-layer takes both self-attended contextual- ized representations H and the codebook as its in- puts and generates codebook-specific information relevant to the speech frames of this contextual rep- resentation. More formally, the operations within encoder layer ENC, can be written as follows:
For our proposed technique, we introduce the following three essential modifications the base architecture: i) Constructing codebooks that can encode accent-specifie information (Section 3.1). Enabling fine-grained integration of accent infor- mation with Conformer-based ASR model using cross-atts cross-attention (Section 3.2). iii) Modifying beam search decoding for inference in the absence of accent labels at test-time (Section 3.3).
A MultHHeadAAtnnaar (h*
H = NormLayer (a A
Č MultiHeadAttnn,
= NormLayer ch
C)
3.1 Codebook Construction
j Convolution C)
Consider M seen accents, which are observed dur- ing training. We generate M codebooks, one per accent, where the ihh codebook learns latent codes specific to the accent. During training, we use a
= NormLayer J)
ht Linear (inearpw(J] (J)
h2 HormLayCrinnnl (J h()

---

where the equations colored in purple highlight our changes to the standard Conformer encoder layer. The above equations can be viewed as a stack of four independent blocks, each having a residual connection and being separated by layer normalization.
Algorithm 1: Inference algorithm that per- forms joint beam-search over all accents. Our modifications to the standard beam search (Meister et al., 2020) are highlighted. Each beam entry is triplet (s, y/ A) where refers to seen accent. score, () is modified scoring function which uses the codebook for accent A during the forward pass.
MultiHeadAttnl K, V) refers to standard multi-head attention module (Vaswani et al., 2017) with Q. K and V denoting queries, keys and values respectively. MulttHeadAAttnsff is a self-attention module where each frame h' attends to every other frame, thus adding contextual information. Convolution is a stack of three convolution lay- ers: depth-wise convolution sandwiched between two point-wise convolutions, each having a single stride. The input and output of the Convolution block are d-dimensional vectors. A position-wiss feed-forward layer Linearpw made up of two lin- car transformations with ReLU activation. This takes d-dimensional output from the convolution module as input and produces a d-dimensional out- put vector with a hidden layer of 2048 dimensions. MultiHeadAAtnag (H, C, c) is our proposed attsstatt module over codebook entries,
Input: x: speech input
V: list of vocabulary tokens
Rmax: maximum hypothesis length
k: maximum beam width
y: output prediction so far
score scoring function
Bo {(0, <sos>, 1) .... (0, <sos>, M)}
for E {1,.... ...... nmax 1) do
3
B
for (s, A)
B 1 do
5
if y.last( = <eos> then
B.add (s, y, A))
continue
for
CVVVV do
8 +
score,(x, A)
where each frame of input H attends to all the en- tries in the codebook e to generate attention scores. These attention scores are further used to gener- ate frame-relevant information Č as a weighted average of codebook entries. We claborate further on the attention computation for single attention head in C.2 Let H2 refer to the jüh frame in H. The attention distribution (ai,1.... (a..1,...a,,p)) a,,P), where ajj is the attention probability given by H, to the 2ith codebook entry, is computed
10
B. B.add y v, A))
B. Btto
12 return B
3.3 Modified Beam-Search Algorithm
Since we do not have access to accent labels at test-time, we rely on either using a classifier to predict the accent or modifying beam-search to accommodate the prediction streams generated by all seen accent choices. Due to large imbalance in the accent distribution during training with certain seen accents dominating the training set, we find the classifier to be incffective during inference. We elaborate on this further in Section
(ag,1, a,,,,..,,,,.... (W (WiH))(W2o))
softmax
(W:
Vd
where Wi. Wi and Wi e IRdxd are learned pro- jection matrices. These attention scores are further used to generate the weighted average of codebook entries in Ć:
P Σ 1=1
Figure shows our inference algorithm that per- forms joint beam search over all the seen accents. Each beam entry is triplet that expands each hy- pothesis using each seen accent. Scores for each seen accent are computed using a forward pass through our ASR model by invoking the codebook specific to the accent. The beam width threshold k is then applied to expanded predictions across all seen accents.
OAK C₁
The cross-attention sublayer is further modified with a residual connection and layer normalization to generate the final bilebook-infusee representa- tions in C.
In our experiments, self-attention uses four attention heads for the encoder and cross- crnss-attention with the codebook uses a single attention head.

---

4 Experimental Setup
Train (in hours)
Dev (in hours)
Test
Accent
(in hours)
4.1 Datasets
Australia
6,95
45.36
4.33
0.46
All our experiments are conducted on the MCV ACv_ACCENT dataset extracted from the "vali- dated" split of the Mozilla Common Voice English (en) corpus (Ardila et al., 2019). Overall, 14 En- glish accents present in MCV ACCCENT are divided into two groups of seen and unseen accents. Table I lists the accents belonging to each of these groups.
6.79
1.21
England
19.51
119.9
3.22
1.65
Scotland
2.69
16.21
0.23
0.16
64.12
400.1
8.32
4.87
Africa
1.71
Hongkong
0.52
India
0.58
Ireland
1.94
Malaysia
0.39
Seen Accents
Unseen Accents
Newzealand
2.11
Australia (AUS)
Africa
(APR)
Malaysia
(MAL)
Philippines
0.90
Singapore
0.64
Canada (CAN)
New
(NW2)
Hong
(HKG)
Zealand
Kong
England (GBR)
Philippines (PHL)
India
(IND)
Table 2: Data splits for experiments. MA-100 and
Scotland (SCT)
Singupore (SGP)
Irelund
(RL)
MA-600 refers datasets MCV_ LEV_ACCENT-ION and MCV_ACCENT-600 datasets respectively.
US
(USA)
Wales
(WLS)
Table 1: List of 5 seen and 9 unseen accents in MCV_ACCENT corpus.
and decoder layers using joint CTC-Attention loss (Kim et al., 2016). We use four attention heads to attend over 256-dimensional tensors. The position-wise linear layer operates with 2048 mensions. We train the model for 50 epochs using 80- 90-dimensional filter-bank features with pitch. In all our experiments, we apply a stochastic depth rate of 0.3 which we found to yield an absolute 2% WER improvement, compared to baseline system without this regularization enabled. Dur- ing inference, we use a two-layer RNN language model (Mikolov et al., 2010) trained for 20 epochs with a batch size of 64. We conducted all our ex- periments on NVIDIA RTX A6000 GPUs.
We create train, dev, and test splits that speaker-disjoint. We construct two train sets, MCV_ACCENT-100 and MCV_ACCENT-600, com- prising approximately 100 hours and 620 hours labeled accented speech, respectively. Since MCV_ACCENT consists of many utterances that correspond to the same underlying text prompts, a careful division into dev/test sets that are dis- joint in transcripts from the train set performed. We train and validate the ASR models only on the seen accents, while the test data consists of both seen and unseen accents. The detailed statistics of our datasets are given in Table 2. For a quick turnaround, most of our experiments were con- ducted on MCV MEV__CCEEN-1000 For experiments in Section 5.3, we use the 620-hour MCV MCV_ACCENT- 600. All the data splits mentioned above are avail- able in codebase, which should enable direct comparisons across accent adaptation techniques. More details about the construction of the datasets are provided in Appendix A.
Experiments and Results
Table shows word error rates (WERs) compar- ing our best system (codebook attend (CA) sys- tem) with five approaches: 1. Transformer base- line (Dong et al., 2018). 2. Conformer baseline (Gu- lati et al., 2020). 3. Adding i-vector features (Chen et al., 2015y* to the filterbank features, input to the Conformer baseline. 4. Conformer jointly trained with an accent classifier using multi-tas task learning (Jicheng et al., 2021). Conformer with Domain Adversarial Training (DAT) (Das et al., 2021b), with an accent classifier at the 1h encoder layer.
4.2 Models and Implementation Details
We use the ESPnet toolkit (Watanabe et al., 2018) for all our ASR experiments. As is standard prac- tice, we further add 3-way speed perturbation to our dataset before training. We use the default con- figurations specified in the train_conforme yaml file provided in the ESPnet toolkit to train Conformer model comprising 12 encoder layers
"Every frame in i-vectors represents a second. Since the imput filterbank features span 25msec, we repeat the same i-vector frame for four consccutive feature frames. Adding, as opposed to concatenating, the i-vector frames was found to perform hetter.
com/exxthss

---

Table 4: Comparison of the zero-shot performance Seen Accents
Aggregated
Unsera Accents
All
Sien
Uneecn
ABS
CAN
UK
SCT
US
AFR
HBK
IND
IRI.
MAl.
NW7
PHI.
SGP
W1.
Trans. (Dong al.. 2018)
22.7
17.3
28.0
18.1
17.8
19.7
18.5
16.3
25.9
32.0
35.4
25.3
36.2
23.5
31.5
35.8
21.0
Conf. (Gulati al., 2020)
18.9
14.0
23.7
13.8
15.0
15.7
134
13.3
21.5
27.2
29.4
21.4
52.2
19.9
26.1
34.7
17.9
I-secter (Chen ol., 2015)
18.9
14.1
23.6
13.9
15.0
16.1
14.6
13.3
21.7
27.2
29.5
21.2
31.7
19.3
27.2
33.8
18.0
MTL Uichhng al., 2021)
18.9
14.1
23.7
14.7
15.1
16.1
13.7
13.2
21.8
28.1
29.1
21.5
33.0
19.4
26.5
34.2
18.1
DAT (Das et 10)
18.7
14.0
23.4
13.3
15.3
15.7
15.5
13.1
21.1
27.0
29.5
21.1
32.2
19.2
26,0
17.9
CA
18.21
13.6
22.9
11.5
14.
14,9
9.7
13.1
21.0
25.7
29.1
20.7
30.9
18.5
25.8
33.7
17.9
Table 3: Comparison of the performance (WER % of our architecture (codebook attend (CA)) with baseline and other techniques on the MCV-ACCENT- 100 dataset. Numbers in bold denote the best across baselines, and the green highlighting denotes the best WER across all experiments. Ties are broken using overall WER. CA: Codebook attend cross-attention applied all layers with 50 entries in each learnable codebook. indicates statistically significant results compared to DAT (at <0.001 using MAPSSWE test (Gillick and Cox. 1989)).
From Table 3, we observe that the Conformer baseline performs significantly better than the Transformer baseline. Adding i-vectors and multi- task training with an auxiliary accent classifier ob- jective perform equally well and are comparable to the Conformer baseline, while using DAT im- proves over the Conformer baseline. Our system significantly outperforms DAT (at p < 0.001 us- ing the MAPSSWE test (Gillick and Cox, 1989)) and achieves the lowest WERs across all the seen and unseen accents. We use 50 codebook entries for each accent and incorporate accent codebooks into each of the 12 encoder layers. Unless speci- fied otherwise, we will use this configuration in all subsequent experiments. Further ablations of these choices will be detailed in Section 5.5.
ning six non-native English accents namely Ara- bic (ARA), Hindi (HIN), Korean (KOR), Mandarin (MAN), Spanish (SPA), and Vietnamese (VIA). Ta- ble 4 shows WERs achieved by our system in comparison to the baseline and other techniques. Our proposed method significantly outperforms all these approaches on every single accent (p 0.001 using the MAPSSWE test (Gillick and Cox, 1989)).
5.2
Effect of Training Data Size
Method
Overall
Seen
Unseen
Conf. (Gulati al., 2020)
9.75
6.04
13.46
I-vector (Chen et al., 2015)
10.05
6.40
13.69
MTL. (Ticheng et al., 2021)
10.02
6.33
13.70
DAT (Das al., 202
9.73
6.12
13.33
CA 50)
9.63
6.22
13.03
5.1
Zero-shot Transfer
CA 200)
9,59
6.20
12.98
CALeaI...aI 500)
9.55
6.19
12.92
Method
All
Accents
Table 5: Comparison of the performance (WER %) of our approach with other methodologies on
ARA
HIN
KOR
MAN
SPA
VIA
Conformer
30.4
30.3
43.5
MCV MCV_ACCENT-600 dataset.
L-vector
33.6
31.0
31.2
27.2
38.0
30.4
43,9
MTT.
30.4
30.1
43.7
Table 5 compares proposed system with DAT and Conformer the 600-hour dataset. Compared to the Conformer baseline and the DAT, the proposed CA approach shows steady improvement over unseen accents, while resulting in a minor drop in performance on the seen accents.
DAT
33.5
30.7
30.8
26.8
38.3
30.1
43,9
CA
32.61
29.5
30.4
26.2
37.1
29.3
42.8
(WER %) of our architecture with other techniques on L2Aretic dataset. t indicates a statistically significant improvement (p <0,001 using MAPSSWE test) using codebook attend (CA) w.r.t. the Conformer baseline.
5.3 Effeet of Number of Parameters
To further validate the efficacy of our proposed approach using accent-specific codebooks, we per- form zero-shot evaluations on the 1.2rrctic dataset. We note here that do not any L2Arctic data for finetuning: our ASR model is trained on MCV ACCENT-100. Such zero-shot evaluation helps ascertain whether our codebooks transfer well across datasets. The L2Arctic dataset (Zhao et al., 2018) comprises English utterances span-
Method
params
Overall
Seen
Coaf.
43M
18.57
14.05
23.67
Coof. wh1 encoder units
46M
18.89
14.02
23.74
Coaf. w/ allienion dim
46M
18.77
14.02
23.51
CAreu.....y 50)
13.57
Table 6: Comparison of the performance (WER %) of our approach with parameter-squivalent variants of the Conformer baseline MCV LCW_AACENN--000
To discount the possibility that improvements

---

using our proposed model could be attributed to an increase in the number of parameters, in Table we compare our proposed system with multiple variants the baseline Conformer model (referred to as Conf. in Table 3) where parameters are in- creased to be commensurate with our proposed model by either (1) Increasing the number of en- coder units (from 2048 + 2320) (2) Increas- ing the dimension used for attention computation (from 256 + 272). We observe slight improve- ment over the standard baseline when the attention dimension is increased. However, compared to all these baselines, our proposed model still shows a statistically significant improvement at <0.001.
Method
Overall
Seen
Unseen
CALec.....a 26)
18.33
13.76
22.89
12) (P= 50)
18.22
13.57
22,86
12 (P 100)
18.36
13.85
22.86
12))
12)
200)
18.41
13.69
23.12
CA2e(t....221) (P 500)
18.39
13.68
23.09
4) 1|l 50)
18.30
13.95
22.64
CAr.ei.....) 50)
18.31
13.86
22.75
CALE (P 50)
18.92
14.24
23.59
CA (P 50)
18.45
13.84
23.05
CAzeci....a) (Pan 50)
18,30
13.65
22.95
Table 8: Comparison the performance (WER %) of different variants of architecture. CA CE(.......)) (P k): Codebook attention applied at all layers from i to with entries per accent codebook. CA ()...( (Prnnd = k): Similar the previous setup. but with codebooks frozen during training. Accent-wiss WER shown in Appendix B and a few select examples highlighted in Appendix C.
5.4 Balanced versus Imbalanced Dataset
Method
Overall
Seen
Unseen
Conformer
19.30
14.73
23.86
CALE( 12 50)
18.88
14.61
23.13
50. Since accent effects can be largely attributed acoustic differences, we see that the early encoder layers closer to the speech inputs benefit most from the codebooks. Adding codebooks only to the last four or eight encoder layers is not beneficial.
Table 7: Comparison of the performance (WER %) of our approach with Conformer baseline on an accent balanced MCV_ sCv_ACCENT--0M dataset.
To check the effectiveness of our approach on a balanced dataset, in Table 7, we compare our proposed system with the Conformer baseline on a 100-hour accent-balance data split. Even on such a balanced dataset, our architecture shows statistically significant improvement (at p=0.005) compared to the baseline.
Randomly initialized codebooks were observed to be as useful as learnable codebooks for self- supervised representation learning in Chiu et al. (2022). Motivated by this result, we experiment with hundomly-initialize accent-specific code- books that are not learned during training. The last row of Table 8 shows that random codebooks only cause slight degradation in performance com- pared to the best performing system, echoing the observations in Chiu et al. (2022).
5.5 Ablation Studies
We present two ablation analyses examining the effect of changing the number of accent-specific codebook entries (P) and the effect of applying cross-attention at different encoder layers.
5.6 Inference with Single Accent
The first five rows in Table 8 refer to the addi- tion of codebooks to all encoder layers via cross- attention with varying accent-specific codebook sizes (P) ranging from 25 to 500. As P increases, the experiments show improved performance on seen accents but degrades on the unseen accents, indicating that the codebooks begin to overfit to the seen accents. Our best-performing system with P 50 performs well on seen accents while also generalizing to the unseen accents. As expected, using lower-capacity codebooks (P 25) shows performance degradation.
To understand the effectiveness of accent-specific codebooks, conduct five experiments by com- mitting to a single seen accent during inference. That is, we decode all the test utterances using a fixed accent label. Table shows results from in- ferring with a single accent across both seen and unseen accents. For the seen accents, the diago- nal contains the lowest WERs indicating that the information learned in our codebooks benefits the accented samples. Furthermore, similar accents, from geographically-close regions, benefit each other. The New Zealand accented English speech achieves the best WERs using Australian accent specific codebooks, Hong Kong, Indian, Philip- pines and Singapore accented test utterances prefer
The next five rows Table refer to codebooks with cross-attention introduced at varying encoder layers. The number of codebook entries is fixed at

---

Seen Accents
Unseen Accents
Accent used
AUS
CAN
UK
SCT
US
AFR
IKG
IND
IRL
MAL
NWZ
PIL
SGP
WLS
Australia
11.5
19.5
17.0
18.1
17.4
22.0
29.8
32.5
24.3
33.7
18.7
30.1
37.8
21.1
Canada
20.5
20.0
15.7
13.5
27.4
32.7
25.7
35.4
21.8
England
13.8
17.7
15.0
14.4
16.2
21.5
27.0
29.9
22.0
32.3
21.1
27.1
34.8
18.0
Scotland
20.7
17.8
19.1
10.2
16.4
24.4
28.2
33.5
22.6
34.4
25.7
29.0
36.6
21.3
us
20.2
14.7
19.4
15.5
13.2
23.4
27.0
28.1
21.7
32.4
24.7
25.8
34.3
22.2
Table 9: Comparison of the performances (WER%) of inferences done using fixed accent labels.
US accented codebooks, and Wales accent achieves its best results using Ingland-sppecffin codebooks. The WER results achieved by our best- performing system in Table 3 are much lower than the best WER results achieved in these single- accent experiments. This indicates that one cannot directly map an unseen accent seen accent and therefore, making this decision in- dependently for each utterance (as we propose to do the joint beam search) is crucial.
Codebooks GBR
AUS
CAN
SCT
US
UK
317
437
174
SCT
7
24
appropriate
US
595
1035
518
1151
1016
APR
340
160
264
202
167
Languages
HMC
56
59
80
116
88
86
99
110
104
5.7 Beam-Search Decoding Variants
229
262
405
244
All the results reported thus far use a joint beam scarch decoding. Table 10 shows a comparison our proposed joint beam search (elaborated in Section 3.3) with other beam-search variants incur- ring varying inference overheads. By in Table 10
49
MWZ
PHL
109
SGP
72
83
114
Method
All
Seen
Unseen
Inference Time
WL.5
43
61
26
Standard search
18.87
14.05
23.67
1.0
5.02
Figure 2: Heatmap showing which codebooks are cho- sen during inference across seen and unseen accents. For example, the third cell in the first shows that 92 out 413 codebook belonging to England during decoding.
split
13611
1.14
Joint heam search
13.57
22.86
Table 10: WER (%) of various inference algorithms described in section 5.7 on MCV SEC_AACCENT-ON setup. Inference time gives relative comparison of the time taken by each decoding variant with the standard beam search as the reference.
utterances used the
equal number of beam slots leading to eventual under-utilization of beam slots. Our proposed joint inference in the B3 setting an effective compro- mise of B3 and B2, incurring similar inference over- heads as B, and achieving a performance closer to the Bj setting.
refers standard beam-search decoding over the Conformer baseline with beam width of k. The setting Bj and B3 refer to running beam-search . times, once for each seen accent and picking the best-scoring hypothesis among all predictions. For B, setting, we use a beam width of l for each seen accent. Naturally, this incurs a large decoding overhead with factor of M increase in inference time and changes the effective beam width to kr. In the B2 setting, divide the beam width into M parts, each occupied by specific accent, thus making the effective beam width k/M. The setting B1 performs the best, but significantly increases inference overhead. The BB setting efficient but under perforre due to all accents being given an
6 Discussion and Analysis
Codebook Utilization: Figure 2 is heatmap showing which accent codebook is used by the joint beam-search algorithm in generating the best ASR hypothesis for the test utterances. Across seen accents, we see diagonal dominance, indicating that seen accents show a preference for their respec- tive codebooks. This effect is especially strong for Australia, England and Scotland accents. US and Canada, on the other hand, have examples evenly

---

divided among each other. Among unseen accents, the Australia-specific codebook is picked up most by New Zealand test utterances.
1.0
0.8
0.4
0.2
0.0
1.5
1.0
0.0
Figure 4: Progression of the probability/entropy across seen accents for single Wales-accented test sample.
Taweapp
Figure 3: Progression of average test entropy of the probability distribution across seen accents.
We hypothesize that this could be due to the lack of strong accent classifier (or a lack of appropriate learning in the gates to capture accent information). Our joint beam-search decoding bypasses this requirement by searching across all seen accents.
Active Accents during Joint Ream-seaache Using joint beam-search decoding, it is possible for samples from different accents get pruned in early iterations and only one or two dominant accents be active from the start. To check for this, we compute the distribution of samples in the beam across the five seen accents and plot the average entropy of this distribution across all test instances in Figure 3. It is clear that four to five seen accents are active until time-step 20, after which certain accents gain more prominence. Figure shows both the probabilities across seen accents appearing in the beam for a single Wales- accented test sample, along with the entropy of this distribution. This shows how nearly all accents are active at the start of the utterance, with England becoming the dominant accent towards the end.
Why do we see Performance Improvements on Unseen Accents? For test utterances from un- seen accents, model is designed to choose (seen) accent codebooks that best fit the underlying (unseen) accent. It is somewhat analogous to how humans use familiar accents tackle unfamiliar ones (Anderson, 2018; Levy et al., 2019). During inference, our model searches through seen accent codebooks and chooses entries that are most like the unseen accents in the test instances.
Conclusion
In this work, we propose new end-to-end tech- nique for accented ASR that uses accent-specific codebooks and eross- -ttss-attention achieve signifi- cant performance improvements on seen and un- seen accents test time. We experiment with the Mozilla Common Voice corpus and show detailed ablations over our design choices. We also em- pirically analyze whether our codebooks encode information relevant to accents. The effective use of codebooks for accents opens up future avenues to encode non-semantic cues in speech that affect ASR performance, such as types of noise, dialects, emotion styles of speech, etc.
Alternatives to Joint Beam-search: We also explore two alternatives to learning accent labels within the ASR model itself: i) We jointly trained an accent classifier with ASR. During inference, this classifier provides pseudo-accent labels across seen accents that we use choose the codebook. ii) We adopted a gating mechanism inspired by Zhang et al. (2021) that adds a learnable gate to each codebook entry. Unlike our current determin- istic policy of picking a fixed subset of codebook entries, the learned gates are trained jointly with ASR to pick a designated codebook entry corre- sponding to the underlying accent of the utterance. During inference, the learned gates determine the codebook entries be used for each encoder layer. Both these techniques performed better than the Conformer baseline but were equivalent in perfor- mance to the DAT approach (Das et al., 2021b).
Acknowledgements
The second and third authors gratefully acknowl- edge financial support from SERB Core Research Grant, Department of Science and Technology, Govt India on accented speech processing.

---

8 Limitations
Nicole Beringer, Florian Schiel, and Peter Regel- Brietzmann. 1998. German regional variants a problem for automatic speech recognition? 5th Inter- national Conference on Spoken Language Processing (ICSLP 1998).
We identify a few key limitations of our proposed approach:
The codebook size a hyperparameter that needs to be finetuned for each task.
Mingming Chen, Zhanlei Yang, Jizhong Liang, Yan- peng Li, and Wenju Liu. 2015. Improving deep neu- ral networks based multi mucti-accent mandarin speech recognition using i-vectors and accent-specific top layer. In INTERSPEECH.
We currently employ accent-specific code- books, one for each accent. This does not scale very well and also does not enable shar- ing of codebook entries across accent code- books. Instead, we could use a single (large) codebook and use learnable gates to pick a subset of codebook entries corresponding to the underlying accent of the utterance.
Yi- Ti-Chen Chen, Zhaojun Yang, Ching-Fent Yeh, Ma- haveer Jain, and Michael L. Seltzer. 2019. Aip- net: Generative adversarial pre-training of accent- invariant networks for end-to-end speech recognition.
Chung-Chern Chiu, James Qin, Yu Zhang, Jiahui Yu. and Yonghui Wu. 2022. Self-supervised learning with uandom-projection quantizer for speech recogni- tion.
Our proposed joint beam-search leads to a 16% increase in computation time at inference. This can be made more efficient as part of future work.
Xiangyun Chu, Elizabeth Combs, Amber Wang, and Michael Picheny. 2021. Accented speech recogni- tion inspired by human perception. arXiv preprint arXiv:2104.0400
Our joint beam-search allows for each utter- ance at test-time commit single seen accent. However, parts an utterance might benefit from one seen accent, while other parts of the same utterance might benefit from a dif- ferent seen accent. Such a mix-and-matee across seen accents is currently not part of our approach. Accommodating for such effects might improve our model further.
Constance M Clarke and Merrill Garrett. 2004. Rapid adaptation to foreign-accented english. The Journal of the Acoustical Society of America, 116(6):3677- 3658.
Amit Das, Kshitiz Kumar, and Jian Wu. 2021a. Multi- dialect speech recognition in english using attention on ensemble of experts. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6244 6248.
Nilaksh Das, Sravan Bodapati, Monica Sunkara, Sun- dararajan Srinivasan, and Duen Horng Chau. 202 lb. Best of both worlds: Robust accented speech recog- nition with adversarial transfer learning.
References
Alëna Aksënova, Zhehuai Chen, Chung-Cheng Chiu, Daan van Esch, Pavel Golik, Wei Han, Levi King, Bhuvana Ramabhadran, Andrew Rosenberg. Suzan Schwartz, et al. 2022. Accented speech recognition: Benchmarking, pre-training and diverse data. arXiv prepriat arXiv:2205.068 000014.
Keqi Deng, Songjun Cao, and Long Ma. 2021. Im- proving accent identification and accented speech recognition under a framework of self-supervised learning.
Linhao Dong, Shuang Xu, and Bo Xu. 2018. Speech- transformer: A no-recurrence sequence-to-sequenc model for speech recognition. In 2018 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5884 5888.
Catherine Anderson. 2018. Essentials of Linguistics. McMaster University.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2019. Common voice: A massively- multilingual speech corpus.
Mohamed Elfeky, Mcysam Bastani, Xavier Velez, Pedro Moreno, and Austin Waters. 2016. Towards acous- model unification across dialects. In 2016 IEEE Spoken Language Technology Workshop (SI.T), pages 628-628.
Mohamed Benzeghiba, Renato De Mori, Olivier Deroo, Stephane Dupont, Teodora Erbes, Denis Jouvet, Luciano Fissore, Pietro Laface, Alfred Mertins, Christophe Ris, et al. 2007. Automatic speech recog- nition and speech variability: A review. Speech com- munication, (1110-11):763-866
Thiago Fraga Silva, Jean Gauvain, and Lori Lamel. 2014. Speech recognition of multiple accented en- glish data using acoustic model interpolation. In 2014 22nd European Signal Processing Conference (EUSIPCO), pages 1781-1785.

---

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2015. Domain-adversarial training of neural networks.
Zhang Jicheng. Peng Yizhou, Pham Tung, Haihua Xu, Hao Huang, and Eng Chng. 2021. EZe-based multi- task learning approach to joint speech and accent recognition.
Ana García-Moral, Rubén Solera-Ureña, Carmen Peléez-Moreno, and Fernando Diaz-de María. 2007. Hybrid models for automatic speech recognition: A comparison of classical ann and kernel based meth- ods. In Advances in Nonlinear Speech Processing, pages 152-160, Berlin, Heidelberg. Springer Berlin Heidelherg.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sama, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Super- vised contrastive learning.
Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2016. Joint etc-attention based end-to-end speech recogni- tion using multi-task learning.
Larry Gillick and Stephen Cox. 1989. Some statisti- cal issues the comparison of speech recognition algorithms. volume 1, pages 532 535 vol.1.
Helena Levy, Lars KONIECZNY, and Adriana HAN- ULÍKOVÁ. 2019. Processing of unfamiliar accents in monolingual and bilingual children: effects of type and amount of accent experience. Journal of Child Language, 46(2):368-392.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data
with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learn- ing, ICML, '06, page 369-376, New York, NY, USA. Association for Computing Machinery.
Bo Li, Tara N. Sainath, Khe Chai Sim, Michiel Bac- chiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yonghui Wu, and Kanishka Rao. 2017. Multi- dialect speech recognition with single sequence-to- sequence model.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han. Shibo Wang. Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolutionaaugmentee trans- former for speech recognition.
Jialu Li. Vimal Manohar, Pooja Chitkara, Andros Tjan- dra, Michael Picheny, Frank Zhang, Xiaohui Zhang, and Yatharth Saraf. 2021a. Accent-robust automatic speech recognition using supervised and unsuper- vised way2vec embeddings.
Tao Han, Hantao Huang, Ziang Yang, and Wei Han. 2021. Supervised contrastive learning for accented speech recognition.
Song Li, Beibei Ouyang, Dexin Liao, Shipeng Xia, Lin Li, and Qingyang Hong. 202 End-to-end multi- accent speech recognition with unsupervised accent modelling. In ICASSP 2021 - 2021 IEEE Interna- tional Conference Acousties, Speech and Signal Processing (ICASSP), pages 6418-6422
Hu Hu, Xuesong Yang. Zeynab Racesy, Jinxi Guo, Gokce Keskin, Harish Arsikere, Ariya Rastrow, An- dreas Stolcke, and Roland Maas. 2020. Redat: Accent-invariant representation for end-to-end asr by domain adversarial training with relabeling.
Clara Meister, Tim Vieira, and Ryan Cotterell. 2020. Best-First Beam Search. Transactions of the Associ- ation for Computational Linguistics, 8:795-809,
Yan Huang, Dong Yu, Chaojun Liu, and Yifan Gong- 2014. Multi-accent deep neural network acoustic model with accent- accent-specific top layer using the kld- regularized model adaptation. In Fifteenth Annual Conference of the International Speech Communica- tion Association. Citeseor.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. volume 2. pages 1045-1048.
Jason Humphrics and Philip C. Woodland. 1997, Us- ing accent-specific pronunciation modelling for im- proved large vocabulary continuous speech recogni- tion. In EUROSPEECH.
Kanishka Rao and Haşim Sak. 2017. Multi-accent speech recognition with hierarchical grapheme based models. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 4815 4819.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira 2021. Perceiver: General perception with iterative atten- tion. In International conference on machine learn- ing, pages 4651 1651-4664. PMLR.
George Saon, Hagen Soltau, David Nahamoo, and Michael Picheny. 2013. Speaker adaptation of neural network acoustic models using i-vectors. 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 55-59. IEEE.
Abhinav Jain, Minali Upreti, and Preethi Jyothi. 2018. Improved accented speech recognition using accent embeddings and multi-task learning. In Interspeech, pages 2458-2458.
Sining Sun, Ching-Feng Yeh, Mci- Hwang, Mari Ostendorf, and Lei Xie. 2018. Domain adversarial training for accented speech recognition.

---

Mehmet Ali Tugtekin Turan, Emmanuel Vincent, and Denis Jouvet. 2020. Achieving multi-accen asr via unsupervised acoustic model adaptation. In INTER- SPEECH 2020.
Train
Dev
(4)
(7)
(5)
8.13 hrs
408,48 hrs
1.01 hrs
Vinit Unni, Nitish Joshi, and Preethi Jyothi. 2020. Cou- pled training of sequence-oo-squeene
(1)
models for accented speech recognition. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 8254 8258.
204.05 05
(3)
hrs
(2)
12. 12.32 hrs
14.01 hrs
(6)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
11.0
Test
Thibault Viglino, Petr Motlicek, and Milos Cernak. 2019. End-to-end accented speech recognition. In INTERSPEECH.
Figure Illustration of the transcript-wir wise overlap be- tween the train, test, and dev sets in terms of dura- tions. (1) represents the duration of the group of utter- ances whose transcripts are present in all three splits. (7) denotes duration of examples having transcripts found only in the train set.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson En- rique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. 2018. Espnet: End-to-end speech processing toolkit.
A Dataset Curation
Felix Weninger, Yang Sun, Junho Park, Daniel Willett, and Puming Zhan. 2019. Deep learning based man- darin accent identification for accent robust asr. In INTERSPEECH.
To build MCV_ACCENT-600, we group the exam- ples from MCV_ACCENT into seven buckets while preserving speaker disjointedness across the train, dev, and test sets.
Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin. Andrea Madotto, Peng Xu, and Pascale Fung. 2020. Learning fast adaptation on cross-accented speech recognition. arXiv preprint arXiv:2003.01901.
The buckets are visualized in Figure 5; bucket (4) refers to utterances that have exactly the same transcript but different speakers appearing across the train and dev sets. We wanted to include some transcript overlap across all combinations of train, test, and dev splits, since the model could learn accent information from samples with the same transcripts and different underlying accents. We note that majority of the dev and test samples are disjoint in both speakers and transcripts from the training set for a true evaluation that does not ben- efit from having seen the same transcripts during training.
Sanghyun Yoo, Inchul Song, and Y. Bengio. 2019. A highly adaptive acoustic model for accurate multi- dialect speech recognition. pages 3716-5720.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual transla- tion. In International Conference on Learning Rep- resentations.
Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Benukhare--Hudiainneee John Levis, and Ricardo Pattierre-Osunn 2018. L2-arctic: non-native english speech corpus. pages 2783-2787.
To create such split, we loop over all the ac- cents, and for every seen accent Qyeens we first filter out examples with transcripts that have been previously dealt with and then split the remaining unique transcripts from aseen into seven buckets. For every bucket 6, transcripts from b are further divided into n groups where n is the number of benefactors for that bucket. As example, for bucket (1) the value of is 3. Let wf be an utter- ance spoken by speaker s³, which is put into the train set by bucket (1). Then to maintain speaker disjointedness, we put all utterances spoken by into the train set. The transcripts of these utter-
Hao Zheng, Zhanlei Yang. Liwei Qiao, Jianping Li, and Wenju Liu. 2015. Attribute knowledge integration for speech recognition based on multi-task learning neural networks. In INTERSPEECH.
Wei Zhou, Haotian Wu, Jingjing Xu, Mohammad Zeineldeen, Christoph Lüscher, Ralf Schlüter, and Hermann Ney. 2023. Enhancing and adversarial: Im- prove with speaker labels. In ICASSP 2023 2023 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5.

---

Arraal
Ground Truih
Exprriment
Sentence
Bas
where i5
youfoda
Australia
your father
DAT
where is
youfeds
CA
where 15
your father
putting pool under this floor was great idea
Base
put it
pool Linto this floor was a ggreek of idee
DAT
put his pool (nderr] this floor was greatt of itea
CA
putting pool under this floor was great idea
Base
will you break for swimring
England
will you breakfast with ne
DAT
will you bright for study
CA
will you
breakfast with
Boor
elsa knittad futiously
Scotland
elsa knitted fur toriously
DAU
elsa knitted fudiously
CA
elsa knitted furtously
long weve seen each other
Base
long since seen each other
US
DAT
longsans would each other
CA
how long since
each other
this might then rricher
Africa
this made then even richer
DAT
this night then rrichard
this made then iricher
he won wor dowide
Base
he won world white potason In his special field
Hongkong
reputation in his spectall Field
DAT
he won world white repetition in his special field
CA
he won worldwide reputation in his special field
Boor
canible nadis music from policcy
can you play nineties music fron paul kelly
India
DAU
canuple ninety is music fron policy
CA
can play nineties misic fron pole kelly
this award is given In three different categories
Base
this award is given In [the dream different categories
Ireland
DAT
this award is given in the tree giffins categords
CA
this award is given ithrea gifting catagories
they just lenter ny nind just.
leyes accid
it just entered my wind at that
Malaysin
DAT
mind that moment eyes accid
Just entered ny nind
content delivery necessary
Base
content delivery mid luke and disincessary
Newzvaland
DAT
content delivery night blue and businecessary
content delivery nnetwork does necessary.
apple google and Facebook are often described as tech giants
appal google
face book often described the
Singapore
appalo google
(face back often described tack
facchoook often described [tech giants you meny layers ironea you are many layers [irony you
google
many layers
how many layers irny are you
Philippines
DAU
minros soldiers
nine rows soldiers
Wales
DAT
of soldiers stood line nine rows of soldiers stood 1ine
CA
Table 11: Comparison of the predictions from the Conformer (labeled as Base), DAT, and our proposed system on a few utterances. is used highlight words that are correctly predicted, whereas highlights predictions that are correct but contain minor mistakes such as unwanted spaces. Similarly. denotes incorrectly predicted words, whereas is used to indicate words that are incorrect but are somewhat closer to the underlying transcript such having similar prefixes.
ances are ignored while processing the remaining buckets and accents. For unseen accent Ounseen. all the examples with transcripts that are not yet processed are put into bucket (6).
B Additional Results
Table 12 shows word error rates (WERs) com- paring all experiments done using MCV -ACEET- 100 dataset. Similarly, Table 13 shows word ror rates comparing all experiments done using MCV_ACCENT-6NN dataset.
Comparison Predictions
The dev and test sets built this way contain around 68 and 130 hours of data. We further ran- domly sample 25% and 15% from these sets, re- spectively. To generate MCV MCV_ MCV_ACCENT-100 split, we randomly sample 14% from the MCV_ACCENT, 600 set.
Table 11 highlights few examples where our pro- posed system performs significantly better than the Conformer baseline and DAT systems.

---

Unseen
Dumal
See
Ceaen
AB8
CAN
UN
SET
168
AFR
HKG
MAAL
NW2
SGP
WLS
Transfommr
22.08
17.34
28.01
18.11
17.81
19.73
15.50
15.51
96
31.99
15.39
25.25
35.25
25.85
31.50
35.78
21.01
15.02
15.74
13.90
32.20
19.85
I Fvectee sm
18.57
14.15
23.56
13.88
15.02
16.07
14.02
13.31
21.06
27.18
29.55
21.18
31.72
19.33
27.22
33.81
17.98
Chnatlier
18.91
14.12
23.69
14.23
15.10
16.08
13.72
13.19
21.0
28.13
29.15
21:46
32.90
19.38
26.51
34.24
13.15
CA (P
18.33
13.76
22.89
13.05
14.90
15:33
10.92
15.12
20.52
26.43
29.29
20.70
31.55
18.56
36.30
33.30
16.58
CA =
13.15
20:95
66
30.87
18.47
25.81
CA 100)
15.24
32.44
18.75
32,93
CP 200)
18.41
13:09
23.12
13,00
1431
15.06
11.10
13.12
21.57
26.78
28.30
20,93
30,95
18.97
25.97
33.17
17.85
CAsen...o9(P 500)
18.39
12:04
14.9%
15.40
21:32
33.13
18.78
25.07
CA
15.24
17:87
CApe 500
18:31
13,86
12.75
13.14
1507
15.76
10.56
13.12
20,50
25.82
78.70
21.00
36.70
18.53
25.59
33.10
18.01
15:36
15.53
13:67
21.50
18.98
CAsem....a
18.68
CAzep....a P 500
18:30
13.65
12.05
12.34
15.19
14,89
11.64
13.06
20.77
25.57
48
20.98
31.88
1866
35.87
33.28
17.82
Table 12: Comparison of the performance(WER Numbers in bold denote the best across baselines, and the green highlighting denotes the best across all the experiments. Ties are broken using overall WER. CAI.E(.....) [P- k):: Codebook attend Cross Attention applied all layers from to j with l entries per accent codebook. CALE().....))) (Pmnd (Prnd k): Similar to the previous setup, but with codebooks frozen during training.
of all the experiments mentioned MCV MCV_ACCEN ACCENT- 100.
Agunpited
Method
Confommer (Basc)
9.75
6.84
13.46
4.95
6.81
7.35
4.42
5.64
11.51
16.30
14.90
12.61
20.43
10.49
15.74
21.26
7.9M
L-eetor
10:55
6.40
13.69
4.67
7.53
7.43
4.06
6.04
12.05
17:45
14.76
13.08
20.31
10.57
15.62
21.22
8.91
1002
13.20
5.30
7.59
5.56
12.19
16.30
5.34
11.76
16.19
14.62
4.67
5.90
11.63
15.64
K.29
CA 200)
9.59
6.20
12.98
4.92
7.47
6.83
3.52
5.91
11.47
15.96
13.87
12.25
19.30
9.98
15.17
20,90
8J08
CA (P 500)
9.55
6.19
1292
4.40
7.60
6.67
2,62
5.99
11.57
15.18
14.13
12.20
17.54
10.00
15.34
20.71
6.39
Table 13: Comparison of the performance( WER % of the experiments mentioned for MCV MCV_ACCENT-600, We
follow the same notation as in Table 12.


Reviews:
[{"review_id": "kP8pjo7fIK", "reviewer": "Reviewer_tMje", "summary": "", "questions": "1. Could you give more details on the model architectures, size, computation for those reported in Table 3, 4, 5?\n2. The codebook construction is quite simple, the fixed number of code entries may not be optimal as the accent data is imbalance. Could you comment on this?\n3. Could you give some statistics on the number distinct accents through several time steps of the modified beam search?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and effort in reviewing our paper and providing us with useful insights. Below, we address all the questions and concerns raised in your review.\n\n---\n\n> **Details on the model architectures, size, and computation for those reported in Table 3, 4, 5.**\n\nTo eliminate the possibility that the improvement in our proposed model is solely due to an increased number of parameters, we conduct baseline experiments in two different ways to make it commensurate with the number of parameters in our proposed model.\n\n`Please note: We use the same model specifications for all the experiments mentioned in Tables 3, 4, and 5. Below, we report numbers only for the setting used in Table 3.`\n\n1. **Baseline**: Baseline conformer model reported in Table 3 (referred to as *Conf.*).\n2. **M1**: Baseline conformer model with increased encoder units (from $2048 \\rightarrow 2320$).\n3. **M2**: Baseline conformer model with increased attention dimension (from $256 \\rightarrow 272$).\n4. **Ours**: Our proposed cross-attention model reported in Table 3 (referred to as *CA*).\n\n| Model Name | Number of Parameters | Overall WER | Seen WER | Unseen WER |\n| :--- | :---: | :---: | :---: | :---: |\n| Baseline | 43M | 18.87 | 14.05 | 23.67 |\n| M1 | 46M | 18.89 | 14.02 | 23.74 |\n| M2 | 46M | 18.77 | 14.02 | 23.51 |\n| Ours |46M | **18.22** | **13.57** | **22.86** |\n  \nEven with an increase in parameters for the baseline models, we still see a **statistically significant improvement** (at $p<0.001$) with our system compared to M2.\n\n---  \n\n> **Fixed number of code entries may not be optimal as the accent data is imbalanced.**\n\nTo check the effectiveness of our approach on a balanced dataset, we created a 100-hour accent balanced data split. The results are as follows:\n\n| Model Name | Overall WER | Seen WER | Unseen WER |\n| :--- | :---: | :---: | :---: |\n| Baseline | 19.30 | 14.73 | 23.86 |\n| Ours | **18.88** | **14.61** | **23.13** |\n\nEven on the balanced dataset, our architecture shows a **statistically significant improvement** (at $p = 0.005$) in comparison to baseline.\n\nIn this work, we use a fixed number of codebooks yielding significant WER reductions. For accent-imbalanced datasets, using a variable number of codebooks per accent might indeed achieve better results. This is a non-trivial modification to the codebook architecture that we leave for future work.\n\n---\n\n> **Statistics on the number of distinct accents through several time steps of the modified beam search.**\n\nPlease refer to [these plots](https://emnlp2023.notion.site/Plots-5db415a2e2f14776b2bb10f064f69a84?pvs=4). Across decoding time-steps, we show 1) the entropy of the probability distribution across seen accents averaged across all test samples and 2) the probability of a seen accented sample appearing in the beam for a single test example. From the entropy plot, it is clear that at least four or all five seen accents are active until time-step 20 after which certain accents gain more prominence. The probability plot shows how (nearly) all accents are in the running until time-step 20, after which it slowly converges to one or two accents.\n\n---\n\n> **In Figure 2, for seen accents, the off-diagonal values are quite close to the diagonal ones, hinting at strong correlations among the learnt accented codebooks.**\n\nThe perceived correlation among the learned codebooks can be attributed to high correlation between certain seen accents. Primarily, as mentioned in our analysis in Section 6, the US and Canada accents in MCV are perceptually very similar, resulting in our model picking Canada's codebooks for US examples and vice-versa. This shared preference between US and Canada leads to certain off-diagonal seen accents’ codebooks appearing to be close to the diagonal one (e.g., Scotland for Canada). This perceived correlation between off-diagonal and diagonal entries would not appear if the choice of codebooks across US and Canada accented utterances skewed towards either US or Canada, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Based on the experimental results, the proposed approach is quite effective for both seen and unseen accents.\n- The paper is generally well-written, with comprehensive ablation studies. The results from Table 7 is interesting, as it potentially highlights the correlation between accents and acoustic features.", "reasons_to_reject": "- The performance reported in Table 3,4 and 5 may not be entirely fair, as the model size, computation complexity and memory usage are not taken into consideration. It could be argued that the better performance is due to larger model size, more computational intensive, etc. \n- The modified beam search algorithm (Algorithm 1) is not guaranteed to search through all the seen accent codebooks, several initial frames with high scores may lead to premature pruning of all low-score accents. And this would behave like using a classifier to select the accent codebook.  \n- In Figure 2, for seen accents, the off-diagonal values are quite close to the diagonal ones, which may hint that there are strong correlations among the learnt accented codebooks.", "questions_to_authors": "1. Could you give more details on the model architectures, size, computation for those reported in Table 3, 4, 5?\n2. The codebook construction is quite simple, the fixed number of code entries may not be optimal as the accent data is imbalance. Could you comment on this?\n3. Could you give some statistics on the number distinct accents through several time steps of the modified beam search?", "paper_topic_and_main_contributions": "- A technique to improve the robustness of ASR for accented speech, with  learnable accented codebooks using cross-attention mechanism.\n- A modified beam search algorithm, which allocate  different beam widths to different seen accents, based on the hypothesis scores.\n- The potential release of train/dev/test splits for multi-accents ASR from common voice corpus.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4WMglEXe31", "reviewer": "Reviewer_ndKt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper. We appreciate your valuable time and effort in reviewing our work. We are happy to provide any additional experiments or clarifications, which might further strengthen our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, the proposed approach is new for accent-aware ASR and the experiments are comprehensive.", "reasons_to_reject": "none", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a method to improve speech recognition accuracy for various accents by introducing accent-specific codebooks which are trained based on known accents. The codebook information is provided to the ASR encoder via cross-attention. During recognition, the authors propose a joint beam search algorithm which iterates over all possible accents. The experimental results indicate small, but statistically significant improvements for seen and unseen accents. Several ablation studies augment the paper.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0HDUQxiw7b", "reviewer": "Reviewer_78aE", "summary": "", "questions": "- [A] How come this method is doing so well for unseen accents? More discussion on this in the paper would be nice to see.\n- [B] Hong Kong, Indian accents should prefer England accent due to historical (and also geographical reasons to some extent) reasons. However, they prefer US. What could be the reason for this?\n- [C] Randomly initialised non-learnable code-books experiments is intriguing. This makes me wonder whether the codebooks are used in the way that we think the model is using or if there is more to it. Would like the authors to comment more about why such a system works.\n- [D] Could the authors add I-vector based system and MTL system results in Table 5 (on 600 hour common voice) and also in Table 4 (for zero-shot setting)? I suspect the MTL/I-vector model to do well when trained on 600 hour dataset. One of them could be the best performing model in this setting.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and effort in reviewing our paper and providing us with constructive feedback. Below, we address all the questions you raised in your review.\n\n---\n\n> **How come this method is doing so well for unseen accents? More discussion on this in the paper would be nice to see.**\n\nFor test utterances with unseen accents, our model is designed to choose (seen) accent codebooks that best fit the underlying (unseen) accent. It is somewhat analogous to how humans use familiar accents to tackle unfamiliar ones (supported by Section 5.4 of the book \"[Essentials of Linguistics](https://ecampusontario.pressbooks.pub/essentialsoflinguistics/chapter/5-3-attitudes-about-accents/)\" by Catherine Anderson and in the paper \"[Processing of unfamiliar accents in monolingual and bilingual children](https://www.cambridge.org/core/journals/journal-of-child-language/article/processing-of-unfamiliar-accents-in-monolingual-and-bilingual-children-effects-of-type-and-amount-of-accent-experience/7B22B1E7C59E024EEBC4025A6E424B29)\" published in the [Journal of Child Language](https://www.cambridge.org/core/journals/journal-of-child-language) by Helena et al). During inference, our model searches through seen accent codebooks and chooses entries that are most like the unseen accents in the test instances.\n\n---\n\n> **Hong Kong, and Indian accents should prefer England accents due to historical (and also geographical reasons to some extent) reasons. However, they prefer US. What could be the reason for this?**\n\nFrom listening to a sample of US accented utterances in the MCV dataset, we find them to be very heterogeneous including both native speakers of English and speakers of English originally from India, China, etc. with native fluency but traces of non-native accents appearing in their speech. This could be one reason for the Indian, Hong Kong accented samples preferring US codebooks.\n\nYou can listen to a few examples from this [anonymized website](https://emnlp2023.notion.site/emnlp2023/Presence-of-heterogenity-in-the-US-split-of-the-Mozilla-Commonvoice-Dataset-1-9147652c526d4baa8c153f6927d9f384).\n\n---\n\n> **Randomly initialized non-learnable code-books experiments are intriguing. This makes me wonder whether the codebooks are used in the way that we think the model is using or if there is more to it. Would like the authors to comment more about why such a system works.**\n\nCodebooks could be made learnable to encode additional information about the underlying accents, or the codebooks could be randomly initialized and frozen during training. The model will learn to adapt to the frozen codebook entries and pick suitable entries specific to different accents. A similar scheme involving randomly initialized frozen codebook entries was previously found to be beneficial for ASR in the BEST-RQ self-supervised algorithm for speech recognition proposed by Chiu et al., “Self-Supervised Learning with Random-Projection Quantizer for Speech Recognition”, ICML 2022.\n\n---\n\n> **Could the authors add I-vector based system and MTL system results in Table 5 (on 600 hour common voice) and also in Table 4 (for zero-shot setting)? I suspect the MTL/I-vector model to do well when trained on 600-hour dataset. One of them could be the best-performing model in this setting.**\n\nBelow we report results using the i-vector and MTL systems in the zero shot setting (Table 4):\n\n| Model Name | Overall | Arabic | Hindi | Korean | Mandarin | Spanish | Vietnamese |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Conformer | 33.3 | 30.4 | 30.4 | 26.9 | 37.9 | 30.3 | 43.5 |\n| DAT | 33.5 | 30.7 | 30.8 | 26.8 | 38.3 | 30.1 | 43.9 |\n| i-vector | 33.6 | 31.0 | 31.2 | 27.2 | 38.0 | 30.4 | 43.9 |\n| MTL | 33.4 | 30.4 | 30.6 | 26.9 | 38.7 | 30.1 | 43.7 |\n| CA | **32.6** | **29.5** | **30.4** | **26.2** | **37.1** | **29.3** | **42.8** |\n\nWe observe that both i-vector and MTL underperform compared to our model CA.\n\nThe i-vector and MTL-based systems for the 600-hour dataset are still running and could not be completed within the rebuttal window. We will share these results during the discussion phase. We would additionally like to present more numbers with our model in the 600-hour setting using larger numbers of codebooks that leads to further reductions in WER.\n\n| Model Name | Overall WER | Seen WER | Unseen WER |\n| :--- | :---: | :---: | :---: |\n| Baseline | 9.75 | **6.04** | 13.46 |\n| DAT | 9.73 | 6.12 | 13.33 |\n| $\\text{CA}_{L \\in (1,2,\\ldots 12)} (P=50)$ | 9.63 | 6.22 | 13.03 |\n| $\\text{CA}_{L \\in (1,2,\\ldots 12)} (P=200)$ | 9.59 | 6.20 | 12.98 |\n| $\\text{CA}_{L \\in (1,2,\\ldots 12)} (P=500)$ | **9.55** | 6.19 | **12.92** |  \n\n> **How does the model perform when the dataset is accent-balanced?**\n\nTo check the effectiveness of our approach on a balanced dataset, we created a 100-hour accent balanced data split. The results are as follows:\n\n| Model Name | Overall WER | Seen WER | Unseen WER |\n| :--- | :---: | :---: | :---: |\n| Baseline | 19.30 | 14.73 | 23.86 |\n| Ours | **18.88** | **14.61** | **23.13** |\n\n\nEven on the balanced dataset, our architecture shows a **statistically significant improvement** (at $p = 0.005$) compared to the baseline.\n\n---\n\n> **The codebooks size may explode if we (eventually) want to model 100s of accents**\n\nIndeed, this is a limitation with the current proposal that we plan to address in future work. One direction we are currently exploring is introducing codebooks with shared/tied entries (analogous to tied-HMMs). Enabling such codebook tying will substantially reduce the number of codebook entries and avoid codebook explosion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear motivation and a well written paper. \n- Really enjoyed the way the authors grouped the works into accent aware and accent-agnostic systems, thereby covering a wide-range of papers!\n- The proposed system is interesting -- codebooks has not been applied to accented E2E ASR and is very suitable for the task. Joint beam search proposed by authors is a neat trick!\n-  The work was systematically executed with experiments with a wide-range of hyperparameters and accents.\n- The system does exceedingly well in a zero-shot setting!", "reasons_to_reject": "- Improvements due to this architecture start to go down as the data increases even to the point where other methods (just Conformer baseline!) start doing well on seen accents.\n- It is unclear how the model would perform when the dataset is accent-balanced.\n- The codebooks size may explode if we (eventually) want to model 100s of accents (for example, consider Africa -- people of Africa alone speak in > 200 accents), if we dedicate one codebook per accent.", "questions_to_authors": "- [A] How come this method is doing so well for unseen accents? More discussion on this in the paper would be nice to see.\n- [B] Hong Kong, Indian accents should prefer England accent due to historical (and also geographical reasons to some extent) reasons. However, they prefer US. What could be the reason for this?\n- [C] Randomly initialised non-learnable code-books experiments is intriguing. This makes me wonder whether the codebooks are used in the way that we think the model is using or if there is more to it. Would like the authors to comment more about why such a system works.\n- [D] Could the authors add I-vector based system and MTL system results in Table 5 (on 600 hour common voice) and also in Table 4 (for zero-shot setting)? I suspect the MTL/I-vector model to do well when trained on 600 hour dataset. One of them could be the best performing model in this setting.", "paper_topic_and_main_contributions": "Existing ASR systems are vulnerable to accent variation, especially so for those accents that are underrepresented in training datasets (e.g., African accented English). This work proposed a system that uses learnable accent codebooks + joint beam search (searches over all accents simultaneously) to solve this issue. The proposed solution outperforms strong baselines (upto 37% in relative WER) on a standard benchmark, i.e, common voice.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 307, Figure 2 is wrongly referenced?"}, "extra_text": {"Missing_References": "There is a rich body of work done during the traditional ASR times on accented speech. Would be nice if the authors cite those as well. Here are some such papers:\n\n- \"German regional variants - a problem for automatic speech recognition?\" by N. Beringer (1998)\n- \"Towards acoustic model unification across dialects\" by M. Elfeky (2016)\n- \"Multi-accent speech recognition with hierarchical grapheme based models\" by K. Rao (2017)\n\nThe authors also missed an influential accented E2E paper: \n\n- \"Multi-dialect speech recognition with a single sequence-to-sequence model\" by B. Li (2018). A similar system would have also been a strong baseline for the system proposed, I feel. "}}]

Author rebuttal:

Meta-review: I concur with the other reviewers and believe this paper's idea is valuable; I recommend acceptance.
Decision: Accept-Main