Automatic Pronunciation Assessment - A Review
Authors: Yassine El Kheir
Ahmed Ali
Shammur Absar Chowdhury
Venue: EMNLP 2023 Findings (2023)
Keywords: computer aided pronunciation training (CAPT)
pronunciation assessment
second language learning
pronunciation error detection
TL;DR: Review about spoken pronunciation assessment approaches

Abstract: Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning over the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight existing limitations, and available resources. This is followed by a discussion of the remaining challenges and possible directions for future work.

Automatic Pronunciation Assessment A Review
Yassine El Kheir, Ahmed Ali and Shammur bsar Chowdhury
Qatar Computing Research Institute, HBKU, Qatar shchowdhury @ hbbu..dd.qq
Abstract
into text regardless of pronunciation mistakes. Ro- bust ASR should perform well with all variation including dialects and non-native speakers. How- ever, MDD should mark phonetic variations from the learner, which may sometimes be subtle differ- ences (Li et al., 20)
Pronunciation assessment and its application in computer-aided pronunciation training (CAPT) have seen impressive progress in recent years. With the rapid growth in language processing and deep learning the past few years, there is a need for an updated review. In this paper, we review methods employed in pronunciation assessment for both phonemic and prosodic. We categorize the main challenges observed in prominent research trends, and highlight exist- ing limitations, and available resources. This followed by a discussion the remaining challenges and possible directions for future work.
Training Resources Recent success in deep learn- ing methods emphasized the need for in-domain training data. Language learners can be divided into two groups: adult secondary (L2) language learners and children language learners the for- mer depends on whether to build system that is na- tive language dependant (L1). At the same time, the latter identifies the need for children's 's voice, which is challenging corpus build (Council III et al., 2019; Venkutasubramaniin al., 2023), even the accuracy for ASR for children is still behind com- pared to adult ASR (Liao et al., 2015). The scarcity and imbalanced distribution of negative mispro- nunciation classes pose significant challenge in training data.
1 Introduction
Computer-aided Pronunciation Training (CAPT) technologies are pivotal in promoting self-directed language learning, offering constant, and tailored feedback for secondary language learners. The ris- ing demand for foreign language learning, with the tide of globalization, fuels the increment in the de- velopment of CAPT systems. This surge has led to extensive research and development efforts in the field (Neri al., 2008; Kang et al., 2018 &; Rogerson- Revell, 2021). CAPT systems have two main us- ages: (i) pronunciation assessment, where the sys- tem is concerned with the errors the speech seg- ment; (ii) pronunciation teaching, where the sys- tem is concerned with correcting and guiding the learner to fix mistakes in their pronunciation.
Evaluation There no clear definition of right or wrong in pronunciation, instead an entire scale from unintelligible to native-sounding speech (Witt, 2012). Given that error in pronunciation is difficult to quantify, it can be split into (a) Objective evalu- ations (i): phonetic segmental; (ūi): prosodic or supra-segmental; and (iii) place or articulation, manner of speech or sub-segmental; (b) Subjective evaluations; in many cases measured through lis- tening tasks followed by human judgment, and can be split into three main classes: (i) intelligibility; (ii) comprehensibility and (iil) accentedness (or lin- guistic native- likeness).eess See Figure I for common pronunciation assessment factors.
This paper addresses the former focusing on pronunciation assessment, which aims to automat- ically score non-native speech-segment and give meaningful feedback. To build such robust pro- nunciation assessment system, the following design aspects should be addressed.
Several studies have summarized advances in pronunciation error detection (Eskenazi, 1999. 2009; Witt, 2012; Li al., 201 Chen and Li, 2016; Zhang et al., 2020; Caro Anzola and Men- doza Moreno, 2023). Eskenazi (1999) investigated
Modelling Mispronunciation detection and diag- nosis (MDD), in many cases, are more challenging to model compared to the vanilla automatic speech recognition (ASR) system, which converts speech

---

Pronunciation Assessment
Phonetic Error
Prosodic Error
Accuracy
Fluency
Completeness
Insertion
Subsitution
Stress
Rythm
Intonation
Word
Sentence
Strese-
Syliable-
Lexical
Figure I: Types of Pronunciation Errors for Assessment
the potentials and limitations of ASR for L.2 pro- nunciation assessment, showcasing its practical im- plementation using an interface developed at CMU. Furthermore, the study reports different automatic scoring techniques, emphasizing modalities of in- teraction, associated algorithms, and the challenges. Witt (2012) presented overview of pronuncia- tion error detection, encompassing various scoring methodologies and assessing commercial CAPT systems. Chen and Li (2016) provided research summary, focusing on phoneme errors and prosodic error detection. More recently, Zhang et al. (2020) provided summary of two automatic scoring ap- proaches (a) ASR-based scoring to calculate confi- dence measures; and (b) acoustic phonetics scoring focusing on comparing classifying phonetic seg- ments using various acoustic features.
to other language skills, learning pronunciation is difficult. Yet, for learners, mastering L2 pronun- ciation is most crucial for better communication. Historically, pronunciation errors (mispronuncia- tions) are characterized by phonetic (segmental) errors and prosodic (supra-segmental) errors (Witt, 2012; Chen and Li, 2016), represented in Fig- 1. This characterization provides some clear distinctions for pronunciation assessment.
2.1 Pronunciation Errors
Phonetic Errors
Phonetic (segmental) errors involve the production of individual sounds, such vowels, and conso- nants, and it includes three errors: insertion, dele- tion, and substitution. This can be attributed to sev- eral factors, including negative language transfer. incorrect letter-to-sed tound conversion, and misread- ing of text prompts (Meng et al., 2007b; Qian et al., 2010; Kartushina and Frauenfelder, 2014; Li et al., 201 6a). For example, Arabic L1 speakers may find it difficult to differentiate between /p/ and /b/ as the phoneme /p/ is non-existent in Arabic, verbs like /park/ and /bark/ might sound similar Arabic LI speakers. Similarly, in Spanish, there are no short vowels, so words like /eat/ and /it/ might sound similar to Spanish LI speakers.
With large transformer-basse pre-trained models gaining popularity, re-visiting the existing litera- ture and presenting comprehensive study of the field is timely. We provide overview of tech- niques adapted for detecting mispronunciation in (a) segmental space, (b) assessing pronunciation with supra-segmal measures, along with (c) dif- ferent data seceraation/aummentatiii
approaches. Unlike previous overview studies, we also cover a handful of (d) qualitative studies bringing to- gether the notion of intelligibility, comprehensive- ness, and accentedness. We note the resources and evaluation measures available the speech com- munity and discuss the main challenges observed within prominent research trends, shedding light on existing limitations. Additionally, we also explore potential directions for future work.
Prosodic Errors
Prosodic features encompass elements that influ- ence the pronunciation an entire word or sen- tence, including stress, rhythm, and intonation. Er- rors related to prosodic features involve the pro- duction of larger sound units. For intelligibility, prosodic features particularly play significant role (Raux and Kawahara, 2002). This is especially true for tonal languages (Dahmen al., 2023) where variation in the pitch can lead words with differ- ent meanings. Prosodic enrors are often language- dependent and categorized by: stress (lexical and sentence), rhythm, and intonation.
2 Nuances of Pronunciation
Pronunciation can be defined as "the way in which word or letter is said, or said correctly, or the way in which language is spoken" Compared
'nttps://dictiin cambridge. ceanbrigge.og/ddcctiiooooo org//ictionary/ englissh/proounciaatinn, Accessed: 2023-01--2

---

Corpus
I Languages
Native Language (1.1)
Dur/Unt
#Speakers
Reported SOTA Results/ / Relevant Stud-
(1.2)
ies
ISLE (Menzel et al., 2000) +
English
German and Italian
18/
46
# (HoseeHisiiii-KKiie et al., 2021).
Accent PCC: 68% (Rasipuram et al., 2015)
FRJ (Minematiu al., English 2004)
Japanese
/68,000
200
Utterance PCC (Luan et al., 2012).
Word Intelligibility (Minematsu al., 2011), 1). Phoneme Erros (Ito et al., 2005)
CU-CHLOF (Meng et al., 2007a)
English
Cantonese and Mandarin
34.6/18,18
210
Phoneme F1 P1-measure: 80.98% (Wu
et al., 2021)
EURONOUNCE (Cyl- Polish wik al., 2009)
German
/721
# Utterance rythm (Wagner, 20144)
iCALL (Chen et al., Mundarin 2015)
24 countrics
142/90,841
305
FAR: 8.65%, FRR: 3.09%: (Li et ul., 2017). Tone Recognition: (Tong et al., 2015)
SingaKids-Mandart Mundarin
Singuporeun (English)
125/79,843
255
PER: 28.51%% Tone Recognition (Tong
+
et al., 2017)
SHEFCE (Ng et al,, English, Can- English, Cantonese
25/
31
Madarin syllabe error rate: 17.3%, En- glish PER: 34.5% (Ng et al., 2017a)
2017b) +
tonese
VoiSTUTOR (Yarra English et al., 2019; Pul et 2022)
Kannada, Malayalam, Tel-
14/26,529
16
Word Intelligibiliiy Accuracy: 96.58% (Anand et al., 2023)
ugu, Tamil, Hindi and Gu-
jarati
EpaDB (Vidal et al., 2019a)*
English
Spanish
/3,200
50
(Sancinetti et al., 2022) reported Min- Cost per phoneme
SELL-CORPUS (Chen English et al, 2019)
Chinese
31.6//
389
Fl-score Accent Detection: Word-level 35%, Sentence-level 45% (Kyriakopou- los al., 2020)
L2-ARCTIC (Zhao
English
Hindi, Korean, Mandarin.
3.6/
24
Fi-score: 2022a)
63.04% (Lin and Wang.
et al., 2018a) +
Spanish, and Arahic
Speechocean762 (Zhang al., 2021b)
English
Chinese
15,000
250
Phone PCC: 65.60% (Chao et al., 2022). 2022)) Word Accuracy PCC: 59 59.80% (Chao et al., 2022). Word Stress PCC: 32.30%
(Do al., 2023), Sentence total score PCC: 79.60% (Chao et al., 2022)
LATIC (ZHANG, Mandarin
Russian, Korean, French, and Arabic
4/2,579
Sentence Accuracy PCC: 69.80% (1.in and Wang. 2023b) Fl-score 70.53% (Algabri al., 2022)
2021) +
Arabic-CAPT (Algabri Arabic et al,, 2022)
India, Pakistan, Indonesia.
2.3/1.611
62
Nepal,
Afghanistan, Nigeria.
Bangladesh,
Uganda
Ara AraVoicel.. 2 (EI. Kheir Arahic ct al., 2023b)
Turkey,
Nigeria,
5.5/7,062
11
F1 F1-score 60.00% (Hl. Kheir et al., 2023b)
Bangladesh,
Indone-
sia, Malaysia
Table I: Widely used datasets. represent publicly available dataset, + is available on request, # relevant study, Dur: total duration in hours, Utt: total number of utterances, SOTA: is the notable reported state-of-the-ee for each corpus. FAR: false acceptance rate, FRR: false rejection rate, PCC: pearson correlation coefficient with human scores
Stress is the emphasis placed on certain sylla- bles in word or sentence. It is articulated by increasing the loudness, duration, and pitch of the stressed syllable. It can be categorized as lexical stress, if the stress is placed on syllables within the word, or sentence stress if the stress placed on words within sentences. Mandarin learners of English have contrastive stress at the word-level that is absent in Korean, Mandarin speakers can have an advantage over Korean speakers in stress
processing of English words (Wang, 2022).
Rythm the pattern of stressed and unstressed syllables in a word or sentence. A language can be classified as either stress-timed or syllable-timed (Ohata, 2004; Matthews, 2014). In stress-timed languages, the duration of stressed syllables tends to dominate the overall time required to complete a sentence. Conversely, in syllable-timed languages, each syllable receives an equal amount of time during production.

---

Intonation refers to the melodic pattern and pitch variations in speech. L2 learners of Vietnamese and Mandarin Chinese encounter significant difficulty in acquiring distinet tones, particularly if their na- tive language lacks tonality. Such tonal languages rely on different pitch patterns to convey distinct meanings, making it challenging for learners to ac- curately grasp and reproduce these tonal variations (Nguyen al., 2014; Chen et al., 2015).
3 Datasets
Obtaining datasets for pronunciation assessment is often challenging and expensive. Most of the available research work focused on private data, leaving only a handful of publicly accessible data to the rescarch community. Table provides an overview of available datasets, indicating English as a popular choice for the target language. Within this handful of datasets, few datasets include phonetic/segmental-ieee transcription and even fewer provide manually rated word and sentence- level prosodic features, fluency along with overall proficiency scores offering insights to learner's L2 speech intelligibility and comprehensiveness (Ar- vaniti and Baltazani, 2000; ?: Cole et al., 2017; Zhang et al., 2021b). More details on datasets and annotation are in Appendix A and B respectively.
2.2 Pronunciation Constructs
The motivation behind mastering L2 pronunciation is to communicate properly in the target language. Most of the time, these successes measured us- ing three pronunciation constructs (Uchihara, 2022) Intelligibiility Comprehensibilltyy and Accent- edness. These are perceived measures, that are partially independent with overlapping features. Intelligibility can be defined using the accuracy of the sound, word, and utterance itself along with utterance-level completeness (Abercrombie, 1949; Gooch al., 2016). Accuracy refers where the learner pronounces each phoneme, or word in the utterance correctly. In contrast, completeness mea- sures the percentage of words pronounced com- pared to the total number of words.
Research Avenues
In this section, we will delve into diverse ap- proaches, old, revised, and current methodologies used for pronunciation modeling of both segmen- tal and supra-segmental features, as illustrated in Figure 2 and Figure 3.
4.1 Classification based on Acoustic Phonetics
Comprehensibility, on the other hand, is de- fined based on the perceived ease or difficulty that listeners experience when understanding L2 speech. Fluency, defined by the smoothness of pronunci- ation and correct usage of pauses (Zhang et al., 2021b), is observed to be one of the key factors that determine the level of comprchensibility, along knowledge and discourse-level organization (Trofimovich and Isaacs, 2012: Saito et al., 2016).
Classifier-based approaches explored both segmen- tal and prosodic aspects of pronunciation. Seg- mental approaches involve the use of classifiers targeting specific phoneme pair errors, utilizing different acoustic features such as Mel-frequency cepstral coefficients (MFCCs) along with its first and second derivative, energy, zero-cross, and spec- tral features ( (Van Doremalen et al., 2009; Huang et al., 2020), with different techniques such as Lin- ear Discriminant Analysis (LDA) (Truong et al., 2004; Strik et al., 2009), decision trees (Strik al., 2009). Prosodic approaches focus on detecting lexical stress and tones, utilizing features such as energy, pitch, duration, and spectral characteris- tics, with classifiers like Gaussian mixture models (GMMs) (Ferrer et al., 2015), support vector ma- chines (SVMs) (Chen and Wang, 2010; Shahin et al., 2016), and deep neural network (DNNs) (Shahin et al., 2016), and multi-distribution DNNs (Li al., 2018a).
with good linguistic- tinguistic-koowldgge
Among the three constructs, accentedness. which is defined as "listeners' perceptions of the degree which I.2 speech is influenced by their native language and/or colored by other non-native features" (Saito al., 2016). It is often confused with both comprehensibility and intelligibility, in- fluencing pronunciation assessment. The accent is an inherent trait that defines a person's identity and is one of the first things that listener notices. It is often observed that most of the unintelligi- ble speech is identified as highly accented whereas highly accented speech is not always unintelligi- ble (Derwing and Munro, 1997; Kang et al., 2018; Munro and Derwing, 1995). Thus accents compli- cate fine-graiee -nn-ggained pronunciation assessment as is harder to pinpoint ssppae--eeemettleeeeeel error.
4.2 Extended Recognition Network (ERN)
ERNs are neural networks used in automatic speech recognition to capture broader contextual informa- tion, they leverage enhanced lexicons in combina- tion with ASR systems. They cover canonical tran-

---

scriptions well error patterns, enabling the detection of mispronunciations beyond standard transcriptions (Meng et al., 2007b; Ronen et al., 1997; Qian et al., 2010; Li et al., 2016b). However, ERNs often depend on experts or hand-crafted er- ror patterns, which are typically derived from non- native speech transcriptions et al., 2010) which makes it language-dependen approach and may limit their generalizability when dealing with unknown languages.
necessitating of speaker's L1 knowledge. Further- more, it offers the advantage of being computation- ally efficient to calculate. However, these scores lack context-aware information that is crucial for accurate pronunciation analysis. To overcome this, Sudhakara et al. (2019) presented a context-aware GOP formulation by adding phoneme state tran- sition probabilities (STP) extracted from HMM model to the GOP score calculation. Furthermore, Shi al. (2020) proposed a context-dependent GOP, incorporating a phoneme duration factor og and phonemes transition factor т. The formulated GOP score combines all the contextual scores as illustrated in Equation 3.
illustrated in (Lo
4.3 Likelihood-basee Scoring and GOP
The initial likelihood-basee MD algorithms aim to detect errors at the phoneme level using pre-trained HMM- MMM-GMM ASR models. Notably, Kim et al. (1997) introduced a set of three HMM-based scores, including likelihood scores, log posterior scores, and sigment-duration-aal
E. = - - p(a)O) log(p(g(O))
scores. Among these three, the log- based posterior scores are widely adopted due to their high correlation with human scores, and are also used to calculate the popular "goodness of pronunciation' (GOP) measure. The GMM-HMM based GOP scores can be defined by the Equation 1.
-(p)
Σ Ee _ Ey
(og(p((())
(3)
COP(p) a.) T())
For sentence accuracy evaluation, common approach is calculate the average GOP scores across phonemes (Kim al., 1997; Sudhakara et al., 2019). However, relying solely on averaging GOP scores at the phoneme level is limited. A re- cent approach in (Sheoran et al., 2023) proposed a combination of phone feature score and audio pitch comparison using dynamic time warping (DTW) with an ideal pronounced speech, score to sess prosodic, fluency, completeness, and accuracy at the sentence level. Inspired by GOP, Tong et al. (2015) proposed Goodness of Tone (GOT) based on posterior probabilities of tonal phones. While efforts have been made to improve the GOP formulation, it is important to acknowledge that the GOP score still has limitations, specifically in its ability to identify specific types mispronun- ciation errors (delction, insertion, or substitution), and also demonstrates a degree of dependency on the language of the acoustic model.
p(0pp) P(p) GOP()) = P()]O) = Σ., p(O)g) P(g)
(I)
denotes a sequence acoustic features, stands for the target phone, and Q represents the set of phones.
These scores are further improved using forced aligmment framework (Kawai and Hirose, 1998). More details are presented in Witt and Young (2000).
4.4 Reformulations of GOP
To enhance further the effectiveness of GOP scor- ing, (Zhang et al., 2008) are first to propose a log- posterior normalized GOP defined as:
GOP-(p) p((ol)p)))) P(p)
p(ollp) max, p(oelg)
(2)
4.5 End-to-End Modeling
Building upon this, Wang and Lee (2012) adopted the GOP formulation and incorporate error pattern detectors for phoneme mispronunciation diagnosis tasks. With the emergence of DNN in the field of ASR, Hu et al. (2013, 2015 bl)bb) demonstrated that using DNN-HMM ASR for GOP yields im- proved correlation scores surpassing GMM-HMM based GOP. The GOP and its reformulation repre- sent a significant milestone. It leverages pre-trained acoustic models on the target language without the
In the new era of DNNs and Transformers, there is significant exploration by researchers in lever- aging the power of these models and training end- to-end pronunciation systems. Li et al. (2017) in- troduced LSTM mispronunciation detector lever- aging phone-level posteriors, time boundary in- formation, and posterior extracted from trained DNNs models on the classification phonetic at- tributes (place, manner, aspiration, and voicing). contrast, Kyriakopoulos et al. (2018) introduced

---

Multi-Vicw Wuulti Task MDD 01-033%
Transfer Learning SSL models 54-61%
Mutti Mutti-Lingua MCD 57-80%
Performancs FT
Data Augmentation (49-56%]
End-to-End Canfermer RNNk [46 98%]
GOP HIVM-DON 45%)
GOP -MM
42%
1990
2012
2020
Oct 2022
March 2023
Figure 2: Overview of the performance different phonetic pronunciation detection models on L.2-ARCTIC
al., 2023a). A subsequent improvement PEP- PANET uses knowledge about phone-level artic- ulation traits with a graph convolutional network (GCN) to obtain more discriminative phonetic em- beddings (Yan et al., 2023b). Recently, Zhang et al. (2023) proposed recurrent neural network trans- ducer RNN-T for L2 phoneme sequence prediction along with extended phoneme set and weakly supervised training strategy differentiate similar- sounding phonemes from different languages.
Performanee PCC Flunay
MulsPA nanweerbe cuas 78-79%
GOP Transformer
and Prosody
feeluning
75-82%
Row Spoech
65 20%
2021
2022
2023
Several approaches have also been proposed for supra-segmental features scoring. Yu et al. (2015), proposed
Figure 3: Overview of the performance of fluency and prosody assessment models on Speechocean762
approach where traditional time-aggregated features are replaced with time- sequence features, such as pitch, to preserve more information without requiring manual feature engi- neering, a BiLSTM model proposed for fluency predictions. Tao et al. (2016); Chen et al. (2018), 8), studied different DNNs models such CNN, BiL- STM, Attention BiLSTM to predict the fluency and prosodic scoring. (Lin and Wang, 2021) uti- lized deep features directly from the acoustic model instead of relying on complex feature computa- tions like GOP scores with a scoring module, incor- porating self-attention mechanism, which is de- signed to model human sentence scoring. More re- cently, (Zhu et al., 2023) proposed BiLSTM model trained to predict the intelligibility score of given phoneme or word segment using an annotated in- telligibility L. speech using shadowing.
siamese network with BiLSTM for pronunci- ation scoring by extracting distance metrics be- tween phone instances from audio frames. A table approach presented in Leung al. (2019) introduced a CNN-RNN-CTE model for phoneme mispronunciation detection without any alignment component dependency. Subsequently, Feng et al. (2020) incorporated character embeddings to en- hance CNN-RNN model. Furthermore, Ye et al. (2022) enhanced the later model using triplet of features consisting of acoustic, phonetic, and lin- guistic embeddings. Subsequently, GOP features extracted from pre-trained ASR are enhanced us- ing a Transformer encoder to predict a range of scores of prosodic and segmental scores (Gong et al., 2022), or using additional SSL representatiin features, energy, and duration within the same ar- chitecture (Chao et al., 2022), using Conformer encoder explored in (Fan et al., 2023). Moreover, PEPPANET is also transformer-basee mispro- nunciation model, but can jointly model the dic- tation process and the alignment process, and provides corresponding diagnostic feedback (Yan
Towards lexical stress detection, several methods have been proposed to improve accuracy and per- formance. Ruan et al. (2019) proposed a sequence- to-sequence approach using the Transformer model upon the need for long- distance contextual infor- mation predict phoneme sequence with stress marks. Furthermore, Korzekwa al. (2020a) intro-

---

duced an attention-based neural network focusing on the automatic extraction of syllable-level fea- tures that significantly improves the detection of lexical stress errors.
leveraging articulatory features as auxiliary tasks phoneme sequence prediction. Furthermore, Kheir et al. (2023b) introduces novel Ll-aware multi- lingual, LI- LI-MultiMDD, architecture for address- ing mispronunciation in multilingual settings en- compassing Arabic, English, and Mandarin using wav2vec-large pre-trained model as the acoustic encoder. LI-MultiMDD is enriched with Ll-aware speech representation, allowing it to understand the nuances of each speaker's native language. SSL models have proven to be effective in pre- dicting fluency and prosodic scores assigned by hu- man annotators. Kim et al. (2022); Lin and Wang (2023a); Yang et al. (2022) fine-tuned wav2vec 2.0 and Hubert predict prosodic and fluency scores. Similarly, another research conducted in (Lin and Wang, 2023a) jointly predicts I.2 phoneme sequence using CTC loss, and predicts prosodic scores using fused acoustic representations with phoneme embeddings. Subsequently Lin and Wang (2023b) introduced fusion of language embed- ding, representation features and build unified framework for multi-lingual prosodic scoring. Re- cently, Chao et al. (2022); Kheir et al. (2023a); Chen et al. (2023), enriched latent speech extracted from SSL models with handcrafted frame- and utterance-level ver-verbal paralinguistic cues such as duration, and energy for modeling Fluency and Prosody scores.
Tone classification has received significant at- tention in Mandarin language learning due to the crucial role that tones play in Mandarin Chinese. To address the challenge several methods have been proposed. One approach involves training a DNN to classify speech frames into six tone classes (Ryant et al., 2014). Inspired by this, DNNs have been used to map combined cepstral and tonal fea- tures to frame-level tone posteriors. These tone posteriors are then fed into tone verifiers to assess the correctness of tone pronunciation (Lin al.. 2018; Li et al., 2018b). Another study utilizes CNN to classify syllables into four Mandarin tones (Chen et al., 60) Similarly, ToneNet, a CNN- based network is introduced for Chinese syllable tone classification using mel -peettcccto as fea- ture representation (Gao et al., 2019). Additionally, BiLSTM model is proposed capture long-term dependencies in acoustic and prosodic features for tone classification (Li al., 2019).
alternative to
4.6 Self-Supervised Models
Motivated by the recent success of self-supervised learning methods (Bacvski et al., 2020; Hsu et al., 2021; Chen et al., 2022; Mohamed et al., 2022) in speech recognition and related downstream tasks such emotion recognition, speaker verification, and language identification (Chen and Rudnicky, 2023; Fan et al., 2020), self-supervised approaches is employed also in this field.
4.7 Unsupervised Approaches
It important to note that the aforementioned ap- proaches for studying mispronunciation detection typically involve the need for expert knowledge, laborious manual labeling, or dependable ASR re- sults, all of which come with significant costs. In contrast, recent years have witnessed considerable endeavors in unsupervised acoustic pattern discov- ery, yielding sub-optimal outcomes. Lee and Glass (2012) initially investigated comparison- based ap- proach that analyzes the extent of misalignment be- tween a student's speech and a teacher's speech. subsequent studies Lee and Glass (2015); Lee et al. (2016), explored the discovery of mispronunciation errors by analyzing the acoustic similarities across individual learners' utterances, with a proposed n- best filtering method to resolve ambiguous error candidate hypotheses derived from acoustic simi- larity clustering. Furthermore, Mao al. (2018) proposed k-means clustering on phoneme-based phonemic posterior-gramt (PPGs) to expand the phoneme set in L2 speech. More recently, Sini et al.
Xu et al. (2021) explored finetuning wav2vec 2.0 on frame-level 1.2 phoneme prediction. A pre- trained HMM-DNN ASR is used to extract time force-alise alieeaaligmmen. To overcome the dependency on time alignment, Peng al. (2021) propose CTC-based wav2vee 2.0 predict L2 phonemes sequences. Building upon this work, Yang et al. (2022) propose an approach that leverages unla- beled L2 speech using momentum pseudo-labeling. In contrasting approach, (Lin and Wang, 2022b) combined wav2vec 2.0 features and phoneme text embeddings in jointly learning framework to predict frame-level phoneme sequence and detect boundaries. Recently EL Kheir et al. (2023a) explored the multi-view representation utilizing mono- and multilingual wav2vec 2.0 encoders to capture different aspects of speech production and

---

(2023) introduced a weighted DTW alignment as an alternative to the GOP algorithm for predicting probabilities and the sequence of target phonemes. Their proposed method achieves comparable re- sults to the GOP scoring algorithm, likewise Anand et al. (2023) explored alignment distance between wav2vec 2.0 representations of teacher and learner speech using DTW, to distinguish between intelli- gible and unintelligible speech.
5 Evaluation Metries
Phoneme Error Rate (PER): is a common metric used in the MD evaluation, measuring the accu- racy of the predicted phoneme with the human- annotated sequence. However, PER might not pro- vide comprehensive assessment of model per- formance when mispronunciations are infrequent which the case for MD datasets.
Hierarchical Evaluation Structure: The hierar- chical evaluation structure developed in (Qian al., 2010), has also been widely adopted in (Wang and Lee, 2015; Li al., 201 6a; EL. Kheir et al., 2023a) among others. The hierarchical mispronunciation detection depends on detecting the misalignment over: what is said (annotated verbatim sequence); what is predicted (model output) along with what should have been said (text-dependent reference quence). Based on the aforementioned sequences, the false rejection rate, false acceptance rate, and diagnostic error rate are calculated, using:
4.8 Data Augmentation
Two major challenges in this field are L2 data scarcity and the imbalanced distribution of neg- ative classes (mispronunciation). To address these challenges, rescarchers have opted for data aug- mentation techniques that are proven to be quite ef- fective in pronunciation assessment. Such methods employed strategies like altering the canonical text by introducing mismatched phoneme pairs while preserving the original word-level speech (Fu et al., 2021). Additionally, a mixup technique utilized in the feature space, leveraging phone-level GOP pooling to construct word-level training data (Fu et al., 2022). Furthermore, the error distance of the clustered SSL model embeddings are employed to substitute the phoneme sound with a similar sound (Zhang et al., 2022b). These latter approaches de- pend the reuse of existing information rather than generating novel instances of mispronuncia- tions. In (Fernandez et al., 2017), voice transfor- mations in pitch, vocal-tract, vocal-source charac- teristics to generate new samples. Furthermore, L2-GEN can synthesize realistic L.2 phoneme se- quences by building a novel Seq2Seq phoneme paraphrasing model (Zhang et al., 2022a). Ko- rzekwa et al. (2020b) proposed augmentation technique by generating incorrectly stressed words using Neural TTS. Furthermore, Korzekwa al. (2022) provided an overview of mispronunciation error generation using three methods, phoneme-2- phoneme P2P relies on perturbing phonetic tran- scription for the corresponding speech audio, text- 2-speech create speech signals that match the syn- thetic mispronunciations, and speech-2-speeee S2S to simulate different aspect of prosodic nature of speech. Recently, SpeechBlender (EL Kheir et al., fine-grained
True acceptance (TA): the number of phones an- notated and recognized as correct pronunciations.
the number of phones both annotated and correctly predicted as mispro- nunciations. The labels are further utilized to measure the diagnostic errors and correct diag- nosis based on the prediction output and text- dependent canonical pronunciation.
True rejection (TR):
False rejection (FR): the number of phones wrongly predicted as mispronunciations.
False acceptance (FA): the number of phones misclassified as correct pronunciations.
As a result, we can calculate the false rejection rate (FRR) that indicates the number of phones recognized as mispronunciations when the actual pronunciations are correct, false acceptance rate (FAR) that indicates phones misclassified as cor- rect but are actually mispronounced, and diagnostic error rate (DER) using the following equations:
FR
FRR = TA+ FR
(4)
FA FAR
(5)
FA+TR DE
DER CD + +E
(6)
2023b) framework is introduced data augmentation pipeline that linearly interpo- lates raw good speech pronunciations to generate mispronunciations at the phoneme level.
Precision, Recall, and F-measure are also widely used as the performance measures for mispronun- ciation detection. These metries are defined as follows:

---

to support from over 100 (Pratap al., 2023), over 1, 000 (Pratap et al., 2023) languages. Tra- ditional research in pronunciation assessments fo- cused on designing monolingual assessment sys- tems. However, recent advancements in multilin- gualism allowed for the generalization of findings across different languages. Zhang et al. (2021a) ex- plored the adaptation of pronunciation assessments from English (a stress-timed language) to Malay (a syllable-ting timed language). Meanwhile, Lin and Wang (2023b) investigated the use of language- specific embeddings for diverse languages, while optimizing the entire network within a unified framework.
TR
Precision
TR PR
Recall
TR + FA = FAR
(8)
F- --measure = Precision Recall
Precision + Recall
Pearson Correlation Coefficient: PCC is widely used to measure the relation of the predicted score of fluency, stress, and prosody with other supra- segmental and pronunciation constructs with sub- jective human evaluation for pronunciation assess- ment. The human scores are typically averaged across all annotators to provide comprehensive score.
Children CAPT: There is a noticeable imbal- ance research between children learning pronun- ciation research papers, for example, reading as- sessments, compared to adults' L.2 language learn- ing. This disparity can be attributed to the scarcity of publicly available corpora and the difficulties in collecting children's speech data.
6 Challenges and Future Look
There are two significant challenges facing advanc- ing the research further: (1) the lack of public re- sources. Table 1 shows handful of L2 languages. With 7,000 languages spoken earth, there is an urgent need for inclusivity in pronouncing the lan- guages of the world. (2) There is a need for a uni- fied evaluation metric for pronunciation learning, this can be used to establish and continually main- tain a detailed leaderboard system, which serves as dynamic and multifaceted platform for track- ing, ranking, and showcasing the advances in the field to guide researchers from academia and in- dustry push the boundaries for pronunciation as- sessments from unintelligible audio to native-like speech. The advent of Al technology represents a pivotal moment in our technological landscupe, offering the prospect of far-reaching and transfor- mative changes that have the potential to revolu- tionize a wide array of services in CAPT. Listing here some of the opportunities:
Dialectal CAPT: One implicit assumption most of the current research in pronunciation as- sessment is that 1.2 language with a standard orthographic rule. However, Cases like dialectal Arabic which is every Arab native language, there is standard orthography. Since speaking as native is the ultimate objective for advanced pro- nunciation learning, there is growing demand for this task.
Conclusion
This paper serves comprehensive resource that summarizes the current research landscape in au- tomatic pronunciation assessment covering both segmental and supra-segmental space. The paper offers insights into the following:
Integration with Conversation AI Systems: The progress made Generative Pre-trained Trans- former (GPT) led to a human-like text-based con- versational AI. Furthermore, low-latency ASR has enhanced the adoption of speech processing in our daily life. Both have paved the way for the devel- opment of reliable virtual tutor CAPT system, which is capable of interacting and providing stu- dents with instant and tailored feedhack, thereby enhancing their pronunciation skills and augment- ing private tutors.
Modeling techniques - highlighting design choices and their effect on performance.
Data challenges and available resources em- phasizes the success of automatic data gener- ation/augmentation pipeline and lack of con- sensus annotation guidelines and labels. The paper also lists available resources to the com- munity along with the current state-of-the-art performances reported per resource.
Importance of standardised evaluation metrics and steady benchmarking efforts.
Multilingual: Recent advancements in end-to- end ASR enabled the development of multi-lingual code-switching systems (Datta et al., 2020; Chowd- hury et al., 2021; Ogunremi et al., 2023). The great progress in SSL expanded ASR capabilities
With the current trend of end-to-end modeling and multilingualism, we believe this study will provide a guideline for new researchers and a foundation for future advancements in the field.

---

Limitations
speech corpus. In Proceedings International Work- shop on Integrating Speech Technology in the (Lan- guage) Learning and Assistive Interface, InStil, pages 10-17.
In this overview, we address different constructs of pronunciation and various scientific approaches for detecting errors, predicting prosodic and flu- ency scores among others. However, we have not included the corrective feedback mechanism of CAPT system. Moreover, the paper does not cover, in detail, the limited literature on CAPT's user study, or other qualitative study involving sub- jective evaluation. With the fast-growing field of pronunciation assessments, it is hard to mention would
Boulder Learning Inc. 2019, Myst corpus. Retrieved July 17, 2019.
Edward Wilder Caro Anzola and Miguel Ángel Men- doza Moreno, 2023. Goodness of pronunciation al- gorithm in the speech analysis and assessment for de- tecting errors in acoustic phoneties: An exploratory review.
Fu-An Chao, Tien-Hong Lo, Tzu-I Wu, Yao-Ting Sung, and Berlin Chen. 2022. 3m: An effective multi- view, multi-granularity. and multi-aspect modeling approach english pronunciation assessment. In 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 575-582.
all the studies and resources. Therefore, also like to apologize for any oversights of corpora or major research papers in this study.
Ethics Statement
We discussed publicly available research and datasets in our study. Any biases are unintended.
Charles Chen, Razvan C Bunescu, Li Xu, and Chang Liu. 2016a. Tone classification in mandarin chinese using convolutional neural networks. In Interspeech. pages 2150-2154.
References
David Abererombie. 1949 Teaching pronunciation. ELT Journal, 3(5):113-122,
Jin- Yu Chen and Lan Wang. 2010. Automatic lexical stress detection for chinese learners' of english. In 2010 7th International Symposium on Chinese Spo- ken Language Processing, pages 407-411. IEEE.
Mohammed Algabri, Hassan Mathkour, Mansour Alsu- laiman, and Mohamed A Bencherif. 2022. Mispro- nunciation detection and diagnosis with articulatory- level feedback generation for non-native arabic speech. Muthematics, 10(15):2727.
Lei Chen, Jidong Tao, Shabnam Ghaffarzadegan, and Yao Qian. 2018. End-to-end neural network based automated speech scoring. In 2018 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 0238-6238. IEEF.
Nayan Anand, Meenakshi Sirigiraju, and Chiranjeevi Yarra. 2023, Unsupervised speech intelligibility as- sessment with utterance level alignment distance be- tween teacher and learner wav2vec-2.0 representa- tions.
Li-Wei Chen and Alexander Rudnicky. 2023. Explor- ing wav2vec 2.0 fine tuning for improved speech emotion recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 1-5. IEEE
Amalia Arvaniti and Mary Baltazani. 2000. Greek tobi: A system for the annotation of greek speech corpora.
In LREC.
Nanc Nancy F. Chen and Haizhou Li. 2016. Computer- assisted pronunciation training: From pronunciation scoring towards spoken language learning. In 2016 Asia-Pacifie Signal and Information Processing As- sociation Annual Summit and Conference (APSIPA). pages 1-7.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self- superviedd learning of speech representations. Advances in neural information processing systems, 12444-1446600
Nancy F Chen, Rong Tong, Darren Wee, Pei Xuan Lee, Bin Ma. and Haizhou Li. 2016b. Singakids- mandarin: Speech corpus of singaporean children speaking mandarin chinese. Interspeech, pages 1545-1549.
Anton Batliner, Mats Blomberg. Shona D' Arey, Daniel Flenius, Diego Giuliani, Matteo Gerosa, Christian Hacker, Martin Russell, Stefan Steidl, and Michael Wong. 2005. The PF STAR Children's 's Speech Cor- pus. In Proc. of Interspeech.
Nancy F Chen, Rong Tong, Darren Wee, Peixuan Lee, Bin Ma, and Haizhou Li. 2015. icall corpus: Man- darin chinese spoken by non-native speakers of euro- pean descent. In Sixteenth Annual Conference of the International Speech Communication Association.
Linda Bell, Johan Boyce, Joakim Gustafson, Mattias Heldner, Anders Lindström, and Mats Wirén. 2005, The swedish nice corpus- - spoken dialogues between children and embodied characters in a computer game scenario. In Proc. of Eurospeech.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.
Patrizia Bonaventura, Peter Howarth, and Wolfgang Menzel. 2000. Phonetic annotation of a non-native

---

Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Se- lected Topics in Signal Processing, 16(6): 155551505--.18.
Tracey M Derwing and Murray Munro. 1997. Accent, intelligibillity and comprehensibility: Evidence from four 1Is. Studies in second language acquisition, 19(1):1-16.
Yu Chen, Jun Hu, and Xinyu Zhang. 2019. Sell-corpus: an open source multiple accented chinese-englise speech corpus for 12 english learning assessment. In ICASSP 2019-2019 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 7425-7429, IEEE
Heejin Do. Yunsu Kim, and Gary Geunbae Lec. 2023, Hierarchical pronunciation assessment with multi- aspect attention. In ICASSP 2023 2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Yassine EL Kheir, Shammur Chowdhury, and Ahmed Ali. 2023a. Multi-Viee Multi-Task Representation Learning for Mispronunciation Detection. In Proc. 9th Workshop on Speech and Language Technology in Education (SLaTE), pages 86-90.
Yu-Wen Chen, Zhou Yu, and Julia Hirschberg. 2023. Multipa: a multi-task speech pronunciation assess- ment system for a closed and open response scenario.
S. Chowdhury, A. Hussein, A. Abdelali, and A. Ali. 2021. Towards one model to rule all: Multilingual strategy for dialectal code-switching Arabic Ast. In- terspeech 2021.
Yassine EL Kheir, Shammur Chowdhury, Ahmed Ali, Hamdy Mubarak, and Shazia Afzal. 2023b. Speech- Blender: Speech Augmentation Framework for Mis- pronunciation Data Generation. In Proc. 9th Work- shop on Speech and Language Technology in Educa- tion (SLaTE), pages 26-30.
Jennifer Cole, Timothy Mahrt, and Joseph Roy. 2017. Srowd-soureing prosodic annotation. Computer Speech & Language, 45:300-325
Maxine Eskenazi. 1999. Using automatic speech pro- cessing for foreign language pronunciation tutoring: Some issues and a prototype.
Morris R Council III, Ralph Gardner III, Gwendolyn Cartledge, and Alana O Telesman. 2019. Improving reading within an urban elementary school: com- puterized intervention and paraprofessional factors. Preventing School Failure: Alternative Education for Children and Youth, 162-774
Maxine Eskenazi. 2009. An overview of spoken lan- guage technology for education. Speech Communi- cation, 51(10):332-844
Maxine Eskenazi, Jack Mostow, and David Graff. 1997. The CMU Kids Corpus LDC97S63. Linguistic Data Consortium.
Natalia Cylwik, Agnieszka Wagner, and Grazyna De- menko. 2009. The euronounce corpus non-native polish for asr-based pronunciation tutoring system. In International Workshop on Speech and Language Technology in Education.
Zhixing Fan, Jing Li, Aishan Wumaier, Zaokere Kadeer, and Abdujelil Abdurahman. 2023. A multifaceted approach to oral assessment based on the conformer architecture. IEEE Access, 1 1128318--88299
Silvia Dahmen, Martine Grice, and Simon Roessig. 2023. Prosodic and segmental aspects of pronun- ciation training and their effects on 12. Languages, 8(1):74.
Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. 2020. Exploring wav2vee 2.0 speaker verifi- cation and language identification. arXiv preprint 2xXiv:2012.06185.
Arindrima Datta, Bhuvana Ramabhadran, Jesse Emond, Anjuli Kannan, and Brian Roark. 2020. Language- agnostic multilingual modeling. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8239- 8243. IEEF
Yiging Feng, Guanyu Fu, Qingcai Chen, and Kai Chen. 2020. Sed-mdd: Towards sentence dependent end-to- end mispronunciation detection and diagnosis. ICASSP 2020 2020 IEEF International Confer- ence Acoustics, Speech and Signal Processing (ICASSP), pages 3492-3496
Katherine Demuth. 1992. Acquisition of sesotho. In Dan Slobin, editor, The Cress-Linguistic Study of Language Acquisition, volume 3, pages 557-638. Lawrence Erlbaum Associates, Hillsdale, N.J.
Raul Fernandez, Andrew Rosenberg, Alexander Sorin, Bhuvana Ramabhadran, and Ron Hoory. 2017. Voice- transformation-batt data augmentation for prosodic classification. 2017 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 5530-5534,
Katherine Demuth, Jennifer Culbertson, and Jessica Alter. 2006. Word-minimality, epenthesis, and coda licensing in the acquisition of english. Language & Speech, 17:4
Luciana Ferrer, Harry Bratt, Colleen Richey, Horacio Franco, Victor Abrash, and Kristin Precoda. 2015. Classification of lexical stress using spectral and prosodic features for computer-assisted language learning systems. Speech Communication, 69:31- 45.
Katherine Demuth and Anne Tremblay. 2007. crusodically-condiionn variability in children's 's production of french determiners. Journal of Child Language, 34:1-29.

---

Kaiqi Fu, Shaojun Gao, Kai Wang, Wei Li, Xiaohai Tian, and Zejun Ma. 2022. Improving non- native word-level pronunciation scoring with phone-level mixup data augmentation and multi-source informa- tion. arXiv preprint arXiv:2203.01822
Wenping Hu. Yao Qian, and Frank Soong. 2015a. An improved
approach to mispronunciation detection and diagnosis of 12 learners' speech. In SLaTE, pages 71-76.
Wenping Hu, Yao Qian, Frank K Soong, and Yong Wang. 2015b. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression elassifiers. Speech Communication, 67: 154-166.
Kaiqi Fu. Jones Lin, Dengfeng Ke, Yanlu Xie, Jin- song Zhang, and Binghuai Lin. 2021. A full text- dependent end to end mispronunciation detection and diagnosis with easy data augmentation techniques. arXiv preprint arXiv:2 104.08428.
Guimin Huang, Qiupu Chen, and Hongtao Zhu. 2020, A mispronunciation detection method of confus- ing vowel pair for chinese students. In Journal of Physics: Conference Series, volume 1693, page 012102. IOP Publishing.
Jun Gao, Aijun Li, and Ziyu Xiong. 2012. Mandarin multimedia child speech corpus: CASS C_HIILD. In International Conference on Speech Database and Assessments (Oriental COCOSDA).
Qiang Gao, Shutao Sun, and Yaping Yang. 2019. Tonenet: A enn model of tone classification of man- darin chinese. In Interspeech, pages 3367-3371.
Akinori Ito, Yen-Ling Lim, Motoyuki Suzuki, and Shozo Makino. 2005. Pronunciation error detection method based on error rule clustering using a deci- sion tree. In Ninth European Conference on Speech Communication and Technology.
Marta Garrote. 2008. CHIEDE: A Spontaneous Child Language Corpus of Spanish. Ph.D. thesis, Universi- dad Autónoma de Madrid, Spain.
Okim Kang, Ron I Thomson,, and John Murphy. 2018. The Routledge handbook of contemporary English pronunciation. Routledge New York, NY.
Yuan Gong, Ziyi Chen, lek-Heng Chu, Peng Chang, and James Glass. 2022. Transformer-based multi- aspect -ultllgraullarity non-native english speaker pronunciation assessment. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7262-7266.
Natalia Kartushina and Ulrich Frauenfelder. 2014. On the effects of 12 perception and of individual differ- ences in 1I production 12 pronunciation. Frontiers in psychology, 5: 1246.
Reginald Gooch, Kazuya Saito, and Roy Lyster. 2016. Effects of recasts and prompts 12 pronunciation development: Teaching english//to korean adult efl learners. System, 60:1 127
Goh Kawai and Keikichi Hirose. 1998. call system us- ing speech recognition to teach the pronunciation of japanese tokushuhaku. In STiLL-Speech Technology in Language Learning.
Roberto Gretter, Marco Matassoni, Stefano Bannò, and Falavigna Daniele. 2020. TLTT-chool:: a corpus of non native children speech. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 378-385, Marseille, France. European Language Resources Association.
Abe Kazemzadeh, Hong You. Markus Iseli, Barbara Jones, Xiaodong Cui, Margaret Heritage, Patti Price, Elaine Anderson, Shrikanth Narayanan, and Abeer Alwan. 2005. TBALL data collection: The making of a young children's speech corpus. In Proc. of Interspeech.
Andreas Hagen, Bryan Pellom, and Ronald Cole. 2003. Children's speech recognition with application to in- teractive books and tutors. In IEEE Workshop on Automatic Speech Recognition and Understanding.
Yassine El Kheir, Shammur Absar Chowdhury, and Ahmed Ali. 2023a. The complementary roles of non-verbal cues for robust pronunciation assessment. arXiv preprint 2309:2230..0733.
Nina Hosseini- Kivennni,, Roberto Gretter, Marco Matas- soni, and Giuseppe Daniele Falavigna. 2021. Exper-
Yassine El Kheir, Shammur Absar Chowdhury, and Ahmed Ali. 2023b. Ll-aware multilingual mis- pronunciation detection framework. arXiv preprint arXiv:2309.07719
iments of asr-based mispronunciation detection for children and adult english learners.
Wei-Ning HsU, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460.
Eesung Kim, Jac-Jin Jeon, Hyeji Sco, and Hoon Kim. 2022. Automatic pronunciation assessment using self-supervised speech representation learning.
Yoon Kim, Horacio Franco, and Leonardo Neumeyer. 1997. Automatic promunciation scoring of specific phone segments for language instruction. In Fifth European Conference on Speech Communication and Technology.
Wenping Hu, Yao Qian, and Frank K Soong. 2013. A new dan-based high quality pronunciation evalua- tion for computer-aided language learning (call). In Interspeech, pages 1886-1890.

---

Daniel Korzekwa, Roberto Barra-Chiconn Szymon Za- porowski, Grzegorz Beringer, Jaime Lorenzo-Tnann Trueba, Alicja Serafinowicz, Jasha Droppo, Thomas Drug- man, and Bozena Kostek. 2020a. Detection of lex- ical stress errors in non-native (12) english with data augmentation and attention. arXiv preprint 1rX1v22012.17788.
Kun Li, Xiaojun Qian, and Helen Meng. 2016a. Mis- pronunciation detection and diagnosis in 12 english speech using multidisttibution deep neural networks. JEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 25()):193-207
Wei Li, Nancy F Chen, Sabato Marco Siniscalchi, and Chin-Hui Lee. 2017. Improving mispronunciation detection for non-native learners with multisource information and Istm-based deep models. In Inter- speech, pages 2759-2763
Daniel Korzekwa, Roberto Barra-Chicote, Szymon Za- porowski, Grzegorz Beringer, Jaime Lorenzo-Tnun Trueha, Alicja Serafnnowcc, Jasha Droppo, Thomas Drug- man, and Božena Kostek. 2020b. Detection of lexi- cal stress errors in non-native (12) english with data augmentation and attention. In Interspeech.
Wei Li, Nancy F Chen, Sabato Marco Siniscalchi, and Chin-Hui Lee. 2018b. Improving mandarin tone mis- pronunciation detection for non-nat non-native learners with soft- target tone labels and bistm- based deep models. In 2018 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 6249 6253. IEEE.
Daniel Korzekwa, Jaime Lorenzo-Trueha, Thomas Drugman, and Bozena Kostek. 2022. Computer- assisted pronunciation training-speettt synthesis is almost all you need. Speech Communication, 142:22- 33.
Wei Li, Nancy F Chen, Sabato Marco Siniscalchi, and Chin-Hui Lee. 2019. Improving mispronunciation detection of mandarin tones for non- non-native learners with soft-target tone labels and blstm- based deep tone models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2012-001 20244
Konstantinos Kyriakopoulos, Kate M Knill, and Mark JF Gales. 2018. deep learning approach to assessing non-native pronunciation of english using phone distances. ISCA.
Konstantinos Kyriakopoulos, Kate M Knill, and Mark JF Gales. 2020. Automatic detection of ac- cent and lexical pronunciation errors spontaneous non-native english speech. ISCA.
Wei Li, Sabato Marco Siniscalchi, Nancy F Chen, and Chin-Hul Lee. 2016b. Using based extended recognition network to detect non-native mandarin tone mispronunciations. In 2016 Asia-Pacific Sig- nal and Information Processing Association Annual Summit and Conference (APSIPA), pages 1-4, IEEF.
Ann Lee, Nancy F Chen, and James Glass. 2016. Per- sonalized mispronunciation detection and diagnosis based on unsupervised error pattern discovery. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6145 45-- 61 49. IEEF.
Hank Liao, Golan Pundak, Olivier Siohan, Melissa Car- roll, Noah Coccaro, Qi-Ming Jiang. Tara N. Sainath, Andrew Senior, Françoise Beaufays, and Michiel Bacchiani. 2015. Large vocabulary automatic speech recognition for children. In Interspeech.
Ann Lee and James Glass. 2012. A comparison-basee approach to mispronunciation detection. In 2012 IEEE Spoken Language Technology Workshop (SI.T), pages 382-387.
Binghuai Lin and Liyuan Wang. 2021. Deep feature transfer learning for automatic pronunciation assess- ment. In Interspeech, pages 4438-4442.
Binghuai Lin and Liyuan Wang. 2022a. Phoneme mis- pronunciation detection by jointly learning to align. In ICASSP 2022-2022 IEFF International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 68226682888822 6626. IEEE.
Ann Lee and James Glass. 2015. Mispronunciation de- tection without nonnative training data. In Sixteenth Annual Conference of the International Speech Com- munication Association.
Binghuai Lin and Liyuan Wang. 2022b. Phoneme mis- pronunciation detection by jointly learning to align. In ICASSP 2022 2022 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 6822-6622666 2226-626.
R. Gary Leonard and George Doddington. 1993. In Web Download, Philadelphia. Linguistic Data Con- sortium.
Wai-Kim Leung. Xunying Liu, and Helen Meng- 2019. Con-rnn-cte based end-to-end mispronunci- ation detection and diagnosis. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 8132-8136. IEEE.
Binghuai Lin and Liyuan Wang. 2023a. Exploiting in-
formation from native data for non- non-native automatic pronunciation assessment. In 2022 IEEE Spoken Lan- guage Technology Workshop (SI.T), pages 708-714 14.
Binghuai Lin and Liyuan Wang. 2023b. Multi-lingual pronunciation assessment with unified phoneme set and language-specific embeddings. In ICASSP 2023- 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Kun Li. Shaoguang Mao, Xu Li, Zhiyong Wu, and He- len Meng. 2018a. Automatic lexical stress and pitch accent detection for 12 english speech using multi- distribution deep neural networks. Speech Communi- cation, 96:28-36.

---

Ju Lin, Wei Li, Yingming Gao, Yanlu Xie. Nancy F Chen, Sabato Marco Siniscalchi, Jinsong Zhang, and Chin-Hui Lee. 2018. Improving mandarin tone recog- nition based on dnn by combining acoustic and artic- ulatory features using extended recognition networks. Journal of Signal Processing Systems, 90: 10:7077-1087.
AbdeIrahman Mohamed, Hung-yi Lee, Lasse Borgholt. Jakob D. Havtorn, Joakim Edin, Christian Igel. Ka- trin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. 2022. Self-supervised speech representation learning: A review. IEEE Journul of Selected Topics in Signal Processing, 117-11117--
Wii-Kit Lo, Shuang Zhang, and Helen Meng. 2010. Automatic derivation of phonological rules for mis- pronunciation detection in computer-assisted pro- nunciation training system. In Eleventh annual con- ference of the international speech communication association.
Murray J Munro and Tracey M Derwing. 1995. For- eign accent, comprehensibility, and intelligibilitty in the speech of second language learners. Language learning, 45(1):73-9) 97.
Ambra Neri, Omnella Mich, Matteo Gerosa, and Diego Giuliani. 2008. The effectiveness of computer as- sisted pronunciation training for foreign language learning by children. Computer Assisted Language Learning, 21(5):393 408.
Yi Luan, Masayuki Suzuki, Yutaka Yamauchi, Nobuaki Minematsu, Shuhei Kato, and Keikichi Hirose. 2012. Performance improvement of automatic pronuncia- tion assessment in noisy classroom. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 428-431.
Raymond W. M. Ng, Alvin C.M. Kwan, Tan Lee, and Thomas Hain. 20 Shefce: A cantonese-english bilingual speech corpus for pronunciation assessment. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5825- 5829.
Shaoguang Mao, Xu Li, Kun Li, Zhiyong Wu, Xunying Liu, and Helen Meng. 2018. Unsupervised discovery of an extended phoneme set in 12 english speech for mispronunciation detection and diagnosis. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2244-6248.
Raymond WM Ng, Alvin CM Kwan, Tan Lee, and Thomas Hain. 2017b. Shefce: A cantonesc-english bilingual speech corpus for pronunciation assessment. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5825- 5829. IEEE
IEEE.
Peter Hugoe Matthews. 2014. The concise Oxford dic- tionary of linguistics. Oxford University Press.
Helen Meng, Yuen Yee Lo., Lan Wang, and Wing Yiu Lau. 2007a. Deriving salient learners' mispronuncia- tions from cross-language phonological comparisons. In 2007 IEEE Workshop on Automatic Speech Recog- nition Understanding (ASRU), pages 437 442.
Thi Thu Trang Nguyen, Do Dat Tran, Albert Rilliard, Christophe d' Alessandro, and Thi Ngoc Yen Pham. 2014. Intonation issues in hmm- based speech synthe- sis for vietnamese. In Spoken Language Technologies for Under-Resourced Languages.
Helen Meng. Yuen Yee Lo., Lan Wang. and Wing Yiu Lau. 2007b. Deriving salient learners' mispronun- ciations from lrnss-language phonological compar- isons. In 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 437- 442. IEFF.
Tolulope Ogunremi, Christopher Manning, and Dan Ju- rafsky. 2023. Multilingual self-supervised speech representations improve the speech recognition of low-resource african languages with codeswitching. In Empirical Methods in Natural Language Process- ing.
Wolfgang Menzel, Eric Atwell, Patrizia Bonaventura, Daniel Herron, Peter Howarth, Rachel Morton, and Clive Souter. 2000. The isle corpus of non-native spoken english. In Proceedings of LREC 2000: Lan- guage Resources and Evaluation Conference, vol 2, pages 9 957-964. European Language Resources As- sociation.
Kota Ohata. 2004. Phonological differences between japanese and english: Several potentially problematic. Language learning. 22:29-41.
Priyanshi Pal, Chiranjeevi Yarra, and Prasanta Kumar Ghosh. 2022. Voistutor 2.0: A speech corpus with phonetic transcription for pronunciation evaluation of indian 12 english learners. In 2022 25th Confer- ence of the Oriental COCOSDA International Com- mittee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O- COCOSDA), pages 1-6.
Nobuaki Minematsu, Koji Okabe, Keisuke Ogaki, and Keikichi Hirose. 2011. Measurement of objective in- telligibility of japanese accented english using erj (en- glish read by japanese) database. In INTERSPEECH. pages 1481-1484.
Nobuaki Minematsu, Yoshihiro Tomiyama, Kei Yoshi- moto, Katsumasa Shimizu. Seiichi Nakagawa, Masa- take Dantsuji, and Shozo Makino. 2004. Develop- ment of english speech database read by japanese to support call research, In Proc. ICA, volume 1, pages 557-560.
R. M. Pascual and R. C. Guevara. 2012. Developing a children's 's filipino speech corpus for application in automatic detection of reading miscues and disfluen- cies. In TENCON 2012 IEEE Region 10 Conference, pages 1-6.

---

Linkai Peng, Kaiqi Fu, Binghuai Lin, Dengfeng Ke. and Jinsong Zhang. 2021. A study on fine-tunig wav2vec2. 0 model for the task of mispronunciation detection and diagnosis. In Interspeech, pages 4448- 4452.
Mostafa Ali Shahin, Julien Epps, and Beena Ahmed. 2016. Automatic classification of lexical stress in english and arabic languages using deep learning. In Interspeech, pages 175-179.
Kavita Sheoran, Arpit Bajgoti, Rishik Gupta, Nishtha Jatana, Geetika Dhand, Charu Gupta, Pankaj Dad- heech, Umar Yahya, and Nagender Aneja. 2023. Pro- nunciation scoring with goodness of pronunciation and dynamic time warping. IEEE Access, 1:15585- 15495.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arum Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam ZaraadZa al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305. 13516.
Xiaojun Qian, Helen Meng, and Frank Soong. 2010. Capturing 12 segmental mispronunciations with joint- sequence models in computer-aided pronunciation training (capt), In 2010 7th International Symposium on Chinese Spoken Language Processing, pages 84- 88. IEEE.
Jiatong Shi, Nan Huo, and Qin Jin. 2020. Context- aware goodness of pronunciation for computer- assisted pronunciation training. arXiv preprint arXiv:2008. 086477
Aghilas Sini, Antoine Perquin, Damien Lolive, and Ar- naud Delhay, 2023. Phone-level pronunciation scor- ing for 11 using weighted-dynamic time warping. In 2022 JEEE Spoken Language Technology Workshop (SI.T), pages 1081-1087 IEEF
Ramya Rasipuram, Milos Ceraa, and Mathew Magimai. -oss. 2015. Hmm- non non-native ac- cent assessment using posterior features.
Antoine Raux and Tatsuya Kawahara. 2002. Automatic intelligibility assessment and diagnosis of critical pronunciation errors for computer-assitte pronunci- ation learning. In INTERSPEECH.
Helmer Strik, Khiet Truong, Febe De Wet, and Catia Cucchiarini. 2009, Comparing different approaches for automatic pronunciation error detection. Speech communication, 51(10):445-85.
Manny Rayner, Nikos Tsourakis, Claudia Baur, Pier- rette Bouillon, and Johanna Gerlach. 2014. CALL- SLT: A spoken CALL system based on grammar and speech recognition. Linguistic Issues in Language Technology, 10(2).
Sweekar Sudhakara, Manoj Kumar Ramanathi, Chiran- jeevi Yarra, and Prasanta Kumar Ghosh. 2019. An improved goodness of pronunciation (gop) measure for pronunciation evaluation with dnn-hmm system considering hmm transition probabilities. In INTER- SPEECH, pages 954-958.
Pamela M Rogerson- 2021. Computer-assisted pronunciation training (CAPT): Current issues and future directions. RELC Journal, 52(1):189-200
Jidong Tao, Shabnam Ghaffarzadegan, Lei Chen, and Klaus Zechner. 2016. Exploring deep learning archi- tectures for automatically grading non-native spon- tancous speech. In 2016 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 614--144.
Orith Ronen, Leonardo Neumeyer, and Horacio Franco. 1997. Automatic detection of mispronunciation for language instruction. In EUROSPEECH.
Yong Ruan, Xiangdong Wang, Hong Liu, Zhigang Ou, Yun Gao, Jianfeng Cheng, and Yueliang Qian. 2019. An end-to-end approach for lexical stress detection based on transformer. arXiv preprint arXiv:/911.046
Rong Tong, Nancy F Chen, and Bin Ma. 2017. Multi- task learning for mispronunciation detection sin- gapore children's mandarin speech. In Interspeech, pages 2193-2197.
Rong Tong. Nancy F Chen, Bin Ma, and Haizhou Li. 2015. Goodness of tone (got) for non-native man- darin tone recognition. In Sixteenth Anmual Con- ference of the International Speech Commaicatioon Association.
Neville Ryant, Jiahong Yuan, and Mark Liberman. 2014. Mandarin tone classification without pitch tracking. In 2014 IEFE international conference on acoustics, speech and signal processing (ICASSP), pages 4868- 4872. IEEF
Pavel Trofimovich and Talia Isaacs. 2012. Disentan- gling accent from comprchensibility. Bilingualism: Language and Cognition, 15(4):905 916.
Kazuya Saito, Pavel Trofimovich, and Talia Isaacs. 2016. Second language speech production: Investigating linguistic correlates of comprehensibility and accent- edness for learners at different ability levels. Applied Psycholinguistics, 37(2)217-220 240.
KP Truong, Ambra Neri, Catia Cucchiarini, and Helmer Strik. 2004. Automatic pronunciation error detection: an phonetic approach.
Marcelo Sancinetti, Jazmín Vidal, Cyntia Bonomi, and Luciana Ferrer. 2022. A transfer learning approach for pronunciation scoring. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 68 6812-6811 16.
Takumi Uchihara. 2022. Is it possible to measure word- level comprchensibility and accentedness as indepen- dent constructs of pronunciation knowledge? Re- search Methods in Applied Linguistics, 1(2): 100011

---

Joost Van Doremalen, Catia Cucchiarini, and Helmer Strik. 2009. Automatic detection of vowel pronunci- ation errors using multiple information sources. In 2009 IEEE Workshop Automatic Speech Recogni- lion & Understanding, pages 580-585. IEEE
Bi-Cheng Yan, Hsin-Wei Wang, and Berlin Chen. 2023a. Peppanet: Effective mispronunciation detection and diagnosis leveraging phonetic, phonological, and acoustic cues. In 2022 IEEE Spoken Language Tech- nology Workshop (SLT), pages 1045-1051.
Lavanya Venkatasubramanian, Vishal Sunder, and Eric Fosler-Lussier. 2023, End-to-end word-level disflu- ency detection and classification in children's 's reading assessment. In ICASSP 2023-2023 IEEF Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 1-5. IEEE
Bi- Ch-nn Yan, Hsin-War Wang, Yi-Cheng Wang, and Berlin Chen. 2023b. Effective sraph-based modeling of articulation traits for mispronunciation detection and diagnosis. In ICASSP 2023 2023 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Jazmín Vidal, Luciana Ferrer, and Leonardo Brambilla. 2019a. Epadb: A database for development of pro- nunciation assessment systems. In INTERSPEECH, pages 589-593.
Mu Yang, Kevin Hirschi, Stephen D Looney, Okim Kang, and John HL Hansen. 2022. Improv- ing mispronunciation detection with wav2vec2- based momentum pseudo-labeling for accented- ness and intelligibility assessment. arXiv preprint 2rXivv2203.15000
Jazmin Vidal, Luciana Ferrer, and Leonardo Brambilla. 2019b. Epadb: a database for development of pro- nunciation assessment systems. Proc. Interspeech 2019, pages 589-593.
Chiranjeevi Yarra, Aparn Srinivasan, Chandana Srini- vasa, Ritu Aggarwal, and Prasanta Kumar Ghosh. 2019. voistutor corpus: A speech corpus of indian 12 english learners for pronunciation assessment. In 2019 22nd Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardisation of Speech Databases and Assessment Techniques (O-COCOSDA), pages 1 1-6, IEEE
Agnieszka Wagner. 2014, Rhythmic structure of utter- ances in native and non- non-native polish. In Proc. 7th International Conference on Speech Prosody 2014, pages 337-341.
Xue Wang. 2022. Segmental versus suprasegmental: Which one is more important to teach? RELC Jour- nal, (3(1):194-202
Wenxuan Ye, Shaoguang Mao, Frank Soong, Wenshan Wu, Yan Xia, Jonathan Tien, and Zhiyong Wu. 2022. An approach to mispronunciation detection and di- agnosis with acoustic, phonetic and linguistic (apl) embeddings. In ICASSP 2022 2022 IEEE Interna- tional Conference on Acousties, Speech and Signal Processing (ICASSP), pages 6827-6831.
Yow-Bang Wang and Lin-Shan Lee. 2012. Improved approaches of modeling and detecting error patterns with empirical analysis for computer-aided pronun- ciation training. In 2012 IEEE international con- ference acoustics, speech and signal processing (ICASSP), pages 5049-5052, IEEE.
Zhou Yu, Vikram Ramanarayanan, David Suendermann- Oeft, Xinhao Wang. Klaus Zechner, Lei Chen, Jidong Tao, Aliaksei Ivanou, and Yao Qian. 2015. Using bidirectional Istm recurrent neural networks to learn high-level abstractions of sequential features for au- tomated scoring of non-native spontaneous speech. In 2015 IEEE Workshop on Automatic Speech Recog- nition and Understanding (ASRU), pages 338-345.
Yow-Bang Wang and Lin-shan Lee, 2015. Supervised detection and unsupervised discovery of pronunci- ation error patterns for computer-assisted language learning. IEÉE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):564555 579.
Silke Witt. 2012. Automatic error detection in pro- nunciation training: Where we are and where we need go. In International Symposium auto- matic detection on errors in pronanciation training, volume 1.
Daniel Zhang, Ashwinkumar Ganesan, Sarah Camp- bell, and Daniel Korzekwa. 2022a. L2-gen: A neural phoneme paraphrasing approach to 12 speech synthe- sis for mispronunciation diagnosis.
Daniel Yue Zhang, Soumya Saha, and Sarah Campbell. 2023. Phonetic rnn-transducer for mispronunciation diagnosis. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 1-5. IEEE.
Silke M Witt and Steve Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech communicattin,, 30(2- 108.
Minglin Wu, Kun Li, Wai-Kim Leung, and Helen Meng. 2021. Transformer based end-to-end mispronuncia- tion detection and diagnosis. In Interspeech, pages 3954-3958.
Feng Zhang, Chao Huang, Frank K Soong, Min Chu. and Renhua Wang. 2008. Automatic mispronuncia- tion detection for mandarin. In 2008 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing, pages 5077-5080. IEEF.
Xiaoshuo Xu, Yueteng Kang, Songjun Cao, Binghuai Lin, and Long Ma. 2021. Explore wav2vee 2.0 for mispronunciation detection, In Interspeech, pages 4438-
Huayun Zhang, Ke Shi, and Nancy F Chen, 202 Mul- tilingual speech evaluation: case studies on english, malay and tamil. arXiv preprint arXiv:2107.03677

---

Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. 2021b. 762 An open- source non-native english speech corpus for pronunci- ation assessment. arXiv preprini arXiv:2104.01378.
Junbo Zhang, Zhiwen Zhang, Yongqing Wang. Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. 2021c. speechocean762: source non-native english speech corpus for pronunci- ation assessment. arXiv preprint
open-
Long Zhang, Ziping Zhao, Chunmei Ma, Linlin Shan, Huazhi Sun, Lifen Jiang, Shiwen Deng, and Chang Gao. 2020. End-to-end automatic pronunciation er- ror detection based improved hybrid cic/attention architecture, Sensors, 20(7):1809.
XIAO ZHANG. 2021. Latic: non-native pre-labelled mandarin chinese validation corpus for automatic speech scoring and evaluation task.
Zhan Zhang, Yuehai Wang, and Jianyi Yang. 2022b. End-to-end mispronunciation detection with simu- lated error distance. Proc. Interspeech 2022, pages 4331.
Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucie, Evgeny Chukkarev-HHudillinen. John Levis, and Ricardo Sitierrez-Osuna. 2018a. L2-arctic: A non-native english speech corpus. In Interspeech, pages 2783-2787.
Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Chukharev-Huuiilineee John Levis, and Ricardo Gutierrez-Osune 201 8b. 1.2-arctie: non-native english speech corpus. Interspeech, pages 2783-2787
Chuanbo Zhu, Takuya Kunihara, Daisuke Saito, Nobuaki Minematsu, and Noriko Nakanishi. 2023. Automatic prediction intelligibiliity of words and phonemes produced orally by japanese learners of english. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1029-1036.

---

Appendix
consists of two sets of data: a phonemic pronuncia- tion set and a prosody set. The phonemic pronunci- ation set contains 460 phonetically- tences, 32 sentences including phoneme sequences difficult for Japanese to pronounce correctly, and 100 sentences designed for the test set. The prosody set contains 94 sentences with various intonation patterns, 120 sentences with various accent and rhythm patterns, and 109 words with various cent patterns. The ERJ corpus annotated with phonemic transcriptions and prosodic markings. The phonemic transcriptions are based on the In- ternational Phonetic Alphabet (IPA). The prosodic markings include information about intonation, ac- cent, and rhythm.
Mispronunciation Assessment and Non-Native Datasets
sen-
In this section, we provide comprehensive overview of existing pronunciation assessment datasets as presented in Table 1.
A.1 ISLE Speech Corpus (Menzel et al., 2000) The ISLE corpus stands out one of the largest speech corpora in terms of duration and offers the advantage of being distributed by ELDA. This corpus focuses on German and Italian accented English, featuring recordings of 23 intermediate- level speakers from each accent group. The par- ticipants, primarily employees and students from project sites in Italy, Germany, and the UK, were selected to achieve balance of native languages (German/Italian) while including a small number of non non-native speakers from other countries (Spanish, French, Chinese) and native British English speak- ers for comparison purposes. The corpus contains readings of both nonfictional autobiographic text (1300 words) and short utterances (1 (1100 words) de- signed to cover common pronunciation errors made by language learners. It offers annotations at both the word and phone levels, making it particularly valuable for developing Computer Assisted Lan- guage Learning systems. The annotation process involved multiple steps, including quality checks, reference transcription, forced alignment, and the addition of canonical pronunciations and stress markings. An emphasis was placed on matching non-English phones to the closest equivalent in the UK English phone set, with occasional input from trained phonetician and native speaker of the speaker's mother tongue for verification and quality improvement purposes.
A.3 Chinese University Chinese Learners of English (CU-CHLOE) (Meng et al., 2007a)
The CU-CHLOE corpus encompasses diverse group of speakers, including 1 110 Mandarin speak- (60 males and 50 females) and 100 Cantonese speakers (50 males and 50 females). It struc- tured into five distinct sections, namely confusable words, minimal pairs, phonemic sentences, the Ae- sop's s Fable "The North Wind and the Sun," and prompts sourced from the TIMIT dataset. Trained linguists have diligently labeled all sections, except for the TIMIT prompts, contributing to approxi- mately 30% of the comprehensive CHLOE data. This expert annotation ensures the corpus provides reliable and precise linguistic information, making a valuable resource for exploring Mandarin and Cantonese speech characteristics and facilitating advancements in research within these languages.
A.4 EURONOUNCE (Cylwik et al., 2009)
EURONOUNCE is a speech corpus developed for an -SRebased pronunciation tutoring system. The annotation conveys a phonetic segmentation, with identifying pronunciation errors, including substi- tutions, insertions, and deletions. The resulting annotations are then reviewed by native speaker of the source language (German) validate the assessment.
A.2 English read by Japanese Corpus (ERJ) (Minematsu al., 2004)
The ERJ corpus (English Read by Japanese) is a database of English speech read by Japanese students. It was created to support research in assister-assisted language learning (CALL). The corpus contains 800 utterances from 202 (100 males and 102 females) Japanese university stu- dents, each of whom read a set of 100 sentences. The sentences were selected to be phonetically bal- anced and to cover a variety of grammatical struc- tures. The corpus is annotated with phonemic tran- scriptions and prosodic markings. The ERJ corpus
A.5 iCALL (Chen al., 2015)
iCALL is comprised of 90,841 spoken statements delivered by 305 individuals, spanning cumula- tive duration of 142 hours. The speaker composi- tion ensures gender equality, encompasses various native languages, and reflects a representative range of adult Mandarin leamner. These oral statements

---

have been transcribed phonetically and evaluated for fluency by proficient native Mandarin speakers.
languages (LIs). Yet, only 150 utterances is manu- ally per speaker to identify three types of segmental mispronunciation errors: substitutions, deletions, and insertions resulting in 3.66 hours.
A.6 Singakids-Mandarinn (Chen et al., 2016b) Migaaiids-Mandarin is comprehensive speech corpus consisting of recordings from 255 Singa- porean children between the ages of 7 and 12. The corpus aims provide a resource for studying Man- darin Chinese pronunciation and language acquisi- tion in Singaporean children. The corpus contains a total of 125 hours of audio data, with 75 hours dedicated to speech. Within this dataset, there are 79,843 utterances, each of which has been metic- ulously annotated by human experts. The annota- tions include phonetic transcriptions, lexical tone markings, and proficiency scoring at the level of in- dividual utterances. The reading scripts used in the corpus encompass a wide range of utterance styles. They cover syllable-level minimal pairs, individual words, phrases, complete sentences, and even short stories. This diversity allows for thorough analy- sis of different aspects of Mandarin pronunciation and fluency in Singaporean children.
A.9 VoiSTUTOR corpus (Yarra et al., 2019)
VOiSTUTOR is a pronunciation assessment corpus of Indian second language (L22 learners learning English. The corpus consists of audio recordings of 16 Indian L.2 learners reading set of 1676 sen- tences. The recordings are accompanied by pho- netic transcriptions, human ratings of pronuncia- tion accuracy on a scale of to 10 for each utter- ance, and binary decisions for seven factors that af- fect pronunciation quality: intelligibility, phoneme quality, phoneme mispronunciation, syllable stress quality, intonation quality, correctness of pauses, and mother tongue influence.
A.10 SELL-CORPUS (Chen et al., 2019)
SELL-CORPUS is a multiple accented speech cor- pus for I.2 English learning in China. The corpus consists of audio recordings of 389 volunteer speak- ers, including 186 males and 203 females. The speakers are from seven major regional dialects of China, including Mandarin, Cantonese, Wu, Min. Hakka, and Southwestern Mandarin. The corpus contains 31.6 hours of speech recordings. Each recording in the corpus contains word-level thographic transcription manually inspected and cleaned by inserting, substituting, or deleting mis- matching characters.
A.7 SHEFCE (Ng et al., 2017b)
SHEFCE (ShefCE) is bilingual parallel speech corpus that focuses on Cantonese and English. It recorded by second language (L.2) English learners in Hong Kong. The corpus consists of recordings from 31 undergraduate postgraduate students, aged 20 30. The corpus includes a total of 25 hours of speech data, with approximately 12 hours recorded in Cantonese and 13 hours recorded in English. The primary goal of this corpus is provide resource for studying the speech pat- terns, pronunciation, and language acquisition of Cantonese-speaking individuals who learning English second language.
A.11 English Pronunciation by Argentinians Database (EpaDB) (Vidal et al., 2019a)
EpaDB consists of English phrases recorded by native Spanish speakers with varying levels of En- glish proficiency. The recordings are annotated with ratings indicating the quality of promunciation at the phrase level. Additionally, detailed phonetic alignments and transcriptions are provided, indicat- ing which phones were actually pronounced by the speakers.
A.8 L2-ARCTIC (Zhao et al., 2018a)
The L2-ARCTIC corpus is specialized speech corpus designed for research in voice conversion, accent conversion, and mispronunciation detection in non-native English. It encompasses a substantial collection 26867 utterances from 24 non-nan native speakers (12 males and 12 females) whose L1 lan- guages include Hindi, Korean, Mandarin, Span- ish, Arabic, and Vietnamese. The recordings were sourced from total of 4 speakers per L11 language, consisting of males and 2 females ensuring bal- anced distribution in terms of gender and native
A.12 Speechocean762 (Zhang et al., 2021b)
Speechocean762 is an extensive dataset specifically designed for pronunciation assessment. It com- prises a total of 5,000 English utterances obtained from 250 non-native speakers. Each utterance in the dataset is associated with five aspect scores at the utterance level, namely accuracy, fluency, com- pleteness, prosody, and a total score ranging from to 10. Additionally, for each word within the ut-
Aversion 5 http:///pp.ongtaaaa..cc/c///rree
relcased in 2020 avalaible:

---

terance, three aspect scores are provided, including accuracy, stress, and a total score ranging from 0 to 10. Furthermore, an accuracy score assigned to each individual phoneme, ranging from to 2. To ensure reliability, each of these scores is annotated by five expert evaluators.
corpora involves linguistics experts transcribing the actual sequence of phonemes spoken by the learner (Bonaventura et al., 2000; Zhao et al., 2018b; Vi- dal et al., 2019b). The resulted transcription is commonly referred as hypothesis annotation. Ad- ditionally, extra tasks can be incorporated, such as providing time boundaries for each pronounced phoneme, to further enhance the annotation pro- cess. This approaches may have limitations in cap- turing non-clear speech instances, such as heav- ily accented pronunciations that may not be eas- ily detected by human annotators. This leads to the second approach, which incorporates scoring- based methods in addition to hypothesis annotation (Zhang et al., 2021c). In this approach, a score is assigned to each phoneme: 0 represents deleted or mispronounced phonemes, indicates heavily accented pronunciation, and signifies good pro- nunciation. This scoring-based approach provides a more comprehensive assessment of pronunciation quality, particularly in cases where clear detection by human annotators may be challenging.
A.13 LATIC (ZHANG, 2021)
LATIC primarily targets non-native learners of Mandarin Chinese. The dataset comprises four hours of recordings involving specifically selected non-native Chinese speakers. The participants' LI's vary, including Russian, Korean, French, and Arabic. Following each audio file, annotators tran- scribed the "closest" transcript and provided mod- em Mandarin annotations after careful listening.
A.14 Arabic-CAPT (Algabri et al., 2022)
Arabic-CAPT is an Arabic mispronunciation de- tection corpus consisting of 62 non non-native Arabic speakers from 20 different nationalities, totaling 2.36 hours of speech data. The Arabic non-native speech is annotated following the guidelines in (Zhao et al., 2018a).
B.2 Supra-segmental Annotation
A.15 Ara VoiceL2 (EL Kheir et al., 2023b)
Limited research has been conducted regarding the annotation of supra-segmental features at the rhythm, stress, and intonation levels such as in (Ar- vaniti and Baltazani, 2000; Chen et al., 2016b; Cole et al., 2017). However, the ultimate objective of an- notating these supra-segmental aspects is ensure the fluency and intelligibility of L2 learners' speech, Hence the most commonly used annotated datasets at the prosodic level provide human- scored words, and sentences based on overall pronunciation qual- ity and fluency. Multiple tiers of human scoring annotations can be applied in this context. This in- cludes providing the accuracy of pronounced words to assess their intelligibility, assigning scores evaluate the positioning of stress within individual words within sentences, and evaluating sentence fluency by considering factors such as pauses, rep- etitions, and stammering in speech as adapted (Zhang et al., 2021c).
Ara VoiceL2 is an Arabic mispronunciation detec- tion corpus comprised of 5.5 hours of data recorded by 11 non-native Arabic speakers. Each speaker recorded a fixed list of 642 words and short sen- tences, making for total of 7,00 7,062 recordings. The corpus is annotated at character level including dia- critics following (Zhang et al., 2021b) guidelines.
A.16 Non-Native Datasets:
Table 2 provides a comprehensive overview of ex- isting non-native datasets that are particularly bene- ficial as they enable the extraction of eттor patterns allowing for a thorough assessment of L2 pronun- ciation. These datasets can also be used to train robust ASR models, from which we extract val- uation features to accurately score L2 speech. Fur- thermore, non-native datasets can enhance existing pronunciation assessment end-to-end approaches.
B Annotation
this section, we provide an overview of the stan- dard approaches to annotate segmental and supra- segmental errors widely used in MDD research.
B.1 Segmental Annotation
Segmental human annotation can be approached from two perspectives. The first and the com- monly utilized approach in most available MDD

---

Corpus
Languages
Dur / #Utt
#Speakers
Demuth
Sesotho Corpus
(Demuth,
Sesotho
98h // 13250
4
1992)
TIDIGITS (Lconard and Doddington. 1993)
English
/ 326
CMU Kids Corpus (Eskenazi al.,
English
/ 5180
76
1997)
CU Children's Read and Prompted
English
100
663
Speech Corpus (Hagen et al., 2003)
CU Story Corpus (Hagen al., 2003)
English
40h / 7062
106
PF-STAR Children's Speech Corpus
English
14.5h
158
(Batliner al., 2005)
TBALL (Kazemzadeh et al., 2005)
English
40h / 5000
256
Swedish NICE Corpus (Bell et al.,
Swedish
5580
2005)
Providence Corpus (Demuth et al.. 2006)
English
363h /
6
Lyon Corpus (Demuth and Tremblay, 2007)
French
185h
4
CHIEDE (Garrote, 2008)
Spanish
8h / 15444
59
CFSC (Pascual and Guevara, 2012)
Filipino
8h
57
CASSCHILD (Gao al., 2012)
Mandarin
23
CALL-SLT (Rayner al., 2014)
German
/ 5000
Boulder Learning-yr Corpus (Boulder Learning Inc, 2019)
English
393h / 228874
1371
TLT-school (Gretter et al., 2020)
English and German
119.1h 19019 26059
6547
Table 2: Non-Native Speech Datasets


Reviews:
[{"review_id": "nrbi4AxOR6", "reviewer": "Reviewer_YAq7", "summary": "", "questions": "No questions on my side.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the encouraging feedback.\n\n---------------------------------------------------------------------------------------------------------------------\nQ1: I would prefer a longer section on evaluation metrics with some quantitative evidence on which metrics are prevalent in the surveyed papers\n\nR1: We agree with your suggestion regarding a longer evaluation metrics section. Due to the limited page number in the initial submission, we couldn't provide more details on the reported evaluation metrics. We plan to address this concern in the camera-ready version of the paper. We will add more details on the hierarchical evaluation metrics, including detailed definition and equation (in main paper) and examples in Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The survey is well-structured and clearly written. \nApart from being a survey it is also a nice introduction to automatic pronunciation assessment for someone new to the field. \nAlso, Figure 2 deserves a special credit.", "reasons_to_reject": "I do not see any major reason to reject this paper.\nI would prefer a longer section on evaluation metrics with some quantitative evidence on which metrics are prevalent in the surveyed papers, but the lack of such information does not disqualify this paper.", "questions_to_authors": "No questions on my side.", "paper_topic_and_main_contributions": "This is a survey paper on automatic pronunciation assessment.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EtwIBmZooG", "reviewer": "Reviewer_NoL5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the feedback.\n\n---------------------------------------------------------------------------------------------------------------------\nQ1: This may be due to length constraints, but the paper requires extensive familiarity with recent work in acoustic modelling. Techniques are mentioned without much explanation. The target audience is not clear, but would seem to be experts in speech who wish to get up to date on automatic pronunciation assessment. Other readers would have to do a good deal of background reading to follow this review. As an example, section 4.2 gives a one paragraph summary of work on 'Extended Recognition Networks (ERN)' without explaining what ERNs are.\n\nR1: The paper serves as a comprehensive introduction to the domain of automatic pronunciation assessment, and offers a valuable starting point for individuals entering this field with a background in speech. We meticulously outlined the diverse facets of pronunciation assessment, covering both phonetic and prosodic aspects of the field. We covered a comprehensive survey of old, revised, and current methodologies employed in modeling pronunciation assessment . Our objective is to introduce readers to a diverse range of existing methods, and resources in automatic pronunciation assessment.\n\n---------------------------------------------------------------------------------------------------------------------\n---------------------------------------------------------------------------------------------------------------------\nQ2: The authors claim 'a discussion of remaining challenges and possible directions for future work' as one of their main contributions. These are addressed only very briefly in section 6 (lines 594-604) as 1) lack of public resources and 2) unified evaluation metrics. \n\nR2: We would like to point it out that in addition to the challenges mentioned above (public resource and evaluation metrics), we also addressed the challenges in handling the following:\nChildren's Speech (lines 636-642) \nThe complexity introduced by the lack of a standard orthography for various languages/dialects (lines 646-648).\nMoreover, we emphasised the pivotal role of integrating Generative AI models to raise the reliability of CAPT tutoring (lines 608-618). \n\nWe highlighted the significance of Multilingualism (lines 619-635), which facilitates not only the extension of knowledge across languages but also the transfer of insights to languages that are resource-scarce (lines 629-632). \n\nWe acknowledge that we've provided an overview of existing challenges and a glimpse into future prospects. The essence of our paper centers on broad challenges rather than delving into specific, tailored ones.\n\n---------------------------------------------------------------------------------------------------------------------\n---------------------------------------------------------------------------------------------------------------------\nQ3: `With large transformer-based pre-trained models gaining popularity, re-visiting the existing literature and presenting a comprehensive study of the field is timely.' However , apart from sections 4.5-4.8, much of the paper is devoted to earlier work.\n\nR3: We would like to draw the reviewer's attention  regarding the content distribution within the paper. Sections of \"previous approaches\" (4.1-4.3) account for approximately half a page, the subsequent sections discussing \"recent approaches\" (4.4-4.8) are significantly more extensive, spanning around three pages, in which we offered a comprehensive depiction of the existing landscape concerning pronunciation assessment approaches in the current context.\n\nFurthermore, within the old/previous approaches, we discussed notable and recent advances in those framework/ techniques. For instance, in Section 4.4, specifically within (lines 315-323) and (lines 328-334), we presented strides in the realm of GOP reformulation. \n\n---------------------------------------------------------------------------------------------------------------------\n---------------------------------------------------------------------------------------------------------------------\n\nQ4: Table 1 includes SOTA results under a range of metrics for a collection of data sets. These aren't really described in enough detail to be useful. Some are defined in Section 5, although superficially, e.g at line 582 'Precision, Recall, and F-measure are widely used to evaluate the performance of MD models.' and some are not explained at all (e.g. 'Min-Cost per phone'). \n\nR4:  We strongly believe that the information presented in Table 1 is a valuable representation of an extensive exploration across various benchmarks and datasets, summarising prevalent works. The table highlights how each dataset in the literature is  associated with distinct (sometimes unorthodox) evaluation metrics, which we have taken great care to meticulously outline in the 'SOTA' column along with concise results. \nIt is important to acknowledge the plethora of evaluation metrics utilised in this field, many of which lack standardisation (e.g., Min-Cost per phone). For brevity, we were unable to provide an exhaustive elaboration on all these evaluation metrics and compensated this limitation by including references for further reading. \nWe focused on the most widely used and recognised metrics such us (Phoneme Error Rate, F-measure, Precision, Recall, and Pearson Correlation Coefficient). These measures are becoming the de-facto metrics in the field over the last few years. \nWe will give more details of these standard metrics in the  camera-ready version; we plan to enhance this section of evaluation metrics by incorporating comprehensive definitions and relevant equations to provide a more thorough understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper assembles a large number of recent and not so recent papers into a systematic presentation that could be of use to speech processing experts who wish to review recent developments in automatic pronunciation assessment.   The literature covered is extensive and covers many aspects of the problem. ", "reasons_to_reject": "This may be due to length constraints, but  the paper requires extensive familiarity with recent work in acoustic modelling.    Techniques are mentioned without much explanation.    The target audience is not clear, but would seem to be experts in speech who wish to get up to date on automatic pronunciation assessment.   Other readers would have to do a good deal of background reading to follow this review.   As an example,  section 4.2 gives a one paragraph summary of work on 'Extended Recognition Networks (ERN)' without explaining what ERNs are.\n\nThe authors claim 'a discussion of remaining challenges and possible directions for future work' as one of their main contributions.   These are addressed only very briefly in section 6 (lines 594-604) as 1) lack of public resources and 2) unified evaluation metrics.    Neither point is profound or original.   The remainder of the section outlines possible applications that would become possible once computer-aided pronunciation training systems were viable.\n\nAnother motivation for the work is `With large transformer-based pre-trained models gaining popularity, re-visiting the existing literature and presenting a comprehensive study of the field is timely.'   However , apart from sections 4.5-4.8,  much of the paper is devoted to earlier work.  This isn't necessarily bad,  but the novelty relative to previous surveys isn't clear.\n\nI have made some notes in the 'typos' sections below.  I would say that without extensive editing the paper is not presentable at EMNLP. \n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an extensive review of recent and past literature in computer aided pronunciation training.   The paper is authoritative:  there is an overview of ' nuances of pronunciation' (section 2) with a collection and summary description of widely used data sets (Table 1).    The  modelling approaches in the 'Research Avenues' (section 4) are well chosen,  although the range and variety of material precludes  in-depth discussion.   When the paper does go into detail (e.g. description of Hierarchical Evaluation Structure in section 5),  the presentation is clear and informative.    \n\nThe main contributions are claimed as:  revisit the literature following recent developments (line 98),  review qualitative studies that address intelligibility, comprehensiveness, and 'accentednes' (line 106), a review of available resources and evaluation measures (line 109), and suggestions for future work.", "excitement": 2, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some unusual bibliography keys (e.g. 'cam', 'sin') and formatting errors (particularly casing) that should be fixed, as well as some incomplete entries (e.g. Caro 2023).\n\nGrammatical error in abstract:  '...for both phonemic and prosodic.'\n\nThe title and/or abstract should make more clear that there is a focus on L2 pronunciation.\n\nTable 1 includes SOTA results under a range of metrics for a collection of data sets.  These aren't really described in enough detail to be useful.   Some are defined in Section 5,  although superficially, e.g  at line 582  'Precision, Recall, and F-measure are widely used to evaluate the performance of MD models.'  and some are not explained at all (e.g. 'Min-Cost per phone').     \n\nSome acronyms (e.g. 'SSL') need definition."}}, {"review_id": "ParhZYbH5Y", "reviewer": "Reviewer_hbGh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the  constructive and positive feedback concerning our paper. Your recognition of its strengths is truly valued. \n\nWe would like to highlight the distinctive contributions that our paper brings to the existing body of literature: Within this paper:\n(i) We offered a comprehensive exploration of pronunciation assessment, encompassing both phonetic and prosodic errors. (ii) Our approach fills a critical void left by previous methodologies that predominantly concentrated solely on the phonetic aspect of pronunciation assessment. (iii) Furthermore, we presented an extensive overview of previous, contemporary, and emerging methodologies in the field of pronunciation assessment, spanning the years 2013 to 2023. (iv) We also meticulously outlined the existing datasets employed in this domain, highlighting their characteristics and the corresponding up-to-date state-of-the-art results we have documented. (v) To provide further value, we included a range of datasets in the appendix that are pertinent to this domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written and provides a focussed but reasonably comprehensive survey of published work on pronunciation assessment.  It will provide a valuable entry point to anybody entering the field of pronunciation assessment.  The information on the available datasets is also useful.  \n", "reasons_to_reject": "There is nothing original in the paper but equally there is nothing to object to.  \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper provides a review of work on pronunciation assessment over the last two decades.  It discusses the types of error that can occur, training and evaluation datasets, and the historical development of assessment methods spanning HMM based likelihood measures to modern self-supervised pre-trained end-to-end models.  It is very high level and essentially a literature review.\n", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Originality**

The presented paper is a very well written literature review of automatic pronunciation assessment. As such it is not original in and of itself.

**Significance**

The paper is a nice introduction to automatic pronunciation assessment for someone new to the field with some descriptive figures and tables thoroughly covering past approaches and describing current and future challenges. This is a useful resource though not something that has much immediate impact on the field.

**Clarity**

This paper is extremely thoughtful and well written.  

**Pros:**
   - A very valuable introduction for those trying to enter the field of automated pronunciation assessment
   - Details current and past approaches as well as future challenges
   - The paper is incredibly well written.

**Cons:**
   - There are no new methods, data, etc., in the paper
   - Limited excitement
Decision: Accept-Findings