Automatic proficiency assessment of Korean speech read aloud by non‐natives using bidirectional LSTM‐based speech recognition

ETRI JournalVolume 42, Issue 5 p. 761-772 ORIGINAL ARTICLEFree Access Automatic proficiency assessment of Korean speech read aloud by non-natives using bidirectional LSTM-based speech recognition Yoo Rhee Oh, Corresponding Author Yoo Rhee Oh yroh@etri.re.kr orcid.org/0000-0002-1557-0538 Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea Correspondence Yoo Rhee Oh, Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea. Email: yroh@etri.re.krSearch for more papers by this authorKiyoung Park, Kiyoung Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorHyung-Bae Jeon, Hyung-Bae Jeon Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJeon Gue Park, Jeon Gue Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author Yoo Rhee Oh, Corresponding Author Yoo Rhee Oh yroh@etri.re.kr orcid.org/0000-0002-1557-0538 Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea Correspondence Yoo Rhee Oh, Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of Korea. Email: yroh@etri.re.krSearch for more papers by this authorKiyoung Park, Kiyoung Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorHyung-Bae Jeon, Hyung-Bae Jeon Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this authorJeon Gue Park, Jeon Gue Park Artificial Intelligence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon, Rep. of KoreaSearch for more papers by this author First published: 15 April 2020 https://doi.org/10.4218/etrij.2019-0400Citations: 8 Funding information This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2019-0-00004, Development of semi-supervised learning language intelligence technology and Korean tutoring service for foreigners). [Correction added on 22nd April 2020, after first online publication: The images in the author biography section were matched to the wrong authors and have been corrected.] AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This paper presents an automatic proficiency assessment method for a non-native Korean read utterance using bidirectional long short–term memory (BLSTM)–based acoustic models (AMs) and speech data augmentation techniques. Specifically, the proposed method considers two scenarios, with and without prompted text. The proposed method with the prompted text performs (a) a speech feature extraction step, (b) a forced-alignment step using a native AM and non-native AM, and (c) a linear regression–based proficiency scoring step for the five proficiency scores. Meanwhile, the proposed method without the prompted text additionally performs Korean speech recognition and a subword un-segmentation for the missing text. The experimental results indicate that the proposed method with prompted text improves the performance for all scores when compared to a method employing conventional AMs. In addition, the proposed method without the prompted text has a fluency score performance comparable to that of the method with prompted text. 1 INTRODUCTION Owing to the rising demand for second-language learning and the advances in machine learning, there has been increase in the need for spoken computer-assisted language learning (CALL) applications [1, 2]. Moreover, with the spread of Korean popular culture overseas [3], the need for Korean language learning has prompted the development of such CALL applications for non-native Korean learners. Among the spoken Korean CALL applications, this paper focuses on an automatic speech recognition (ASR)–based proficiency assessment for non-native Korean speech. Non-native speech significantly degrades the performance of the ASR used in a spoken CALL owing to the pronunciation variabilities in non-native speech [4, 5]. Consequently, numerous research results have been reported on automatic proficiency assessment methods for non-native speech that is read aloud [6-13] and for spontaneous speech [14-17]. However, there has been limited research on proficiency assessment of non-native Korean speech [18]. Moreover, most research has been focused on the analysis of pronunciation variabilities in non-native Korean speech. For instance, [19, 20] analyzes the pronunciation variabilities of Korean spoken by Japanese and Chinese learners using contrastive and corpus-based statistical approaches. In addition, [21, 22] examine the correlation of the proficiency and analytic scores of non-native Korean speech in a read and a spontaneously spoken corpus. Fewer studies have been conducted on automatic proficiency assessment of non-native Korean speech. That is, [20] presents an ASR system for Korean utterances by Chinese speakers by modeling a non-native pronunciation dictionary and [23] examines the performance of an oracle pronunciation assessment of Korean utterances by Chinese speakers by using manual transcription. Moreover, [24] proposes a pronunciation assessment method for non-native Korean utterance using a selection of pronunciation scoring features. In addition, [25] proposes a pronunciation training method for non-native speakers by using self-imitating feedback based on generative adversarial network (GAN). This study aims to develop an automatic spoken proficiency assessment system for Korean utterances read aloud by non-native speakers. To this end, we present two spoken proficiency assessment methods: (a) a method with prompted text and (b) a method without prompted text. The proposed method with the prompted text is designed for utterances where the prompted text is known because the user speaks the text provided by a CALL system. The proposed method comprises (i) a speech feature extraction step, (ii) a forced-alignment step, and (iii) a linear regression-based proficiency scoring step. That is, the speech features are extracted from the input speech, and then the speech features and the prompted text are decoded into two time-aligned sequences by forced-alignment using the bidirectional long short–term memory (BLSTM)–based native acoustic model (AM) and the BLSTM-based non-native AM, respectively. The native AM is trained with a native speech corpus, whereas the non-native AM is trained with a non-native speech corpus. The latter's data are augmented using a speed perturbation method [26] and a frequency and time masking method [27], as the number of non-native data is far less than that of native data. Next, the two time-aligned sequences are converted into proficiency scoring features and then these features are fed into five linear regression–based proficiency scoring models, which output five scores (holistic impression of proficiency, segmental accuracy, phonological accuracy, fluency, and pitch and accent). The proposed method without the prompted text is designed for utterances when the prompted text is not known because the user reads the text from his or her textbook. To replace the missing prompted text, a Korean ASR step is additionally performed before the forced-alignment step. In the Korean ASR step, the speech features of the input speech are decoded into a subword-based text by a Viterbi decoding using a BLSTM-based AM and an n-gram–based language model. The AM for the Korean ASR is trained with native and non-native speech corpora and its augmented data. Next, the recognized subword-based text is converted into word-like text by performing a word un-segmentation (WUS) method [28]. Subsequently, the word-like recognized text is fed into the forced-alignment step as the input text. The contributions of this paper are summarized as follows: A Korean proficiency assessment system: An automatic proficiency assessment system of non-native Korean speech has not yet been thoroughly presented. Therefore, this paper presents an automatic spoken proficiency assessment system for Korean utterances read by non-native speakers. Use of BLSTM-based AMs: For better performance, BLSTM-based AMs are employed instead of the Gaussian mixture model (GMM)–based AM or the feed forward deep neural network (DNN)–based AM. From the proficiency assessment experiments, the proposed method employing BLSTM-based AMs improves the averaged correlation coefficients across five scores by 0.052 when compared to a system employing GMM-based AM. In addition, BLSTM-based AMs improve the performance for segmental accuracy by 0.016 when compared to DNN-based AMs. Use of speech data augmentation: For better performance, a speed perturbation method [26] and a frequency and time masking method [27] are utilized. From these ASR experiments, it is found that the use of augmentation methods improves the error rate reduction (ERR) by 9.34% and 16.14% for native speech and non-native speech, respectively. Use of WUS: The WUS method [28] is utilized for the proposed Korean proficiency assessment method without prompted text. It matches the basic unit between the subword-based text that is decoded from the speech recognition step and the word-based text that is entered at the proficiency assessment step. From the proficiency assessment experiments, the subword-based text degrades the performance by 0.015 when compared to the word-based text for the proficiency assessment step. Performance comparison of the proposed methods The performance of a proficiency assessment with prompted text is compared to a proficiency assessment without prompted text, which has not been performed previously. First, the proposed methods, with and without prompted text, attain comparable performance scores for fluency. As the fluency score is solely based on speaking rates, it can be concluded that its performance is robust though prompted text is not presented. Second, the proposed method without prompted text demonstrates reduced performance for the holistic, segmental accuracy, and phonological accuracy scores. Even though the ASR-decoded text is incorrect, the ASR-decoded text could achieve a high acoustic score and this could lead to a misestimation of proficiency scores that are highly related to acoustic scores. The organization of this paper is as follows. Section 2 describes a Korean read speech corpus for a spoken proficiency assessment and Section 3 presents the proposed proficiency assessment method with prompted text and the method without prompted text. Section 4 shows performance comparisons of the proposed proficiency assessment methods for Korean speech read by a non-native. Finally, we conclude our findings in Section 5. 2 A KOREAN READ SPEECH CORPUS FOR A PROFICIENCY ASSESSMENT The corpus of Korean read speech for the spoken proficiency assessment consists of 2500 utterances by 50 non-native speakers and 500 utterances by 10 native speakers. Each speaker utters 50 sentences and the speech data are recorded at a rate of 16 kHz. In particular, all utterances were carefully recorded so as not to have reading errors such as insertions or deletions. The non-native speakers were Asians from China, Japan, Cambodia, Vietnam, and the Philippines. The gender and spoken language proficiency levels were evenly distributed among the speakers. Moreover, the 50 sentences were designed for the spoken language proficiency assessment and contained various phonological rules, of which a detailed description is provided in [21]. Each non-native utterance was annotated by four human experts for the five proficiency area scores: holistic impression of proficiency (sholistic), segmental accuracy (ssegment), phonological accuracy (sphonology), fluency (sfluency), and pitch and accent (spitch). Each score was measured on a scale of 1–5 with 1, 2, 3, 4, and 5 indicating very poor, poor, acceptable, good, and perfect, respectively. Accordingly, each utterance from the non-native corpus was annotated with 20 scores. Conversely, each native utterance was not annotated but was recorded as a 5 (top score). The detailed description of the proficiency score is found in [21]. Throughout this study, a proficiency score ( ) of the i-th utterance for the score type (t) is calculated as an average as follows: (1)where t represents either holistic, segment, phonology, or fluency and pitch. is the score of the i-th utterance for type t by the r-th human expert. Furthermore, we use a 10-fold cross-validation test method owing to the small size of the speech assessment corpus. 3 PROPOSED SPOKEN PROFICIENCY ASSESSMENT METHODS WITH AND WITHOUT PROMPTED TEXT Figure 1 shows the overall procedure of the proposed automatic proficiency assessment methods with and without prompted text for Korean speech read aloud by a non-native speaker. The dotted areas of the figure indicate additional components for the proposed method without prompted text. FIGURE 1Open in figure viewerPowerPoint Comprehensive procedure for the proposed automatic proficiency assessment methods with and without prompted text for Korean speech read aloud by a non-native. First, the proposed method with prompted text consists of (a) the speech feature extraction step, (b) the forced-alignment step, and (c) the proficiency scoring step. The proposed method without prompted text consists of (a) the speech feature extraction step, (d) the Korean ASR step, (b) the forced-alignment step, and (c) the proficiency scoring step. The additional components of the proposed method without prompted text are depicted using a dotted line The proposed method with prompted text consists of (a) the speech feature extraction step, (b) the forced-alignment step, and (c) the proficiency scoring step. In the speech feature extraction step, we initially extracted the 40-dimensional log mel filterbanks for each 10 ms analysis frame of the speech input. Next, we stacked the features of the previous seven frames and the subsequent seven frames, which resulted in a 600-dimensional speech feature for each 10 ms analysis frame. In the forced-alignment step, the speech features and prompted text were decoded using a forced-alignment algorithm using the BLSTM-based native AM and the BLSTM-based non-native AM, respectively, which resulted in two time-aligned sequences. Each time-aligned sequence contained: (a) the start and end time for each word of the prompted text, (b) the start and end time for each phoneme of the prompted text, (c) the acoustic scores of the null hypothesis (H0) [29], and (d) the acoustic scores of the alternative hypothesis (H1). In the proficiency scoring step, the proficiency scoring features were extracted from two time-aligned sequences. Then, the five proficiency scores were calculated using the proficiency scoring features and the five linear regression–based proficiency scoring models. A proficiency score, st, was calculated with a corresponding linear regression–based proficiency scoring model, , by performing the following equation: (2)where t is one of holistic, segment, phonology, fluency, or pitch, and and indicate the number of features and the i-th feature value, respectively. In addition, and bt indicate the i-th coefficient and the bias of . When a user reads aloud not the prompted text of a CALL system but his or her own text, the proposed method without prompted text consisting of (a) the speech feature extraction step, (b) the Korean ASR step, (c) the forced-alignment step, and (d) the proficiency scoring step can be used. In the speech feature extraction step, we extracted a 600-dimensional speech feature for each frame, similar to the proposed method with prompted text. In the Korean ASR step, the speech features were decoded into the subword-based text by a Viterbi decoding algorithm using the BLSTM-based AM and an n-gram–based language model. Then, the subword text was converted into the word-like text utilizing a WUS method [28]. In the forced-alignment step, the speech features and the recognized word-like text were decoded into the two time-aligned sequences. This step is similar to that of the proposed method with prompted text except for the input text. In the proficiency scoring step, we calculated the five scores, similarly as for the proposed method with the prompted text. Briefly, the main differences between the proposed methods with and without the prompted text are: (a) use of the Korean ASR step and (b) the input text of the forced-alignment step. The following subsections provide detailed descriptions of the BLSTM-based AMs used in the forced-alignment step, speech data augmentation, the proficiency scoring features and models, as well as BLSTM-based AMs and language model for the Korean ASR step. 3.1 BLSTM-based native and non-native acoustic models of the forced-alignment step This subsection explains the BLSTM-based native AM and the BLSTM-based non-native AM that were used in the forced-alignment step. The proposed method uses the BLSTM-based native AM to determine whether a speech input is similar to native speech and uses the BLSTM-based non-native AM to obtain more accurate time information for the prompted or recognized text. The BLSTM-based native AM consists of one input layer, five BLSTM layers, a fully connected layer, and a soft-max layer. It was trained with 670k clean and native Korean utterances (738 hours) using a Kaldi toolkit [30]. Because any noisy or augmented speech could increase the confusion of similarity detection between speech input and native speech, no data augmentation was applied. Meanwhile, the BLSTM-based non-native AM has the same structure as the native AM and was trained with 347k non-native Korean utterances (529.4 hours) and with the augmented non-native speech data. The augmented speech data included five additional copies provided by performing speed perturbation [26] and frequency and time masking [27], which are described in Section 3.2. 3.2 Speech data augmentation for the BLSTM-based acoustic models This subsection explains two speech data augmentation methods that were used to train the BLSTM-based AMs, whereas data augmentation techniques are commonly utilized for acoustic modeling to increase the quantity of training data and to reduce overfitting [26]. A speed perturbation method [26] was used with speech factors of 0.9 and 1.1 by using a SoX tool [31]. In addition, a frequency and time masking method was adopted from [27] and modified as shown in Algorithm 1. As shown in the pseudocode, the main divergence from the work of [27] is that the parameters of [27] were randomly selected by using the parameters, rF, rT, mF,max, and mT,max, which were set empirically as 0.15, 0.15, 2, and 2, in this study. Training data have increased sixfold after speed perturbation followed by the frequency and time masking. 3.3 Proficiency scoring features and linear regression–based scoring models This subsection explains the proficiency scoring features and the linear regression–based proficiency scoring models of the proficiency scoring step. The proficiency scoring features were extracted from the two time-aligned sequences of the forced-alignment step and they were classified into frequency- or duration-related features, syllabic features, and acoustic features, as shown in Tables 1, 2, and 3, respectively. The frequency- or duration-related features were adopted from [32] and extracted using the duration and the number of each token. The syllabic features were extended from [17, 33] and extracted using the duration of each vowel, consonant, syllable, syllable onset, and syllable coda. The acoustic features were extracted using the duration and the acoustic scores for each phoneme, for which features in the form of Lx were extended from [17]. TABLE 1. The frequency or duration related features [32] Item Description Numwds No. of non-silence words Numsil No. of silence Longpfreq No. of longp Wdpchk Mean of (no. of words of chk) Wdpchkmeandev Mean deviation of (no. of words of chk) Globsegdur Dur. of segment uttsegdur Dur. of speech segment Segdur Dur. of speech segment without pauses Secpchk Mean of (dur. of chk) Longpmn Mean of (dur. of longp) Secpchkmeandev Mean deviation of (dur. of chk) Silmeandev Mean deviation of (dur. of silences) Longpmeandev Mean deviation of (dur. of longp) Silstddev Standard deviation of (dur. of silences) Longpstddev Standard deviation of (dur. of longp) Wpsec Numwds/Segdur Wpsecutt Numwds/uttsegdur Silpwd (Dur. of silences)/Numwds Silmean (Dur. of silences)/Numsil Silpsec (Dur. of silences)/Wpsec Longpwd (Dur. of longp)/Numwds Tpsec (Dur. of unique words)/Segdur Tpsecutt (Dur. of unique words)/uttsegdur longp: Long pause, a silence with a duration longer than 0.2 second. chk: Speech chunk, a speech segment that is interrupted by a long pause. dur: Duration, which is measured in seconds. TABLE 2. The syllabic features [17, 33] Item Description percentX (Duration of X)/(Duration of phonemes) meanX Mean of (Duration of X) stddevX Standard derivation of (Duration of X) varcoX Variance of (Duration ofX) rPVIX nPVIX X: one of {Vowel, Consonant, Syllable, Onset, Coda}. xk: Duration of the k-th X. : No. of X. TABLE 3. The acoustic features [17] Item Description AMtoks,H0 Acoustic score of tokens AMtoks,H1 Anti-model acoustic score of tokens AMwds,H0 Acoustic score of non-silence tokens AMsil,H0 Acoustic score of silence words SLLR Sentence-level log-likelihood ratio L 1, X L 2, X L1/Numwds L 3, X L1/(No. of syllables) L1/(No. of phones) L 4, X L1/uttsegdur L 5, X L 6, X L4,X/R L 7, X L5,X/R X: one of {H0, H1, WLLR}. R: uttsegdur/(No. of phones). ti: the duration of the i-th word. For the linear regression–based proficiency scoring model, , we use the training set of the non-native and native speech assessment corpus. For each utterance, the 600-dimensional speech features were extracted and the two time-aligned sequences were obtained by the forced-alignment algorithm applied to the prompted text or the recognized word-like text. In addition, the target score of each utterance for st were prepared as the average value described in (1). Then, was trained with the two time-aligned sequences and the target score of the training data using a toolkit R [34]. Accordingly, we obtained the five linear regression models for the five proficiency scores, sholistic, ssegment, sphonology, sfluency, and spitch. 3.4 BLSTM-based AM and language model for the Korean ASR step This subsection explains the BLSTM-based AM and the n-gram–based language model for the proposed method without prompted text. Unlike the BLSTM-based native AM and BLSTM-based non-native AM of Section 3.1, the BLSTM-based native and non-native AM used in the Korean ASR step was intended to include as many pronunciation variabilities as possible to recognize both native and non-native speech. The BLSTM-based AM had the same structure as the BLSTM-based AMs for the forced-alignment step and it was trained with 3097k native Korean utterances (3439 hours), the 347k non-native Korean utterances, the 1000-hour augmented native speech data, and the 1000-hour augmented non-native speech data. The 1000-hour augmented native speech data were obtained by utilizing speed perturbation and frequency and time masking, and by performing a random sampling of the 1000-hour data. Moreover, the 1000-hour augmented non-native speech data were obtained in the same manner. It is noted that random sampling was used to supplement limited computational resources. The n-gram–based language-model is a back-off trigram of 541k subwords, in which a subword is a commonly used unit of a language model for an agglutinative language such as Korean [28]. The language model was trained with approximately 30 GB of text data including newspapers, web-sourced texts, etc. The text data were first preprocessed using a text normalization method and a word segmentation (WS) method [28], and then the most frequent 541k subwords were obtained. Next, the back-off trigram for the 541k subwords was trained with the preprocessed text data using a SRILM toolkit [35, 36]. 4 EXPERIMENTS To evaluate the proposed automatic spoken Korean proficiency assessment methods, two metrics, Pearson's correlation and root mean square error (RMSE), were used for comparing the target score of (1) and the score of the proficiency assessment system. The Pearson correlation is a metric commonly used to evaluate the performance of proficiency assessment methods, while the RMSE is used to evaluate the performance of each score because the score distribution is not balanced. We also present the averaged values over the 10-fold cross-validation test for each metric. 4.1 Performance comparison of the proposed spoken proficiency assessment method with prompted text In this subsection, we discuss the experiments using the proposed spoken Korean proficiency assessment method with prompted text: (a) comparison of the BLSTM-, the DNN-, and the GMM-based AMs and (b) comparison when using different text pre-processing methods. To compare the performance of the proposed proficiency assessment method with the prompted text, we developed a proficiency assessment method employing the GMM-based native AM and the GMM-based non-native AM, of which GMM-based AM was extensively used in an earlier study [17]. To this end, we trained the GMM-based native AM with the 670k clean and native Korean utterances and the GMM-based non-native AM with 347k non-native Korean utterances by using the HTK toolkit [37]. As a speech recognition feature, we used the 39-dimensional mel-frequency cepstral coefficient (MFCC) [38]. Next we developed another proficiency assessment method employing the DNN-based native and non-native AMs, as DNN-based AM is increasingly being adopted in this research area [17]. We trained the DNN-based native AM with the 670k clean and native Korean utterances and the DNN-based non-native AM with 347k non-native Korean utterances. The structure of DNN-based AMs consisted of one input layer, five DNN layers, a fully connected layer, and a soft-max layer. Except for the model structure, the training procedure was identical to the BLSTM-based AMs. Table 4 shows that the proficiency assessment system employing the proposed method with prompted text using the BLSTM-based AMs improved the averaged correlation across the five scores by 0.052 and 0.007 when compared to the systems using the GMM-based AMs and using the DNN-based AMs, respectively. Moreover, it can be seen from the first row of the table that the performance of the proposed system is comparable with that of inter-rater scores. It can be seen from Table 5 that the proposed proficiency assessment system improved the averaged RMSEs for most score ranges when compared to the system using the GMM-based AMs. It is also noted that the proposed system significantly improved the RMSEs for ssegment and sphonology when compared to the system using the DNN-based AMs. Therefore, it can be concluded that the BLSTM-based AMs significantly improve performance when compared to the GMM-based AMs and the DNN-based AMs. TABLE 4. Comparison of the averaged Pearson correlation coefficients of the automatic spoken Korean proficiency assessment systems employing the proposed method with prompted text using the BLSTM-based AMs, using the DNN-based AMs, and using the GMM-based AMs, respectively, for the 10-fold cross-validation test of the non-native speech assessment corpus. The first row presents the range of correlation coefficients of the inter-rater scores as an upper boundary AM s holistic s segment s phonology s fluency s pitch Human 0.589–0.706 0.545–0.673 0.615–0.665 0.611–0.715 0.383–0.518 GMM 0.683 0.643 0.664 0.855 0.506 DNN 0.745 0.709 0.723 0.856 0.542 BLSTM 0.752 0.726 0.735 0.857 0.540 TABLE 5. Comparison of the averaged RMSEs of each score range of the automatic spoken Korean proficiency assessment systems employing the proposed method with the prompted text using the BLSTM-based AMs, the DNN-based AMs, and using the GMM-based AMs, respectively, for the 10-fold cross-validation test of the non-native speech assessment corpus ≤1.5 ≤2.5 ≤3.5 ≤4.5 ≤5 The proposed method using the GMM-based AMs s