A study on English speech acclimatization based on accent conversion for
non-native speaker
Yutao ZHANG(1) , Takuro SASAKI(1) , Yukoh WAKABAYASHI(2) , Takahiro FUKUMORI(1) , Takanobu NISHIURA(1)(1) Ritsumeikan University, Japan, gr0320fr@ed.ritsumei.ac.jp, is0215kk@ed.ritsumei.ac.jp, fukumori@fc.ritsumei.ac.jp,

nishiura@is.ristumei.ac.jp(2) Tokyo Metropolitan University, Japan / Ritsumeikan University, Japan, wakayoko@fc.ritsumei.ac.jp

ABSTRACT
It is considered that listening and understanding English speech of native speakers are hard tasks for manyJapanese. Therefore, we carry out a research on converting the English speech spoken by native speakers basedon emphasis of the feature which is familiar to listening to Japanese. We believe the converted speech could
be understood more easily by Japanese listeners for English education. We called the research English speechacclimatization. In this paper, we proposed an accent conversion method as an English speech acclimatization
method. Specifically, we convert the accent of native speakers into that of Japanese speakers. The acousticfeatures are time stretched to match the same phoneme duration with the English speech of Japanese speaker.
Then the pitch change in English speech of native speaker is replaced with the that in English speech of Japanesespeaker. Experiments are conducted to evaluate the performance of proposed method based on the English
speech before and after conversion. To confirm the effectiveness on words recognition and comprehensionof the sentence, subjective evaluation experiments are conducted respectively. As the result of the evaluation
experiments, the effectiveness of proposed method was confirmed.
Keywords: Speech acclimatization, Accent conversion, Pitch, Phoneme duration1 INTRODUCTION
In times of globalization and internationalization, the opportunities for Japanese people to communicate inEnglish have increased. Nevertheless, because of the difference between speech perception processed by nativespeakers of difference languages, the communication in English is not an uncomplicated task for many Japanes.
According to the various countries’ TOEIC average score in terms of 2017 Report on Test Taskers Worldwide
provided by ETS (Educational Testing Service), Japanese has lower language proficiency in English [1].
To improve English communication skills for Japanese, we considered a training method based on the ideathat the ability of English listening would be improved by carrying on listening training with appropriate English speech according to each level of English listening proficiency. We believed by gradually eliminating thedifference between actual native English speech and the perception by non-native speaker, we can expect a goodtraining effect by the sense of accomplishment that trainees can gain easily in this kind of training. In orderto achieve this training, it is necessary to create English speech in graded listening difficulty. Meanwhile, theprimary goal is to create the English speech with the lowest listening difficulty for Japanese trainee. In otherwords, the aim of this study is to convert the English speech spoken by native speaker to better understand byJapanese. We called it English speech acclimatization.
In this paper, we propose an accent conversion method as an English speech acclimatization. Accent isvariously defined by researches, such as phoneme duration, pitch change, sound pressure, rhythm, etc. In thesefeatures, phoneme duration plays an important role in speech intelligibility [2]. Pitch change is an importantelement in Japanese language perception [3]. Based on these researches, we focus on phoneme duration andpitch change to convert accent in this study. In our method, acoustic features (the pitch change which is vocal
cord information and the MFCC (Mel-Frequency Cepstrum Coefficients) which is vocal tract information) in the
English speech of native speaker are time-stretched to keep the same phoneme duration with that in the English

6876

speech of Japanese speaker. Then the pitch change in English speech of native speaker is replaced with the thatin English speech of Japanese speaker. In the rest of this paper, we will discuss about the differences between
the accent in English speech of native speaker and Japanese speaker in Section 2. In Section 3, we introduce
the method of accent conversion proposed in this paper. In Section 4, experiments are conducted to evaluate
the performance of proposed method based on the English speech before and after conversion.

2 ACCENT IN ENGLISH SPEECH OF NATIVE SPEAKER AND JAPANESE SPEAKER
Accent is a complex expression via the prosodic features such as loudness, pitch and phoneme duration.
Generally, English accent belongs to stress accent which mainly depends on loudness, while Japanese accent isconsidered to belong to pitch accent which mainly depends on pitch change. Figs. 1, 2 and 3 show spectrogramsof the English speech of native speaker and the English speech of Japanese speaker. It is clear that the pitchchange is different when native speaker and Japanese speak the same English speech. Meanwhile, it is also
remarkable that the durations of voiced periods differ from each other. In the English speech of Japanesespeaker, there are many pauses (unvoiced parts) which are relatively spaced at regular intervals, while in theEnglish speech of native speaker, there are less pause. In conclusion, the differences between the accent ofEnglish and Japanese appear in the same way on English speech of native speaker and Japanese speaker. The
English speech of native speaker and Japanese speaker has difference phoneme duration and pitch change. Basedon this phenomenon, we believe that it is possible to realize the English speech acclimatization by converting
the accent in English speech of native speaker to that in English speech of Japanese speaker which Japaneseaccustomed.

3 SPEECH ACCLIMATIZATION BASED ON ACCENT CONVERSION
3.1 Overview of accent conversion
As we mentioned before, in order to achieve the English speech acclimatization, the accent in English speechof native speaker is replaced by that in English speech of Japanese speaker. In this paper, the accent refers tophoneme duration and pitch change. The pitch is operated through the operation of the fundamental frequency(F0). The outline of the accent conversion algorithm in the experiment is shown in Fig. 4. Here, f1 indicatesthe pitch of the native speech and f2 indicates the pitch of non-native speach. The acoustic features we used in
our proposed method are MFCC and pitch. The pitch extraction is performed by SWIPE (Sawtooth Waveform
Inspired Pitch Estimator) [4]. Based on SWIPE, the frequency that makes the maximum correlation value
between SWIPE kernel and the amplitude spectrum is calculated as the pitch. In order to keep the phoneme
duration of native speech and non-native speech consistent, the pitch and MFCC of native speech are timestretched to match the non-native speech. Then the stretched pitch is converted to mimic the non-native speech.
Finally, the converted MFCC and pitch are used to synthesize the converted speech with MLSA filter [5].
3.2 Feature time stretch
In order to keep the phoneme duration of native speech and non-native speech consistent, the acoustic features(pitch and MFCC) of native speech are linearly time stretched for each phoneme. The duration of each phoneme
is extracted with the speech segmentation toolkit of Julius [6]. The example of time stretch is shown in Fig. 5.
The time stretched feature can be defined as follows:
m01 (t) = (1at=bt= t/kk= (T2bt )m1 (at ) + bt m1 (at + 1),(2)

bt/kc,

bt/kc,1)/(T1

(1)
(3)

1),(4)

6877

(a) English speech of native speaker.

(b) English speech of Japanese speaker.

Figure 1. Spectrograms of the speech “When I came, he greeted me warmly.".

(a) English speech of native speaker.

(b) English speech of Japanese speaker.

Figure 2. Spectrograms of the speech “Is it John who writes poetry?".

(a) English speech of native speaker.

(b) English speech of Japanese speaker.

Figure 3. Spectrograms of the speech “I saw her this morning and invited her to dinner.".

6878

Figure 4. Overview of accent conversion algorithm.
Here, b·c represents a floor function. m01 (t) represents the tth element in the time stretched feature. m01 () is thetime stretched feature linearly interpolated between the samples taken from the m1 () and has the same duration
time as m2 (). at and at + 1 indicate two samples participates in interpolation, bt indicates the interpolation ratio
and k indicates the stretching factor. T1 and T2 correspond to the length of m1 () and m2 () respectively.
3.3 Pitch conversion
The time stretched pitch with the algorithm mentioned in Subsection 3.2 is converted as follows:log( f1!2 ) = {log( f2 )µ2 }s10+ µ10 .
s2(5)Here, µ10 and s10 indicate the time average and standard deviation of log( f10 ) and µ2 and s2 indicate thetime average and standard deviation of log( f2 ). f10 is the pitch which is time sched from f1 with the algorithm
in Subsection 3.1. The converted pitch f1!2 , which maintains the change of f2 , can be acquired through thenormalization with the mean and variance of f10 .
Finally, the acclimatized speech is resynthesized from the pitch and MFCC with MLSA filter [5].

4 EVALUATION EXPERIMENT
Two subjective evaluation experiments (Experiment A and B) were carried out to evaluate the effectivenessof English speech acclimatization by accent conversion. In Experiment A, dictation tests were conducted inorder to confirm the words recognition. In Experiment B, the multiple-choice tests were conducted based on
the response test in the TOEIC to confirm the effectiveness on comprehension of the sentences. In these experiments, the speech before and after accent conversion were compared according to the correct answer rate of
the questions. Note that speech distortion caused by speech synthesis interferes fair comparison between native
English speech and converted one in terms of comprehension of sentences and word recognition. In order toequalize the effect of the distortion, the native English speech used in these experiments was resynthesized withthe acoustic features extracted from the native English speech database.
4.1 Experiment condition
Since it is considered that the recognition performance will be improved when the utterance has been heard
several times before, in the experiments, two sets of evaluation speech were used to prevent the subjects fromlistening to same utterances. The speech before and after conversion was assigned to different sets, meanwhile
the number of pre-conversion speech and post-conversion speech in each set was the same. 10 subjects listened
to the speech of set 1 and remaining 10 subjects listened to the speech of set 2 with headphone. The sampling
frequency of the evaluation speech was 16 kHz, and number of quantization bits was 16 bits.

6879

Figure 5. Example of accoustic feature converion by time stretch.
4.1.1 Experiment AThe conditions of Experiment A are shown in Table 1. We used the UME-ERJ database [7] which contains
the English speech of Japanese speakers and native speakers. The sentences of the evaluation speech wereselected from the database. Subjects were asked to listen to 12 periods of speech and dictate all the words(67 words) they heard. Meanwhile, the inflections of words, such as past tense of verb, incorrectly dictated bysubjects would not be judged as mistakes. We also avoided the speech includes names when we chose it fromdatabase. The accuracy of dictation was calculated.
Table 1. Condition of Experiment A.
Native speech
Non-native speech
Question form
Times of listening
for each speech
Subjects

Native speaker English speechin UME-ERJ database [7]Non-native speaker English speechin UME-ERJ database [7]Dictation (12 questions)
220 Japanese students

4.1.2 Experiment B
Table 2 shows the experiment conditions of Experiment B. The speech in the response test of TOEIC support
material was used as the English speech of native speaker in this experiment. Meanwhile, a Japanese malein 20s was required to speak the same sentences as the English speech of Japanese speaker. We prepared 12questions in this experiment. In each question, as an example shown in Table 3, 20 subjects were asked tolisten to the speech and choose the best response they thought for the speech. The number of choices in eachquestion was 3.

6880

Table 2. Condition of Experiment B.
Native speech
Non-native speech
Question form

Speech inTOEIC support material [8]Speech spoken by a Japanese male in 20s
Multiple-choice test (12 questions)

Table 3. An example of response test in Experiment B.
SpeechQuestion

Whose file is this on my desk?
(A) No, that’s not my desk.
(B) I’ve been looking for that. Sorry!
(C) Mr. Smith called ten minute ago.

4.2 Experiment result
4.2.1 Experiment A
The experiment results are shown in Table 4. In the average of set 1 and set 2, the accuracy of dictations
with the pre-conversion speech is 59.6%, and that with the post-conversion speech is 63.7%, showing an improvement of 4.1%. This tendency also can be confirmed in each set of the experiment. The results indicate
the effectiveness of accent conversion on the improvement of word recognition.
Table 4. The accuracy with and without accent conversion [%].
ExperimentA
Experiment
B

Set 1
Set 2
Average
Set 1
Set 2
Average

Pre-conversion speech
56.8
62.4
59.6
55.0
51.7
53.3

Post-conversion speech
61.0
66.5
63.7
66.7
68.3
67.5

4.2.2 Experiment B
The experiment results are shown in Table 4. In the average of set 1 and set 2, the improvement of accuracy
is 14.2%. Meanwhile, the improvement in set 1 is 11.7%, and the improvement in set 2 is 16.6%. The results
show that the accent conversion in proposed method can improve the performance of sentence comprehension.
4.3 Analysis
According to the improvement rate of each experiment, the improvement rate of the experiment A is larger
than that in the experiment B. The results indicate that the proposed method is more effective in the sentencecomprehension than the word recognition. The reason should be that the proposed method converts accent notjust for words, but for the whole sentence. Improving the performance of word recognition is just a part of theadvantage of this method.
As the feedback of the subjects who had a high accuracy for the converted speech, the sufficient phoneme
duration and acquainted pitch change are useful for understanding the English speech. On the other hand, as
the feedback of the subjects who had a high accuracy for the post-conversion speech, there was an opinion thatthe converted speech is difficult to understand since it sounds like unnatural speech. Therefore, it is necessary

6881

to carry out the study of reducing the feeling of unnatural in post-conversion speech in the future.

5 CONCLUSION
In this paper, we proposed an English speech acclimatization method that converts accent of English speechof native speaker into the accent of Japanese speaker. As the result of the subjective evaluation experiment, theeffectiveness of the English speech acclimatization with accent conversion has been confirmed. In the feature,
we will focus on the applications such as controlling the conversion rate of the accent, aiming to synthesize thespeech suitable for each level of English listening proficiency.

ACKNOWLEDGEMENTS
This work was supported by R-GIRO (Ritsumeikan Global Innovation Research Organization) funded by
Ritsumeikan University, JST COI, and JSPS KAKENHI Grant Numbers JP18K19829, JP19H04142.

REFERENCES[1] Educational Testing Service, "2017 report on test takers worldwide," 2017. https://www.etsglobal.org/Aboutus/News/2017-Report-on-Test-Takers-Worldwide[2] K. Tajima, R. Port, and J. Dalby, "Effects of temporal correction on intelligibility of foreign-accented English," Journal of Phonetics, vol. 25, no. 1, pp. 1–24, 1997.
[3] K. Hirose, "Speech Prosody and CALL," Journal of the Phonetic Society of Japan, vol. 9, no. 2, pp. 38–46,
2005.
[4] A. Camacho, and J. G. Harris, "A sawtooth waveform inspired pitch estimator for speech and music," The
Journal of the Acoustical Society of America, vol. 124, no. 3, pp. 1638–1652, 2008.
[5] S. Imai, K. Sumita, and C. Furuichi, "Mel log spectrum approximation (MLSA) filter for speech synthesis,"Electronics and Communications in Japan (Part I: Communications), vol. 66, no. 2, pp. 10–18, 1983.
[6] A. Lee, and T. Kawahara, "Recent development of open-source speech recognition engine julius," AsiaPacific Signal and Information Processing Association, pp. 131–137, 2009.
[7] N. Minematsu, Y. Tomiyama, K. Yoshimoto, K. Shimizu, S. Nakagawa, M. Dantsuji, and S. Makino, "Development of English speech database spoken by Japanese learners," In Reprint of the COCOSDA Workshop, pp. 76–81, 2001.
[8] Y. Mori, "New TOEIC test strategy royal road listening edition [in Japanese]," Asahi Press, Japan, 2008.

6882