2018 26th European Signal Processing Conference (EUSIPCO)

Automatic speech pronunciation correction
with dynamic frequency warping-based
spectral conversion
Nobukatsu Hojo, Hirokazu Kameoka, Kou Tanaka, Takuhiro Kaneko
NTT Communication Science Laboratories, NTT Corporation, Japanhojo.nobukatsu@lab.ntt.co.jp

Abstract—This paper deals with the problem of pronunciation
conversion (PC) task, a problem to reduce non-native accents inspeech while preserving the original speaker identity. Although
PC can be regarded as a special class of voice conversion (VC), a
straightforward application of conventional VC methods to a PC
task would not be successful since with VC the original speaker
identity of input speech may also change. This problem is due tothe fact that two functions, namely an accent conversion function
and a speaker similarity conversion function, are entangled
in an acoustic feature mapping function. This paper proposes
dynamic frequency warping (DFW)-based spectral conversion
to solve this problem. The proposed DFW-based PC converts
the pronunciation of input speech by relocating the formants
to the corresponding positions in which native speakers tend
to locate their formants. We expect the speaker identity ispreserved because other factors such as formant powers are
kept unchanged. in a low frequency domain evaluation results
confirmed that DFW-based PC with spectral residual modelingshowed higher speaker similarity to original speaker while
showing a comparable effect of reducing foreign accents to aconventional GMM-based VC method.
Index Terms—Accent conversion, dynamic frequency warping,voice conversion

I. I NTRODUCTION
This paper deals with the problem of pronunciation conversion (PC), a problem to reduce accent in speech while
preserving the speaker identity of the original speech. Accents are differences in pronunciation by a community ofpeople from a national or regional geographical area, or asocial grouping [1]. It is known that differences by accentsare manifested in the differences such as the formants andtheir trajectories [2], [3] or pitch intonation and duration
parameters [4], [1]. Reducing these accents from speech while
preserving the original speaker identity will be beneficial for
applications such as language educational systems for second
language learners [5], [6], [7] and teleconference scenarios
where people with different nationalities participate.
If we consider an accent as one of non/para-linguistic
information, PC can be regarded as a special class of a voice
conversion (VC) problem, a problem to convert non/paralinguistic information while preserving linguistic information.
For VC, data-driven statistical methods have been successfully
introduced during the last two decades [8], [9], [10]. One
successful VC method is based on Gaussian mixture models(GMMs) [9], [10]. Most of the conventional VC methods
first train a mapping function between the acoustic featuresof source and target speech using parallel data, i.e. a pair oftime-aligned feature sequences of source and target speech. At
test time, a feature sequence of the input speech is converted

ISBN 978-90-827970-1-5 © EURASIP 2018using the trained mapping function. A direct application ofthese methods to the PC task would not be successful because
the original speaker identity may also change. This problem
is because two functions, the accent conversion function andthe speaker identity conversion function, are entangled in an
acoustic feature mapping function.
This work is based on a belief that a dynamic frequency
warping (DFW)-based spectral conversion approach can be areasonable solution to this problem. The DFW-based spectral
conversion approach was originally proposed [11], [12] mainly
for the purpose of improving naturalness of converted speech.
Since one of the dominant differences in accented speech
appears in formant frequency trajectories [2], [3], we expectthat pronunciation can be corrected via frequency warping
if the formants can be relocated to the positions in which
native speakers tend to locate their formants. In this way, we
expect that the original speaker identity will not be affected
since DFW does not convert the other factors such as formant
powers and spectral tilts.
The main purpose of this work is to investigate the effectiveness of DFW-based spectral conversion for the PC task.
Furthermore, there are two points that we want to investigate.
One is the effectiveness of spectral power interpolation. DFW
only allows us to deform source spectra in the frequency
direction and does not have an ability to reduce the gap inspectral powers. Since the peakiness of each formant is also
expected to be an important factor that characterizes pronunciation quality, we would also want to deform source spectrain the power direction. However, if we convert source spectra
exactly to target spectra, the speaker identity will no longer bepreserved. Thus, there is a trade-off between the pronunciation
quality and the voice similarity to a source speaker. This tradeoff is investigated by subjective evaluation. The other point isconcerned with the modeling of frequency warping functions.
The idea of the method in [12] is to associate a frequency
warping function with each Gaussian of the GMM that models
a joint distribution of source and target spectra. The frequency
warping function is obtained from a pair of source and target
spectra averaged over all the frames assigned to the same
Gaussian. We found that the averaged spectra tend to be oversmoothed and so the obtained warping function will also beover-smoothed. To avoid this, we propose a method that firstextracts frequency warping functions from pairs of source andtarget spectra frame-by-frame, treats the obtained frequency
warping functions as features to be predicted, and models thejoint distribution of source spectra and the frequency warping2324

2018 26th European Signal Processing Conference (EUSIPCO)

between a frequency warped source spectrum and a target
spectrum;

Source Mel-cepstraTraining
Source Speech

Source SpectraAnalysis &Alignment
Target Speech
(DTW)

DFW

DFW
Vector

Target Spectra

GMM
Trainingŵt = arg min
w1 ,··· ,wF

Mel-cepstra
AnalysisF0, ap
Converted Speech

Spectra

Frequency
Warping(1)

where wf takes a frequency index wf ∈ {1, · · · , F } inan ascending order. By restricting the warping path to be
wf +1 − wf ∈ {0, 1, 2} for each f ∈ {1, · · · , F − 1}, asmoothed and fixed length DFW vector can be extracted.
Although it is possible to use the l2 norm of a log spectral
difference to define D(x, y), the obtained warped spectra tend
to have plateau because of the power difference of spectral
peaks between source and target speakers. We concern thatthe plateau can degrade harmonics and quality of converted
speech, which is the reason we define the distance introducing
the frequency derivative distance term as follows in this work;!
"
D xf , yf =|| log xf − log yf ||2+ γ||ẋf − ẏf ||2 ,(2)ẋf = log xf +1 − log xf ,(3)ẏf = log yf +1 − log yf(4)

DFW Vector
Estimation

DFW Vector

Synthesis
Converted Spectra

Fig. 1. Architecture of the proposed accent conversion method.

Fig. 2. Example of DFW spectra (top) and extracted DFW vector (bottom).
The blue and red line shows the result using γ = 0 and γ = 25.0 in eq. (2),respectively.

functions using a GMM. In this paper, we particularly focus
on the problem of spectral conversion only. However, it canbe used in combination with prosody conversion methods [5],[13].
II. DYNAMIC F REQUENCY WARPING - BASED ACCENTC ONVERSION
The proposed method consists of training process andconversion process. We show the overall architecuture of the
proposed method in Fig. 1.
A. Dynamic frequency warping with frequency derivative distance
The proposed method first finds an optimal warping of
frequency axis with DFW [14], [15]. Let us denote the timealigned source spectra by X = {xt }Tt=1 and the target spectraF
by Y = {y t }Tt=1 . Here xt = [xf,t ]F
f =1 and y t = [yf,t ]f =1
respectively denote the source and target spectrum. t, f denotethe frame and frequency indices, respectively. The warping
function ŵt = [ŵf,t ]F
f =1 , which we call the DFW vector, canbe obtained as the path that minimizes the spectral distanceISBN 978-90-827970-1-5 © EURASIP 2018

D(xwf ,t , yf,t ),

f =1

GMM

Conversion
Input Speech

FX

where and γ indicates the weight for the frequency derivative
term. This simple spectral distance is expected to have asimilar effect to correlation based DFW [16] or DFW based on
the histogram of spectral peaks [17]. An example of spectral
warping using frequency derivative distance is illustrated in
Fig.2. We can see that the spectral plateau is eliminated byintroducing the frequency derivative distance term.
Since we would want to eliminate the speaker identity
information from the DFW vector as much as possible, we
found it necessary to apply vocal tract length normalization(VTLN) [18] to the source and target speech spectra before
performing DFW.
B. GMM training and DFW vector estimation
For estimating DFW vectors from the source spectra, we
model the joint distribution of the source spectra and the DFW
vectors using a GMM. In order to avoid overfitting in modelinghigh dimensional vectors, we compress the dimension of the
T Tfeature vectors and constitute a joint vector z t = [mTt , dt ] ,where mt is the mel-cepstrum extracted from the source
spectrum xt at frame t. The vector dt is derived as follows;
dt =DCT(wt − wb ),(5)

where wb = [1, · · · , F ]T and DCT(·) denotes the discretecosine transform. We model the joint feature as follows;p(z) =IX
i=1αi N (z; µi , Σi ),IX

αi = 1, αi > 0,

(6)

i=1

where N (z; µ, Σ) denotes the normal distribution with themean vector µ and the covariance matrix Σ. αi denotes aweight of class i, and I denotes the total number of theGaussian mixtures. Given a training set of {mt , dt }Tt=1 pairs,
we train the GMM parameters using the EM algorithm.

2325

2018 26th European Signal Processing Conference (EUSIPCO)

At test time, DFW vectors are estimated using the trained
GMM and the input spectra. The mapping function [10] isgiven byF (m) = E[d|m]
=

I
X

(d)

hi (m)[µi(dm) !

+ Σi(mm) "−1Σi(m)(m − µi

)]

i=1

(7)
(m)
(mm) "
αi N m; µi , Σihi (m) = #I,!
(m)(mm) ", Σjj αj N m; µj

!

(m)

(8)

(d)

where µi and µi denote the mean vectors of class i for themel-cepstra and DCT of DFW vectors. The estimated DFW
vector is derived as follows;!
"w̃t = iDCT F (mt ) + wb ,
(9)

(d)

Fig. 3. Example of DFW vectors reconstructed from µi . Six out of 16
Gaussian components are shown for simplicity.

where iDCT(·) denotes the inverse discrete cosine transform.
Since the estimated w̃t takes a continuous value, they arefloored to be an integer frequency index to convert spectraas ỹ t = [xw̃f,t ,t ]F
f =1 .
C. Spectral residual modeling
Note that DFW only has an ability to deform source spectrain the frequency direction and does not have an ability to fill
the gap in spectral powers. Since the power of each formant
is also expected to be an important factor that characterizes
pronunciation quality, we would also want to deform source
spectra in the power direction. However, if we convert source
spectra exactly to target spectra, the speaker identity will nolonger be preserved. Thus, there is a trade-off between the
pronunciation quality and the voice similarity to a source
speaker. Here, we also consider predicting the power differences between the warped spectra and the target spectra so that
we can add the predicted differences to the warped spectra attest time.
The residual spectra r t = [rf,t ]F
f =1 is defined as thedifference between the target spectra and the warped source
spectra as follows;
rf,t =

yf,t
xŵf,t ,t(10)This residual spectra is extracted for each frame and constructa joint vector st = [mTt , q Tt ]T , where q t denotes the DCTof log r t . This joint vector is modeled by another GMM andused to estimate residual spectra r̃ in the same manner for
estimating the residual vector. The output converted spectra(r)(r)
ỹ t = [ỹf,t ]F
f =1 is derived as follows;(r)

λỹf,t = ỹf,t · r̃f,tFig. 4. Subjective evaluation results.

conventional VC and proposed methods. We used an Indian
male speaker as the source (hereafter, the SRC) and an
American male speaker as the target (the TGT). The dataset
consisted of 65 pair utterances (5.2 minutes). We used 20
utterances for evaluation and the others for training. Table. I
shows the statistical information of the corpus. The data
were sampled at 16kHz, then 25 mel-cepstral coefficients,
fundamental frequency (F0 ), and aperiodicities were extracted
every 5ms by using the STRAIGHT analysis system [19].
To obtain parallel utterances, we used dynamic time warping(DTW) to align mel-cepstral sequences of the source and target
speakers [20]. We evaluated and compared the following 6
spectral conversion methods.

(11)

where λ denotes the weight for spectral residual modeling.
III. E XPERIMENTAL EVALUATION
A. Experimental conditions
We evaluated pronunciation similarity and speaker identityof the converted speech to compare the performance of theISBN 978-90-827970-1-5 © EURASIP 2018

2326

GMM-MCEP: The conventional GMM-based VC method
using mel-cepstral feature s [10].
• DFW-CENTROID:
DFW-based spectral conversion
method. DFW functions were derived from centroid
spectra pair of each Gaussian.
• PROP-0: The proposed method without spectral residualmodeling (λ = 0.00).
• PROP-1: The proposed method with spectral residualmodeling (λ = 0.33).
• PROP-2: The proposed method with spectral residualmodeling (λ = 0.67).
•

2018 26th European Signal Processing Conference (EUSIPCO)(a) SRC(b) PROP-1(c) GMM-MCEP(d) TGT
Fig. 5. Example of spectrogram of SRC, PROP-1, GMM-MCEP and TGT for a utterance “Going forward.”. We show frequency region of 0 - 5 kHz toclarify the formant transition.
TABLE IS TATISTICAL INFORMATION OF THE SPEECH CORPUS
Speakers
Total speech length
Number of utterances
Average number of words per sentence•

2 males
5.2 minutes
65 sentences
5.8 words

PROP-3: The proposed method with spectral residualmodeling (λ = 1.00).

Although DFW-CENTROID was implemented emulating theconventional DFW-based VC method [12], details such as
vocoder for synthesizing speech was not the same. This is inorder to reduce the factors considered in the experiments. We
evaluated 4 proposed methods with different spectral residual
weight in order to evaluate the trade-off of the pronunciation
quality improvement and speaker identity degradation.
For DFW-CENTROID, we first train a GMM that models
joint distribution of source and target spectra using line spectral pairs (LSP) features. Then, a pair of source and target
centroid spectra was reconstructed from the mean vector ofeach Gaussian. The frequency warping function was obtained
from the pair of source and target centroid spectra using
the DFW method in sec. II-A. The conversion procedure
was similar to the proposed method except for using LSPs
instead of mel-cepstra as input spectral features. We used 25
dimensional LSP for this method.
For the proposed methods, we used 25 dimensional melcepstra and 25 dimensional DCT of DFW vectors. We usedmel-spectra for DFW vector extraction so that the warping
function gets higher frequency resolution in a low frequency
domain. We set γ = 25.0 in eq. (2), which was determined
experimentally. We show the results without conducting VTLN
before DFW vector extraction, because the results were not
improved. We consider this is because the vocal tract lengthof SRC and TGT was almost same.
For all of the methods, the number of mixture components
for GMMs was 16. All of the speech samples were synthesized
by STRAIGHT vocoder [19], using converted spectra by eachmethod, F0 and aperiodicity of the SRC speaker.
Fig. 3 shows the example of warping functions recon(d)structed from µi of the trained GMM of PROP-0. We can seethat these functions can warp frequency by 200 Hz around 1−3
kHz frequency region, which we expect is enough warping
width to relocate formants to reduce accents.

ISBN 978-90-827970-1-5 © EURASIP 2018

B. Subjective evaluation
Two subjective evaluations were conducted. We evaluatedthe pronunciation similarity to the TGT and speaker similarity to the SRC by DMOS tests. The subjects rated thepronunciation and the speaker similarity using a 5–point scale:“5” for absolutely similar, “4” for similar, “3” for fair, “2”for dissimilar “1” for absolutely dissimilar . There were tenparticipants who were well-educated English speakers.
Fig. 4 shows the pronunciation and speaker similarity scoresobtained in the subjective evaluations with confidence intervals of 5%. First we compare the results of GMM-MCEP
and PROP-0. We can see that GMM-MCEP shows higher
pronunciation similarity score. This result shows that DFWbased spectral warping without spectral residual modelingcannot reduce the foreign accents as much as conventional
VC methods. However, if we compare GMM-MCEP and
PROP-1, we can see that pronunciation similarity scores werecomparable while the speaker similarity score was higher in
PROP-1. These results confirmed that DFW-based PC with
spectral residual modeling showed higher speaker similarity
to original speaker while showing a comparable effect of
reducing foreign accents to a conventional GMM-based VC
method. We then compare DFW-CENTROID and PROP-0.
We can see both the score is improved by PROP-0. These
results confirm that the DFW modeling by the proposed
method was effective. We then compare the results between
the proposed methods. Although the pronunciation similarity
score by PROP-1 was higher than PROP-0, those by PROP-2
and PROP-3 were comparable with PROP-1 while the speakersimilarity score degrades. These results show that the weighthigher than 0.67 was not effective for PC in our experimental
conditions.
Fig. 5 shows examples of spectrogram of SRC, PROP1, GMM-MCEP and TGT for a utterance “Going forward.”
included in the test set. From these examples, we can see thatthe formant frequency trajectory around 1 kHz by TGT is
located in higher frequency region than SRC. We can also seethat this formant frequency trajectory is converted to higher
frequency by PROP-1. It is also confirmed that the spectraby PROP-1 is more similar to SRC than GMM-MCEP, which
resulted in higher speaker similarity to SRC in our subjective
evaluations.
IV. C ONCLUSION
This paper investigated the effectiveness of DFW-based
spectral conversion for pronunciation conversion task, a prob-2327

2018 26th European Signal Processing Conference (EUSIPCO)

lem to reduce accent in speech while preserving the speaker
identity of the original speech. The proposed method converts
the pronunciation of input speech by relocating the formants
to the corresponding positions in which native speakers tend
to locate their formants. We expect the speaker identity ispreserved because other factors such as formant powers are
kept unchanged. Subjective evaluation results confirmed that
DFW-based pronunciation conversion with spectral residualmodeling showed higher speaker similarity to original speaker
while showing a comparable effect of reducing foreign accents
to a conventional GMM-based VC method. It is worth investigating whether utilizing neural network-based methods [21],[22], [23] for DFW vector estimation is effective with larger
size of speech corpus. Future works include extending theexperiments to other speaker, accent and language pairs.

ISBN 978-90-827970-1-5 © EURASIP 2018

R EFERENCES[1] Qin Yan and Saeed Vaseghi, “A comparative analysis of uk and us
english accents in recognition and synthesis,” in Proc. ICASSP 2002.
IEEE, 2002, vol. 1, pp. I–413.
[2] Jonathan Harrington, Felicity Cox, and Zoe Evans, “An acoustic phonetic study of broad, general, and cultivated australian english vowels,”
Australian Journal of Linguistics, vol. 17, no. 2, pp. 155–184, 1997.
[3] Catherine I Watson, Jonathan Harrington, and Zoe Evans, “An acoustic
comparison between new zealand and australian english vowels,” Australian journal of linguistics, vol. 18, no. 2, pp. 185–207, 1998.
[4] John C Wells, Accents of English, vol. 1, Cambridge University Press,
1982.
[5] Daniel Felps, Heather Bortfeld, and Ricardo Gutierrez-Osuna, “Foreign
accent conversion in computer assisted pronunciation training,” SpeechCommunication, vol. 51, no. 10, pp. 920 – 932, 2009, Spoken Language
Technology for Education.
[6] Daniel Felps, Christian Geng, and Ricardo Gutierrez-Osuna, “Foreign
accent conversion through concatenative synthesis in the articulatory domain,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 20, no. 8, pp. 2301–2312, 2012.
[7] Daniel Felps and Ricardo Gutierrez-Osuna, “Developing objective
measures of foreign-accent conversion,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 18, no. 5, pp. 1030–1040, 2010.
[8] Masanobu Abe, Satoshi Nakamura, Kiyohiro Shikano, and Hisao
Kuwabara, “Voice conversion through vector quantization,” Journalof the Acoustical Society of Japan (E), vol. 11, no. 2, pp. 71–76, 1990.
[9] Yannis Stylianou, “Applying the harmonic plus noise model in concatenative speech synthesis,” IEEE Transactions on speech and audio
processing, vol. 9, no. 1, pp. 21–29, 2001.
[10] Tomoki Toda, Alan W Black, and Keiichi Tokuda, “Voice conversion
based on maximum-likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 15, no. 8, pp. 2222–2235, 2007.
[11] Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano, “Voice
conversion algorithm based on gaussian mixture model with dynamic
frequency warping of straight spectrum,” in Proc. ICASSP 2001. IEEE,
2001, vol. 2, pp. 841–844.
[12] Daniel Erro, Asunción Moreno, and Antonio Bonafonte, “Voice conversion based on weighted frequency warping,” IEEE Transactions on
Audio, Speech, and Language Processing, vol. 18, no. 5, pp. 922–931,
2010.
[13] Kwansun Cho and John G Harris, “Towards an automatic foreign accent
reduction tool,” in Proc. 3rd Int. Conf. Speech Prosody, 2006.
[14] Hélene Valbret, Eric Moulines, and Jean-Pierre Tubach, “Voice transformation using psola technique,” Speech communication, vol. 11, no.
2-3, pp. 175–187, 1992.
[15] Noriyasu Maeda, Banno Hideki, Shoji Kajita, Kazuya Takeda, andFumitada Itakura, “Speaker conversion through non-linear frequency
warping of straight spectrum,” in Sixth European Conference on SpeechCommunication and Technology, 1999.
[16] Xiaohai Tian, Zhizheng Wu, Siu Wa Lee, and Eng Siong Chng,“Correlation-based frequency warping for voice conversion,” in Chinese
Spoken Language Processing (ISCSLP), 2014 9th International Symposium on. IEEE, 2014, pp. 211–215.
[17] Elizabeth Godoy, Olivier Rosec, and Thierry Chonavel, “Voice conversion using dynamic frequency warping with amplitude scaling, forparallel or nonparallel corpora,” IEEE Transactions on Audio, Speech,
and Language Processing, vol. 20, no. 4, pp. 1313–1323, 2012.
[18] Jordan Cohen, Terri Kamm, and Andreas G Andreou, “Vocal tract normalization in speech recognition: Compensating for systematic speakervariability,” The Journal of the Acoustical Society of America, vol. 97,
no. 5, pp. 3246–3247, 1995.
[19] Hideki Kawahara, Jo Estill, and Osamu Fujimura, “Aperiodicity extraction and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis systemstraight,” in Second International Workshop on Models and Analysis of
Vocal Emissions for Biomedical Applications, 2001.
[20] Yannis Stylianou, “Statistical methods for voice quality transformation,”
Proc. European Speech Communication Association, Madrid, Spain,
1995, pp. 447–450, 1995.
[21] Toru Nakashika, Tetsuya Takiguchi, and Yasuo Ariki, “Voice conversion
using speaker-dependent conditional restricted boltzmann machine,”
EURASIP Journal on Audio, Speech, and Music Processing, vol. 2015,
no. 1, pp. 8, 2015.
[22] Lifa Sun, Shiyin Kang, Kun Li, and Helen Meng, “Voice conversion
using deep bidirectional long short-term memory based recurrent neuralnetworks,” in Acoustics, Speech and Signal Processing (ICASSP), 2015
IEEE International Conference on. IEEE, 2015, pp. 4869–4873.
[23] Shariq Mobin and Joan Bruna, “Voice conversion using convolutional
neural networks,” arXiv preprint arXiv:1610.08927, 2016.

2328