PROCEEDINGS of the23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Voice conversion model for estimation of transfer characteristic
in auditory feedback
Shota MORITA1; Daiki KAWAMOTO1; Teruki TOYA2
12

Fukuyama University, Japan

Japan Advanced Institute of Science and Technology, JapanABSTRACT
We propose a voice conversion model for estimation of transfer characteristic in auditory feedback. Aspeaker feels a sense of discomfort when the speaker listens to recorded speaker’s speech sound. Thephenomenon is caused by the different transfer pathways. The recorded speech sound contains just air
conducted (AC) sound from the speaker. On the other hand, the speech sound by the speaker has AC and bone
conducted (BC) sounds in auditory feedback. There is a problem that the sound source position and thelistening point cannot be underspecified. In this paper, we proposed a voice conversion model of AC and BC
voices for estimation of transfer characteristic in auditory feedback. Converted voice was obtained bysynthesizing AC and BC voices with time delay, AC/BC ratio, and reverberation time as parameters. BCvoice was obtained by filtering from AC voice. The transfer characteristic in auditory feedback was estimatedby subtracting spectrum of converted voice from that of AC voice.
Keywords: Voice conversion, Auditory feedback, Air-conducted speech, Bone-conducted speech,

1. INTRODUCTION
When we listen to our recorded speech, we have slightly different listening sense. The different
sense is caused by auditory feedback (AF) in air-conducted (AC) and bone-conducted (BC) speech. We
listen to our own AC and BC speech simultaneously when we speak, which means speakers perceive
their own voices from their production systems while communicating through speech [1]. Thismechanism is referred to as AF. The AF has an important role of the monitoring loop for pronunciationin speech chain [2]. The speech chain is considered as a mechanism of information pathways in humanspeech communication. Delayed auditory feedback (DAF) is one of the techniques for investigating
the importance of such monitoring loop. We cannot smoothly speak in DAF experiment. It is caused bytime delay of AC speech [3]. Here, the recorded own speech contains just AC speech.
The transfer function between AC and self-perceived own (SPO) voices cannot be measured. ACspeech can be easily record by ordinary microphones. However, the s SPO voices at the listening point
in auditory perception cannot be measured by microphone as physical measurement. Therefore, thetransfer function should be estimated by other approaches. If we can obtain the transfer characteristics
between AC and SPO voices, the transfer characteristics will be breakthrough for hearing of tonedeaf.
The transfer function from AC to SPO voices were estimated using equalizers by peak and shelffilters [4]. The SPO voice was estimated from AC speech by voice conversion using equalizers. Thus,the transfer function was estimated from AC and SPO voices. This estimate approach from AC andSPO voices are very useful to estimate the transfer function in AF. However, the approach cannot
clarify the detail of the pathway between AC and SPO voices. Voices in AF for estimating thetransfer function should be converted from AC voice based on the model by AC and BC voices using
conventional studies. The studies on different pathways with AC and BC voices can be hints toclarify the transfer characteristics with AC and SPO voices. Computer simulation can be one of theuseful approaches to estimate the transfer function of BC voice in human head. Fujisaka et al.
analyzed BC sound wave propagation using FDTD method for computer simula tion [5]. The transfer

6630

1

s-morita@fukuyama-u.ac.jp

2

yattin_yatson@jaist.ac.jp

function between AC and BC was estimated using singing voice [6]. In the experiment, AC and BC
singing voices were recorded by Supercardioid type and piezo contact microphones, respectively.
They showed the frequency characteristics of AC and BC singing voices. The AC voices weredelayed about 0.4-0.8 ms from BC voices in AF [7]. In the experiment, BC voice was recorded onskin around left mastoid process by acceleration sensor, vocal-fold vibration was recorded on skinaround left thyroid by acceleration sensor, and AC voice was recorded on left pinna by small
microphone. The delay time between AC and BC voices in AF was calculated using the times on
three recorded times.
In this paper, we propose voice conversion model based on AC and BC voices for estimation oftransfer characteristic in AF. The voice conversion model can be used to obtain the SPO voice from ACvoice.

2. ESTIMATION OF TRANSFER CHARACTERISTICS IN AUDITORY FEEDBACK
2.1 Estimation of Transfer Characteristics Using Voice Conversion ModelWe mention that our concept for the estimation of transfer characteristic in auditory feedback in thissection. One of the approaches to estimate the transfer characteristics in AF is the estimation usingvoice conversion from AC voice [4]. Estimation approaches of transfer characteristics using AC andSPO voices are very useful. In the research, SPO voices were obtained by an equalizer. In this research,we obtained SPO voice based on the voice conversion model of pathway from AC and BC voices. We
explain the model in the next subsection.
2.2 Voice Conversion Model
Pathways for SPO voice by AC and BC voices should be considered to establish the model of voice
conversion in AF. We propose a voice conversion model from AC and BC sounds. The outline of themodel is shown in Fig. 1.

Figure 1  Voice conversion modelBC voice is converted from recorded AC voice by filtering. The filter is constructed based onspectra of AC and BC singing voices that was referred from Won & Berger [6]. The filter characteristic
is obtained by subtraction AC spectrum from BC spectrum as referred in Fig. 1 in [6]. The filter
characteristic for conversion from AC spectrum ܵ௔௖ ሺ݂ሻ to BC spectrum ܵ௕௖ ሺ݂ሻ was obtained asfollows,‫ ܨ‬ൌ ܵ௕௖ ሺ݂ሻ െ  ܵ௔௖ ሺ݂ሻ(1)
where f is frequency, F is the obtained filter characteristic. The obtained filter characteristic wasshown in Fig. 2. The filter for conversion to BC voice consist of a low pass filter with cutofffrequency 200 Hz, and two bandpass filters of 250-500 and 750-1500 Hz, based on the Fig. 2. BCvoice was obtained by these three filters.
Three parameters of time delay td , AC/BC power balance ABR, reverberation time tr were used in
the model to convert SPO voice from AC and BC voices. The time delay was set as the time delay ofAC voice from BC voice, based on the results in [7]. The AC/BC power ratio ABR was set to fix thepower difference of AC and BC voices. The ABR was calculated as follows,‫ ܴܤܣ‬ൌ ͳͲ  ͳͲ ൬ܲ௔௖൰ሾሿܲ௕௖(2)where ܲ௔௖ is the power of AC voice and ܲ௕௖ is the power of BC voice. The artificial reverberation
using reverberation time tr was set to assumed that the effect of reflection in head. Schroeder’s room6631

Figure 2  Filter characteristics for conversion from AC voice to BC voice
impulse response [8] were implemented as artificial reverberation. However, the enormity of theeffect of the reflection is not cleared. Thus, the effect of the reflection in head will be clarified by anexperiment using the model.
Arrival time of recorded AC voice was delayed using time delay td from arrival time of BC voice.
In the next process, the delayed AC and converted BC voices were fixed these power balance, thenthese AC and BC voices were summed. After that, the SPO voice was obtained by convolving theimpulse response as reverberation.
2.3 Application of Voice Conversion
Applications of voice conversion based on voice conversion model in subsection 2.2 was created by
MATLAB. The applications consisted of recorded and conversion parts in Fig. 3 and 4, respectively.
Speakers’ vowels of /a/, /i/, /u/, /e/, /o/ can be recorded each three times when they select the name inthe left box in Fig. 3, and push the button of “Recording Start.” Recording time was determined up to3 s. Waveform window is shown on the top for checking a waveform of the recorded own vowel afterrecorded. Recorded condition is shown as checked box on the right bottom in Fig. 3.

Figure 3  Screenshot of application for recording6632

Figure 4  Screenshot of application for voice conversion
The recorded own vowels by the recording application in Fig. 3 are converted to obtain theself-perceived vowels by the application of voice conversion in Fig. 4. Delay times td were set
0.00-0.14 s at a certain interval 0.02 s. AC/BC ratios ABR were set 100-0 dB. ABR = 0 dB means ACand BC power level are equal to each other. Reverberation times tr were set 0.00 to 0.40 s. A
recorded own vowel are selected from left panel, then three parameters are fixe d on the top panel.
After clicking the “set” and “play” buttons, converted vowel was played. After fixing theseparameters to best self-perceived vowel and clicking the “complete” button, the values of parametersand converted vowel are saved automatically to new .mat and .wav files, respectively.

3. EXPERIMENTS
3.1 Experimental Condition
The proposed voice conversion model from AC voice in AF was tested in the experiment.
Four male speakers aged 19-22 participated in the experiments. They are native Japanesespeakers.
The AF experiments consisted of recording and voice conversion sessions. Figure 5 shows aschematic diagram of the recording system in AF experiments.

Figure 5  Schematic diagram of the recording system in AF experiments
Experiments were conducted in a small room. Speaker’s voices were recorded through a microphone(Rode NT1 KIT) and routed through an audio interface (Roland QUAD -CAPTURE) to a PC (AppleMacBook Pro, with OS macOS High Sierra 10.13.8). The recording application in Fig. 3 was used torecord the vowels. The sampling frequency was 8 kHz and the number of quantizing bits was 16.
Figure 6 shows a schematic diagram of the voice conversion system in AF experiments. The vowels
were routed through the interface, presented to the listeners (participants) through an openheadphone (AKG K712) The application of voice conversion as shown in Fig. 4 was used to convert
vowels. The system latency did not concern because sessions of recording and voice conversion were6633

Figure 6  Schematic diagram of the voice conversion system in AF experiments
separated.
Speakers uttered five vowels of /a/, /i/, /u/, /e/, /o/ using recording application in recording
session. Each vowel was uttered three times in each speaker. The distance between microphone andspeaker were set around 0.2 m.
In the voice conversion session, a participant converted own recorded vowels to self-perceived
vowels using voice conversion application. Sequential order of vowels in a certain conversion trial
was “/a/, /i/, /u/, /e/, /o/.” The trial was repeated totally three times. In each stimulus, the participant
fixed three parameters to get close to the self-perceived voice. The participant can utter the samevowel while listening to converted own vowel when fixing these parameters in the experiment.
3.2 Results
The results of AF experiments were shown in Tab. 1. These results were calculated by average ofparticipants and trials number. The results of these parameters were different in each vowel. Effectsof time delay from BC voice were around 0.07 s. The results showed the possibility of time delay insection of ear canal entrance and auditory periphery because the large time lag was shown between
the time delay in [7] and the results in Tab. 1. AC/BC ratios of /a/ and /o/ were smaller than theothers. The results implied that the effects of BC voice in /a/ and /o/ were larger than /i/, /u/, and /e/.
Reverberation times of /a/, /u/ and /e/ were over 0.1 s. It is assumed that the converted vowel fromAC and BC voices was perceived like two separate stimuli because of the effects of time delay invoice conversion. However, it is assumed that the converted vowel, where the room impulse
response was convolved, was heard as one unified stimulus. The results implied that thereverberation affects the voice conversion model from two signals like AC and BC voice to a signal
like self-perceived voice.
The spectra of recorded and converted voice /e/ by a participant were shown in Fig. 7. For theanalysis, the window size was set 2,048 and Hanning window was used. The higher frequency
components were drastically attenuated in the converted (SPO) voice compared with the recorded(AC) voice.
Table 16634

Results of AF experiments

Time delay [s]AC/BC ratio [dB]

Reverberation time [s]/a/0.070

37.5

0.115/i/

0.070

48.3

0.062

/u/

0.077

47.5

0.130

/e/

0.057

67.9

0.153

/o/

0.075

37.9

0.072

Figure 7  Spectra of recorded and converted vowels /e/ by a participant
The transfer characteristics between recorded and converted (self -perceived own) voice can beobtained by subtracting spectrum of converted voice from spectrum of recorded AC voice. This isalmost the same process with estimate transfer function from AC and equalized voices [4]. The
calculated transfer characteristics of voice /e/ by a participant were shown in Fig. 8. The transfer
characteristics showed low-pass characteristics.

Figure 8  Transfer characteristic of vowels /e/ by a participant

4. CONCLUSIONSThis paper proposed the voice conversion model by AC and BC voices for estimating transfer
characteristics between recorded and SPO voices in AF. Time delay, AC/BC ratio and reverberation
time were considered as parameter in the model. The results show the possibility that the voice
conversion model by AC and BC voices can be used to obtain the SPO voice from recorded AC voice.
Moreover, the results showed the possibility to estimate the transfer characteristics. The
individuality and affect reverberation should be considered in the future work.

6635

REFERENCES
1. Denes PB, Pinson EN. The speech chain, 2nd ed. New York, USA: Waveland Press, Inc.; 1993.
2. Postma A. Detection of errors during speech production: a review of speech monitoring models.
Congnition 2000; 77: p.87-131.
3. Toya T, Ishikawa D, Miyauchi R, Nishimoto K, Unoki M. Study on effects of speech production during
delayed auditory feedback for air-conducted and bone-conducted speech. Journal of Signal Processing
2016; 20(4): p.197-200.
4. Won SY, Berger J, Slaney M. Simulation of one's own voice in a tow-parameter model. Proc ICMPC
2014; 4-8 August 2014; Seoul, South Korea.
5. Fujisaka Y, Nakagawa S, Tonoike M. Analysis of bone conducted sound wave propagation in humanhead. Proc. ICA 2004; 4-9 April 2004; Kyoto, Japan 2004. p. I683-I686.
6. Won SY, Berger J. Estimating transfer function from air to bone conduction using singing voice. Proc.
ICMC 2005; 5-9 September 2005, Barcelona Spain 2005.
7. Ochiai Y, MURAKAMI T. Measurement of delay times between air conduction sounds and bone
conduction sounds for auditory feedback. Proc. student society of IEICE Tokyo branch; Japan 2018. p.
140. (in Japanese)
8. Schroeder MR. Modulation transfer functions: definition and measurement. Acoustica 1981; p.179-182.

6636