An Evaluation of Target Speech for a Nonaudible
Murmur Enhancement System in Noisy
Environments
Sakura Tsuruta, Kou Tanaka, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura
Graduate School of Information Science, Nara Institute of Science and Technology (NAIST)
E-mail: [tsuruta.sakura.tl0, ko-t, tomoki, neubig, ssakti, s-nakamura]@is.naist.jp Tel: +81-743-72-5265

Abstract—Nonaudible murmur (NAM) is a soft whispered
voice recorded with NAM microphone through body conduction.
NAM allows for silent speech communication as it makes it
possible for the speaker to convey their message in a nonaudible
voice. However, its intelligibility and naturalness are significantly
degraded compared to those of natural speech owing to acoustic
changes caused by body conduction. To address this issue,
statistical voice conversion (VC) methods from NAM to normal
speech (NAM-to-Speech) and to a whispered voice (NAM-toWhisper) have been proposed. It has been reported that theseNAM enhancement methods significantly improve speech quality
and intelligibility of NAM, and NAM-to-Whisper is more effective
than NAM-to-Speech. However, it is still not obvious which
method is more effective if a listener listens to the enhanced
speech in noisy environments, a situation that often happens insilent speech communication. In this paper, assuming a typicalsituation in which NAM is uttered by a speaker in a quiet
environment and conveyed to a listener in noisy environments,
we investigate what kinds of target speech are more effective for
NAM enhancement. We also propose NAM enhancement methods
for converting NAM to other types of target voiced speech.
Experiments show that the conversion process into voiced speech
is more effective than that into unvoiced speech for generating
more intelligible speech in noisy environments.

I. INTRODUCTION
Speech communication is one of the most fundamental communication methods in our daily life, and the advancement oftechnology such as cell phones has enabled us to communicate
with each other whenever and wherever. However, it has also
created the awareness of existence of some situations where
we hesitate to talk; e.g., we would have trouble maintaining
privacy in a crowd, or speaking itself can annoy others in quiet
environments such as a library or an office.
In recent years, silent speech interfaces have attracted attention as a new speech communication style [1] that allows
us to talk while keeping silent. One of the promising silent
speech media is nonaudible murmur (NAM) [2]. NAM is a
soft whispered voice recorded with NAM microphone throughbody conduction. It is produced by articulating respiratory
sounds without using vocal-fold vibration. One of the biggest
drawbacks of NAM is that its acoustic properties are very
different from those of natural voices because of the essential
properties of body-conductive recording, i.e., lack of radiation
characteristics from the lips and effect of the low-pass characteristics of soft tissues. Consequently, its naturalness andintelligibility are significantly degraded.
To address this issue, statistical NAM enhancement methods
have been proposed [3]. In these methods, acoustic parameters

978-616-361-823-8 © 2014 APSIPA

of NAM are converted into those of air conducted normal
speech (NAM-to-Speech) or a whispered voice (NAM-toWhisper) by using statistical voice conversion (VC) techniques[4][5]. It has been reported in [3] that NAM-to-Whisper
outperforms NAM-to-Speech because 1) NAM-to-Whisper can
avoid the F0 and unvoiced/voiced information prediction from
NAM, which is an essentially difficult prediction process, and2) spectral conversion accuracy in NAM-to-Whisper is higher
than that in NAM-to-Speech because spectral parameters of
NAM are more similar to those of a whispered voice thanthose of normal speech. However, because they were comparedto each other through listening tests under only a quiet
environment, it is still not obvious which method is more
effective if a listener listens to the enhanced speech in noisy
environments, a situation that often happens in real silent
speech communication.
In this paper, we investigate what kinds of target speech
are more effective for enhancing intelligibility of NAM.
Silent speech communication is often used to support speechtelecommunication between persons in very different environments. NAM will be used by a person in a relatively quiet
environment to speak to the other while keeping silent. On theother hand, the other person is not always in such a special
environment. In this paper, as one of the typical situations insilent speech communication, we assume the situation where
NAM uttered by a speaker in a quiet environment needs to beconveyed to a listener in noisy environments.
II. NAM ENHANCEMENT METHODS BASED ON
STATISTICAL VOICE CONVERSIONIn statistical NAM enhancement methods, acoustic featuresof NAM uttered by a speaker are converted into those of more
intelligible and natural voices, such as normal speech or awhispered voice, uttered by the same speaker. These methods
consist of training and conversion processes. In the training
process, we train a Gaussian mixture model (GMM) to model
the joint probability density of the source and target acoustic
features [6] using a corresponding joint feature vector set
generated by performing automatic time frame alignment fora parallel data set consisting of NAM and target speech. In theconversion process, the source acoustic features are converted
into the target acoustic features based on maximum likelihood
estimation of speech parameter trajectories considering global
variance (GV) [5].

APSIPA 2014

A. Conversion from NAM to Normal Speech (NAM-to-Speech)
Spectral segment features [7] of NAM are used as thesource features. On the other hand, spectral and excitation
features of normal speech are used as the target features,
where F0 including unvoiced/voiced (U/V) information andaperiodic components [8] are used as the excitation features.
We separately train three GMMs for converting the spectral
segment features of NAM to the individual target features ofnormal speech, i.e., the spectral, F0 , and aperiodic features. In
the conversion process, the spectral segment features of NAM
are converted into each of the target features of normal speechusing the corresponding GMM. After a mixed excitation signal[9] is designed according to the converted F0 and aperiodic
features, the converted speech is generated by filtering themixed excitation signal with the converted spectral features.
B. Conversion from NAM to Whisper (NAM-to-Whisper)
The whispered voice is totally unvoiced speech. It sounds
much more natural and intelligible compared to NAM. Therefore, naturalness and intelligibility of NAM are significantly
improved by NAM-to-Whisper conversion. As the source
features, the spectral segment features of NAM are used as in
NAM-to-Speech. On the other hand, as the target feature, only
the spectral features of the whispered voice are used because
white Gaussian noise can be always used as an excitation
signal in synthesizing a whispered voice. Therefore, we needto train a single GMM for converting the spectral segment
features of NAM into the spectral features of the whispered
voice. In the conversion process, we perform the spectral
conversion in the same manner as in NAM-to-Speech.
III. INVESTIGATION OF TARGET SPEECH TO
IMPROVE INTELLIGIBILITY IN NOISYENVIRONMENTS
The improvements of intelligibility of NAM through NAM
enhancement systems is the most important factor to make itpossible to use NAM in silent speech communication. Moreover, because NAM is a special speaking style, environmental
situations where a speaker uses NAM will be relatively limited,e.g., the speaker utters in NAM to avoid annoying others inquiet environments such as in a library. On the other hand,the environmental situation of the listener is not limited atall. It is highly possible that the listener calls a speaker innoisy environments. Therefore, it is worthwhile to investigate
intelligibility of the enhanced speech in various environments.
In this paper, we investigate what kinds of target speechyield larger improvements in intelligibility of NAM in somenoisy environments. Although NAM-to-Whisper outperforms
NAM-to-Speech in a very quiet environment such as insoundproof room as reported in [3], it is possible that NAMto-Speech conversely outperforms NAM-to-Whisper in noisy
environments because F0 patterns of voiced speech could be
useful as an acoustic cue to perceive speech sounds. We also
propose voicing of a whispered voice and electrolaryngeal(EL) speech [10] as new forms of target voiced speech.
A. Voicing of Whispered VoiceAs reported in [3], spectral conversion accuracy of NAMto-Whisper is higher than that of NAM-to-Speech. Therefore,ED&ĞĂƚƵƌĞĞǆƚƌĂĐƚŝŽŶ
^ƉĞĐƚƌĂůƐĞŐŵĞŶƚĨĞĂƚƵƌĞƐ

ŽŶǀĞƌƐŝŽŶ
ǁŝƚŚ'DDƐƉEDͲƚŽͲtŚŝƐƉĞƌ EDͲƚŽͲ>ŽŶǀĞƌƐŝŽŶǁŝƚŚ'DDĨϬĨŽƌEDͲƚŽͲ^ƉĞĞĐŚ

ŽŶǀĞƌƐŝŽŶ
ǁŝƚŚ'DDĂƉĨŽƌEDͲƚŽͲ^ƉĞĞĐŚ

&ϬƉĞƌŝŽĚŝĐĐŽŵƉŽŶĞŶƚƐ

^ƉĞĐƚƌƵŵ

DŝǆĞĚĞǆĐŝƚĂƚŝŽŶŐĞŶĞƌĂƚŝŽŶ

&ŝůƚĞƌŝŶŐ

ǆĐŝƚĂƚŝŽŶƐŝŐŶĂů
ŽŶǀĞƌƚĞĚƐƉĞĞĐŚ

Fig. 1. Conversion process of proposed methods

we propose another NAM enhancement method to convert
NAM to a new target voice generated by the spectral feature of
the whispered voice and the excitation features of the normal
speech as shown in Fig. 1. The spectral features are determinedwith the GMM for NAM-to-Whisper. On the other hand,the excitation features are predicted with GMMs in NAM-toSpeech. We also implement continuous F0 patterns (CF0 ) [10]as the F0 features because it yields significant improvements
in prediction accuracy of F0 patterns. The enhanced speech isdenoted as “converted whisper+CF0 .”B. Electrolaryngeal Speech
Although “converted whiper+CF0 ” is totally voiced speech,
a combination of the spectral feature of unvoiced speech(i.e., the whispered voice) and the excitation features ofvoiced speech (i.e., the normal speech) causes a mismatchof unvoiced/voiced information between the spectral and excitation features. This mismatch might cause adverse effects
on intelligibility because we never use a voice suffering fromsuch mismatch in actual speech communication.
To avoid this mismatch, we also propose a conversion
method from NAM to another new target voice based on
EL speech [10]. The EL speech is produced by one of thealternative speaking methods for laryngectomees who have removed their vocal codes. It is produced by articulating voiced
excitation signals artificially generated from an electrolarynx.
Consequently, EL speech is totally voiced speech. Moreover, it
has been reported that intelligibility of EL speech is relatively
high [11]. The conversion process of this method is also
shown in Fig. 1. In the training process, we record EL speech
produced by the same speaker as in NAM recording. Then,
we train another GMM for converting the spectral segment
features of NAM into the spectral features of EL speech. In theconversion process, the spectral features are determined withthe trained GMM. On the other hand, the excitation features
are determined with the same GMMs as used in generating“converted whisper+CF0 .” The enhanced speech is denoted as“converted EL+CF0 .”IV. E XPERIMENTAL E VALUATIONS
A. Experimental Conditions
We recorded normal speech, a whispered voice, and EL
speech using an air-conductive microphone, and NAM usinga NAM microphone as well as an air-conductive microphone.
The speaker was one Japanese male. He uttered 50 sentences
included in the ATR phonetically balanced sentence set [12].
He also uttered the most unfamiliar 160 words included in
the familiarity controlled word lists 2007 [13] in NAM. Allof these words consisted of 4 Japanese mora. The sampling
frequency was set to 16 kHz.
In the NAM enhancement methods, the 0th through 24th
mel-cepstral coefficients were used as the spectral parameters.
STRAIGHT analysis [14] was used for normal speech andEL speech. Mel-cepstral analysis was used for the whispered
voice and NAM. As the excitation parameters, F0 and aperiodic components were extracted from normal speech withSTRAIGHT analysis. The shift length was set to 5 ms. Current± 4 frames were used for extracting the spectral segment
feature of NAM.
We conducted a 5-fold cross validation test in which 40
utterance pairs were used for training, and the remaining 10
utterance pairs were used for evaluation. We train three GMMs
for spectral conversion, 1) GMM SP for converting into thespectral features of the normal speech, 2) GMM WH for
converting into those of the whispered voice, and 3) GMM EL
for converting into those of the EL speech. We also train
three GMMs for the excitation prediction, 1) GMM F0 for
converting into the F0 features of the normal speech, 2)
GMM CF0 for converting into the continuous F0 featuresof the normal speech, and 3) GMM AP for converting into
the aperiodic features of the normal speech. The numbers of
mixture components was set to 64 for the spectral conversion,
32 for the F0 and CF0 prediction, and 16 for the aperiodic
prediction. Note that the air-conducted NAM simultaneously
recorded with (body-conducted) NAM was used to improveaccuracy of time frame alignment between NAM and the target
voices. Table 1 and Table 2 show conversion accuracy of the
trained GMMs evaluated by the cross-validation test.
We conducted an opinion test on listenability using a 5scaled opinion score (1: Bad, 2: Poor, 3: Fair, 4: Good, and 5:
Excellent) as in [11]. The following 8 kinds of speech samples
were evaluated:• NAM: the original NAM
TABLE I
SPECTRAL CONVERSION ACCURACY OF EACH MODEL.
MEL-CEPSTRAL DISTORTION IS CALCULATED WITHOUT
POWER INFORMATION
Mel-cepstral distortion

GMM SP
4.2 dB

GMM WH4.5 dBGMM EL
5.4 dBTABLE IIEXCITATION FEATURE PREDICTION ACCURACY OF EACH
MODEL
Aperiodic distortion
U/V error rate
F0 correlation coefficient

GMM F0
N/A
8.9 %
0.43

WH: the original whisper
EL: the original EL speech• SP: the original normal speech• CVSP: the enhanced speech by NAM-to-Speech using
GMM SP, GMM F0 , and GMM AP• CVWH: the enhanced speech by NAM-to-Whisper using
GMM WH• CVWH+CF0 :the enhanced speech “converted
whisper+CF0 ” using GMM WH, GMM CF0 , and
GMM AP• CVEL+CF0 : the enhanced speech “converted EL+CF0 ”
using GMM EL, GMM CF0 , and GMM AP
These samples were presented to listeners using a headphone in a soundproof room assuming three conditions: 1) a
quiet environment without external noise, 2) a noisy environment by adding office noise so that SNR was set to 0 dB,
and 3) another noisy environment by adding office noise sothat SNR was set to -10 dB. The SNR was set to these values
considering that A-weighted sound pressure level of speechpresented from cell phones (55 dB(A)), that of the officenoise (55 dB(A)), and that of the babble noise (65 dB(A)).
The number of listeners was 12. Each listener evaluated 240
samples consisting of 10 samples from each kind of speechand each condition.
We also conducted a dictation test on intelligibility. We
evaluated speech samples of NAM, CVSP, CVWH, andCVWH+CF0 using utterances of the unfamiliar words. We
considered two environments, the quiet environment without
external noise and the noisy environment by adding 0 dB
SNR of the office noise. The number of listeners was 4. Eachlistener evaluated 160 samples consisting of 20 samples fromeach kind of speech and environment.
••

GMM CF0
N/A
13.5 %
0.53

GMM AP
3.6 dB
N/A
N/A

B. Experimental Results
The result of the opinion test on listenability is shown in
Fig. 2. For the original speech, SP has the highest score and
NAM has the lowest score in every environment. On the other
hand, the score of WH is higher than that of EL in the quiet
environment, they become almost equal as the external noiselevel increases. This result shows that listenability of unvoiced
speech (i.e., the whispered voice) is more sensitive to theexternal noise level than that of the voiced speech.
For the converted speech, we can see that most of theNAM enhancement methods yields significant improvements
in listenability in every environment. The score of CVSP is
almost equal to or better than that of CVWH. Moreover, ina comparison between CVWH and CVWH+CF0 , we can see
a tendency that the score of CVWH is higher than that of
CVWH+CF0 in the quiet environment but they are reversed
as the noise level increases. Namely, the voiced speech is morerobust against external noise in terms of listenability. It isinteresting that the enhanced speech CVWH+CF0 outperforms
the original whispered voice WH in the noisy environment
with -10 dB SNR. On the other hand, the score of CVEL+CF0
is much lower than the other types of enhanced speech. Thisis caused by poor spectral conversion accuracy as shown in
Table 1.
Table 3 shows the result of the dictation test on intelligibility. The score shows mora correct rate. It can be observed that

Quiet environment: 0 dB: -10

for generating intelligible enhanced speech in noisy conditions.
V. C ONCLUSIONSIn this paper, we have investigated what kinds of target
speech are more effective for generating intelligible speech
by nonaudible murmur (NAM) enhancement assuming oneof the typical situations of silent speech communication,
where NAM is uttered by a speaker in a quiet environment
and the enhanced speech is conveyed to a listener in noisy
environments. The experimental result has demonstrated thatthe conversion process from NAM into voiced speech suchas normal speech or voicing of a whispered voice is morerobust to external noise than that into unvoiced speech suchas the whispered voice. We plan to investigate more about
target speech for making the enhanced speech more intelligible in NAM enhancement. We will also implement several
techniques to improve intelligibility of normal speech in noisy
environments for NAM enhancement.
ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Numbers: 26280060 and 23240023.
R EFERENCES

Fig. 2. Result of opinion test on listenability (top figure shows quiet environment, middle one shows SNR = 0 dB, bottom one shows SNR = -10 dB)

NAM is less intelligible but the NAM enhancement methods
are capable of effectively improving its intelligibility. The
intelligibility of CVWH is greatly degraded in the noisy environment. On the other hand, degradation of the intelligibilityof CVSP and CVWH+CF0 caused by the external noise ismuch smaller than that of CVWH. This tendency is the sameas observed in the previous test on listenability; i.e., voiced
speech is more robust to external noise than unvoiced speech.
Moreover, we can also see that CVSP is again better than
CVWH, in cotrast to the results reported in [3]. It is possiblethat this inconsistency is caused by speaker differences ordifferences of utterances to be evaluated. Investigating thismore thoroughly is future work.
From these results, it is demonstrated that the NAM enhancement methods converting NAM to voiced speech such as
normal speech or voicing of the whispered voice are effective
TABLE IIIR ESULT OF DICTATION TEST ON INTELLIGIBILITY
Mora correct rate
Mora correct rate
(Quiet environment)(SNR = 0 dB)
NAM
55.0%
42.5%
CVSP
67.8%
63.1%
CVWH
61.6%
55.0%
CVWH+CF0
65.3%
62.5%

[1] B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, “Silent speech interfaces,” Speech Commun, vol. 52, no. 4, pp. 270287, 2010.
[2] Y. Nakajima, H. Kashioka, N. Cambell, and K. Shikano, “Non-audiblemurmur(NAM) recognition,” IEICE Trans. Inf. Syst., vol. E89-D, no. 1,
pp. 1-8, 2006.
[3] T. Toda, M. Nakagiri, K. Shikano, “Statistical voice conversion techniques
for body-conducted unvoiced speech enhancement,” IEEE Trans. Audio,
Speech, Lang. Process., vol. 20, no. 9, pp. 2505-2517, Sep. 2012.
[4] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Trans. Speech Audio Process., vol.
6, no. 2, pp. 131-142, Mar. 1998.
[5] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum likelihood estimation of spectral parameter trafectory,” IEEE
Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2222-2235, Nov.
2007.
[6] A. Kain and M.W. Macon, “Spectral voice conversion for text-to-speechsynthesis,” in Proc. ICASSP, Seattle, WA, pp. 285-288, May.1998[7] T. Toda, A.W. Black, K. Tokuda, “Statistical mapping between articulatory movements and acoustic spectrum using a Gaussian mixture model,”Speech Commun., vol. 50, no. 3, pp. 215-227, 2008.
[8] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction andcontrol using mixed mode excitation and group delay manipulation
for a high quality speech analysis, modification and synthesis system
STRAIGHT,” in Proc. MAVEBA, Firenze, Italy, pp. 2266-2269, Sep. 2006.
[9] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum likelihood
voice conversion based on GMM with STRAIGHT mixed excitation,” inProc. INTERSPEECH, Pittsburgh, PA, Sep.2006, pp. 2266-2269.
[10] K. Tanaka, T. Toda, G. Neubig, S. Sakti, S. Nakamura, “A Hybrid
Approach to Electrolaryngeal Speech Enhancement Based on Noise
Reduction and Statistical Excitation Generation,” IEICE Transactions on
Information and Systems, Vol. E97-D, No. 6, pp. 1429-1437, Jun. 2014.
[11] H. Doi, T. Toda, K. Nakamura, H. Saruwatari, K. Shikano, “Alaryngeal
Speech Enhancement Based on One-to-Many Eigenvoice Conversion.,”IEEE Trans. Audio, Speech, Lang. Process., vol. 22, no. 1, pp. 172-183,
Jan. 2014.
[12] Y. Sugisaka, K. Takeda, M. Abe, S. katagiri, T. Umeda, and H. Kuwabata, “A large-scale Japanese speech database,” inProc. ICSLP90, Kobe,
Japan, Nov. 1990, pp.1089-1092.
[13] T. Kondo, S. Amano, S. Sakamoto, and Y. Suzuki, “Spoken WordIntelligibility of Young and Old Adults with Familiarity-Controlled Word
Lists 2007 (FW07),” Proc. ICP 2008, PS-Tue-pm-157, 2008.
[14] H. Kawahara, H .Katayose, A. de Cheveigne, and R. D. Patterson,“Fixed point analysis of frequency to instantaneous frequency mapping
for accurate estimation of F0 and periodicity,” in Proc. EUROSPEECH,
Budapest, Hungary, pp. 2781-2784, Sep. 1999.