NOISE SUPPRESSION METHOD FOR BODY-CONDUCTED SOFT SPEECHENHANCEMENT BASED ON EXTERNAL NOISE MONITORING
Yusuke Tajiri†

Tomoki Toda‡

Satoshi Nakamura†† Graduate School of Information Science, Nara Institute of Science and Technology, Japan‡ Information Technology Center, Nagoya University, Japan{tajiri.yusuke.tk0, s-nakamura}@is.naist.jp, tomoki@icts.nagoya-u.ac.jp

ABSTRACTThis paper presents a novel approach to suppressing adverse effects
of external noise on body-conducted soft speech for silent speechcommunication in noisy environments. Nonaudible murmur (NAM)
microphone as one of the body-conductive microphones is capable of detecting very soft speech. However, body-conducted soft
speech easily suffers from external noise owing to its faint volume.
To address this issue, the proposed method additionally uses an airconductive microphone to detect only an external noise signal anduses the detected external noise signal to suppress its effect on thebody-conducted soft speech. A semi-blind source separation technique is applied to the proposed method for estimating a linear filter
to suppress the noise components without voice activity detection.
Experimental results demonstrate that the proposed method yields 10
dB SNR improvements in 80 dBA noisy conditions and also yields
significant improvements in sound quality of body-conducted soft
speech.
Index Terms— silent speech communication, nonaudible murmur microphone, noise suppression, external noise monitoring,semi-blind source separation
1. INTRODUCTION
Speech communication plays a principal role in our daily life as themost efficient human communication method. In recent decades,
thanks to mobile phone or other devices, we have come to be able
to talk with each other beyond limitations of distance and location.
However, there still exist some situations where we hesitate to talkwith others using those devices; e.g., we have difficulty in talking about private information in a crowd; and speaking itself would
sometimes annoy others in quiet environments such as in a library.
Recently, silent speech interfaces [1] have attracted attention asa technology to achieve a new style of speech communication. They
enable us to talk with each other without the necessity of emitting an
audible acoustic signal. To detect silent speech, several sensing devices have been explored as alternatives to a usual air-conductive microphone, such as body-conductive microphones [2, 3], electromyography [4], ultrasound imaging [5], and so on.
We especially focus on nonaudible murmur (NAM) microphone[3] as one of the body-conductive microphones capable of detecting
silent speech. It has been originally developed to detect an extremely
soft whispered voice called NAM, which is so quiet that people
around the speaker barely hear its emitted sound. Figure 1 shows astructure of the NAM microphone and its setting position. The NAM
This work was supported in part of JSPS KAKENHI Grant Numbers:
15K12064 and 26280060.

978-1-4799-9988-0/16/$31.00 ©2016 IEEE

5935

Fig. 1. Setting position and structure of NAM microphone.
microphone is capable of detecting various types of speech, such as
NAM, a whispered voice, a soft voice, and normal speech, from thissetting position through only the soft tissues of the head. It is also
more robust against external noise owing to its noise-proof structure compared to the usual air-conductive microphone. Althoughsevere degradation of speech quality is caused by essential mechanisms of body conduction [6], the body-conducted speech detectedwith the NAM microphone still has a great potential to be used as acommunication medium if people get used to hear its special soundquality. Moreover, there are several attempts to directly address thisquality-degradation issue by developing body-conducted speech enhancement techniques to convert it into air-conducted speech using
a statistical voice conversion technique [7, 8].
There still remain some issues to be addressed in order to makeit possible to practically use the body-conducted soft speech detected with the NAM microphone for silent speech communication.
Although the previous work uses the body-conducted soft speechrecorded in a sound-proof room [7], external noise usually exists
in real environments. Even though the NAM microphone is robust
against external noise, it cannot completely block external noise signals. Therefore, their effect on the body-conducted speech signal is
not ignored. In particular, when detecting soft speech like NAM or awhispered voice, its body conducted speech signal significantly suffers from external noise because power of soft speech is too small.
To address this issue, several enhancement methods additionally using the air-conducted noisy speech signal detected with theusual air-conductive microphone have been proposed. For example, the direct filtering method [9] and the statistical enhancement
method [10] have been proposed although these methods actuallydeal with speech enhancement with a bone-conductive microphone
under heavy noisy conditions. Inspired by these methods, a bodyconducted soft speech enhancement method additionally using airconducted soft speech detected under noisy conditions has been proposed and its effectiveness has been reported in [11]. On the other
hand, it has also been reported that adaptation of the statistical enhancement model to each external noise condition is essential. However, it is not straightforward to accurately adapt the model to arbi-

ICASSP 2016

Table 1. SNR of NAM detected with each of air- and bodyconductive microphones under several noisy conditions
Noise
SNR (air) [dB] SNR (body) [dB]office 50 dBA
-12.7
12.7
crowd 60 dBA
-17.1
10.1
booth 70 dBA
-27.4
5.5
station 80 dBA
-36.9
-2.8
Fig. 2. Setting position of body- and air-conductive microphones.
trary noisy conditions, and consequently model adaptation performance is usually limited. Therefore, it is worthwhile to develop afront-end noise suppression technique robust against any external
noisy conditions for reducing the external noise components as muchas possible.
In this paper, we propose a noise suppression method for the
body-conducted soft speech based on external noise monitoring using the air-conductive microphone. Because power of emitted soft
speech such as NAM is too small, it is possible to detect only an
external noise signal by setting the air-conductive microphone to aplace not close to the speaker’s mouth, e.g., a place just close to
the setting position of the NAM microphone. The proposed method
effectively uses the air-conducted external noise signal to suppress
its components detected with the NAM microphone, which is called
the body-conducted external noise signal in this paper. A semi-blindsource separation technique [12] is used to estimate a linear filter
to suppress the body-conducted external noise signals without voiceactivity detection. Several experimental evaluations are conducted,
demonstrating that the proposed method is capable of significantly
suppressing the body-conducted external noise signals to enhance
the body conducted soft speech and the semi-blind source separation
technique is more effective compared to other alternative techniques.
2. EXTERNAL NOISE MONITORING USING
AIR-CONDUCTIVE MIROPHONE
NAM is an extremely soft whispered voice and it is relatively difficult to be detected with the usual air-conductive microphone inparticular under noisy conditions because of its faint volume. Considering this property of NAM, the proposed method uses the airconductive microphone to detect only the external noise signal, andthen uses the detected signal to suppress the body-conducted external noise signal. It is expected that a proper setting position of the
air-conductive microphone will be 1) far away from the speaker’s
mouth in order to reduce components of the emitted NAM signal detected with the air-conductive microphone as much as possible and2) close to the setting position of the NAM microphone in order todetect the air-conducted external noise signal of which acoustic characteristics are similar to those of the body-conducted external noise
signal detected with the NAM microphone. Thus in this paper we
set the air-conductive microphone as shown in Fig. 2.
As a preliminary experiment, we recorded some types of external noise signals with the NAM microphone and the air-conductive
microphone, and also separately recorded the NAM signal with both
microphones. Then, the signal-to-noise ratio (SNR) of the NAM signal detected with each microphone was calculated. Table 1 shows
results. We can see that SNR of the body-conducted NAM is muchhigher than that of the air-conducted NAM thanks to the noise-proof
structure of the NAM microphone but it starts to significantly decrease when the sound pressure level of external noise is higher than5936

60 dBA. Namely, it is expected that the noise suppression process is
not necessary up to 60 dBA of the sound pressure level of external
noise but it is essential under more than 60 dBA noisy conditions.
We can also see that the SNR of the air-conducted NAM signal isvery low under more than 60 dBA noisy conditions (i.e., booth 70
dBA and station 80 dBA). Therefore, in such noisy conditions, thesignal detected with the air-conductive microphone will be well approximated by only the external noise signals without including theemitted NAM signal. Please note that this preliminary experimentignores the Lombard reflex [13] that is also observed in NAM [14];i.e., as the external noise level is higher, power of the emitted NAM
is usually larger. Therefore, actual SNR is expected to be higher
than shown in Table 1 in particular under 70 dBA and 80 dBA noisy
conditions.
Considering these results, we assume the following mixing process in the proposed method:
x1 (t)x2 (t)=≈s1 (t) + a(t) ∗ s2 (t)s2 (t)(1)(2)

where x1 (t) is an observed signal detected with the NAM microphone, x2 (t) is an observed signal detected with the air-conductive
microphone, s1 (t) is a clean body-conducted NAM signal, s2 (t) is
an air-conducted external noise signal, and a(t) is a transfer function
to compensate acoustic differences between the external noise signal
detected with the air-conductive microphone and that detected withthe body-conductive microphone. In the proposed noise suppression
method, the clean body-conducted NAM signal s1 (t) is extracted
from the observed signals x1 (t) and x2 (t) by estimating the transfer
function a(t). This problem is equivalent to a classical acoustic echo
cancellation (AEC) problem [15], i.e., the observed signal x2 (t) isregarded as a reference signal and the transfer function a(t) is regarded as an echo path. Therefore, typical AEC methods such as the
Least Mean Square (LMS) algorithm [16] and the Recursive Least
Squares (RLS) algorithm [16] can be used in the proposed method.
3. NOISE SUPPRESSION METHOD BASED ON
SEMI-BLIND SOURCE SEPARATION
The typical AEC methods are originally designed to cancel only farend echo, and they don’t work well when both the far-end and nearend speakers utter simultaneously. This situation is well known as
double-talk. Therefore, double-talk detection is usually needed to remove the corresponding data segments including near-end speaker’s
voices from the observation data used for estimating the echo path.
The same problem occurs in the proposed method, i.e., the situation
where NAM is uttered is regarded as double-talk. Therefore, it isnecessary to perform voice activity detection (VAD) of NAM andto remove the detected voice activity segments from the observation
data to estimate the transfer function a(t).
In this paper, we apply a semi-blind source separation (semiBSS) technique to the proposed method in order to achieve doubletalk free noise suppression. Let us assume an observed signal vector

x(ω, τ ) = [x1 (ω, τ ), x2 (ω, τ )]⊤ consisting of the observed signals
and a source signal vector s(ω, τ ) = [s1 (ω, τ ), s2 (ω, τ )]⊤ consisting of the clean body-conducted NAM signal s1 (ω, τ ) and the airconducted external noise signal s2 (ω, τ ), where ω is frequency bin
index, τ is time frame index, and ⊤ is vector transpose. Because theobserved signals are modeled as convolutive mixtures in the time
domain as shown in Eqs. (1) and (2), the observed signal vectorx(ω, τ ) is modeled as instantaneous mixture in the frequency domain as follows:x(ω, τ ) = A(ω)s(ω, τ )(3)where A(ω) is a (2 × 2) mixing matrix, which is assumed to betime-invariant in this paper. BSS is a technique to automaticallyfind a (2 × 2) un-mixing matrix W (ω) that can separate the source
signals from the observation signals as follows:y(ω, τ ) = W (ω)x(ω, τ )(4)where y(ω, τ ) = [y1 (ω, τ ), y2 (ω, τ )]⊤ is the separated signal vector. Independent component analysis (ICA) [17] is often used toestimate the un-mixing matrix. In the proposed method, one of thetwo source signals (i.e., s2 (ω, τ )) is known by the external noisemonitoring as shown in Eq. (2). Therefore, we can formulate theproposed method using semi-BSS rather than BSS. Some elementsof the un-mixing matrix W can be explicitly given as follows:[]
w11 (ω) w12 (ω)
W (ω) =.
(5)0
1
Therefore, it is necessary to estimate only a part of the un-mixing
matrix, an un-mixing vector w = [w11 (ω), w12 (ω)]. In this paper,
we use ICA based on natural gradient [18] to estimate the un-mixing
vector. The un-mixing vector is iteratively updated as follows:
∆w
w(ω)

=
←

η{w(ω) − ⟨φ(y1 (ω, τ ))y(ω, τ )H ⟩τ W (ω)} (6)w(ω) + ∆w(7)where H is Hermitian transpose, η is a step-size parameter, ⟨·⟩τ is atime average operator, and φ(y1 (ω, τ )) is a nonlinear function likea polar function given byφ(y1 (ω, τ )) = tanh(|y1 (ω, τ )|) exp(jarg(y1 (ω, τ ))).

(8)

To handle scaling indeterminacy in ICA, a projection back method[19] is also applied. Consequently, the estimated clean bodyconducted NAM signal is given byy1 (ω, τ ) = x1 (ω, τ ) +w12 (ω)x2 (ω, τ ).
w11 (ω)(9)

4. EXPERIMENTAL EVALUATIONS
4.1. Experimental conditions
We simultaneously recorded clean body- and air-conducted NAM
signals with the NAM microphone and the air-conductive microphone, respectively in a sound-proof room. The setting positionof these microphones was the same as shown in Fig. 2. We also
recorded body- and air-conducted signals of the following 3 kinds ofnoise using the same microphone settings by presenting them froma loud speaker in the sound-proof room.
• crowd60dB: 60 dBA crowd noise• booth70dB: 70 dBA booth noise

5937• station80dB: 80 dBA station noise
The sound pressure levels of the individual noises were measured
by a sound level meter placed at around the speaker’s head. The
noise signals recorded with the NAM microphone and the airconductive microphone were superimposed on the clean body- andair-conducted NAM signals. The resulting signals were effectively
used in objective evaluations because some distortion measures
between the processed body-conducted NAM signal and the clean
body-conducted NAM signal could be calculated as evaluation metrics. On the other hand, the Lombard reflex was ignored in thesesimulated signals as mentioned in Section 2. Therefore, we also
recorded the noisy NAM signals with both the NAM microphone
and the air-conductive microphone while presenting the noise signals. The recorded signals more accurately simulated the noisy
NAM signals detected in real conditions including the Lombard reflex although the distortion measures could no longer be calculated.
Therefore, they were used in a subjective evaluation.
Fifty sentences in a phoneme balanced sentence set were utteredin NAM. The sampling frequency was set to 16 kHz. The window
length of STFT was set to 64 ms and the shift length was set to 32
ms. For the semi-BSS, the step-size parameter η was set to 0.01. The
number of iterations was varied from 5 to 200.
To investigate the effectiveness of the proposed method using
noise suppression based on the external noise monitoring, the following 7 methods were evaluated.
• unprocessed: no noise suppression• upper bound: upper bound of time-invariant linear filter inthe proposed method• LS w/ VAD: least squares (LS) with ideal VAD in the proposed method• NLMS w/ VAD: AEC based on the normalized LMS [20]with ideal VAD in the proposed method• APA w/ VAD: AEC based on the affine projection algorithm(APA) [21] with ideal VAD in the proposed method• RLS w/ VAD: AEC based on the RLS with ideal VAD in the
proposed method• semi-BSS: the semi-BSS with no VAD in the proposed
methodIn upper bound, we used the transfer function estimated by the LS
algorithm using the recorded noise signals detected with the NAM
microphone and the air-conductive microphone before their superimposition on the clean NAM signals. In LS w/ VAD and the AEC
algorithms (i.e., NLMS w/ VAD, APA w/ VAD, and RLS w/ VAD),
we performed double-talk detection based on ideal VAD results,
which were determined by applying VAD to a clean air-conducted
NAM signal with high SNR recorded with another air-conductive
microphone set to a place just close to the speaker’s mouth. LS w/VAD uses the batch-type filter estimation process while the AEC
algorithms use the adaptive filter estimation process. On the other
hand, in semi-BSS, the filter estimation process was performed incompletely unsupervised manner without VAD.
4.2. Objective evaluations
Noise suppression performance was evaluated using SNR and melcepstral distortion (MCD) calculated using the 1st through 24th melcepstral coefficients between the estimated body-conducted NAM
signals and the clean body-conducted NAM signals.
First, we evaluated the SNR improvements yielded by semiBSS. We also investigated whether or not the assumption used in20

6

unprocessed
upper boundsemi-BSS w/ ideal datasemi-BSS

50
100 150 200
Number of iterations

Signal-to-noise ratio [dB]40

NLMS w/ VAD
APA w/ VAD
RLS w/ VAD

semi-BSS

10
50
booth70dB

station80dB

Fig. 4. SNR of body-conducted NAM.

8

Mel-cepstral distortion [dB]8

unprocessed
upper boundLS w/ VAD

15

crowd60dB

unprocessed
upper boundsemi-BSS w/ ideal datasemi-BSS

50
100 150 200
Number of iterations

6.0

unprocessed
upper boundLS w/ VAD

5.5

the proposed method (shown in Eq. (2)) holds. Figure 3 shows
the improvement in SNR yielded by semi-BSS. SNR is significantly
improved even after only five iterations compared to unprocessed.
More iterations make SNR improvements close to that by upperbound. Moreover, in Fig. 3, ideal data indicates results of using
the air-conducted signals including only external noise signals withno superimposition (i.e., simulated data when x2 (t) is completely
equivalent to s2 (t) in Eq. (2)). We can see that there is almost no
difference between semi-BSS and semi-BSS w/ ideal data. This reveals that the assumption in Eq. (2) actually holds in the proposed
external noise monitoring method.
Then, we evaluated performance of individual noise suppression
algorithms. The number of iterations in semi-BSS was set to 200.
For the individual AEC algorithms, their parameters were optimized(except for the number of constrains in APA) so that SNR was maximized. The optimized parameter settings are shown in Table 2.
Figures 4 and 5 show SNR and MCD of the body-conducted NAM
signals estimated by each method, respectively. We can see that SNR
and MCD are significantly improved by the proposed method using
any noise suppression algorithm. Especially, semi-BSS yields significantly large improvements in both SNR and MCD under 80 dBA
noisy conditions even though no VAD is needed.
4.3. Subjective evaluation
We conducted an opinion test on sound quality using a 5-point opinion scale, such as 1: very bad, 2: bad, 3: fair, 4: good, and 5:
excellent. We evaluated the clean body-conducted NAM signals,
the unprocessed noisy body-conducted NAM signals, and the bodyconducted NAM signals estimated by semi-BSS. The number of listeners was 10. Each listener evaluated 18 samples for each method.
Figure 6 shows the result. The sound quality of estimated bodyconducted NAM signals is significantly improved by the proposed
method under 70 dBA both and 80 dBA station noisy conditions. It
is interesting that the result under 70 dBA noisy condition is sim-

5938

NLMS w/ VAD
APA w/ VAD
RLS w/ VAD

semi-BSS

5.0
4.5
4.0
3.5
3.0
crowd60dB(a) booth70dB(b) station80dB
Fig. 3. Improvement in SNR yielded by semi-BSS-based noise suppression.

booth70dB

station80dB

Fig. 5. Mel-cepstral distortion of body-conducted NAM.
ilar to that under 80 dBA noisy condition. This is because several
acoustic changes are caused by the effect of Lombard reflex and theNAM signal itself tends to be more intelligible as the external noiselevel increases. Therefore, the Lombard reflex doesn’t cause any adverse effects in the proposed method and it is actually helpful for
improving quality of the estimated NAM signal.
5. CONCLUSIONSThis paper has presented a noise suppression method for the bodyconducted soft speech based on external noise monitoring using the
air-conductive microphone. We have shown that only an external
noise signal can be detected with the air-conductive microphone bysetting it to a place far away from the speaker’s mouth and the detected signal can be effectively used to estimate a linear filter to suppress the external noise components observed in the body-conductedsignals. We have also applied a semi-blind source separation technique to the proposed method to make it possible to estimate thelinear filter without voice activity detection. Experimental results
have demonstrated that the proposed method yields significant improvements in both estimation accuracy and sound quality of bodyconducted soft speech.
Mean opinion score (MOS)

10

Signal-to-noise ratio [dB]

Signal-to-noise ratio [dB]

Table 2. Parameter settings for individual AEC algorihtnms
NLMS
Filter length: 64 ms
Step size: 0.01(60dBA), 0.05(70dBA), 0.5(80dBA)
APA
Filter length: 64 ms
Step size: 0.01(60dBA), 0.05(70dBA), 0.25(80dBA)
Number of constrains: 2
RLS
Filter length: 16 ms
Forgetting factor: 0.999(80dBA), 1(60dBA, 70dBA)

5
Confidence interval (95%)

4
3
2
1
clean

unprocessed semi-BSS unprocessed semi-BSS
booth70dB
station80dB

Fig. 6. Result of subjective evaluation on speech quality of bodyconducted NAM.

[16] S. Haykin, Adaptive filter theory, Prentice Hall, fourth edition,
2002.

6. REFERENCES[1] B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, andJ. S. Brumberg, “Silent speech interfaces,” Speech Communication, vol. 52, no. 4, pp. 270–287, 2010.
[2] S-C. Jou, T. Schultz, and A. Waibel, “Adaptation for softwhisper recognition using a throat microphone,” Proc. INTERSPEECH, pp. 1493–1496, 2004.

[17] A. Hyvarinen, J. Karhunen, and E. Oja, Independent component analysis, John Wiley & Sons, 2004.
[18] S. Amari, “Natural gradient works efficiently in learning,” Neural Computation, vol. 10, no. 2, pp. 251-276, 1998.

[3] Y. Nakajima, H. Kashioka, N. Cambell, and K. Shikano, “NonAudible Murmur (NAM) recognition,” IEICE Transactions on
Information and Systems, vol. E89-D, no. 1, pp. 1–8, 2006.

[19] N. Murata, and S. Ikeda, “An on-line algorithm for blind source
separation on speech signals,” Proc. International Symposiumon Nonlinear theory and its Application, vol. 3, pp. 923–926,
1998.

[4] T. Schultz and M. Wand, “Modeling coarticulation in EMGbased continuous speech recognition,” Speech Communication, vol. 52, no. 4, pp. 341–353, 2010.

[20] J. Nagumo, and A. Noda, “A learning method for system identification,” IEEE Transactions on Automatic Control, vol. 12,
no. 3, pp. 282-287, 1967.

[5] T. Hueber, EL. Benaroya, G. Chollet, B. Denby, G. Dreyfus,
M. Stone, “Development of a silent speech interface driven by
ultrasound and optical images of the tongue and lips,” SpeechCommunication, vol. 52, no. 4, pp. 288–300, 2010.

[21] K. Ozeki, and T. Umeda, “An adaptive filtering algorithm using an orthogonal projection to an subspace and its properties,”
Electronics and Communication in Japan, vol. 67-A, no. 5, pp.
19-27, 1984.

[6] T. Hirahara, M. Otani, S. Shimizu, T. Toda, K. Nakajima,
and K. Shikano, “Silent-speech enhancement using bodyconducted vocal-tract resonance signals,” Speech Communication, vol. 52, no. 4, pp. 301–313, 2010.
[7] T. Toda, M. Nakagiri, and K. Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 9, pp. 2505–2517, 2012.
[8] T. Toda, K. Nakamura, H. Sekimoto, and K. Shikano, “Voice
conversion for various types of body transmitted speech,” Proc.
ICASSP, pp. 3601–3604, 2009.
[9] Z. Liu, Z. Zhang, A. Acero, J. Droppo, and H. Huang,“Direct filtering for air- and body-conductive microphones,”
Proc. IEEE 6th Workshop on Multimedia Signal Processing,pp. 363–366, 2004.
[10] A. Subramanya, Z. Zhang, Z. Liu, and A. Acero, “Multisensory
processing for speech enhancement and magnitude-normalized
spectra for speech modeling,” Speech Communication, vol. 50,
no. 3, pp. 228–243, 2008.
[11] Y. Tajiri, K. Tanaka, T. Toda, G. Neubig, S. Sakti, S. Nakamura, “Non-Audible Murmur Enhancement based on Statistical Conversion using Air- and Body-Conductive Microphones
in Noisy Environments,” Proc. INTERSPEECH, pp. 2769–
2773, 2015.
[12] S. Miyabe, T. Takatani, Y. Mori, H. Saruwatari, K. Shikano,
and Y. Takakura, “Double-talk free spoken dialogue interface
combining sound field control with semi-blind source separation,” Proc. ICASSP, pp. 809–812, 2006.
[13] J. C. Junqua, “The Lombard reflex and its role on human listeners and automatic speech recognizers,” The Journal of the
Acoustical Society of America, vol. 93, no. 1, pp. 510–524,
1993.
[14] T. Toda, K. Nakamura, T. Nagai, T. Kaino, Y. Nakajima, and K.
Shikano, “Technologies for processing body-conducted speechdetected with Non-Audible Murmur microphone,” Proc. INTERSPEECH, pp. 632–635, 2009.
[15] A. Gilloire, M Vetterli, “Adaptive filtering in subbands withcritical sampling: analysis, experiments, and application toacoustic echo cancellation,” IEEE Transactions on Signal Processing, vol. 2, no. 8, pp. 148-151, 1995.

5939