JAIST Repository
https://dspace.jaist.ac.jp/

Title

A study on restoration of bone‑conducted speech
in noisy environments with LP‑based model andGaussian mixture model

Author(s)Phung, Nghia Trung; Unoki, Masashi; Akagi, Masato

Citation

Journal of Signal Processing, 16(5): 409‑417

Issue Date

2012‑09

Type

Journal Article

Text version

publisher

URL

http://hdl.handle.net/10119/10825

RightsCopyright (C) 2012 信号処理学会. Phung Nghia
Trung, Masashi Unoki and Masato Akagi, Journal of
Signal Processing, 16(5), 2012, 409‑417.

Description

Japan Advanced Institute of Science and Technology

Journal of Signal Processing, Vol. 16, No.5, pp.409-417, September 2012

PAPER

A Study on Restoration of Bone-Conducted Speech in Noisy
Environments with LP-based Model and Gaussian Mixture Model
Phung Nghia Trung, Masashi Unoki and Masato Akagi
School of Information Science, Japan Advanced Institute of Science and Technology
1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan
E-mail: {ptnghia.unoki.akagi}@jaist.ac.jp

Joumalof
Signal Processing

Journal of Signal Processing, Vol. 16, No.5, pp. 409-417, September 2012

PAPER

A Study on Restoration of Bone-Conducted Speech in Noisy
Environments with LP-based Model and Gaussian Mixture Model
Phung Nghia Trung, Masashi Unoki and Masato Akagi
School of Information Science, Japan Advanced Institute of Science and Technology
1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan
E-mail: {ptnghia.unoki.akagi}@jaist.ac.jp

Abstract

The restoration of bone-conducted speech is a very important issue that enables robust
speech communication in extremely noisy environments. We proposed a method of blind restoration in
our previous studies based on a scheme of linear prediction with a method of training and prediction basedon the simple recurrent neural network. However, prediction based on neural networks is not suitable
for training with large corpora, which is necessary for real applications. The over-training problem withsimple recurrent neural networks makes it difficult to train various kinds of bone-conducted speech in one
session. In addition, it is difficult to adapt the neural network model to bone-conducted speech in unknown
noisy environments to build an open dataset restoration of bone-conducted speech. Thus, a method of
training and prediction based on the Gaussian mixture model was used in this research, instead of a neuralnetwork. A method of re-estimating the residual ratio in the scheme of linear prediction is also proposed.
We also investigated how the proposed method works to restore bone-conducted speech in extremely noisy
environments. Objective and subjective evaluations were carried out to evaluate the improvements insound quality and the intelligibility of restored speech. The results revealed that our proposed method
outperformed previous methods in both human hearing and automatic speech recognition systems even in
extremely noisy environments.

Keywords: bone-conducted speech, Gaussian mixture model, linear prediction, speech intelligibility

1.

Introduction"

Speech communication in noisy environments still
remains a challenge. There have been many models
and algorithms to reduce noise in noisy speech, butthere is still a lack of efficient models and algorithms
in extremely noisy environments. Bone-conducted(BC) speech in extremely noisy environments is stable against surrounding noise so that it is able tobe efficiently used for communication instead of airconducted (AC) speech [1].
However, there are two main drawbacks to BC
speech, Le., degradation due to bone conduction andchanging speaker pronunciations due to surrounding
noise, which is referred to as the Lombard effect.
While the Lombard effect is a typical problem, whichis the same as that in AC speech in noisy environments, another is its critical effect on the quality ofspeech. When signals are transmitted through bone
conduction, they are complexly affected by a loss of

Journal of Signal Processing, Vol. 16, No.5, September 2012

sound quality and intelligibility of speech. The degradation varies for different pick-up points (i.e., BC microphone positions), speakers, and the way syllables
are pronounced. This is because the characteristics ofbone conduction vary for different measuring positions
and the distribution of frequency components varies
with speakers who pronounce syllables differently.
There have been many studies on the restoration of BC speech in the literature to overcome thedegradation in BC speech caused by bone conduction. However, the results have still been limited. For
example, a model for restoring BC speech based oncross spectrum has been proposed [2], and long-term
Fourier transform (LTF) has been applied to restore
BC speech [3]. These methods seem the simplest andmost straightforward methods of restoring BC speech,
but they have yielded restored signals with artifacts
such as musical noise and echoes and only achievedslight improvements in voice quality [4]. In addition,these methods have been difficult to apply to blind409

.I LP -+LSF ,y(n)BCspeech

LP analysis(Eqs.5-6)

J Prediction I., Conversion ,1(LSF-SRN) 1

J LSF-+LP I·1 Conversion I

The LP residual ACIBC ratio is fixed as aconstant ky(n)LP analysis(Eqs.5-6)

Prediction(LSF-GMM)(Eq.16)LP -+LSF
Conversion

LSF-+LP
Conversion

J Estimation of the average LP residual II!
LPReSynthesis(a)BC speech

Deriving
H-1(z)(Eq.9)

AC/BC ratio (Eqs. 10-12)

speech-Deriving
H-1(z)(Eq.9)

I

+
(b)

x(n)
Restored

LPResynthesisx(n)RestoredspeechFig. 1 Block diagrams of BC-speech blind restoration based on LP scheme: (a) Our previous model and (b) our
proposed model
restoration.
The approach of using the modulation transfer
function (MTF) to restore BC speech has been proposed [4] to overcome the drawbacks with previous
methods. The MTF-based model has better restoration abilities and yields restored signals with better
intelligibility than the cross-spectrum and LTF methods. However, the quality of speech is still limited and
it is still difficult to predict the model's parameters inblind restoration.
Body-transmitted speech (which is like BC speech)
restoration based on the Gaussian mixture model(GMM) has been proposed [5], which has been
adopted from a technique of voice conversion. In
general, GMM is flexible and available for training
with huge amounts of data. Therefore, it is easy totrain GMM under various conditions and to adapt thetrained models under various conditions to those ofother unknown conditions. This is an advantage ofthe use of GMM in the voice conversion. However,
due to the difficulty of estimating the fundamental
frequency (FO) from these signals, this approach has
only been efficiently applied to unvoiced speech suchas whispered speech.
We proposed a scheme of linear prediction (LP)to restore BC speech in previous studies (6]. Insteadof long-term processing as in traditional methods, weused short-term frame-based processing in this model,
which might be used in real-time practical applications. The inverse filter was built only based on line
spectral frequency (LSF) coefficients without FO estimation. We used a simple recurrent neural network(SRN) to blindly predict the LSF of AC speech fromBC speech after a training process.
The experimental results presented in our previous

410

paper revealed that our method of LSF-SRN, based on
the LP scheme, could adequately improve the quality
and intelligibility of BC speech, and it could also efficiently be applied to blind-restoration and real-time
applications. However t this method also had threeoutstanding problems that needed to be solved to further improve the quality and intelligibility of speech.
Vu et al. [6] assumed that the residual ratio was aconstant. However, these values changed from frame
to frame in the time domain in our current analysis,
and thus they should be optimized for each frame.
The learning method we used in our previous
restoration model was SRN but it is impractical touse SRN for training huge corpora. Due to the overtraining problem with SRN t it is not suitable to train
various kinds of BC speech in one training session.
Another problem with SRN training is that it is difficult to adapt the SRN model to BC speech in unknown
noisy environments. This problem makes it difficult tobuild an open dataset BC speech restoration in noisy
environments.
We only evaluated the performance of an LP
scheme for BC speech restoration in a clean environment in previous studies. Thus, we needed to confirm
whether the LP scheme (both LSF-SRN and our currently proposed LSF-GMM) was useful for restoring
BC speech in noisy environments.
We solved all three outstanding problems to improve the performance of the LP scheme in the restoration of BC speech in this study. Instead of using the
fixed residual ratio for all frames, we approximatedthese ratios as the ratios of the averaged LP residualsof AC /BC frame pairs. To overcome the drawbacks
with SRN, we used GMM to train the joint vector of
LSF and the average LP residual of Be and associ-Journal of Signal Processing, Vol. 16, No.5, September 2012

ated AC speech, since the GMM-based approach canbe used to adapt the trained models of BC speech under various noisy conditions to those of other unknown
noisy conditions and then to restore BC speech in unknown noisy environments. The LSF and averaged
residual of AC speech were then predicted by using
those of BC speech and the trained parameters.
The rest of this paper is organized as follows. The
next section describes the restoration of BC speechbased on the LP scheme. Section 3 explains the problems with the current LP scheme for the restoration of
BC speech. Section 4 explains our improved method.
Section 5 presents evaluations and Section 6 concludes
with a summary and mentions future work.
2.

Restoration of BC Speech Based on LP
Scheme

2.1

Definition of LP scheme- L ax(i)x(n - i)

(1)

i=l

P

y(n)

=

- L ay(i)y(n - i)

(2)

i=l

where x(n) and y(n) are the predicted signals, P is
the LP order, x(n - i) and y( n - i) are the previous
observed values, and ax(i) and ay(i) are the i-th LP
coefficients where i = 1,2,···, P. The residual is obtained by using the error between the current and thepredicted samples.

gx(n) = x(n) - x(n)gy(n) = y(n) - y(n)

(3)
(4)

Here, x( n) and y( n) are represented by the LP modelin the z-domain as:p-Gx(z)= X(z) L ax(i)z-i,ax(O) = -1 (5)i=Op-Gy(z) =Y(z)LSF inverse filtering in LP scheme

Let us assume that the mathematical descriptionof transfer function h( n) from x(n) to y( n) is an Morder FIR filter. It is represented in the z domain
as:
M

L

(8)

Vu et al. [4] demonstrated that the inverse filter could
be represented using LSF parameters.

p

=

2.2

H(z) = Y(z) =h(i)z-i
X(z)i=OThe flow of the LP scheme for the blind restoration
ofBC speech [6] is outlined in Fig. 1. We compute the
LSF parameters of BC speech and corresponding AC
speech to train SRN in the training phase. We haveto predict the LSF parameters of AC speech based on
those of BC speech in the restoration phase and thecorresponding trained SRN parameters. The residual
ratio is fixed as a constant, k. This residual ratio andthe LP parameters restored from LSF parameters areused to derive the inverse filtering function to convert
BC speech to associated AC speech.
Let x(t) and y(t) be AC and associated BC speech
signals. Using LP analysis, the discrete signals, x( n)and y(n), can be represented as:x(n)where X(z) and Y(z) are the z-transforms of x(n) andy(n). Here, Gx(z) and Gy(z) are the z-transforms of
the LP residuals gx(n) and 9y(n). In Vu et al. (6],the residual ratio of x( n) and y( n) in z domain (orfrequency domain) was defined as gain k:

L ay(i)z-i, ay(O) = -1 (6)i=OJournal of Signal Processing, Vol. 16, No.5, September 2012

H-1(z) = k Uy(z) + Vy(z)Ux(z) + Vx(z)(9)

Here, (Uy(z), Vy(z» and (Ux(z), Vx(z» are a symmetric polynomial and an anti-symmetric polynomial for
BC and AC speech that are determined from the LSF
coefficients. Inverse filtering therefore depends on the
LSF coefficients of AC and BC speech and gain k.
3.

Problems with Current LP Scheme for
Restoration of BC Speech

The latest method using the LP scheme to restore
BC speech is known as the method of LSF-SRN as in
Vu et al. [6]. This approach is based on the supposition that the LP residual is related to the source information (glottal information) of speech, and this kindof information may remain unchanged in both AC andBC speech signals. Therefore, the inverse restoration
function is built up with a fixed value of the averaged
LP residuals ratio of AC and BC speech. However,in our current analysis shown in Fig. 2, these averagevalues change from frame to frame in the time domain.
The learning method used in the previous LP
scheme was SRN. SRN and other neural-networkbased training techniques can be used efficiently withsmall corpora, but when the size of the training corpus increases, the time taken for training will greatly
increase. This makes it impractical to use SRN for
training huge corpora. In addition, we had to separately train the joint LSF vectors of AC/BC speech foreach specific condition in our previous blind restoration of BC speech due to the over-training problem
with SRN. When we extended the method of blind
BC speech restoration to various kinds of noisy environments, SRN did not seem to be suitable for training. Another problem with SRN training is that itis difficult to adapt the SRN model to Be speech inunknown noisy environments. This problem makes it411

I~~tl---I.':.--,-"·.l---1ifHRt~~~·• • ;-----j·.:l-o.4o"---I.1oo----2.....
oo-~300LU!:.:~4~0':-0----'-:50=0:---:::600=--'.g 0.2,

OI---v>I!_(IIIo-JIIH!

< -0.2L.:...;.----.:..J....:..:..:..;.;......;..;~::....:...:..:~::!lt:.~~~::::__;;:=___:;;;;oTlme(ms)

c:~~
101

m

0

=-------=::-::~:1 i ~t .;,;·»?~Wif"',[i~sfi~;": ------'---:----i.--:......

100

200

300

400

500

04
m- . 0600

Tlme[ms)-0i:i't';±T.~
r
£

-5'

5

10

15

200

400

-0

20

25

30

Frame number

600
800
Time (ms)

1000

.BC

7• •~

1200

1400

i~

n~
£
I!

10

20

30

40

50

60

70

Frame number

Fig. 2 (a) and (b): AC speech, (c) and (d): BC speech, and (e) and (f): Variations in k for /a/ (left panel) andfor /nukumori/ (right panel) of a male speakerdifficult to build an open dataset BC speech restoration in noisy environments. Here, statistical models
such as GMM might be a better solution for training
the joint LSF vectors of AC/BC speech for various
kinds of clean and noisy speech.
We only evaluated the performance of the LP
scheme to restore BC speech in clean environments
in previous studies. The goal of BC speech communication is especially to use it in noisy speech environments. Therefore, we should confirm whether the
LP scheme (both LSF-SRN and our proposed LSFGMM) is useful for restoring BC speech in noisy environments.

4.

Improved Method

4.1

Re-estimation of residual ratio

Vu et ale [4] investigated change in gain k in thefrequency domain and assumed gain k would be constant in the time domain. We investigated the change
in gain k in the time domain in this research. The
results we obtained from analysis are given in Fig. 2,
where we can see the change in k in the time domain,
especially in long syllables, is considerable.
The residuals ratio in the frequency domain can be
fixed as an average constant factor over all frequencies[4]. Therefore, instead of using constant gain k for all
frames [4], we compute the average LP residuals foreach frame of AC/BC speech to better estimate gain
k. Gain k is computed as:(10)N

Gy(eiw )= ~ L Gy(eiWi )

(12)

i=l

Here, N is the number of frequency bins in FFT analysis.
4.2

LSF-GMM training and prediction

GMM is one of the most efficient methods of training in voice conversion [7]. GMM is suitable for training with huge amounts of data and is used to adapt
the noise model in other unknown models in speech
recognition [8]. Thus, GMM can be used to adapt
the trained model of BC speech under various specificconditions to those of other unknown conditions.
We used GMM for the training phase in this study
instead of SRN, which was used in the original LP
scheme [4].
This section presents the procedure for training
and prediction in our proposed LSF-GMM BC speech
blind restoration.
4.2.1 Procedure for training
The source (BC speech) and target (clean AC
speech) vector are presented in two time sequences,i.e., X = [X1!X2'''',XN) and Y = {YbY2,"',YN},
where N is the number of frames. The Xi and Yi are
D-dimensional feature vectors for the i-th frame. For
each frame of AC /BC speech, we add one average LP
residual coefficient computed with Eq. (10) to the
LSF vectors to compute the joint AC/BC vector. The
source and target vector of each frame are therefore
replaced as:

where

Xi
(11)

412

[LSF:Z: b LSF:z:2,"', LSF:z:p, G:z:(eiW )],(13)Yi = [LSFYbLSFY2, ... ,LSFyp,Gy(ejW»]'(14)Journal of Signal Processing, Vol. 16, No.5, September 2012

Joint source-target vector Z = [Zl' Z2, ••. ,znl where
Zq = (xT, yIlT. The distribution of Z is modeled by
GMM, as in Eq. (15).
Mp{z) =

L omN{z; J.Lm, Em) = p{x, y)

(15)
m=l
where M is the number of Gaussian components.
N{z; J.Lm, Em) denotes the 2D dimension normal distribution with the mean J.Lm and the covariance matrix Em. am is the prior probability of z having
been generated by component m-th, and this satisfies 0 ~ Q m S 1, E!:=l am = 1. The parameters(om' J.Lm' Em) for the joint density p{ x, y) can be estimated using the expectation maximization algorithm[7}.

Table 1 Equipment and setup for recording
Measurement site
Soundproof room
Number of pick-up points 5
10
Number of speakers
RecorderMARANZ, PMD671
PCM
Coding method
Sampling frequency
48 kHzSample size
16 bits
Number of channels
2 (Left:AC, Right:BC)
Mic. A for AC speech
SONY, C536P
Mic. power supply A
SONY, AC148F
Mic. B for BC speech
TEMCO, HG-17
TEMCO, SK-lMic. C for BC speechMic. amp. Band C
Handmade
Speakers (4 set)
JBL, CM62

4.2.2 Procedure for prediction
The transformation function that converts source
feature x to target feature y is given by Eq. (16).

F{x) =E(ylx) =!

Soundproof room

yp(Ylx)dy

I R
I

~
~

M

=

L Pm{X)(JL~ + E~(E~)-l{X - J.L~))m=l

J yet)(16)................. _------_.-. .... '"where

Pm (x)=

OmN{x; J.L~, E~)
M

I

I

I

(17)

L omN{x; J.L~, E~)

Fig.3

Environment for recording AC/BC speech

m=l

(18)

Em =

J.L:nx[ J.L¥:

J.L;;tJ.Lflt1(19)

The Pm (x) is the probability of x belonging to the mth Gaussian component. We use Eq. (16) to predict
vector Y' of clean AC speech from vector X' of BC
speech. After that, we separate the LSF coefficientsand the average residuals. Gain k is then computed asin Eq. (10) and the inverse filter to restore BC speech
is finally computed as in Eq. (9).
We used diagonal covariance GMM in our experiments. The chosen number of Gaussian components
M, which should be selected to be sufficiently large
if we have sufficient data for training, was 15. The
frame size was set to be large enough at 256 ms andthe step was 128 ms. The use of large frames assisted
our method in real-time applications. The order of LP
analysis P was chosen to be 20 in all experiments.
5.

Evaluations

5.1

Data preparation and experimental setup

We evaluated the proposed model for BC speech
restoration in clean environment in our previous stud-

Journal of Signal Processing, Vol. 16, No.5, September 2012

ies [4, 6]. BC speech is especially used for noisy speechenvironments. Therefore, we should confirm whether
the LP scheme is useful for restoring BC speech innoisy environments.
We investigated both our LSF-SRN method [4, 6]
and our proposed LSF-GMM model in both clean andnoisy environments.
The speech data used in our evaluation is afamiliarity-controlled Japanese speech dataset thatwas recorded from 17 speakers, including 10 males and7 females. All speakers were native Japanese graduate
students.
Figure 3 outlines the environment we used to construct the database. BC speech was collected atfive different pick-up points on the head and face (1:mandibular angle, 2: temple, 3: philtrum, 4: forehead, and 5: calvarium). Different microphones wereused at pick-up points from 1 to 4 and at pick-up point

5.
In this work, we only used BC speech that wascollected at the farthest pick-up point from the mouth,Le., pick-up point 5 (calvarium).
The microphone was positioned in front of themouth to record AC speech. Original speech wastransmitted from the mouth to the microphonethrough air, which is the air-conduction process. The

413

further the distance from the mouth to the microphone, the greater the effect the air-conduction process had on observed AC speech.
It is known that the recording environments for AC
and BC speech always differ. To compare the qualityof AC and BC speech, we should use AC and BC microphones that have recording environments that are
as similar as possible. We defined the term observed
noisy AC speech in this paper, which is noisy AC
speech dependent on the environment, on the recording quality of the AC microphone, on the position ofthis microphone in front of the mouth, and on thenoise source. We then recorded clean AC speech totrain BC speech, and observed noisy AC speech incomparative testing; both were recorded closely infront of the mouth, as seen in Fig. 3.
The list of equipment and the setup are listed inTable 1.
Our dataset contains 100 Japanese words and 100
Japanese syllables from Japanese word lists in four
different familiarity ranges (Rl, R2, R3, and R4)[10]. The noisy data contains three kinds of noise,
which are factory, pink, and white noise.
Each kind of noise has three levels, in which thesound pressure level (SPL), called the noise level inthis paper, is low (35 dB), medium (55 dB), and high(75 dB). It is known that the widely used signal tonoise ratio (SNR) depends on the signal being investigated while SPL, which is used to describe the noise
source, is independent of the signal being investigated.
We wanted to control the effects of independent noise
sources on AC and BC speech in our experiments;
therefore, we used SPL instead of SNR.
Objective evaluations were carried out for all low,
medium, and high noise levels of factory noise andsubjective evaluations were only undertaken with thehighest noise level, Le., the factory noise of 75 dB.
5.2 Objective evaluations
We used the log spectral distortion (LSD), linear prediction coefficients distance (LPCD), Melfrequency cepstral coefficients distance (MFCCD),
and perceptual evaluation of speech quality (PESQ) to
objectively and comparatively evaluate the proposed
method. LSD was defined [4], PESQ was defined [9],LPCD, and MFCCD was defined similarly to those in
Vu et al. [4].
p

LPCD

=

L (a:x:(i) - ay(i»2

(20)

i=l

Q

MFCCD

L (c:x:(i) - Cy(i»2

(21)

i=l

where a:x:(i) & ay(i) and Cx(i) & Cy(i) are the LP coefficients and Mel-frequency cepstral coefficients (MFCC)

414

of the source and target speech for evaluation. The P
is the LP order and Q is the cepstral order. The LSD
and PESQ are objective evaluations of voice quality
for hearing while LPCD and MFCCD are objective
evaluations of the voice quality for speech recognition.
From Fig. 4, we can see that non-blind LSF is basically the best method and non-restored BC is theworst. The two blind restoration methods of LSFSRN and LSF-GMM approximately reach the restoration of LSF. This might be because we trained enoughdata and the predicted LSFs in blind restorations approximated the LSFs of clean AC speech.
The proposed LSF-GMM outperformed LSF-SRN
as well the LTF methods in the LSD and PESQ tests
related to speech quality for human hearing. The proposed LSF-GMM outperformed LSF-SRN as well the
LTF methods in the LPCD and MFCCD tests related to performance in automatic speech recognition(ASR). Therefore, the proposed LSF-GMM outperformed both previous LSF-SRN and LTF methods inboth human hearing (LSD and PESQ tests) and ASR
systems (LPCD and MFCCD tests). Note that in ourdataset, the AC and BC speech were recorded withdifferent microphones, as in Fig. 3. The recording microphone for AC speech was a Sony C536P, which was
directional and expensive, while the recording microphone for BC speech was a Temco, which was inexpensive and commercially available. The observed signals
recorded with the Temco microphones were smeared
due to surrounding noise while those recorded with theSony C536P had much less smear due to surrounding noise. Thus, the quality of observed AC speech
was much better than that of BC speech. In addition, our recording position for observed noisy AC
speech may have been too close to the mouth and insufficient to emphasize the effect of the air-conduction
process on observed AC speech. Therefore, the quality of observed noisy AC speech was better both fororiginal and restored BC speech. This finding does
not conflict with the previous results [1], in which BC
speech is more stable against surrounding noise thanAC speech. It only supports a supplementary idealthat BC speech is only more robust against noise thannoisy AC speech under similar recording environment
conditions, which have to be carefully chosen and setup.
5.3 Subjective evaluations
Due to time limitations, we only conducted a subjective evaluation, in which we evaluated the recognition scores for the LP-scheme-based methods in only
high-level factory noise (75 dB).
The subjective tests were carried out with sevensubjects who had normal hearing. All were native
Japanese graduate students.
The speech signals of 96 Japanese syllables, extracted from our dataset, were played in random order

Journal of Signal Processing, Vol. 16, No.5, September 2012

18
16

(a)

(b)

';-",,1 Non-Blind LSF_

I - I Non-Blind LSF_
Observed AC

3.5

!

~

iii'10
~

3

0

c

...J

~LSF-SRN~LSF-GMM

4

Observed AC

12

(J)

IJBILTF

4.5~LSF-SRN~LSF-GMM

14

_SC

5

_BC
I-ILTF

8

m2.5

6

2

4

1.5 .

Q.

i··

2
0

75dB

55 dB

35 dB

0.5

Clean

25

35 dB

Clean

_BC
I!!!E!ILTF(d)~LSF-SRN~LSF-GMM

20

55 dB

30

_BC
I-ILTF

(e)

- -

75 dB

~LSF-SRN~LSF-GMM

25

1--,,'

1/".""",., Non-Blind LSF__

Observed AC

Non-Blind LSF
Observed AC

20
CD

u
c::

8c:: 15

::

as

c 15
0
0

0

u.

~ 10

::::'!

10
5

0

5

75 dB

55 dB

35 dB

Clean

0

75 dB

55 dB

35 dB

Clean

Fig.4 Results of objective evaluation of factory noise: (a) LSD, (b) PESQ, (c) LPC distance (LPCD), and (d)MFCC distance (MFCCD)
in the tests. The subjects had not heard these syllables previously and they had not been trained before
the experiment. They were asked to listen to eachword only once and write down what they heard in
Hiragana to avoid training effects in determining syllables with lower familiarity. We used six types of audio: AC speech, BC speech, and four types of restored
signals using the four models (LTF, blind LSF-SRN,
blind LSF-GMM, and non-blind LSF). Intelligibility
could generally be evaluated using the average recognition accuracy scored by all subjects.
Figure 5(a) shows the average scores for recognition accuracy under clean conditions and Figure 5(b)shows those under the noisy factory conditions of 75
dB. The non-blind LSF model was also the best for the
subjective evaluation followed by the blind LSF-GMM
model. The subjective evaluation confirmed that our
improved method of restoring BC speech, LSF-GMM,
outperformed LSF-SRN and the other previous methods of restoring BC speech.
As mentioned in previous sub-section, we comparatively evaluated our proposed method and observed
noisy AC speech rather than actual high noisy AC

Journal of Signal Processing, Vol. 16, No.5, September 2012speech. The observed noisy AC speech was recordedwith an extremely high quality microphone that was
too close to the mouth. Therefore, the effectivenessof BC speech in comparison with AC speech was notdemonstrated in the results we obtained from our evaluation. The effectiveness of BC speech was maintained as in previous results [1], in which BC speech
was more stable against surrounding noise than AC
speech. If we had used actual noisy AC speech, which
had been recorded with a general (non-directional) microphone far from the speaker, instead of the observed
noisy AC speech in this study, the recognition rate fornoisy AC would have been drastically reduced.
It is easy to see that BC speech restored with
our improved method was more intelligible than theoriginal BC speech as well as BC speech restored byother previous methods. Consequently, our improved
method was robust against both degradation due tobone conduction and changing speaker pronunciations
due to surrounding noise, which is referred to as theLombard effect.

415

100~(-~~C-~----~--~----~~ r.===-~----~
90

80

.I

70

S!

60

e
j 50

'1

I_'LTF
C:=J Non-BUnd LSF
C:=J LSF-SRN
I.-'LSF-GMM_CleanAC

100~~-)~~m--~-n-oi~se----~----~~ ~===-B-C----~[!!!I!i!!JLTF.... :
90

C:=J Non-BIlnd LSF
C:=J LSF-SRN
C:=J LSF-GMM

60l,mll,","'I.. ' Observed AC

70_CleanAC

....

:.

30

I

!I ..

!I·

i···

40

a:

"·1
...1

I

30

I

20

20
10

234
Familiarity range

2

3

4

Familiarity range

Fig. 5 Results of subjective evaluation: Scores for word recognition
6.

Conclusion

We solved all three remaining problems to improve
the performance of the LP scheme in restoring BC
speech. Instead of using constant residual gain for
all frames, we estimated the average gain for eachframe. We used GMM for training instead of neural network, which made our model easier to use fortraining under various conditions. We also conducted
experiments under both of clean and noisy environments. The experimental results indicated that our
improved approach outperformed the previous methods in both human hearing quality and ASR. It also
demonstrated its robustness to both degradation due
to bone conduction and changing speaker pronunciations due to surrounding noise, which is referred to as
the Lombard effect.
We intend to study how to adapt the trained GMM
model to Be speech in various kinds of noisy environments in the future, and therefore, our proposed
methods should be able to be applied to restoring
open-dataset BC speech in noisy environments. We
also plan to rebuild the dataset for AC/BC speech,
in which there are many kinds of observed AC speechfrom near to far from the mouth, recorded with thesimilar commercially available microphones to thosefor BC speech, to obtain a balanced evaluation of both
BC and AC speech.
Acknowledgments
This work was supported by a Grant Program
made by the Yazaki Memorial Foundation for Science and Technology. It was also supported by Scope(071705001) of the Ministry of Internal Affairs andCommunications (MIC), Japan.

416

References[I] S. Kitamori and M. Takizawa: An analysis of bone conducted speech signal by articulation tests, IEICE Trans.
Vol. J72-A No. 11, pp. 1764-1771, 1989.
[2] S. Ishimitsu, H. Kitakaza, Y. Tshuchibushi, H. Yanagawa
and M. Fukushima: A noise-robust speech recognition system making use of body-conducted signals, Acoust. Sci &Tech., Vol 25, pp. 166-169, 2004.
[3] T. Tamiya and T. Shimamura: Reconstruct filter designfor bone-conducted speech, Proc. ICSLP 2004, Vol. II, pp.
1085-1088, 2004.
[4] T.T. Vu, K. Kimura, M. Unoki and M. Akagi: A study on
restoration of bone-conducted speech with MTF-based andLP-based models, J. Signal Processing, Vol. 10, No.6, pp.
407-417, 2006.
[5] M. Nakagiri, T. Toda, H. Kashioka and K. Shikano: Improving body transmitted unvoiced speech with statisticalvoice conversion, Proc. ICSLP-2006, pp. 2270-2273, 2006.
[6] T. T. Vu, G. Seide, M. Unoki and M. Akagi: Method of
LP-based blind restoration for improving intelligibility ofbone-conducted speech, Proc. Interspeech 2007, pp. 966-969,2007.
[7] A. Kain and M. W. Macon: Spectral voice conversion for
text-to-speech synthesis, Proc. ICASSP-1998, Vol. 1, pp.
285-288, 1998.
[8] M. Ida and S. Nakamura: HMM composition-based rapid
model adaptation using a priori noise GMM adaptation
evaluation on Aurora2 corpus, Proc. ICSLP-2002, pp. 437440, 2002.
[9] Y. Hu and P. Loizou: Subjective evaluation and comparisonof speech enhancement algorithms, Speech Communication,
Vol. 49., pp. 588-601, 2007.
[lOJ Database for speech intelligibility testing using Japaneseword lists, NTT-AT, March 2003.

Journal of Signal Processing, Vol. 16, No.5, September 2012

Phung Nghia Trung
received his B.E. in electronic& telecommunication engineeringfrom the Hanoi University of Technology in 2002 and his M.E.
in electronic & telecommunication engineering from the Vietnam National University, Hanoi,
in 2007. He has been with theThai Nguyen University of Information and Communication Technology from 2003. He has also
been a Ph.D. candidate at the
School of Information Science of
the Japan Advanced Institute of
Science and Technology (JAIST) since 2009. His main researchinterest is speech signal processing.

of Phonetics Science at the University of Amsterdam. He is amember of the Institute of Electronics, Information and Communication Engineers (IEICE) of Japan, the Acoustical Societyof Japan (ASJ), the Institute of Electrical and Electronic Engineering (IEEE), the Acoustical Society of America (ASA), andthe International Speech Communication Association (ISCA).
Dr. Akagi received the IEICE Excellent Paper Award fromthe IEICE in 1987, the Best Paper Award from the ResearchInstitute of Signal Processing in 2009, and the Sato Prize for
Outstanding Papers from the ASJ in 1998, 2005, 2010 and 2011.
Professor Akagi is currently the president of the ASJ.
(Received November 07, 2011; revised February 14, 2012)

Masashi Unoki
received his
M.S. and Ph.D. (Information Science) from the Japan Advanced
Institute of Science and Technology (JAIST) in 1996 and 1999.
His main research interests are in
auditory motivated signal processing and the modeling of auditory
systems. He was a Japan Society for the Promotion of Science(JSPS) research fellow from 1998
to 2001. He was associated with
the ATR Human Information Processing Laboratories as a visiting
researcher from 1999-2000, and he
was a visiting research associate at the Centre for the Neural
Basis of Hearing (CNBH) in the Department of Physiology at
the University of Cambridge from 2000 to 200l. He has beenon the faculty of the School of Information Science at JAIST
since 2001 and is now an associate professor. He is a member of the Research Institute of Signal Processing (RISP), theInstitute of Electronics, Information and Communication Engineers (IEICE) of Japan, and the Acoustical Society of America(ASA). He is also a member of the Acoustical Society of Japan(ASJ), and the International Speech Communication Association (ISCA). Dr. Unoki received the Sato Prize from the ASJ
in 1999 and 2010 for an Outstanding Paper and the Yamashita
Taro ''Young Researcher" Prize from the Yamashita Taro Research Foundation in 2005.

Masato Akagi
received his
B.E. from Nagoya Institute ofTechnology in 1979, and his M.E.
and Ph.D. Eng. from the Tokyo
Institute of Technology in 1981
and 1984. He joined the Electrical Communication Laboratories of Nippon Telegraph and Telephone Corporation (NTT) in 1984.
From 1986 to 1990, he worked
at the ATR Auditory and Visual
Perception Research Laboratories.
Since 1992 he has been on the faculty of the School of Information
Science of the Japan Advanced Institute of Science and Technology (JAIST) and is now a full
professor. His research interests include speech perception, themodeling of speech perception mechanisms in human beings,
and the signal processing of speech. During 1998, he was associated with the Research Laboratories of Electronics at MIT
as a visiting researcher, and in 1993 he studied at the Institute

Journal of Signal Processing, Vol. 16, No.5, September 2012417