Proceedings, APSIPA Annual Summit and Conference 201812-15 November 2018, Hawaii

Bottleneck feature-mediated DNN-based feature
mapping for throat microphone speech recognition
Takahito Suzuki*, Jun Ogata†, Takashi Tsunakawa*, Masafumi Nishida* and Masafumi Nishimura**
Shizuoka University, Shizuoka, Japan
E-mail: suzuki.takahito.14@shizuoka.ac.jp†National Institute of Advanced Industrial Science and Technology, Japan

Abstract—Throat microphones are more robust toenvironmental noises than usual acoustic microphones such as
close-talk microphones because they detect speech signals throughskin vibrations rather than by air transmission. Throat
microphones, however, cannot be used in conventional speech
recognition systems because their acoustic characteristics aremuch different from those of the acoustic microphones. In thisstudy, we propose a deep neural network (DNN)-based feature
mapping method for throat microphone speech recognition. To
utilize a large amount of training data recorded by acoustic
microphones and effectively reduce the acoustic mismatch
between the throat and acoustic microphones, we tried to use thebottleneck features to mediate between them. Evaluation results
for a large-vocabulary speech recognition task of Japanese freeconversation revealed that the proposed system had a 45.8%
lower character error rate (75.5% → 40.9%) than the typical
MFCC system trained from the acoustic microphone data.

I.

INTRODUCTION

Deep neural networks (DNN) with high discrimination
performance have recently been applied to speech recognition,and the recognition accuracy was reported to be almost thesame level as that of a human’s transcription of an English
telephone conversation (Switchboard) [1]. However,recognition performance is still much degraded in highly nonstationary noise environments and needs to be further improved.
As a method to suppress external noise, a throat microphone,which closely adheres to the throat and receives the vibrations
directly from the skin by the piezo element, can be used [2][3].
It is more robust to external noise than a standard acoustic
microphone, which receives vibrations of air. However, there
is a big difference in acoustic characteristics between throat
and acoustic microphones. Therefore, the recognition accuracyof a throat microphone deteriorates when using an acoustic
model trained by speech collected with an acoustic microphone
because of acoustic mismatch. Furthermore, it is difficult totrain acoustic models from scratch with the speech of a throat
microphone because it can use only a limited amount of data.
To solve this acoustic mismatch of the throat microphone inspeech recognition, various methods have been proposed [4][6]. Lin et al. [6] proposed a DNN-based feature mapping fromthe throat microphone to the acoustic microphone by using
acoustic models of a conventional acoustic microphone. Itsfeature transformation is from the mel-frequency cepstralcoefficient (MFCC) of the throat microphone to the MFCC of
the acoustic microphone.

978-988-14768-5-2 ©2018 APSIPAIn distant-talking speech-recognition task, Hiwaman et al.
[7] proposed a method of DNN-based feature mapping from
the MFCC feature space of a single distant microphone (SDM)
to the bottleneck feature space of an individual headmicrophone (IHM) to suppress acoustic mismatch between
IHM and SDM. The bottleneck feature (BNF) is extracted
from the bottleneck layer of DNN that is trained to discriminate
phonemes. The BNF has intrinsic information of phonemes
and is more effective for phoneme discrimination thanconventional features such as MFCC. In the work of Hiwaman
et al. [7], the weights of DNN for feature mapping were
initialized to the weights of DNN for extracting the BNF of anIHM and then fine-tuned with the MFCC of an SDM as theinput signal and the BNF of the IHM as the supervised signal.
Das et al. [8] reported that a DNN that was trained with Englishalignments and then retrained with a limited number of Turkishalignments had higher Turkish phoneme discrimination
accuracy than a DNN that was randomly initialized and thentrained with only Turkish alignments. Hence, feature mapping
accuracy was assumed to be improved by devising the initial
weights of DNN as done by Hiwaman et al. [7].
Inspired by the Hiwaman’s work, we introduce the BNFbased feature mapping approach into throat-microphone
speech-recognition task. In this study, we propose a DNNbased feature mapping from the MFCC of throat-microphone
speech to the BNF of acoustic-microphone speech. By using
this feature mapping as a pre-processing, a normal acoustic
model trained from large-sized acoustic-microphone data canbe applied on the following recognition process. Furthermore,
we study an initialization technique of DNN to improve thefeature mapping.
The rest of this paper is organized as follows. Section 2describes the method of DNN-based feature mapping for throat
microphone speech recognition, Section 3 describes conditions
and results of recognition experiments in a Japanese large
vocabulary continuous speech recognition (LVCSR) task, andSection 4 discusses conclusions and future works.
II.
A.

PROPOSED METHODOverview of our throat mic. speech recognition system

Fig. 1 shows the block diagram of the throat-microphone
speech-recognition system. First, this system extracts a 13dimensional MFCC of throat microphone input by applying

1738

APSIPA-ASC 2018

Proceedings, APSIPA Annual Summit and Conference 201812-15 November 2018, Hawaii

Fig. 1 Block diagram of our throat-microphone speech-recognition system (TM: throat microphone, AM: acoustic microphone)

cepstrum mean normalization (CMN) and then splices 4 frameson each side of the current frame. After that, the current frame
is compressed to 40 dimensions with linear discriminant
analysis (LDA), and maximum linear likelihoodtransformation (MLLT) is applied. Then, feature-space
maximum linear likelihood regression (fMLLR) is also applied.
In this study, we refer to the 40-dimensional feature vector as
throat microphone (TM)-MFCC. To estimate the fMLLR
transform, the alignment information of the throat microphone
is obtained by using a GMM-HMM trained only with throat
microphone speeches. Next, the system carries out the feature
mapping from the 440-dimentional feature vector obtained bysplicing ±5 frames of the TM-MFCC to the BNF of the acoustic
microphone (AM-BNF). The feature vector estimated by theDNN-based mapping is input to the recognition system that has
the GMM-HMM trained by AM-BNFs taken from a typicalspeech corpus.
B.

Training of DNN for extracting BNF

DNN for extracting BNF<Softmax> Output layer<Sigmoid> 1024 units

AM-BNF

42 units <Sigmoid>
Initialize weights<Sigmoid> 1024 units<Sigmoid> 1024 units<Sigmoid> 42 units<Sigmoid> 1024 units<Sigmoid> 1024 units<Sigmoid> 1024 units<Sigmoid> 1024 units<Sigmoid> 1024 units<Sigmoid> 1024 units

AM-MFCC
40x11

TM-MFCC
40x11Fig. 2 Architectures of DNN for extracting BNF and for feature mappingC.

The architecture of the DNN for extracting the BNF is shownin Fig. 2. The acoustic microphone (AM)-MFCC is extracted
from speech data of the acoustic microphone as well as the TMMFCC in pre-processing. Then the 440-dimensional features
obtained by splicing ±5 frames of the AM-MFCC are used fortraining of the DNN as an input signal. The state alignments of
the acoustic-microphone speeches are estimated by using aGMM-HMM trained only with the AM-MFCCs and used fortraining of the DNN as a supervised signal. The DNN is pretrained by stacked denoising auto-encoders (SdA) and finetuned. Then a 42-dimentional BNF is estimated from themiddle layer of the DNN.

DNN for feature mapping

Training of DNN for feature mapping

Fig. 3 shows the training process of DNN for the feature
mapping (FM-DNN). The AM-BNF and TM-MFCC are
extracted from parallel data simultaneously recorded by theacoustic and throat microphones. The input signal for trainingof the FM-DNN is the 440-dimensional features obtained bysplicing ±5 frames of the TM-MFCC, and the supervised signal
is the AM-BNF. The FM-DNN has the same architecture as
the DNN for extracting the BNF cut up to the bottleneck layer
as shown in Fig. 2. The weights of the FM-DNN are initialized
to the weights of the DNN for extracting BNF and fine-tuned.

Fig. 3 Pipeline for training the DNN for feature mapping (TM: Throat microphone, AM: Acoustic microphone, FM-DNN: DNN for feature mapping)1739

Proceedings, APSIPA Annual Summit and Conference 2018III.

12-15 November 2018, HawaiiEXPERIMENTS

Datasets
Parallel data used for training feature mapping was recorded
simultaneously with a throat microphone (NANZU SH – 12jK)
and an acoustic microphone (Sony EXM - CS 3) using a multitrack recorder (ZOOM R24). The sampling rate of the audio
data is 16,000 Hz. We recorded 8 male speakers reading 504
Japanese phoneme balanced sentences for about 3 hours. Thethroat microphone speech data of this parallel data was also
used for training the GMM-HMM of the throat microphone.
As test data, free conversations of 10 male speakers for about
19 minutes were recorded. The contents of these freeconversations were group discussions. Each group consistedof three people. The speech data has less external noise
because the conversations were recorded in a quiet
environment. The speakers of the test data are not included inthe training data.
Approximately 240 hours from the Corpus of Spontaneous
Japanese (CSJ) were used for training data of the GMM-HMMof the acoustic microphone, and about 114 hours from the CSJ
were used for training the DNN to extract the BNF.

TABLE I
CHARACTER ERROR RATE (CER) OF CONVENTIONAL METHODS AND

A.

B.

Experimental conditions
Kaldi [9] was used for feature extraction, acoustic modeltraining, and recognition experiments. Kaldi+PDNN [10] wasalso used to train the DNN to extract the BNF. In this training,the mini batch size was set to 256, the initial learning rate to0.01, and number of iterations to 8. Keras was used to train theFM-DNN. In this, the mini batch size was set to 4096, and theinitial learning rate to 0.001, and number of iterations to 100.
The 3-gram language model was used and was generated fromtranscripts of the CSJ.
Experimental results
Three types of experiments were conducted to evaluate theeffectiveness of the proposed throat-microphone speechrecognition system.

PROPOSED METHOD

To evaluate the performances of four conventional systems
and the proposed system, we conducted five recognition
experiments.
(1) A system using the TM-MFCC as the input feature and a
GMM-HMM trained with the AM-MFCC (AM GMMHMM) as an acoustic model(2) A system using the BNF of the throat microphone as theinput feature and a GMM-HMM trained with the AMBNF (AM Tandem) as an acoustic model(3) A system using the TM-MFCC as the input feature and aGMM-HMM trained with the TM-MFCC (TM GMMHMM) as an acoustic model(4) A system using the BNF of the throat microphone as theinput feature and a GMM-HMM trained with BNF of thethroat microphone (TM Tandem) as an acoustic modelType of input feature

CER [%]AM GMM-HMM

MFCC

74.6

(2)

AM Tandem

BNF

96.6

(3)

TM GMM-HMM

MFCC

52.4

(4)

TM Tandem

BNF

48.9

(5)

AM Tandem(Proposed)

Mapped feature

42.2(5) A system using the feature mapped by the DNN as theinput feature and a GMM-HMM trained with the AMBNF (AM Tandem) as an acoustic model (proposedsystem)
Each system recognized some speech data of the throat
microphone. The BNFs input to systems (2) and (4) are
extracted from the DNNs trained with the acoustic and throat
microphones, respectively. In these experiments, the FM-DNN
is randomly initialized. Table I shows experimental results.
System (1) had a character error rate (CER) of 74.6% becauseof a big acoustic mismatch between the acoustic and throat
microphones. Furthermore, system (2) had a 96.6% CERbecause the DNN trained with the AM-MFCC cannot
transform the TM-MFCC accurately into a feature space of theAM-BNF. Meanwhile, system (5) shows feature mapping
could extract features that suppress the mismatch with the AMBNF. The GMM-HMM trained with a large amount of theacoustic microphone’s speech data has higher phoneme
discrimination performance than the GMM-HMM trained with
only about 3 hours of throat microphone data. As a result, theproposed system (5) had higher recognition accuracy thansystems (3) and (4).
C-2. FEATURE MAPPING METHODSC.

C-1. COMPARISON WITH CONVENTIONAL SYSTEMS

Acoustic model(1)

To verify the effectiveness of the proposed feature mappingfrom the TM-MFCC to AM-BNF, we experimented with four
mapping methods as shown in Table II. TM-BNF is extracted
from DNN trained only with throat microphone.
In all methods, the architecture of hidden layers of the FMDNN is the same as that shown in Fig. 2, and the weights ofFM-DNN were randomly initialized. In speech recognition,
the input feature is obtained from the mapped feature byapplying CMN, LDA, and fMLLR as well as the TM-MFCC.
When the feature input to the FM-DNN is the MFCC, thenumber of units of the input layer is 440 in order to input thefeature obtained by splicing ±5 frames of the TM-MFCC. For
the same reason, when the feature input to the FM-DNN is theBNF, the number of units of the input layer is 462. When theoutput feature is the MFCC, the supervised signal is a 13dimentional MFCC of the acoustic microphone by applying
CMN and hence the number of units of the output layer is 13.
The acoustic model is the GMM-HMM trained with the AMMFCC. Meanwhile, when the output feature is the BNF, thesupervised signal is a 42-dimentional BNF of the acoustic
microphone by applying CMN and hence the number of units

1740

Proceedings, APSIPA Annual Summit and Conference 201812-15 November 2018, Hawaii

IV. CONCLUSIONTABLE II
CHARACTER ERROR RATE (CER) FOR EACH METHOD OF FEATURE MAPPINGFROM MFCC OR BNF OF THROAT MICROPHONE TO MFCC OR BNF OF

In this study, we proposed DNN-based feature mappingfrom the MFCC of a throat microphone to the BNF of an
acoustic microphone by using an acoustic model trained by alarge amount of the BNF of an acoustic microphone. The
proposed method performed much better than the acoustic
models trained only with speech data of a throat microphone oran acoustic microphone. Moreover, our feature mapping
method achieved higher recognition accuracy than theconventional feature mapping method that transforms theMFCC of the throat microphone into the MFCC of the acoustic
microphone. We also found that feature mapping accuracy wasimproved by initializing the weights of the FM-DNN using theweight of the DNN for extracting the AM-BNF instead of
initializing randomly. For future work, we will try to improvefeature mapping accuracy and conduct experiments in noisyenvironments.

ACOUSTIC MICROPHONE

Feature mapping method

CER [%](1) TM-MFCC to AM-MFCC

48.0(2) TM-BNF to AM-MFCC

59.1(3) TM-BNF to AM-BNF

51.1(4) TM-MFCC to AM-BNF (Proposed)

42.2

TABLE III
CHARACTER ERROR RATE (CER) OF USING FEATURE ESTIMATED BY FMDNN INITIALIZED FOR EACH METHOD
Initialization method

CER [%](1) Random

42.2(2) TM-DNN

41.8(3) AM-DNN (Proposed)

40.9

of output layer is 42. The acoustic model is the GMM-HMM
trained with the AM-BNF. Table II shows experimental results.
In the results, the proposed method (4) achieved a lower CER
than the conventional feature mapping from the TM-MFCC to
the MFCC of the acoustic microphone [6]. Methods (2) and(3) using the TM-BNF as the input signal had higher CERs thanmethods (1) and (4) using the TM-MFCC.
C-3. INITIALIZATION OF FM-DNN
To assess whether initializing the weights of the FM-DNN
reduces CER, we experimented with three initialization
methods.
(1) Random initialization (Random)(2) Weights of DNN for extracting the BNF of the throat
microphone (TM - DNN)
(3) Weights of DNN for extracting the BNF of the acoustic
microphone (AM - DNN) (Proposed)
We conducted recognition experiments using features
mapped by each DNN. The acoustic model is the GMM-HMM
trained by the AM-BNF. Table III shows the experimentalresults. The accuracy of feature mapping was improved byinitializing the weights of the FM-DNN using the weight of theDNN for extracting the BNF instead of initializing randomly.
Finally, by using the proposed method (3), the CER wasreduced to 40.9% by initializing with the weight of the DNN
for extracting the AM-BNF.
From the above, the proposed method had a lower CER(40.9%) than the conventional methods. On the other hand, inthe results of recognition of acoustic microphone speech data
recorded simultaneously with the test data using the GMMHMM trained with the AM-MFCC, the CER was 29.0%. Whenthe test data with less external noise was used, the throat
microphone still had inferior recognition accuracy to theacoustic microphone.

ACKNOWLEDGMENTS
This work was supported by JSPS KAKENHI Grant
Numbers (16H01817) and (16K01543).
REFERENCES[1] G. Saon et al., “English Conversational Telephone SpeechRecognition by Humans and Machines,” Proc. Interspeech 2017,
pp. 132–136, 2017.
[2] T. Dekens, W. Verhelst, F. Capman, F. Beaugendre, “ImprovedSpeech Recognition in Noisy Environments by Using a Throat
Microphone for Accurate Voicing Detection,” Signal Processing
Conference, pp. 1978–1982, 2010.
[3] W. Amano, K. Noguchi, R. Takeda, K. Honma, “AutomaticSpeech Recognition Using Throat Microphone Under HighlyNoisy Environments,” journal of EICA, pp. 182–186, 2014, inJapanese[4] A. Shahina, B. Yegnanarayana, “Mapping Speech Spectra fromThroat Microphone to Close-Speaking Microphone: A Neural
Network Approach,” EURASIP Journal on Advances in Signal
Processing, vol. 2007, no. 2, pp. 1–10, 2007.
[5] K. Vijayan K. Sri Rama Murty, “Comparative Study of Spectral
Mapping Techniques for Enhancement of Throat MicrophoneSpeech,” Twentieth National Conference on Communications, pp.
1–5, 2014.
[6] S. Lin, T. Tsunakawa, M. Nishida, M. Nishimura, “DNN-based
Feature Transformation for Speech Recognition Using Throat
Microphone,” APSIPA ASC 2017, pp. 596–599, 2017.
[7] I. Himawan et al., "Learning Feature Mapping Using Deep
Neural Network Bottleneck Features for Distant Large
Vocabulary Speech Recognition," ICASSP, pp. 4540–4544, 2015.
[8] A. Das, M. Hasegawa-Johnson, “Cross-lingual Transfer Learning
during Supervised Training in Low Resource Scenarios,”
INTERSPEECH, pp. 3531–3535, 2015.
[9] D. Povey et al., “The Kaldi Speech Recognition Toolkit,” IEEE
2011 Workshop on Automatic Speech Recognition andUnderstanding. IEEE Signal Processing Society, 2011[10] Y. Miao, "Kaldi+PDNN: Building DNN-based ASR Systems
with Kaldi and PDNN," arXiv:1401.6984, 2014.

1741