୍⯡♫ᅋἲே 㟁Ꮚ᝟ሗ㏻ಙᏛ఍                      ಙᏛᢏሗ‫ٻ‬‫ٻڇڮڞڤکڪڭگڞڠڧڠٻڡڪٻڠگڰگڤگڮکڤٻڠڣگ‬                     信学技報‫ٻۏۍۊۋۀڭٻۇڼھۄۉۃھۀگٻڠڞڤڠڤ‬社団法人 電子情報通信学会‫ٻڮڭڠڠکڤڢکڠٻکڪڤگڜڞڤکڰڨڨڪڞٻڟکڜٻکڪڤگڜڨڭڪڡکڤ‬                TECHNICAL‫ٻڄڍڌڈڑڌڋڍڃٻڎڑڈڑڌڋڍګڮ‬THE INSTITUTE OF ELECTRONICS,REPORT OF IEICE.
‫ٻ‬INFORMATION AND COMMUNICATION ENGINEERS

コクレオグラムとスペクトログラムを用いた深層学習音声認識
アンドロスチャンドラ†

サクリアニ

サクティ†

ミルナアドリアーニ††中村哲†† 奈良先端科学技術大学院大学†† インドネシア大学
あらまし 本論文では、対数メル尺度やスペクトログラムにコクリオグラムを加えた特徴ベクトルを用いた、Deep

Neural Network（DNN）, Convolutional Neural Network（CNN）による音声認識システムを提案する。TIMIT 音素認識タスクにおいて、スペクトログラム―コクリオグラムを用いた CNN で、スペクトログラムのみを用いた、CNNに対して 8.2%、DNN に対して 19.7%の性能向上を示した。
キーワード

深層学習：素性組み合わせ、コクリオグラム

Deep Learning-based ASR
using Cochleogram and Spectrogram Features Combination
Andros TJANDRA† , Sakriani SAKTI† , Mirna ADRIANI†† , and Satoshi NAKAMURA†† Nara Institute of Science and Technology†† Universitas Indonesia
Abstract This paper proposes various possibilities to combine cochleogram features with log-mel ﬁlter banks
or spectrogram features within the DNN and CNN framework. Performance was evaluated on TIMIT phoneme
sequence recognition task.

The best accuracy was obtained by high-level combination of two dimensional

cochleogram-spectrogram features using CNN, achieved up to 8.2% relative phoneme error rate (PER) reduction
from CNN single features or 19.7% relative PER reduction from DNN single features.
Key words Deep learning, feature combination, cochleogram, DNN and CNN

1. Introduction

Deep neural network - HMM (DNN-HMM) hybrid systems
have been proven to be superior compared to the conventional GMM-HMM model. As DNNs are less sensitive todata correlation and the increase in the input dimensionality
than GMMs, they allow us to exploit a richer set of features.
Recent research has also shown that auditory features basedon gammatone ﬁlters are promising to improve robustness ofASR systems [1]. Another alternative to DNNs is the use
convolutional neural networks (CNNs). CNNs with two dimensional log-mel ﬁlter banks or spectrogram input features
have shown improvements over DNNs [2]. Although, CNN
framework has shown to give many advantages, various features and combination within CNN framework have not been
widely explored.
In this work, we attempt to explore the two dimensionalfeatures derived from gammatone ﬁlter, which are also called
cochleograms within NN-HMM framework. Furthermore, we
also investigated the possibilities to combine cochleogram
features with spectrogram features. In particular, we combine within low and high levels of CNNs, which we call lowlevel and high-level feature combination. As comparison, wealso construct the similar conﬁguration with DNN in which
the features were vectorized into one dimensional features.

2. Neural Networks

DNN is a neural network which has many hidden layers
between input and output layers. Compared to traditional
neural networks with one layer, DNNs have a greater capacity to learn and generalize to more complex datasets [3].
Another type of neural networks is CNN. CNNs are neuralnetworks that combine values between local receptive ﬁelds,
shared weights, and perform sub-sampling. In CNN, the convolutional layers consist of multiple ﬁlters which convoluted
across a given input or previous layer output and the poolinglayers try to sub-sample the value from certain area. Using
convolution and pooling, CNN has spatial-temporal connectivity and local translation invariance for the given input.
2. 1 Cochleogram
In speech recognition, spectrogram is the widely used features obtained via fast Fourier Transform. In this work, we
also explore gammatone-based ﬁlter. Cochleogram constructa time-frequency representation of the input signal to mimic
the components from the cochlea of human hearing system.
To construct a cochleogram, gammatone ﬁlter is used:
g(t) =

atn−1 cos (2πfc t + φ)
e2πbt

(t  0),(1)

where a deﬁnes the value for amplitude, n deﬁnes the order—1—This article is a technical report without peer review, and its polished and/or extended version may be published elsewhere.

Copyright ©2016 by IEICE

of the ﬁlter, b deﬁnes the bandwidth, fc is the central frequency (in kHz) and φ for phase (which usually we set into0). According to [4], b = 1.019 ∗ 24.7 ∗ (4.37 ∗ fc + 1). In ourexperiments, we down-sample the frequency into frequency
bands with equivalent rectangular bandwidth (ERB) scale.

3. Feature Combination

表 1 Comparisons of DNN and CNN using diﬀerent features interms of phoneme error rates on TIMIT core test set.
Features

PER(%)DNN CNN

Mel

26.58

23.24

Coch26.78

23.65

Mel+Coch (Low)

26.02

22.61

Mel+Coch (High) 24.89

21.34

nected hidden layer and softmax layer on the top. For DNN
high-level feature combination, we use 2 diﬀerent stacks of 5
fully connected hidden layer, 1 fully connected for transition
between high level feature with softmax layer, and softmax
layer on the top. For CNN low-level feature combination, weuse 2 convolution and pooling layer and 2 fully connected hidden layer with softmax layer on the top. For CNN high-levelfeature combination, we use 2 diﬀerent stacks of 2 convolution and pooling layer and 2 fully connected hidden layer
with softmax layer on the top.

5. Experiment Results

図 1 a) Low level feature combination for DNN b) High levelfeature combination for DNN

In low level features combination, we convert the speech
into a 2D feature representation. In our case, we convert thespeech into mel-ﬁlterbank spectrogram and cochleogram. In
this approach, we do concatenation and the result is matrix
features xcomb = [xspec ; xcoch ] ∈ R2f ×t . Figure 1.a shows
the detail for the DNN with low level feature combination.
We vectorized the matrix features into 1D vector R2f t , thenused a using Stacked Denoising Autoencoder (SDAE) to pretrain the weights W = [W1 , ..., Wk ] and followed by ﬁnetuning. To apply low-level feature combination for the CNN,
we change the multiple fully connected layers into multiple
convolutional and max-pooling layers respectively and feed
it into a fully connected hidden layer with the softmax layer.
In high level feature combination, we split our model into
2 diﬀerent stacks of hidden layers. For DNN models in Figure 1.b, we separate input features and build two stacks ofseveral hidden layers. On the left stacks and right stacks, theweight parameters [W1spec , .., Wjspec ] and [W1coch , .., Wkcoch ]are trained only with spectrogram and cochleogram featuresand hcoch
into
respectively. In the end, we concatenate hspecjjhf1 ull and put softmax layer. The same architecture is also
applied in the high-level CNN model by replacing each fully
connected layers with convolution and max-pooling layers.

4. Experimental Setup
4. 1 Corpus and Front-End
Phone recognition experiments were perfomed on theTIMIT dataset. We extracted the context window by using a 25-ms Hamming window with 10-ms step size. Then,the spectrogram and cochleogram speech features are generated by a Fourier-transform-based ﬁlter-banks and gammatone ﬁlter. In our experiments, we set gammatone ﬁlter
parameter into 29 frequency bands from 20 Hz to 20.000
Hz, into equivalent rectangular bandwidth (ERB) scale. For
each moving window result, we average across time domainthen we apply 14 context window to the left and right. For
mel-spectrogram features, we also use 29 frequency bands.
Following the TIMIT s5 recipe in Kaldi, the acoustic model
consists of 1943 tied triphone states.
4. 2 Framework
For DNN low-level feature combination, we use 6 fully con-Table 1 shows performance comparisons of various systems
in terms of phoneme error rates (PER) on TIMIT core test
set. As can be seen, both low-level and high-level features
combination within DNN and CNN framework provided improvements in recognition accuracy. The best performances
are 21.34% which was obtained by high-level combinationof two dimensional cochleogram-spectrogram features within
CNN framework.
Overall, the combination of spectrogram and cochleogram
features provided consistent improvements over single features. We hypothesize that this may be because cochleogramwith ERB scale of the gammatone ﬁlter could support thebetter representation at lower frequency. Therefore, combining the strengths of spectrogram and cochleogram features
into a single system, lead to a more accurate ﬁnal result.

6. Conclusion

In this paper, we explored the use of cochleogram featuresin the deep-learning framework. Furthermore, we also investigated various possibilities of cochleogram and spectrogram
feature combination. The results reveal that 2D featureswith CNN performed better than 1D features with DNN.
The best accuracy was obtained by high-level combinationof two dimensional cochleogram-spectrogram features using
CNN, achieved up to 8.2% relative PER reduction from CNN
single features or 19.7% relative PER reduction from DNN
single features.
謝辞 Part of this work was supported by Microsoft CORE
10 Project as well as JSPS KAKENHI Grant Numbers
24240032 and 26870371.
文
献[1] J. Qi, D. Wang, Y. Jiang, and R. Liu, “Auditory features
based on Gammatone ﬁlters for robust speech recognition,”
IEEE ISCAS, pp.305–308, 2013.
[2] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, and G. Penn,“Applying convolutional neural networks concepts to hybrid
NN-HMM model for speech recognition,” ICASSP, pp.4277–4280, 2012.
[3] Y. Bengio, “Learning deep architectures for AI,” Foundations and trends in Machine Learning, vol.2, no.1, pp.1–127,2009.
[4] B.R. Glasberg and B.C. Moore, “Derivation of auditory
ﬁlter shapes from notched-noise data,” Hearing research,
vol.47, no.1, pp.103–138, 1990.

—2—